#1
  1. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    95
    Rep Power
    144

    Crawl Delay Robots.txt


    Keep finding odd things in robots.txt files this month.

    The latest one is:

    User-agent: *
    Crawl-delay: 2

    It's on a small 16 page website selling bungalows to pensioners.

    I've had a look round on the internet and the general consensus seems to be that only big sites need to do this.

    The main question is how harmful could this be from an SEO point of view?

    I ask because they sent over a snapshot of their analytics which shows them losing pretty much all of their organic traffic from December last year to now.
  2. #2
  3. My views are my own ;)
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jun 2012
    Location
    Clifton Park, New York
    Posts
    1,192
    Rep Power
    1616
    I don't think the loss of traffic could be attributed to crawl delay... Not saying it's needed but I have never seen it really harmful.

    I'll ask around to find if anyone saw crawl delay harming a website...
    Everything will be ok in the end

    If it's not ok, it's not the end
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Sep 2016
    Posts
    860
    Rep Power
    2073
    The robots command Crawl-delay: is now obsolete.

    Originally it was intended to stop excessive crawling by bots. Technology and hardware have surpassed the need to use this command.

    Googlebot actually ignores this command now.
  6. #4
  7. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    95
    Rep Power
    144
    Interesting, thanks for the responses.

    Fact is they have so much else wrong with the site I was just wondering if this could've been a contributing factor.
  8. #5
  9. Super Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,357
    Rep Power
    2161
    Originally Posted by IvanDrago
    Interesting, thanks for the responses.

    Fact is they have so much else wrong with the site I was just wondering if this could've been a contributing factor.
    It is just a directive, so would not contribute to how a site ranks. I would remove it now as there shouldn't be a problem with the way that Google crawls.
    Always happy to help in private message. | Learn about my latest project Can You Get Automated Pages To Rank In Google?
    Useful Links: Useful SEO Tools | GTmetrix | FileOptimizer | Cloudflare

Similar Threads

  1. Robots.txt file crawl delay
    By PInder325 in forum New User SEO Questions and Answers
    Replies: 3
    Last Post: Feb 9th, 2016, 09:09 AM
  2. Robots.txt: Allowing search engine spiders to crawl site?
    By VikesRule in forum Web Design, Coding and Programming
    Replies: 2
    Last Post: Apr 20th, 2009, 04:29 PM
  3. Multiple sites on one domain.. do robots crawl their dir's?
    By warehousemultim in forum Google Optimization
    Replies: 0
    Last Post: Mar 19th, 2009, 01:22 PM
  4. When do the Crawlers / Robots / Spiders Crawl the website
    By wasim_@_drushti in forum Google Optimization
    Replies: 3
    Last Post: Apr 26th, 2006, 03:32 AM

IMN logo majestic logo threadwatch logo seochat tools logo