#1
  1. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2017
    Posts
    179
    Rep Power
    212

    Crawl Delay Robots.txt


    Keep finding odd things in robots.txt files this month.

    The latest one is:

    User-agent: *
    Crawl-delay: 2

    It's on a small 16 page website selling bungalows to pensioners.

    I've had a look round on the internet and the general consensus seems to be that only big sites need to do this.

    The main question is how harmful could this be from an SEO point of view?

    I ask because they sent over a snapshot of their analytics which shows them losing pretty much all of their organic traffic from December last year to now.
  2. #2
  3. My views are my own ;)
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jun 2012
    Location
    Clifton Park, New York
    Posts
    1,208
    Rep Power
    1723
    I don't think the loss of traffic could be attributed to crawl delay... Not saying it's needed but I have never seen it really harmful.

    I'll ask around to find if anyone saw crawl delay harming a website...
    Everything will be ok in the end

    If it's not ok, it's not the end
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Posts
    1,015
    Rep Power
    2271
    The robots command Crawl-delay: is now obsolete.

    Originally it was intended to stop excessive crawling by bots. Technology and hardware have surpassed the need to use this command.

    Googlebot actually ignores this command now.

    Comments on this post

    • Digital Marketeer agrees : Yep
  6. #4
  7. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2017
    Posts
    179
    Rep Power
    212
    Interesting, thanks for the responses.

    Fact is they have so much else wrong with the site I was just wondering if this could've been a contributing factor.
  8. #5
  9. Super Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,363
    Rep Power
    2206
    Originally Posted by IvanDrago
    Interesting, thanks for the responses.

    Fact is they have so much else wrong with the site I was just wondering if this could've been a contributing factor.
    It is just a directive, so would not contribute to how a site ranks. I would remove it now as there shouldn't be a problem with the way that Google crawls.
    Always happy to help in private message. | Learn about my latest project Can You Get Automated Pages To Rank In Google?
    Useful Links: Useful SEO Tools | GTmetrix | FileOptimizer | Cloudflare
  10. #6
  11. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jul 2017
    Posts
    1
    Rep Power
    0
    Crawl-Delay is parameter used in file named "robots.txt" to stop the "Crawler" or "Spider" to crawl through the web page very frequently. This parameter is used mostly in the case where your website is containing large number of web-pages. It is also used for a website which frequently updated for e.g. FACEBOOK, TWITTER etc.

    Syntax for using Crawl delay :


    User-agent: Googlebot-Image Disallow: Crawl-delay: 14

    ** This means the Gooblebot will wait for 14 sec before crawling through the next page.
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2017
    Posts
    75
    Rep Power
    183
    Crawl Delays are obsolete.

Similar Threads

  1. Robots.txt file crawl delay
    By PInder325 in forum New User SEO Questions and Answers
    Replies: 3
    Last Post: Feb 9th, 2016, 09:09 AM
  2. Robots.txt: Allowing search engine spiders to crawl site?
    By VikesRule in forum Web Design, Coding and Programming
    Replies: 2
    Last Post: Apr 20th, 2009, 04:29 PM
  3. Multiple sites on one domain.. do robots crawl their dir's?
    By warehousemultim in forum Google Optimization
    Replies: 0
    Last Post: Mar 19th, 2009, 01:22 PM
  4. When do the Crawlers / Robots / Spiders Crawl the website
    By wasim_@_drushti in forum Google Optimization
    Replies: 3
    Last Post: Apr 26th, 2006, 03:32 AM

IMN logo majestic logo threadwatch logo seochat tools logo