#1
  1. Mr. Goober Guy ;)
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2004
    Location
    Tampa, Florida
    Posts
    1,320
    Rep Power
    24

    From Yahoo - Robots.txt on your site is blocking Yahoo! Slurp crawler


    I recieved this e-mail from yahoo... Not sure if anyone else has gotten one, but thought it was odd considering they removed/de-indexed/banned it from their search engine for using a Java link to link to another site of mine on every page...(this is an assumption as it was the only thing this high ranking, at the time, mortgage site was doing that could be considered wrong...lol) so I said "screw you" and banned the the slurp in return.

    Dear Webmaster,

    We have noticed that the Yahoo! Search crawler, Slurp, is disallowed on
    your website(http://mydomainname.com), while other search engine
    crawlers are allowed. As a major search engine, Yahoo! can help many
    users find your website, but disallowing Slurp keeps pages out of
    Yahoo! Search index. To enable more users to find your website, we
    would request you to allow Slurp to crawl your website by editing your
    Robots.txt file.

    The following line(s) in your /robots.txt file blocks Slurp from your
    site. Please remove this line so people will be able to find your site
    content in Yahoo! Search results.

    User-agent: Slurp
    Disallow: /

    For further information see:
    http://help.yahoo.com/help/us/ysearch/slurp/slurp-02.html

    If you have had any issues with Slurp in the past, we would like to
    work with you in addressing these issues. For feedback or any other
    information, please contact us at crawl-support@yahoo-inc.com.

    Best regards,

    Yahoo! Crawler Support Team

    ==================== Useful Instructions: ====================

    Below are a few features taken from our "Yahoo! Search Tips for
    Webmasters: Saving Bandwidth" posting,
    [ http://www.ysearchblog.com/archives/000078.html ]
    that Yahoo!'s crawler supports that you can use to help save bandwidth
    while ensuring that we get the latest content from your site:

    Gzipped Files: Yahoo! crawler supports gzipped files to reduce
    bandwidth requirements. On average, you will get a 75% savings when you
    enable compression for your site. Also you should be happy to know that
    if our crawler has any trouble with a compressed page, it will re-fetch
    the uncompressed version. In practice, it does encounter a small
    percentage of decompression failures.

    Smart Caching: Yahoo! crawler can use HTTP mechanisms designed to help
    reduce network usage when a client (that's us) repeatedly fetches a web
    file that has not changed. In particular, our crawler often sends the
    If-Modified-Since request-header field [ see section 14.25 of RFC 2616
    at http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.25 ]
    when making repeat requests. The advantage of this is that a response
    of 304 from web server doesn't include your page content, so it uses up
    less bandwidth than a full 200 response.

    Crawl-Delay: To reduce the speed at which our crawlers make requests to
    your site, a webmaster can use a special robots.txt directive
    "Crawl-delay: xx".

    See http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

    This allows a webmaster to manage their bandwidth without restricting
    content on their site from our crawler. A safe value for this would be
    a delay that would allow us to fetch every page on your site in about
    five days. So a five second delay [ Crawl-delay: 5.0 ] would be fine
    for a site with 2,000 pages, but not for a site with 100,000 or more.

    For additional reference on robots.txt
    see http://www.searchengineworld.com/robots/robots_tutorial.htm

    To funny to me. Maybe I'll remove it and see if I "get back in"....... I'll have to think about it since I'm mad at them....

    Comments on this post

    • Jammer agrees : Bigtime!
    Cheerios!

    New to SEO? See the FAQ!

    My Disclaimer:
    Don't Listen To Me - I know nothing!
  2. #2
  3. <-- No I am not Kermit
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Mar 2005
    Location
    Fort Worth, TX
    Posts
    1,015
    Rep Power
    19
    Sounds like you have a few choiced available to you:

    * Leave the robots.txt file so that Y never crawls your site - you'll not be indexed or see SERPs for that site in Y.
    * Let Y crawl and either stay banned or be added back and subsequently banned - you'll not be indexed or see SERPs for that site in Y.
    * Let Y crawl and be reindexed not banned - you'll be indexed and possibly see SERPs for your site.

    The first two options have the same result, but allowing Y to crawl has the potential of staying indexed. Choice is yours, though.

    Comments on this post

    • gchaney agrees : lol...Yep, think I picked door number 3 ;) Just thought the letter itself was humerous given the circumstances of the sites fall....lol
    SEO Tips

    SEO Consulting $50/hr - IM me for details *** PayPal available
  4. #3
  5. Protecting wit's mice
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2005
    Location
    Jammerland
    Posts
    464
    Rep Power
    17
    Originally Posted by gchaney
    at the time, mortgage site was doing that could be considered wrong...lol) so I said "screw you" and banned the the slurp in return....

    ...To funny to me. Maybe I'll remove it and see if I "get back in"....... I'll have to think about it since I'm mad at them....

    You rule, just kick their butt 'till they are offering to pay you to crawl your site!!
  6. #4
  7. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2004
    Location
    Northern NJ, USA
    Posts
    322
    Rep Power
    15
    LOL.... Think I'll ban slurp just to get the bleeding-heart email from Y! and hang it on my wall.

    I'm curious. Did they send the email to the admin contact in your DNS record?

    If so, that's a bit spammy to me. I'd have to blacklist them on my mail server so they get a delivery failure notice "message rejected: user has blacklisted mail from crawl-support@yahoo-inc.com" <hehehehe>

    /*tom*/
  8. #5
  9. Mr. Goober Guy ;)
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2004
    Location
    Tampa, Florida
    Posts
    1,320
    Rep Power
    24
    Originally Posted by longcall911
    LOL.... Think I'll ban slurp just to get the bleeding-heart email from Y! and hang it on my wall.

    I'm curious. Did they send the email to the admin contact in your DNS record?

    If so, that's a bit spammy to me. I'd have to blacklist them on my mail server so they get a delivery failure notice "message rejected: user has blacklisted mail from crawl-support@yahoo-inc.com" <hehehehe>

    /*tom*/
    Yep... sent to the e-mail listed in the DNS who is. We'll see what happens.

Similar Threads

  1. Need N/A SE Rank explained
    By Ramses357 in forum Google Optimization
    Replies: 21
    Last Post: Jul 12th, 2005, 08:06 PM
  2. Help - Yahoo Only Indexes My Homepage
    By Sweat in forum BING/Yahoo Search Optimization
    Replies: 10
    Last Post: Jan 21st, 2005, 06:22 PM
  3. Any tips for making Yahoo Spider go more than one level deep on a dynamic site?
    By Lionzshare in forum BING/Yahoo Search Optimization
    Replies: 4
    Last Post: Oct 28th, 2004, 01:58 PM
  4. Is there any problem with Yahoo OR with US only?
    By rasesh in forum BING/Yahoo Search Optimization
    Replies: 3
    Last Post: Oct 16th, 2004, 01:06 AM
  5. Small Hidden Text
    By Phoenix in forum Google Optimization
    Replies: 4
    Last Post: Apr 30th, 2003, 04:20 PM

IMN logo majestic logo threadwatch logo seochat tools logo