#1
  1. No Profile Picture
    Learn every day
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Aug 2004
    Location
    Denver
    Posts
    136
    Rep Power
    15

    Robots.txt file?


    What, exactly, is a robots file used for? I don't have one, and all the bots look for one. Aren't they just to keep the bots out of parts of your site? If you don't have one they'll just index the whole site, yes?

    Is there any compelling reason to have one?

    Thanks for any and all answers!

    Victor
  2. #2
  3. Supercalifragilistic
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    May 2003
    Location
    Friendswood, TX
    Posts
    1,018
    Rep Power
    18
    Correct. There are many reasons why you may want to restrict robots from indexing parts of your site.

    You can find out more here: http://www.robotstxt.org/wc/robots.html
    Last edited by SEO_AM; Jan 30th, 2005 at 09:14 PM. Reason: Removed live link
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Sep 2004
    Posts
    758
    Rep Power
    15
    In some very rare cases I have observed that the lack of a robots.txt file did produce some indexing problems.

    Some time ago I analyzed a web site published on a very bad hosting server. The web server was misconfigured and it didn't correctly reply with a standard http header + 404 status code to requests for missing files.

    The result was that some spiders refused to index the site.

    So, although that case was a rare exception, I always advise webmasters to create a robots.txt file, even if empty or containing instructions to esplicitly allow the indexing:

    User-agent: *
    Disallow:
    Please, please, help tsunami survivors making a donation: UNICEF - Red Cross

Similar Threads

  1. SSI: Why and How to Use Server Side Includes
    By amabaie in forum SEO Chat Articles
    Replies: 0
    Last Post: May 20th, 2004, 09:07 PM
  2. If there is no robots.txt file...
    By gleeloyd in forum Google Optimization
    Replies: 11
    Last Post: May 7th, 2003, 04:01 PM

IMN logo majestic logo threadwatch logo seochat tools logo