Robots.txt file?

What, exactly, is a robots file used for? I don't have one, and all the bots look for one. Aren't they just to keep the bots out of parts of your site? If you don't have one they'll just index the whole site, yes?

Is there any compelling reason to have one?

Thanks for any and all answers!

Victor

Correct. There are many reasons why you may want to restrict robots from indexing parts of your site.

You can find out more here: http://www.robotstxt.org/wc/robots.html

In some very rare cases I have observed that the lack of a robots.txt file did produce some indexing problems.

Some time ago I analyzed a web site published on a very bad hosting server. The web server was misconfigured and it didn't correctly reply with a standard http header + 404 status code to requests for missing files.

The result was that some spiders refused to index the site.

So, although that case was a rare exception, I always advise webmasters to create a robots.txt file, even if empty or containing instructions to esplicitly allow the indexing:

User-agent: *
Disallow: