Yet another robots.txt question

[Jef Patat]

Hello,

I've been surfing the net and the forum but I'm still confused. In the past someone advised me it was better to use an empty robots.txt file to get rid of the 404 loads in my logs. Ok, this works fine. But now I've been reading it is better to:
1. not have a robots.txt if you don't want to exclude anything
2. have one, but empty
3. have one and use this:
User-agent: *
Disallow:
Now, what is true and what isn't?

I prefer to allow all robots everything, if necessary I can exclude a file to make them happy with the file, a stupid image or something like that.

Kind regards, Jef

My site can be found at http://jefpatat.freefronthost.com for the moment the robots.txt is still empty

[Wit]

Hi Jef,

All three methods are valid and have the same effect, although only method 2 and 3 get rid of the 404 errors in your logs. Choose any method you like...

[Jef Patat]

I've read somewhere that an empty robots.txt file is not valid, which is true if you check it in a validator. Won't this harm? Don't robots hate invalid robots.txt?

[LowLevel]

The most safe method is the number 3, that is a robots.txt file containing just:

User-agent: *
Disallow:

This way, it will not generate a 404 (not found) error and it will give spiders full access to the site.

[shaikmunnaf]

robots usefull to seo how it is u can install robotics is automatically used