Robot.txt file

Today i was going thru an article where it has been said that use robot.txt file is important...I would like to know how can we effectively use robot.txt file and for what purpose? For indexing?

Hi sp,

It's robots.txt and it goes in your domain root.
It's only useful function is to deny access to files or folders on your domain but it will only work for spiders that obey robots.txt (no use for stopping spambots or site stealers etc - they mostly ignore it).
Google and all major SE's obey it though, so it can be handy if you wish to keep them out of any section(s) of your site.

If that's your plan you can deny as follows:

User-agent: *
Disallow: fred.html
Disallow: /thisfolder
Disallow: /folder/this-sub-folder

Line 1 will keep them out of any url's called "fred.html" and you can even specify specific useragents if you want to.

If you have no wish to deny any areas then the only use for a robots.txt file would be to stop your server stat's showing denied requests for robots.txt - in which case you could upload a robots.txt file with only these lines:

User-agent: *
Disallow:

Just to clarify one thing from you ClickyB.

Is this the way it is supposed to work ...?

user-agent: *
disallow: fred.html

Is this supposed to block all URL's having fred.html in it, like abcfred.html ; xyzfred.html ; 123fred.html ?

And whereas putting:

user-agent: *
disallow: /fred.html

Will block only the fred.html file and allow others such as abcfred.html ; xyzfred.html ; 123fred.html ?

Thanks for your explanation in advance

Hi,

is not a valid robots.txt line. The text after Disallow: must start with a slash.


will block any URL starting with /fred.php. In other words, it will block /fred.php, /fred.php3, /fred.php?city=brussels,... It will not block /abcfred.php or /mike/fred.php.

Jean-Luc

Quite right Jean-Luc... I made a silly mistake there (sleep-deprivation is a dangerous thing - lol).
Thanks for setting the record straight

...I'm going back to bed now