question on robot.txt file

I would like to know if it would benifit my sites if i added a robots.txt file at the moment als i have is a tag in the html header

<meta name="robots" content="index,follow">

i have this on every page i only have mabe two pages that i do not want indexed. Those pages i have noindex/follow

are thier any crawlers that do not follow the tag i am using???

kapsat

Yes you should have a robots.txt file. See Darrins for a start:

http://www.seochat.com/robots.txt

Why? Lower bandwidth consumption and fewer server "spikes". Finally, if you serve text adverts via a script which includes click tracking, you don't want your advertiser stats to reflect "bot" clicks and they will follow/click unless you have a robots.txt file.

The next entry prevents images in the G cached version of the page:

User-Agent: Googlebot-Image
Disallow: /


HTH

thanks steve,
so if i want all of my pages indexed by all robots my robots.txt file would read?

User-Agent: *
Disallow:

and if i have one file i do not want to be indexed it would read?

User-Agent: *
Disallow: /filename.htm


Now if this is correct do i still leave the tag:

<meta name="robots" content="index,follow"> in the header of all my pages?

and finaly when making a robots.txt file do i create this with ...say... note pad? and FTP it to my server

or do i need to write it in html???

please let me know if i am understanding all of this correctly

kapsat

one more thing if i understand what i have read so far

the example you provided

User-Agent: Googlebot
Disallow: /

this example would tell googly not to spider any pages on my site???

kapsat

Kapsat

Using your example would stop GOOGLE from spidering ALL pages on your site, whereas would mean GOOGLE can spider anything.

thanks

kapsat