What is best robot txt coding to insure spiders index all pages?

Hello,

I a SEM newbie and wanted to know what is the absolute best robot txt coding one should use to insure Google and the major SE's spiders index all of my sites pages?

I've seen the following used:

<meta content="index,follow" name="robots">

<meta name="robots" content="all">

<meta content="all" name="robots">

Which one is best? Or are they all equally good and does it even matter which one I use?

Next, is there an even better one than these three? If so, what is it?

Lastly, is there any other key coding/files I need to insure spiders index all of my sites page? If so, please list with enought details on how to implement for us newbies!

Many thanks,

Randall

That's the one I use,
<meta name="robots" content="index,follow,all">

You also need to use the robots.txt file in your root directory to keep some robots like Scooter (Altavista) wants to see it there.

Open your notepad in Windows or emacs in Linux and paste this code:

User-agent: *
Disallow:

and save it as "robots.txt" when you upload it chmod 755 it.

Hey biz, does that code matter for GGOOGGLLEE?

As far as I know having the robots tags and robots.txt doesn't hurt, BUT if you are using Google AdSense on any of your pages, then Google requires to use a special User-Agent: type to be used in the robots.txt file.

An important reminder here: Please don't try to spam in that .txt file, as most of the idiot spammers tried to do long ago.

Robots only read that text file to find out their rights to, which folders and pages they are allowed to index.

Nothing else will be read than the

User-agent:
Disallow:

lines by the robots.

I am saying this, because in the past years I have seen some robot.txt files full of keywords. Also Firestarter has found one last week and posted it under the Strange robots.txt post on this board.

This thread is like 100 years old... But it is really highly ranked in Google, so people are probably still coming here for search engine optimization help...

In regards to Robot.txt. There have been MANY MANY updates to the Google algo since this post was put out here in 2003. You should REALLY look for more recent info before trusting this old info.

Myself - I don't put much stock in Robot.txt at all for search engine optimization. At this point in 2008, Google is able to pretty much figure out things for itself. The only thing I've ever needed for is to block of certain parts of my site from being indexed. But that's about it.

Do not need a robot.txt to get google to index everything. You just need to submit a site map with all urls to google so that they will now what all your urls are!

There's no point at all using robots.txt or the robots meta tag to tell a bot to follow links, that's their default behaviour anyway, you should only be using it to stop bots if you need to.

The best way to ensure that bots can find all your site's pages is to design your navigation to be easy to follow. Just remember that bots can't execute java script or fill in forms so any content only reachable only by those methods will be inaccessible to bots.

(EDIT: lol, this thread IS old isn't it, never mind, posted on it now.)

The Excellent Info form JJMcClure should be used.


On a more techi basis the robots txt I generally use is,
User-Agent: *
Allow: /

Hope this helps!

I agree with Biz, especially about the spamming. They are getting more and more sticky about it and why risk it after all the work we do to get where we are. Good luck.