#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jul 2003
    Location
    California, USA
    Posts
    4
    Rep Power
    0

    Question What is best robot txt coding to insure spiders index all pages?


    Hello,

    I a SEM newbie and wanted to know what is the absolute best robot txt coding one should use to insure Google and the major SE's spiders index all of my sites pages?

    I've seen the following used:

    <meta content="index,follow" name="robots">

    <meta name="robots" content="all">

    <meta content="all" name="robots">

    Which one is best? Or are they all equally good and does it even matter which one I use?

    Next, is there an even better one than these three? If so, what is it?

    Lastly, is there any other key coding/files I need to insure spiders index all of my sites page? If so, please list with enought details on how to implement for us newbies!

    Many thanks,

    Randall
  2. #2
  3. simple-biz.com
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2003
    Location
    Thailand, Phuket
    Posts
    223
    Rep Power
    16
    That's the one I use,
    <meta name="robots" content="index,follow,all">

    You also need to use the robots.txt file in your root directory to keep some robots like Scooter (Altavista) wants to see it there.

    Open your notepad in Windows or emacs in Linux and paste this code:

    User-agent: *
    Disallow:

    and save it as "robots.txt" when you upload it chmod 755 it.
    Last edited by simple-biz; Jul 16th, 2003 at 12:32 AM.
  4. #3
  5. Modsky
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2003
    Location
    canada
    Posts
    1,793
    Rep Power
    18
    Hey biz, does that code matter for GGOOGGLLEE?
  6. #4
  7. simple-biz.com
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2003
    Location
    Thailand, Phuket
    Posts
    223
    Rep Power
    16

    Lightbulb


    As far as I know having the robots tags and robots.txt doesn't hurt, BUT if you are using Google AdSense on any of your pages, then Google requires to use a special User-Agent: type to be used in the robots.txt file.

    An important reminder here: Please don't try to spam in that .txt file, as most of the idiot spammers tried to do long ago.

    Robots only read that text file to find out their rights to, which folders and pages they are allowed to index.

    Nothing else will be read than the

    User-agent:
    Disallow:

    lines by the robots.

    I am saying this, because in the past years I have seen some robot.txt files full of keywords. Also Firestarter has found one last week and posted it under the Strange robots.txt post on this board.
    Last edited by simple-biz; Jul 16th, 2003 at 12:08 AM.
  8. #5
  9. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2007
    Location
    Rochester, NY
    Posts
    269
    Rep Power
    80
    This thread is like 100 years old... But it is really highly ranked in Google, so people are probably still coming here for search engine optimization help...

    In regards to Robot.txt. There have been MANY MANY updates to the Google algo since this post was put out here in 2003. You should REALLY look for more recent info before trusting this old info.

    Myself - I don't put much stock in Robot.txt at all for search engine optimization. At this point in 2008, Google is able to pretty much figure out things for itself. The only thing I've ever needed for is to block of certain parts of my site from being indexed. But that's about it.
  10. #6
  11. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2008
    Location
    Melbourne Australia
    Posts
    103
    Rep Power
    18
    Do not need a robot.txt to get google to index everything. You just need to submit a site map with all urls to google so that they will now what all your urls are!
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2007
    Posts
    325
    Rep Power
    13
    There's no point at all using robots.txt or the robots meta tag to tell a bot to follow links, that's their default behaviour anyway, you should only be using it to stop bots if you need to.

    The best way to ensure that bots can find all your site's pages is to design your navigation to be easy to follow. Just remember that bots can't execute java script or fill in forms so any content only reachable only by those methods will be inaccessible to bots.

    (EDIT: lol, this thread IS old isn't it, never mind, posted on it now.)
  14. #8
  15. Joe
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2008
    Location
    Belfast, Northern Ireland
    Posts
    343
    Rep Power
    25
    The Excellent Info form JJMcClure should be used.


    On a more techi basis the robots txt I generally use is,
    User-Agent: *
    Allow: /

    Hope this helps!
  16. #9
  17. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Dec 2008
    Posts
    36
    Rep Power
    10
    I agree with Biz, especially about the spamming. They are getting more and more sticky about it and why risk it after all the work we do to get where we are. Good luck.

Similar Threads

  1. google.com does but google.co.uk doesn't index pages
    By neil_rutherford in forum Google Optimization
    Replies: 6
    Last Post: Dec 16th, 2003, 02:06 PM
  2. My index page is not in the index... but my internal pages are!
    By mariobox in forum Google Optimization
    Replies: 0
    Last Post: Sep 15th, 2003, 04:37 PM
  3. does changing internal pages affect your index serp?
    By whyme in forum Google Optimization
    Replies: 9
    Last Post: Sep 7th, 2003, 08:40 PM
  4. Loss of pages in the new index
    By tonyfelice in forum Google Optimization
    Replies: 2
    Last Post: May 11th, 2003, 10:56 AM

IMN logo majestic logo threadwatch logo seochat tools logo