#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2003
    Location
    Edinburgh, Scotland, UK
    Posts
    61
    Rep Power
    17

    Robots.TXT how to?


    Hi,

    I have been reading about exculding individual files using Rebots.txt and I am wondering how easy it is do for google?

    If I have a file in my ROOT folder called nospider.htm , how would I tell google not to spider it?

    I dont want google to follow the links on this page either.

    How should I format the robots.txt file? I want to keep nospider1.htm , nospider2.htm and nospider3.htm non indexed, and I dont want google to follow the links either.

    Thanks!

    Gary Whittle.
  2. #2
  3. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jan 2003
    Posts
    475
    Rep Power
    17
    Also remember, there are some "rogue" robots that dont listen to a robots.txt file...

    -Josh
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2003
    Location
    Edinburgh, Scotland, UK
    Posts
    61
    Rep Power
    17
    Hi again,

    I am going to add a robots.txt file at some point. Another few questions

    Is the robots.txt file private? Or can anyone come along and type www.xyz.com/robots.txt and see the contents?

    Should the file robots.txt be named ROBOTS.TXT , Robots.txt or robots.txt? Does the casing acutally matter with the filename?

    Should I include any code on the page(s) of the site which tell the search engine to look for robots.txt or do they all check for it automatically?

    Thanks,

    Gary.
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2003
    Location
    Edinburgh, Scotland, UK
    Posts
    61
    Rep Power
    17
    Thanks Darrin,

    Your a great help. Your work is most appreciated,

    Gary.
  8. #5
  9. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2003
    Location
    London
    Posts
    122
    Rep Power
    17
    This is a useful resource for anyone wanting to learn more about using the 'robots.txt' file http://www.searchengineworld.com/robots/robots_tutorial.htm.

    Well I use if from time to time anyway, especially the validator ( http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ), which I would recommend to anyone making changes to their 'robots.txt' file as typos can cause chaos.

    Hope this helps.

Similar Threads

  1. robots.txt file
    By clasione in forum Web Design, Coding and Programming
    Replies: 9
    Last Post: May 23rd, 2004, 10:07 AM
  2. robots.txt file
    By atlinx in forum Google Optimization
    Replies: 7
    Last Post: Jun 9th, 2003, 12:42 PM
  3. If there is no robots.txt file...
    By gleeloyd in forum Google Optimization
    Replies: 11
    Last Post: May 7th, 2003, 04:01 PM
  4. styles.css excluded in robots.txt
    By izzetz in forum Web Design, Coding and Programming
    Replies: 9
    Last Post: Mar 12th, 2003, 05:29 AM

IMN logo majestic logo threadwatch logo seochat tools logo