|
|||||||||
|
|||||||||
|
|||||||||
 |
|||||||||
| |
||
|
|
|
|
|||||||||
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
|
|
|
| |||||||||
 |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
SEO Chat Forums Sponsor:
|
|
|
|
#1
February 25th, 2003, 02:14 PM
|
|||
|
|||
|
Robots.TXT how to?
Hi,
I have been reading about exculding individual files using Rebots.txt and I am wondering how easy it is do for google? If I have a file in my ROOT folder called nospider.htm , how would I tell google not to spider it? I dont want google to follow the links on this page either. How should I format the robots.txt file? I want to keep nospider1.htm , nospider2.htm and nospider3.htm non indexed, and I dont want google to follow the links either. Thanks! Gary Whittle.
__________________
www.your-loan.net
|
|
#2
February 25th, 2003, 03:05 PM
|
||||
|
||||
|
The best way is to use the following for Googlebot ONLY:
Code:
User-agent: Googlebot Disallow: /nospider.htm Disallow: /nospider1.htm If however you wanted to block ALL ROBOTS, then use the following syntax: Code:
User-agent: * Disallow: /nospider.htm Disallow: /nospider1.htm Only the listed files would be blocked. The first example can also work for other robots so long as you specify the correct "User Agent".
__________________
Darrin J. Ward, a Professional SEO Consultant and Original Founder of SEO Chat (this site), Google Dance Tool & some other cool stuff! * Rankings Reporter - Track your Website's Keyword Rankings in Google & Yahoo. * ChatButton - Free AJAX Chatboxes to embed onto any Webpage - super-easy copy/paste setup!.
|
|
#3
February 25th, 2003, 03:39 PM
|
|||
|
|||
|
Also remember, there are some "rogue" robots that dont listen to a robots.txt file...
-Josh
|
|
#4
February 25th, 2003, 03:42 PM
|
||||
|
||||
|
Quote:
Correct, especially "Email Harvesters".
|
|
#5
February 25th, 2003, 04:14 PM
|
|||
|
|||
|
Hi again,
I am going to add a robots.txt file at some point. Another few questions  Is the robots.txt file private? Or can anyone come along and type www.xyz.com/robots.txt and see the contents? Should the file robots.txt be named ROBOTS.TXT , Robots.txt or robots.txt? Does the casing acutally matter with the filename? Should I include any code on the page(s) of the site which tell the search engine to look for robots.txt or do they all check for it automatically? Thanks, Gary.
|
|
#6
February 25th, 2003, 04:17 PM
|
||||
|
||||
|
1. They check for it automatically
2. The file IS CASE SENSITIVE. It must be all lower case "robots.txt" and it MUST be located in the root directory i.e. www.seochat.com/robots.txt 3. This file is NOT private and can be viewed by ANYBODY.
|
|
#8
February 25th, 2003, 07:23 PM
|
||||
|
||||
|
This is a useful resource for anyone wanting to learn more about using the 'robots.txt' file http://www.searchengineworld.com/robots/robots_tutorial.htm.
Well I use if from time to time anyway, especially the validator ( http://www.searchengineworld.com/cgi-bin/robotcheck.cgi ), which I would recommend to anyone making changes to their 'robots.txt' file as typos can cause chaos. Hope this helps.
|
|
| Viewing: SEO Chat Forums > Other > HTML Coding > Robots.TXT how to? |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
 |
|
|
|
|
|
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
Linear Mode
