Altavista emailed me about Robots.txt

For those that may find this interesting........


Looks like you need to upload a robots txt file. Here is some information regarding the issue.
Your server needs to respond with a 404 (page cannot be found) or a 200 OK http status with the proper robots.txt file uploaded to the root server. If you have your site set up to respond with a custom 404 page, it has to respond with a 404 http status instead of a 200 OK.

Monday through Friday, AltaVista sends an indexing spider robot program(Scooter) to crawl the html source code of your URL (WebPage address) to collect data. But before it visits your page, the spider checks to see if it's allowed to visit by looking for a robots.txt file in your root directory that might disallow indexing anywhere on your website.

A robots.txt file may list private areas of your website you would not want a search engine to index (for example: members only areas or customer data). If the spider does not find a robots.txt file or finds nothing on the file that disallows AltaVista's spider to index your page, it will request your URL from your server.

If your server responds incorrectly with invalid robots protocol, it will stop AltaVista from indexing or listing your page, and infoSpider URL Status will report an error.

ROBOTS.TXT ERRORS THAT CAUSE URL INDEXING PROBLEMS

1) 302 Error: Server auto redirecting URL location of /robots.txt

2) 403 error forbidding access

3) robots.txt file content Disallows indexing section of site where

submitted URL is located

4) robots.txt syntax errors:

Variable * used in the Disallow line

User-agent with wrong case letter A: "User-Agent" is wrong:

User-agent is correct

5) Submitted URL has meta robots noindex tag in place

ROBOTS.TXT RESOURCES

Web Server Admin's Guide to Robots.txt:

http://www.robotstxt.org/wc/exclusion-admin.html

All About Robot Spiders: http://www.searchtools.com/robots/

Robots.txt Validator:

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Robots.txt Generator: http://www.submitcorner.com/Tools/Robots/

AltaVista Express Inclusion assures that your WebPage URL address gets

revisited each day by its spider in an to attempt to re-index. Please remove anything that would send AltaVista's indexing spider an error code and then the next indexing attempt will report success.

Please consider the following links to AltaVista's own help pages which

provide useful insights on How AltaVista Works:

Frequently Asked Questions by Webmasters :

http://www.altavista.com/help/search/faq_web

Webmaster Search: http://www.altavista.com/help/search/help_wms

AltaVista's Submission Policy: http://addurl.altavista.com/addurl/new#rls

AltaVista Contact Form: http://www.altavista.com/help/contact/intro_help