|
|||||||||
|
|||||||||
|
|||||||||
 |
|||||||||
| |
||
|
|
|
|
|||||||||
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
|
|
|
| |||||||||
 |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
SEO Chat Forums Sponsor:
|
|
|
|
#1
November 11th, 2003, 10:41 AM
|
|||
|
|||
Robot Clueless
I have a few questions about robots.
I just loaded a robot.txt to my root to prevent the errors from my logs: User-agent: * Disallow: I realize that allows everything, but I'm just getting started here. Q1. Do I need both robots.txt and Robots Meta tags on my pages? Q2. I want to Disallow nasty robots. Is there a list of bad bots that I can use to add to my robots.txt file? (I think I saw a list somewhere here in the forums). Q3. Will a robots.txt file and/or Robots Meta tags help the good bots to index me faster? TIA for your help.
|
|
#2
November 11th, 2003, 05:58 PM
|
|||
|
|||
|
You don't NEED a robots.txt file, but it sure helps. As for Q2, most "bad" robots wont obey, but if you go to www.webmasterworld.com/robots.txt you will get a pretty good list of bad bots and some that aren't bad, just annoying like Google Image bot and the Way BAck Machine. Try googling "robot user agents" to find out more.
|
|
| Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Optimization > Robot Clueless |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
 |
|
|
|
|
|
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
Linear Mode
