|
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
|
|
Lose your application development headaches. Start developing and deploying applications with Advantage Database Server today. Download a 30-day trial for Free!
|
|
#1
|
|||
|
|||
|
Google spider me too much!
I have something that starts to look like a problem. I run a small forum with something like 10-20 topics per day. We have about 5,000 topics in total. Google eat almost 80% of my bandwidth meaning about 6Gb per month.
I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem? I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference. What I would like to do is to ask Googlebot to visit only one time per day... The site is www.thailandsforum.se |
|
#2
|
||||
|
||||
|
You can use the crawl-delay option within robots.txt to slow down googlebot:
Code:
User-agent: Googlebot Crawl-delay: 10 I'd also check the logs and confirm that it is really Google, and not a scraper or other undesirable pretending to be the googlebot. Last edited by JagNet : October 11th, 2007 at 07:24 AM. |
|
#3
|
||||
|
||||
|
Quote:
Check the source link:http://www.google.com/support/webmasters/bin/answer.py?answer=48620 |
|
#4
|
||||
|
||||
|
Is the Spider and the Bot the same thing? I get a daily visit via the bot so im assuming the bot and the spider are the same?
|
|
#5
|
|||
|
|||
|
Or simply move bot to banned category! Crapy, but works
|
|
#6
|
||||
|
||||
|
Log in your Webmaster Console at Google and select a slower crawl rate.
__________________
Happy to be #1 Google Slurper |
![]() |
| Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Google spider me too much! |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|
|
|
|
|