Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat ForumsSearch Engine StrategiesSearch Engine Spiders

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
Lose your application development headaches. Start developing and deploying applications with Advantage Database Server today. Download a 30-day trial for Free!
  #1  
Old October 11th, 2007, 07:07 AM
Hanuman07 Hanuman07 is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Sep 2007
Posts: 6 Hanuman07 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 25 m 31 sec
Reputation Power: 0
Google spider me too much!

I have something that starts to look like a problem. I run a small forum with something like 10-20 topics per day. We have about 5,000 topics in total. Google eat almost 80% of my bandwidth meaning about 6Gb per month.

I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem?

I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference.

What I would like to do is to ask Googlebot to visit only one time per day...

The site is www.thailandsforum.se

Reply With Quote
  #2  
Old October 11th, 2007, 07:22 AM
JagNet's Avatar
JagNet JagNet is online now
Smoke me a kipper...
Click here for more information
 
Join Date: Aug 2007
Posts: 1,457 JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 3 Weeks 6 Days 6 h 44 m 42 sec
Reputation Power: 23
You can use the crawl-delay option within robots.txt to slow down googlebot:
Code:
User-agent: Googlebot
Crawl-delay: 10
where the delay is the number of seconds between pages crawled.

I'd also check the logs and confirm that it is really Google, and not a scraper or other undesirable pretending to be the googlebot.

Last edited by JagNet : October 11th, 2007 at 07:24 AM.

Reply With Quote
  #3  
Old October 12th, 2007, 01:30 AM
weblaunchphxx's Avatar
weblaunchphxx weblaunchphxx is offline
Permanently Banned
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Jul 2007
Posts: 375 weblaunchphxx User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 Days 20 m 12 sec
Warnings Level: 10
Number of bans: 1
Reputation Power: 0
Quote:
Originally Posted by Hanuman07
I have something that starts to look like a problem. I run a small forum with something like 10-20 topics per day. We have about 5,000 topics in total. Google eat almost 80% of my bandwidth meaning about 6Gb per month.

I am using a robots.txt file but it does not seem to do the trick. I want Google to spider but not the whole site every hour. Anyone here who have solved this problem?

I am using Googles webmastertool, but changing the speed which googlebot spider makes no difference.

What I would like to do is to ask Googlebot to visit only one time per day...

The site is www.thailandsforum.se

Check the source link:http://www.google.com/support/webmasters/bin/answer.py?answer=48620

Reply With Quote
  #4  
Old October 20th, 2007, 09:41 PM
Dart Shop's Avatar
Dart Shop Dart Shop is offline
Dart Shop
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Oct 2007
Location: Sydney Australia
Posts: 111 Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 1 h 55 m 42 sec
Reputation Power: 2
Is the Spider and the Bot the same thing? I get a daily visit via the bot so im assuming the bot and the spider are the same?

Reply With Quote
  #5  
Old March 12th, 2008, 07:05 PM
Amnesia Amnesia is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Mar 2008
Posts: 9 Amnesia User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 45 m 23 sec
Reputation Power: 0
Or simply move bot to banned category! Crapy, but works

Reply With Quote
  #6  
Old April 27th, 2008, 05:40 AM
Doodlebug's Avatar
Doodlebug Doodlebug is offline
Don't Panic!
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Apr 2007
Location: UK
Posts: 288 Doodlebug User rank is Corporal (100 - 500 Reputation Level)Doodlebug User rank is Corporal (100 - 500 Reputation Level)Doodlebug User rank is Corporal (100 - 500 Reputation Level)Doodlebug User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Week 2 Days 8 h 31 m 23 sec
Reputation Power: 4
Log in your Webmaster Console at Google and select a slower crawl rate.
__________________
Happy to be #1 Google Slurper

Reply With Quote
Reply

Viewing: SEO Chat ForumsSearch Engine StrategiesSearch Engine Spiders > Google spider me too much!


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

 Free IT White Papers!
 
Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.

 
Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.

 
Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.

 
Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five "checkpoints" for any implementation plan that will help make IP communications a successful proposition.

 
Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.

 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 2 hosted by Hostway