Discuss 65.214.44.138 crashed my site twice in the Ask Optimization forum on SEO Chat. 65.214.44.138 crashed my site twice Ask, formerly Ask Jeeves, Optimization forum discussing Ask.com. Discuss Ask, including how to achieve a high rank and get traffic from the engine.
BuySellLinks.com is a new Text Link Advertising Brokerage by the founder of LinkAdage that offers high-end text links and a premier membership base. We screen all advertisers and publishers and their web pages to make sure we are offering a service that works for everyone. Our plug-in is super easy to install but if you are hands off, w e offer free installation.
Tired of dealing with shady text link buyers and/or sellers? Try BuySellLinks.
Receive the tools necessary to be the rock star of your field. Our 12-month program teaches you the evolving world of multi-channel marketing as well as the complex issues and opportunities found in the industry.
Posts: 134
Time spent in forums: 1 Day 20 h 27 m 16 sec
Reputation Power: 8
65.214.44.138 crashed my site twice
last night the ask bot came to my site and hit it about 600 times in about 10 seconds. It basically brought my site down. Has anyone experienced this or know why this would of happenned?
Posts: 217
Time spent in forums: 1 Day 6 h 55 m 10 sec
Reputation Power: 9
Quote:
Q: Can I control the crawler request rate from Teoma spider to my site?
A: Yes. We support the "Crawl-Delay" robots.txt directive. Using this directive you may specify the minimum delay between two successive requests from our spider to your site.
I think this is what I've been looking for.
q1. What should be the minimum delay?
q2. Is there any example of robots.txt for this particular directive?
q2.2. Is there any example of robots.txt for Ask/Teoma?
Posts: 217
Time spent in forums: 1 Day 6 h 55 m 10 sec
Reputation Power: 9
q1. robots.txt is not case sensitive right?
q2.
Quote:
User-agent: Teoma
Crawl-delay: 300
#value in seconds
300 seconds seems awfully long. Also bandwidth is not a problem to me, but how fast do Google and Yahoo crawl? I'd like to set the number so that Ask/Teomo crawls slightly more slowly that these two.
Posts: 6,063
Time spent in forums: 2 Months 6 Days 10 h 52 m 26 sec
Reputation Power: 24
Hehe - don't ask ME how quickly Gbot crawls. I'm not the average player here
When Gbot DOES come to visit, it hits my pages every three seconds on average. Sometimes more than one file a second.
But if you set the crawl delay to 1 second, you may end up with AJ/Teoma visiting your pages every second. What good will that do?
I'd devide my average site update interval by the number of my pages, and divide by two again, and use that. Unless that's greater than 300 (which in my case it probably is - LOL).
Anyway, bot frequencies can be found in your bare logs. Lots of info in there, but even more non-info....
Posts: 217
Time spent in forums: 1 Day 6 h 55 m 10 sec
Reputation Power: 9
Quote:
Originally Posted by Wit
When Gbot DOES come to visit, it hits my pages every three seconds on average. Sometimes more than one file a second.
But if you set the crawl delay to 1 second, you may end up with AJ/Teoma visiting your pages every second. What good will that do?
3 seconds sounds reasonable to me. Ask/Teoma did cause some problem, but Google never did. Neither did Yahoo. So, I'm guessing that, as long as I let Ask/Teoma to slow down to the point that it crawls slightly more slowly than Google and Yahoo, I think the problem will be resolved.
Posts: 224
Time spent in forums: 2 Days 1 h 17 m 27 sec
Reputation Power: 8
I had a crawler crash my site like that once. Turned out to be my fault, since I had an error in my .htaccess file that resulted in a recursive redirect. I would check your server logs and see if the same URL is being requested over and over, and if so then you might have a similar problem.
Posts: 1,303
Time spent in forums: 6 Days 4 h 19 m 18 sec
Reputation Power: 12
Irish...contact your hosting company, or read your hosting company's FAQs about where your log files are. If the host doesn't provide you with a log analysis tool, then you can always pick up an inexpensive one like bothunter.
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
A lot of bandwidth each month (10G+) but 1-10 visitors 
Linear Mode
