Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Spiders
Reload this Page

Spider/Robot Names

Discuss Spider/Robot Names in the Search Engine Spiders forum on SEO Chat.
Spider/Robot Names Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old February 13th, 2003, 05:27 PM
cuttsgroup's Avatar
cuttsgroup cuttsgroup is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2003
Location: PA, USA
Posts: 14 cuttsgroup User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 11 m 11 sec
Reputation Power: 0
Spider/Robot Names
What are the spider/robot names I should be look for in my log files?

I think I have seen here that googlebot is Google - Is that the only Google one? He's been on my site for 4-days.

1) What are the other ones I should be for?
2) When and/or how often should I be looking for them?
Reply With Quote
  #2  
Old February 13th, 2003, 06:15 PM
Darrin Ward's Avatar
Darrin Ward Darrin Ward is offline
Founder, SEOChat.com :)
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Dec 2002
Location: Miami, Florida
Posts: 1,444 Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 18 h 38 m 35 sec
Reputation Power: 15
There are loads and loads of them

Also look out for "Slurp", which is the Inktomi spider.

"ZyBorg" is WiseNut.
"Fast-WebCrawler" is Fast.
"Scooter" is Altavista.

etc....

A complete list anyone?
__________________
Darrin J. Ward, the Original Founder of SEO Chat (this site), Google Dance Tool & some other cool stuff! Read my: Blog or Twitter.

* Rankings Reporter - Free PHP Script to Track your Website's Keyword Rankings in Google & Yahoo.
Reply With Quote
  #3  
Old February 13th, 2003, 06:44 PM
cuttsgroup's Avatar
cuttsgroup cuttsgroup is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2003
Location: PA, USA
Posts: 14 cuttsgroup User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 11 m 11 sec
Reputation Power: 0
slurp... that's the other one that on my site with googlebot all this week.

Is there any way to know how long or how deep they will crawl on a given adventure?
Reply With Quote
  #4  
Old February 14th, 2003, 05:39 AM
SpiralCris's Avatar
SpiralCris SpiralCris is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2003
Location: NC
Posts: 120 SpiralCris User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 7
Quote:
Originally posted by "cuttsgroup"

slurp... that's the other one that on my site with googlebot all this week.



"slurp" is Inktomi
__________________
www.spiralxdesign.com
Reply With Quote
  #5  
Old February 14th, 2003, 10:55 AM
Darrin Ward's Avatar
Darrin Ward Darrin Ward is offline
Founder, SEOChat.com :)
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Dec 2002
Location: Miami, Florida
Posts: 1,444 Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level)Darrin Ward User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 18 h 38 m 35 sec
Reputation Power: 15
Quote:
Originally posted by "cuttsgroup"

Is there any way to know how long or how deep they will crawl on a given adventure?

Not really, just keep an eye on them and hopefully they will go as deep as possible. If you have a good few inbound links and also a good text based navigation menus, then they should crawl everything (so long as it's not all dynamic).
Reply With Quote
  #6  
Old February 15th, 2003, 05:20 AM
SpiralCris's Avatar
SpiralCris SpiralCris is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2003
Location: NC
Posts: 120 SpiralCris User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 7
Here's one that started spidering me a few month's ago:

Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html ( info@szukacz.pl )

Aparently it's only for Polish speaking sites but it visits me daily

-cris
Reply With Quote
  #7  
Old July 15th, 2003, 11:39 PM
simple-biz's Avatar
simple-biz simple-biz is offline
simple-biz.com
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Feb 2003
Location: Thailand, Phuket
Posts: 223 simple-biz User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 5 h 48 m 27 sec
Reputation Power: 7
Send a message via Yahoo to simple-biz Send a message via Skype to simple-biz
An extensive list of User-Agents/Browsers, Web Spiders, Robots is at:

http://www.pgts.com.au/pgtsj/pgtsj0208d.html
and

http://www.netsys.com/cgi-bin/display_article.cgi?1193
Reply With Quote
  #8  
Old July 20th, 2005, 04:37 PM
SleepingSandMan SleepingSandMan is offline
Permanently Banned
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jul 2005
Posts: 10 SleepingSandMan User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 56 m 31 sec
Warnings Level: 10
Number of bans: 1
Reputation Power: 0
that's cool ponsensibility loomer spun shorewark
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Spider/Robot Names

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump



 Free IT White Papers!
 
How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.
Request Your Free Technology Downloads!
 
Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.
Request Your Free Technology Downloads!
 
Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.
Request Your Free Technology Downloads!
 
Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.
Request Your Free Technology Downloads!
 
Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.
Request Your Free Technology Downloads!
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2009 by Developer Shed. All rights reserved. DS Cluster 6 hosted by Hostway. 
Stay green...Green IT