Inktomi Indexing...

I've never had Inktomi spider me for more than 20 hits per day... Last night I had 1,477 hits from them. Seems odd... I thought Inktomi was a paid submission engine (which I never paid), or is it just the top rankings are paid?

- Shawn

Shawn, thats good news buddy. They are a paid engine, but Slurp also spiders sites like the Googlebot if it likes your content enough. It probably means that you are considered some of the top documents in the Ink database. It also has to do with the amount of time your site has been online, which as I know has been many years. The longer the better, and older sites are usually in the index without having to have paid. Consider it attributed to the site age, new backlinks, increased popularity in the results themselves, and overall good HIGH value content.

Inktomi comes to my site daily and hits like 20+ of my pages. I never paid them a penny.

I don't have 1500 pages but it gets a majority of my pages over a few day span.

Definitaly an age thing. I get around 1,500 hits a day from Ink. In January I had around 25,000 visits by Ink but I think thats because it keeps requesting a new session id for every page. Never paid them a penny and have over 5,000 pages in the index. Site is about 6 years old or so.

Pk, 6 years huh? Yeah that would plenty of time for Slurp to find a good portion of your site.
I have been trying to gauge how long it actually takes in terms of site age in order to get picked up permantantly by Slurp. I haven't seen anything under 2 years that gets spidered on a frequent basis unless it gets linked heavily by sites that spidered by Slurp. But again it all depends. The spectrum is wide.
As any interesting idea to follow up. If you get spidered by Inktomi on regular basis and have NOT PAID to be included into Inktomi database - how old is your site?

how do you check how many of your pages are included in INK ?

Actually upon a further look it's 9 years. It seems that I'm in there for good. Always get around 1500 daily hits. Is it normal to have two Slurp bots indexing the site everyday??? I have always had two bots.

I'll be doing Inktomi research soon so I'll be around your part of the woods more often Ben

I posted this about a month ago. This is how I do it



Read up in this thread for more ideas http://forums.seochat.com/t6382/s.html

my new forums is only a few weeks old and ink has visited twice in as many days, never paid for this site to be included but i have paid for other sites in the past

Well, I must officially be on their "I like you list"... As I had another 1,480 hits from Inktomi yesterday. Still not as much as Googlebot (3,446 hits from them yesterday), but sometimes I think Googlebot visits more often than it needs.

- Shawn

This is going to be another dumb question I can see it LOL.

How do you know which bots/spiders are visiting your site? Is it through your log files, or web stats, or is there a specific tool that does this for you?

At the lowest level, it's through your log files... the robots will have a user agent which is logged (in most cases) to your web logs...

A few examples:

Googlebot:
www.digitalpoint.com 64.68.82.168 - - [02/Feb/2004:00:00:45 -0800] "GET /products/isp/screenshots.html HTTP/1.0" 200 13481 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Googlebot (because of AdSense):
www.digitalpoint.com 64.68.87.69 - - [03/Feb/2004:04:36:04 -0800] "GET /lists/75016.html HTTP/1.0" 200 13193 "-" "Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)"

Inktomi:
www.digitalpoint.com 66.196.90.163 - - [03/Feb/2004:08:10:40 -0800] "GET /lists/19800.html HTTP/1.0" 200 14773 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)"


- Shawn

I see..... msnbot/0.* in my logs. I didn't know msn did spidering. I thought that Inktomi did that for them.

Everyone has their own bot... the msnbot I think is going to be for Microsoft's upcoming search engine they want to use to compete with Google. Just little hits here and there from msnbot so far...

Why pay Inktomi (Yahoo) when they can go it themselves?

- Shawn

Thanks Shawn! I've seen the raw log files, and can see Google bot and Inktomi slurp

Is there any free software out there which will analysise your raw log files, because it'll take me ages to go through this file!