The Search Engines Lie About Index Sizes?

Fantomaster pointed me to a fantastic blog belonging to a professor of IR research in France - Jean Véronis. Sadly I cannot read and comprehend French, but he has several English entries on the subject of search engine index sizes:

• Google - aixtal.blogspot.com/2005/02/web-googles-missing-pages-mystery.html
• Yahoo! - aixtal.blogspot.com/2005/03/web-yahoo-indexes-more-pages-than.html
• MSN - aixtal.blogspot.com/2005/02/web-msn-cheating-too.html

The findings are very impressive and his research appears solid to me.

Basically, his findings are that MSN and Google are both inflating their index size, that you can find a "truer" number of results at Google by typing the term you're seeking into the engine twice (i.e. search for "string string" instead of "string"), and that Yahoo! probably has the largest index size of major search engines, along with the highest level of honesty.

If any SEOChatters are versed in Francais, please bring back any other tidbits of information you find at his site - these posts alone are enough to make me sign up for a French class.

Interesting information. Makes one wonder if Google and Microsoft (and even NBC) will try to keep this guy's information hush-hush fearing backlash in their stocks.

I can see it now:

"Billions and billions indexed."

Thanks McDonalds.

ROFLOL!!! More like, McGoogle's?

It's funny, but I just read another report that SEs UNDERestimate their index sizes.

Not that I believe that (because what would be the point?), but still...

LOL. Well, I guess we can therefore follow the President's theory (which he used for his tax plan)...

Some say the index is overestimated, some say it's underestimated... therefore it's probably about right.

All kidding aside, to my knowledge, the only real benefit to knowing the exact size of the index is for that SE's advertising/marketing campaign and stock value. For website SEO, the focus is on those sites which rank well on SERPs for keyword/keyword phrases. But the actual index size is interesting to know if own that SE's stock.

Actually index size is very useful for a lot of the calculations done with my tools and for purposes of identifying keywords, calculating term weight on a page, etc.

Perhaps this is why Google & MSN mis-represent...

If you just check some of your bigger sites you will notice patterns.

I have a 10k pages site that is showing 23k pages.
It first happened just before google doubled their index from 4 billiom to 8 billion about 6 months ago.
It started indexing all different formats of sites,, pdf, etc and also just showing silly amounts of untrue pages.

Nice find Rand,

really interesting blog, and yes his research does look good, he documents all his experiemnts really well, so you can see exactly what's going on there.

I particularly like the "trustrank" lots of noise for nothing: aixtal.blogspot.com/2005/05/google-trustrank-beaucoup-de-bruit.html

He says the technical report from Stanford is also co-authored by jan Pederson. Obviously Yahoo! wouldn't want Google to have a part of this for a start. He says that it's possible that Google tried to get in there quick by applying for patent.

He also points out that the date that it was submitted the 16 septembre 2003.

He also talks about the search engine meeting in Boston. His blog is mostly though about the constitutions translation. However his work written in english is really relevant.

It's interesting to read those articles. I don't have time to pick at it (you know me), but I'll have a look later. Anyway, looks nice!