Lazy Spider.

My site has been deep crawled twice in the last 4 days. Each time the crawl has indexed about 16,000 pages (aprox 65% of total pages). Unfortunately it appears to have indexed the same pages on both occassions, ignoring two important areas of the site that have recently (last 6 weeks) been added. Is there anyway or anything I can use to gaurantee Google will deep crawl the whole site.

Apicius

No gaurantee's, Google doesn't always deep crawl an entire site.

Google's crawler was not on many of the sites I work on early yesterday for several hours so they may have had a problem and did a crawler restart, that may account for the double crawl of content on your site. I haven't checked what the bots collected after they came back.

Do you have a sitemap? How about really large pages (HTML not images) that may place links outside Googles reach? Seem to recall seeing that beyond a certain point in the page (100K), the spider doesn't go (and since all my pages are a lot smaller than that I can't say for certain).

Cheers,

Cheers,