Google Crawling bot

[Cove]

Is there any website you can go to to simulate googles bot crawling your site to see if you have any problems that prevent it from stopping early? I have 27,000 pages and it only crawls about 1500 a day and dont know if it runs into pages that stop it because there are no links for it to follow anymore if you know what I mean?

[tstolber]

You can use Google Webmaster Tools -> Health -> Fetch as Googlebot.

The crawl rate can also be set, and it is sometimes possible to set a higher crawl rate but its usually best to let Google determine your crawl frequency. Its unlikely that you will have all 27K pages crawled in one day. Your indexing stats sound normal.

Google will give you crawl errors in webmaster tools.

You can also check your server logs.

Everything sounds normal here though.

[imexperttools]

1,500 pages per day is totally normal. It is because your site has low traffic that is why Google do not put high priority when crawling your site. I have several sites, popular sites got up to 10,000 of pages crawled everyday while less popular ones only have around 2,000 - 3,000 pages crawled daily. So it is totally normal.

[yoursanmol]

if you are getting crawl problem then you have to use google webmaster tool for good result.

Thanks
Anmol

[Cove]

Thanks for the reply, the last question I have is in google errors it says i have 6,000 errors but they are pages I deleted a long time ago that it cannot find.

[david0019]

Webmaster Tools -> Health -> Fetch as Googlebot is best way