Great community. Great ideas.
Welcome to SEOChat, a community dedicated to helping beginners and professionals alike in improving their Search Engine Optimization knowledge. Sign up today to gain access to the combined insight of tens of thousands of members.
May 16th, 2013
Feb 19th, 2013, 06:38 PM
Google Crawling bot
Is there any website you can go to to simulate googles bot crawling your site to see if you have any problems that prevent it from stopping early? I have 27,000 pages and it only crawls about 1500 a day and dont know if it runs into pages that stop it because there are no links for it to follow anymore if you know what I mean?
Feb 19th, 2013, 08:20 PM
You can use Google Webmaster Tools -> Health -> Fetch as Googlebot.
The crawl rate can also be set, and it is sometimes possible to set a higher crawl rate but its usually best to let Google determine your crawl frequency. Its unlikely that you will have all 27K pages crawled in one day. Your indexing stats sound normal.
Google will give you crawl errors in webmaster tools.
You can also check your server logs.
Everything sounds normal here though.
Feb 19th, 2013, 09:31 PM
1,500 pages per day is totally normal. It is because your site has low traffic that is why Google do not put high priority when crawling your site. I have several sites, popular sites got up to 10,000 of pages crawled everyday while less popular ones only have around 2,000 - 3,000 pages crawled daily. So it is totally normal.
Feb 20th, 2013, 04:36 AM
if you are getting crawl problem then you have to use google webmaster tool for good result.
Feb 20th, 2013, 11:57 AM
Thanks for the reply, the last question I have is in google errors it says i have 6,000 errors but they are pages I deleted a long time ago that it cannot find.
Feb 26th, 2013, 06:45 AM
Webmaster Tools -> Health -> Fetch as Googlebot is best way
By sepoto in forum Google Optimization
Last Post: Aug 22nd, 2011, 09:10 PM
By pbsnarayanan in forum Google Optimization
Last Post: May 14th, 2010, 12:29 PM
By joer80 in forum Google Optimization
Last Post: Nov 14th, 2005, 12:19 PM
By wrkalot in forum Google Optimization
Last Post: Jan 22nd, 2004, 12:21 AM