#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2008
    Posts
    84
    Rep Power
    7

    Google Crawling bot


    Is there any website you can go to to simulate googles bot crawling your site to see if you have any problems that prevent it from stopping early? I have 27,000 pages and it only crawls about 1500 a day and dont know if it runs into pages that stop it because there are no links for it to follow anymore if you know what I mean?
  2. #2
  3. SEO Consultant
    SEO Chat Genius (4000 - 4499 posts)

    Join Date
    Jul 2004
    Location
    Minneapolis, MN, USA
    Posts
    4,206
    Rep Power
    966
    You can use Google Webmaster Tools -> Health -> Fetch as Googlebot.

    The crawl rate can also be set, and it is sometimes possible to set a higher crawl rate but its usually best to let Google determine your crawl frequency. Its unlikely that you will have all 27K pages crawled in one day. Your indexing stats sound normal.

    Google will give you crawl errors in webmaster tools.

    You can also check your server logs.

    Everything sounds normal here though.

    Comments on this post

    • SEO_AM agrees : We have several hundred thousand pages. Google averages about 6,000 pages a day.
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2013
    Posts
    4
    Rep Power
    0
    if you are getting crawl problem then you have to use google webmaster tool for good result.

    Thanks
    Anmol
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2008
    Posts
    84
    Rep Power
    7
    Thanks for the reply, the last question I have is in google errors it says i have 6,000 errors but they are pages I deleted a long time ago that it cannot find.
  8. #5
  9. No Profile Picture
    Banned
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2013
    Location
    Ahmedabad
    Posts
    13
    Rep Power
    0
    Webmaster Tools -> Health -> Fetch as Googlebot is best way

Similar Threads

  1. Google Crawling
    By sepoto in forum Google Optimization
    Replies: 4
    Last Post: Aug 22nd, 2011, 09:10 PM
  2. Difference between Google fresh crawling and main crawling
    By pbsnarayanan in forum Google Optimization
    Replies: 1
    Last Post: May 14th, 2010, 12:29 PM
  3. Slow in google crawling?
    By joer80 in forum Google Optimization
    Replies: 3
    Last Post: Nov 14th, 2005, 12:19 PM
  4. Google crawling SEOChat?
    By wrkalot in forum Google Optimization
    Replies: 3
    Last Post: Jan 22nd, 2004, 12:21 AM

IMN logo majestic logo threadwatch logo seochat tools logo