#1
  1. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2015
    Location
    Warrington, UK
    Posts
    168
    Rep Power
    249

    Spike in our crawl stats


    Hi Everyone,

    I have, in the past, noticed large spikes in the Crawl Stats in WMT. This doesn't seem to affect anytin at all but I wondered how mornal it was to occasionally see these spikes.

    Average Crawl is around 13,000 pages/day
    The spike is up at 91,000 pages in one day
    Total pages indexed in the Index Status is 13,279

    Should we be concerned at all?

    HJJ
  2. #2
  3. Super Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,598
    Rep Power
    2896
    Have you seen a rise in indexed pages?

    Also have you added a large amount of content? Do you have any risk of duplicate content on the site?
    Always happy to help in private message. | Learn about my latest project Can You Get Automated Pages To Rank In Google?
    Useful Links: Useful SEO Tools | GTmetrix | FileOptimizer | Cloudflare
  4. #3
  5. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    16,979
    Rep Power
    0
    I would believe it is normal but how many pages do you have and like Thomas noted how many are indexed?
  6. #4
  7. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2015
    Location
    Warrington, UK
    Posts
    168
    Rep Power
    249
    Total pages indexed is 13,279

    We do run risk of duplicate as we have pages for each car model and spec, therefore there are hundreds ( maybe thousands ) of pages that are all very similar but each page is for a different car.

    We can't use canonical as even though these pages could be seen as duplicated, they do hold their own purpose.

    We do have canonical in place on other article pages for times when a user visits from facebook at the URL slug has been amended to indicate they are a facebook referral.

    Our index is ever increasing as cars come and go and I implement 404's on a regular basis - is there a good way of combating this?

    Thanks

    HJJ
  8. #5
  9. Super Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,598
    Rep Power
    2896
    Originally Posted by HappyJackJunior
    Total pages indexed is 13,279

    We do run risk of duplicate as we have pages for each car model and spec, therefore there are hundreds ( maybe thousands ) of pages that are all very similar but each page is for a different car.

    We can't use canonical as even though these pages could be seen as duplicated, they do hold their own purpose.

    We do have canonical in place on other article pages for times when a user visits from facebook at the URL slug has been amended to indicate they are a facebook referral.

    Our index is ever increasing as cars come and go and I implement 404's on a regular basis - is there a good way of combating this?

    Thanks

    HJJ
    So the total pages you have created is?
  10. #6
  11. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    16,979
    Rep Power
    0
    How many pages does your domain have?
  12. #7
  13. Super Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,598
    Rep Power
    2896
    Any news on this HappyJackJunior?
  14. #8
  15. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2015
    Location
    Warrington, UK
    Posts
    168
    Rep Power
    249
    Apologies for the delay...

    I've counted up 28,244 URLs in our sitemap.

    Average Crawl is around 13,000 pages/day
    The spike is up at 91,000 pages in one day
    Total pages indexed in the Index Status is 13,279

    Sitemap: https://www.carkeys.co.uk/sitemap-index.xml
    Last edited by HappyJackJunior; Dec 22nd, 2015 at 08:38 AM. Reason: Addition of sitemap
  16. #9
  17. Super Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,598
    Rep Power
    2896
    Originally Posted by HappyJackJunior
    Apologies for the delay...

    I've counted up 28,244 URLs in our sitemap.

    Average Crawl is around 13,000 pages/day
    The spike is up at 91,000 pages in one day
    Total pages indexed in the Index Status is 13,279

    Sitemap: https://www.carkeys.co.uk/sitemap-index.xml
    I've just done a full crawl of your website with ScreamingFrog and i managed to get 24,532 pages (foolishly had some excludes from a previous crawl, will do another one tomorrow to see what I get that way) with 1173 redirects (change your pinterest share url from and that'll remove 220 of them https://pinterest.com/pin/create/but...n_Engine_1.jpg to https://wwwpinterest.com/pin/create/...n_Engine_1.jpg)

    You've also got a few broken images here: https://www.carkeys.co.uk/car-review...a/prius-estate


    Back to the topic, you've not seen any issues in the past with a loss of rankings or any fluctuations? 91,000 pages would be google crawling your website four times in a single day, which I don't believe it's something that would be done.
  18. #10
  19. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2015
    Location
    Warrington, UK
    Posts
    168
    Rep Power
    249
    We haven't seen any flux. I have noticed that we aren't disallowing /search in robots.txt however - could this be a possible reason and the search queries are being indexed? Moz is returning these search query URL's as duplicate, so they are able to be crawled currently.

    I will get the pinterest issue sorted with the dev guys - thanks for pointing that out.

    We naturally will have a lot of redirects as a result of the type of business we are in, with ever changing models, cars etc.

    The crawling as settled now, just every so often, it spikes in at around 90K, normal is around 7-8K/day

Similar Threads

  1. Are these Crawl Stats Pretty bad??
    By Halen_zhau in forum Google Optimization
    Replies: 2
    Last Post: Jun 7th, 2011, 08:34 AM
  2. Crawl Stats: Google Webmaster Tools
    By anandmistry in forum Google Optimization
    Replies: 12
    Last Post: Mar 29th, 2010, 07:42 AM
  3. Something wrong in google crawl stats?
    By feras80 in forum Google Optimization
    Replies: 2
    Last Post: Aug 31st, 2008, 12:32 AM
  4. Google PageRank -- Crawl Stats
    By narsaw in forum Google Optimization
    Replies: 1
    Last Post: Jul 5th, 2007, 01:31 PM
  5. Google Sitemaps - no crawl stats/page analysis
    By lkmbrd in forum Google Optimization
    Replies: 2
    Last Post: Mar 19th, 2006, 10:00 AM

IMN logo majestic logo threadwatch logo seochat tools logo