#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Mar 2014
    Posts
    20
    Rep Power
    0

    Screaming Frog Doesn't Crawl All Pages?


    My Screaming Frog (spider tool) report did not contain a certain page on my site, and I'm thinking how many other pages did it not crawl? I thought the point of it was to crawl ALL pages, and gives you a status on the tags and such?
    Here is the page that I noticed it did not crawl. phillongdenver.com/exotic-new-inventory/vehicle-details.htm
    Any thoughts?
    Last edited by markroberts; Jan 25th, 2016 at 11:58 AM.
  2. #2
  3. Dinosaur
    SEO Chat Hero (2000 - 2499 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    2,079
    Rep Power
    3325
    Is it linked to from anywhere on your site ?
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Mar 2014
    Posts
    20
    Rep Power
    0
    I honestly don't know... Does Mozbar tell me that information? Is that possibly why it SF didn't crawl it? Because that's how it works: It crawls based on link pages? Thanks for your time!
  6. #4
  7. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    17,294
    Rep Power
    0
    Ya... Why isn't it indexed in Google?
  8. #5
  9. Dinosaur
    SEO Chat Hero (2000 - 2499 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    2,079
    Rep Power
    3325
    Screaming frog takes your starting page and follows links to find more, if there is no links then of course it wont find it.
    If there was a link in the past then google may know about the page and keep checking its still there but running SF as of today it wont have a clue as it does not store history.
  10. #6
  11. Super Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    May 2015
    Location
    United Kingdom
    Posts
    1,120
    Rep Power
    1610
    Is it being blocked by your robots.txt? I'll have a look at this tomorrow
    Always happy to help in private message. | Freelance SEO https://thomasharvey.me
    Useful Links: Useful SEO Tools | Wordpress Hosting | Magento Hosting | FileOptimizer | Cloudflare
  12. #7
  13. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2009
    Location
    Detroit, MI USA
    Posts
    113
    Rep Power
    305
    As far as I can tell, it doesn't seem to be included on your sitemap. I'd double check that for sure, and if you need to add it there, do so, then resubmit that to Google/Bing/etc
  14. #8
  15. I love SEOChat.com
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2016
    Posts
    39
    Rep Power
    29
    Here below are some broken link checkers you might wanna try, why just stick to screaming frog?

    xenu
    dr link checker (new and good)
    anybrowser dot com
    there is a wordpress plugin too (for WP blogs)
    W3C checker
    iwebtool
    linkchecker (see github)

    there are a few more..

    Cheers!
  16. #9
  17. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2015
    Posts
    3
    Rep Power
    0
    I found the same problem here https://moz.com/community/q/can-t-cr...-what-is-wrong. May be it will help you.
  18. #10
  19. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2015
    Posts
    7
    Rep Power
    0
    First of all, crawling and indexing are quite separate, so there will always be some disparity. URLs might be crawled, but it doesn’t always mean they will actually be indexed in Google.

    Comments on this post

    • Test-ok agrees

Similar Threads

  1. Duplicate Content in Screaming Frog
    By superherostuff in forum New User SEO Questions and Answers
    Replies: 4
    Last Post: Sep 28th, 2015, 11:56 AM
  2. Screaming Frog making my head hurt
    By ThomasHarvey in forum New User SEO Questions and Answers
    Replies: 1
    Last Post: May 13th, 2015, 12:18 PM
  3. Screaming Frog Tool: Your Thoughts
    By GraceLoveSEO in forum Search Engine Optimization
    Replies: 11
    Last Post: May 13th, 2015, 09:33 AM
  4. Finding influencers via Screaming Frog
    By PBK in forum New User SEO Questions and Answers
    Replies: 1
    Last Post: May 7th, 2014, 08:16 AM
  5. Screaming Frog vs. IIS SEO Toolkit
    By kacunn45 in forum Search Engine Optimization
    Replies: 1
    Last Post: Oct 4th, 2013, 03:31 PM

IMN logo majestic logo threadwatch logo seochat tools logo