#1
  1. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    6
    Rep Power
    0

    Screaming Frog shows different results when I scan directly from the sitemap


    Can anyone give me any clue why does screaming frog SEO spider show only 460 HTML pages when I crawl my website but when i change Mode to List and crawl it directly from the sitemap using Upload -> Download Sitemap Index it shows more then 50 000 HTML pages.
  2. #2
  3. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2012
    Posts
    364
    Rep Power
    328
    That is defintety something to take a closer look at...it is possible that there may be an SEO issue and that you may have orphan pages or an accessibility issue where a number of your pages can't be crawled. A starting point for you to troubleshoot further is to pull the inlinks (under reports section or bulk export in screaming frog) and see how the pages that you're not seeing in the crawl are internally linked withing in the list mode...
    "...using this: sages act out of constructs but they do not rely on them"
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    1,081
    Rep Power
    2276
    Sounds similar to another thread / question....

    Where one of our members, "Chedders", stated....

    The Free Version of Screaming Frog will only process 500 pages

    Perchance you using the free version???
  6. #4
  7. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    6
    Rep Power
    0
    No no I have a full version, when i tested it on other websites it works just fine and shows thousands of pages but not on my website.

    Originally Posted by KnowOneSpecial
    Sounds similar to another thread / question....

    Where one of our members, "Chedders", stated....

    The Free Version of Screaming Frog will only process 500 pages

    Perchance you using the free version???
  8. #5
  9. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    1,081
    Rep Power
    2276
    Mind sharing the site name. We might be more helpful.
  10. #6
  11. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    6
    Rep Power
    0
    Sure, the website is https://www.avto-shina.ru/. For en example a UK version of the website https://www.mytyres.co.uk/ ScreamingFrog crawls everything just fine but for some reason it doesn't for the Russian version.
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    1,081
    Rep Power
    2276
    Well the sites are not identical by a long shot.

    I can see what you have done. You have created 11 sites, and using rel alternate / hreflang tags to target different languages, Nice, but I can see that you have not completely finished porting the sites.

    You have a much more comprehensive navigation system on the co.uk site. In addition a second additional footer nav, so on the co.uk site you basically have 3 footer nav menus. Two for on-site it looks like and the 3rd for your other alternate sites.

    Finish porting the site, and implement your rel alternate and hreflang tags on the second site. Also I looked at a couple of the others, and they appear to be in need of the same attention.

    It is also possible you have created orphaned pages as whiterabbit has suggested.
    Last edited by KnowOneSpecial; Aug 18th, 2017 at 01:06 AM.
  14. #8
  15. Dinosaur
    SEO Chat High Scholar (3500 - 3999 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,888
    Rep Power
    6518
    I have not checked the site but the other thing to remember is screaming frog obeys nofollow tags if it finds a nofollow tag it wont follow it so there could be 100's of pages not crawled if there is a block.
    Personally I find that a good thing, when your checking a site as it confirms how google will see the site.
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  16. #9
  17. Dinosaur
    SEO Chat High Scholar (3500 - 3999 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,888
    Rep Power
    6518


    Sorry just ran the site in screaming frog

    Why so many redirects ? almost 10% of the pages are being redirected somewhere.
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  18. #10
  19. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    5
    Rep Power
    0
    no I have a full version, when i tested it on other websites it works just fine and shows thousands of pages but not on my website.

    Comments on this post

    • KnowOneSpecial : bot huh
  20. #11
  21. Dinosaur
    SEO Chat High Scholar (3500 - 3999 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,888
    Rep Power
    6518
    If you have a site map of all the URLS on your site and run that in screaming frog then that's how many pages you should see.
    If you point screaming frog at your home page then it will follow all follow links it can find but that is very different to a site map as it requires a link to a page in order to find it.
    These pages we call orphan pages as there is no route to them so no parents, in a well constructed site I agree the 2 should be the same but things get broken and sometimes a tool like screaming frog can indicate you have a problem.

    I am still really concerned about all the errors on the site though, even some 500 errors, that to me does not indicate a well managed site.

    Comments on this post

    • 5upernem agrees
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  22. #12
  23. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    6
    Rep Power
    0
    Thank you all for your help. Internal linking was the problem and we are working on fixing it. We added a sitemap.html on the home page so the crawler can crawl all the pages but that is not the final solution. The website has a lot of problems so there is a lot work to be done.

Similar Threads

  1. Some screaming frog SEO questions
    By Philipo22 in forum New User SEO Questions and Answers
    Replies: 6
    Last Post: Aug 16th, 2017, 05:24 AM
  2. Screaming Frog and 302 redirect
    By dpuk44 in forum Search Engine Optimization
    Replies: 2
    Last Post: Jul 6th, 2017, 02:10 PM
  3. Screaming frog shows site title as h1
    By Philipo22 in forum New User SEO Questions and Answers
    Replies: 19
    Last Post: Apr 7th, 2017, 07:01 PM
  4. Screaming Frog Crawl
    By HappyJackJunior in forum Search Engine Optimization
    Replies: 1
    Last Post: Jun 27th, 2016, 02:39 PM
  5. Screaming Frog vs. IIS SEO Toolkit
    By kacunn45 in forum Search Engine Optimization
    Replies: 1
    Last Post: Oct 4th, 2013, 03:31 PM

IMN logo majestic logo threadwatch logo seochat tools logo