#1
  1. Online Business Builder
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Dec 2010
    Location
    India
    Posts
    697
    Rep Power
    367

    Multiple sitemaps generator for larger website


    I wanted generate larger sitemap.xml in which every section should have its own sitemap.xml and all these sub sitemaps should be listed in one sitemap.xml (index sitemap). Is it possible to generate it automatically?

    I have successfully been using Xenu, gsite and screaming frog and they can handle thousands of (or may be millions) URLs but I am not sure if I can do classification using these tools.

    I thought anyone may have faced similar issue and found solution of it.
    Lead, follow or leave the way...
  2. #2
  3. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Aug 2013
    Posts
    247
    Rep Power
    208
    Do the URL paths in the site architecture allow for segmenting them out by looking for and pulling out certain strings?

    If the parts of the site look like:

    http://www.example.com/category1/sub...ry1/page1.html
    http://www.example.com/category2/sub...ry1/page1.html

    and/or

    http://www.example.com/blog/category1/article1.html
    http://www.example.com/blog/category2/article1.html

    then you might be able to use the advanced features in Xenu/Screaming to crawl specific parts of the site one at a time OR you could use a script to pull out just parts of one big crawl one at a time.

    Hope that helps.

    Comments on this post

    • jamespayne agrees
  4. #3
  5. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Apr 2011
    Posts
    967
    Rep Power
    1550
    The one thing I have against systems like screaming frog is that they can index a lot of junk while missing some truly important pages.

    I create the script/code which reads through databases (either parts of it or all of it) to generate sitemaps so we know exactly what is in each sitemap without doubt, even when we're talking about thousands of pages.

    Comments on this post

    • LBachmann agrees : Correct. I acknowledge your reply.
  6. #4
  7. No Profile Picture
    Contributing User

    Join Date
    Dec 2014
    Posts
    33
    Rep Power
    3
    Best thing to create sitemap for huge websites is to break them into categories and then use some script/code to read them from database. This way all the sections of the website would get pulled out in xml file and same can be read by bots when get called in a single sitemap.xml file using coding. You can use many online tools available.
  8. #5
  9. Website and SEO tools

    Join Date
    May 2006
    Posts
    159
    Rep Power
    12
    Originally Posted by LBachmann
    Best thing to create sitemap for huge websites is to break them into categories and then use some script/code to read them from database. This way all the sections of the website would get pulled out in xml file and same can be read by bots when get called in a single sitemap.xml file using coding. You can use many online tools available.
    I am late to the party, but A1 Sitemap Generator supports:

    • Analysis filters where you through text-strings or regular expressions can define "limit-analysis-urls-to" and "exclude-urls-from-analysis" URL patterns
    • Output filters where you through text-strings or regular expressions can define "limit-sitemap-urls-to" and "exclude-urls-from-sitemap" URL patterns
    • Adding addition start search paths


    It is an "advanced" feature, so you will have to switch off "easy mode", but beyond that, it sounds to match your needs - i.e. you can split your sitemap crawls up into smaller chunks and create a sitemap for each. (And then automate/schedule the whole process including upload and pinging at night through the command-line support.)
  10. #6

  11. Join Date
    May 2005
    Location
    Denver, Colorado
    Posts
    60
    Rep Power
    13
    You might try the tools at http://xml-sitemaps.com. You can either create a sitemap right on that page, or buy their cheap xml sitemap generator and have it create updated sitemaps regularly for you. It also creates images and video sitemaps as well as others.

    It pings the search engines after creating the sitemap if you want it to. I have used it for years.

    Hope it helps

Similar Threads

  1. Free XML Sitemap Generator Tool: Website Have 500+ URLs
    By anandmistry in forum Search Engine Optimization
    Replies: 2
    Last Post: Apr 22nd, 2010, 07:12 AM
  2. Sitemap Generator
    By TBP Levi in forum Google Optimization
    Replies: 6
    Last Post: Feb 12th, 2007, 03:39 PM
  3. Replies: 2
    Last Post: Aug 7th, 2006, 10:44 AM
  4. Replies: 0
    Last Post: Jul 30th, 2006, 01:04 AM

IMN logo majestic logo threadwatch logo seochat tools logo