#1
  1. No Profile Picture
    Contributing User

    Join Date
    Nov 2007
    Posts
    100
    Rep Power
    12

    Question XML Sitemp Issue - Expert Advice


    Hi All,

    I have submitted XML sitemap for my site by following the limit of 50,000 URLs per xml sitemap and there are 8 XML sitemap in and index file having approx 50,000 URLs for each XML sitemap that is also apprx 10 MB filesiize.

    Sitemap also dont' show any error/warnings at webmaster tools but even after 2 months there are only 7000 URls get indexed from all sitemaps.

    My question is that does breaking 50,000 urls for each sitemap into 20,000 URLs or less may help in faster indexing through webmaster tool.

    Also, I am aware that we need offline promotion to faster crawling and indexing so please advice me for the better way to use xml sitemap for faster crawling and indexing at webmaster tools.
  2. #2
  3. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    16,980
    Rep Power
    0
    Originally Posted by coolslko
    I am aware that we need offline promotion to faster crawling and indexing.
    that answers this:

    Originally Posted by coolslko
    7,000 vs 50,000 X 8 = 400,000
    A sitemap aids Googlebot's ability to quickly find pages but certainly doesn't help it rank them... if a page would be ranked below position 1,000 there isn't much point indexing them in a hurry.

    Getting links to each page or getting links to a silo of pages will afford rapid indexing.
  4. #3
  5. No Profile Picture
    Contributing User

    Join Date
    Nov 2007
    Posts
    100
    Rep Power
    12

    Question


    Originally Posted by fathom
    that answers this:



    A sitemap aids Googlebot's ability to quickly find pages but certainly doesn't help it rank them... if a page would be ranked below position 1,000 there isn't much point indexing them in a hurry.

    Getting links to each page or getting links to a silo of pages will afford rapid indexing.
    I am concerned about indexing first and not ranking as of now and I just want to ask that -

    1. Does it really help in faster indexing (comparing to current situation) if keep less number of URLs like 20,000 or less per sitemap XML comparing to the current xml sitemap having 50000 urls each...
    2. Also, should I need to submit each XML sitemap file seperately to Google or should I submit an index file having URLs of all XML sitemaps in it.?
    3. will it be a good practice or not?


    You can check our site (technologyvs.co.uk)
  6. #4
  7. Dinosaur
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    5,146
    Rep Power
    7343
    You also have to remember just because the page has been indexed does not mean it will remain in the index. Google often drops pages when they deem fit. If I look at any of my sites the number of pages indexed goes up and down, and sometimes without any changes being made so quite why it varies is always a strange one.

    However, despite you uploading a site map IMHO it will not help get all your pages indexed, yes I agree its letting google know they are available but it does not force google to index them. Google determines how many pages to crawl per visit on partly your servers ability to serve the pages. Google will attempt not to take out your server by bombing it with requests so if your site is slow google will detect that and slow down its requests to your site ability .

    I have over recent months stopped using site maps altogether as I said in a previous post, I have not seen any effect over sites with site maps. Your time is better spent on content and site structure than producing these XML files. And before anyone jumps down my throat its just my opinion I am voicing.

    Comments on this post

    • fathom agrees : Anyone? Don't you mean me? I agree with you 100% on your opinion.
    • Doodled agrees
  8. #5
  9. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    16,980
    Rep Power
    0
    Originally Posted by coolslko
    I am concerned about indexing first and not ranking
    But if ranked below position #1000 that is the same as not being indexed.

    Dump 7 of your sitemaps and 350,000 pages and watch you get indexed faster.

    7,000 to 50,000 in no time.
    Last edited by fathom; Jun 24th, 2015 at 12:28 PM.
  10. #6
  11. No Profile Picture
    Contributing User

    Join Date
    Nov 2007
    Posts
    100
    Rep Power
    12
    Originally Posted by fathom
    But if ranked below position #1000 that is the same as not being indexed.

    Dump 7 of your sitemaps and 350,000 pages and watch you get indexed faster.

    7,000 to 50,000 in no time.
    Sorry..I didn't get you..!!

    Please tell me what to do for faster indexing..? Do you mean I should break the sitemap and use multiples of them with putting less number of URLs in each sitemap like i mention before or what?
  12. #7
  13. Dinosaur
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    5,146
    Rep Power
    7343
    I will try and explain it a little better.

    At no point as far as I am aware has google said they will index every page of every website. It will index as much as it thinks is appropriate for their users not the webmaster. How many pages will be depend on your site and the contents and not your sitemaps.It may look at each page over time but that does not mean it will make the index.

    If you really have 8 x 50,000 pages of content so 400,000 pages then I bet a hell of a lot of it will be duplicated or autogenerated. Google I would suspect would recognize that and only index a sample of the pages.

    The best way as far as I know of getting a page into the index is to get users viewing it / linking to it etc. i.e. get the page being talked about including social channels so the page is trending and being talked about then the page will I am sure make the index.

    400,000 pages though that users are not interested in I doubt will ever make the cut and may never see the light of day as far as the index is concerned.

    An example, say I had a website all about widgets, and I had 400,000 types of widgets so created a page for each I doubt many would make the index as they are pretty much all the same but just variants.
    On the other hand a site like wikipedia where each page is unique and full of interesting stuff that people want to read with 1,000's of links external and internal then the vast majority of the pages will get indexed.

    So in short your facing this problem the wrong way round. Get your SERPS higher and maybe more pages will then get indexed.
  14. #8
  15. No Profile Picture
    Contributing User

    Join Date
    Nov 2007
    Posts
    100
    Rep Power
    12

    Exclamation


    Originally Posted by Chedders
    I will try and explain it a little better.

    At no point as far as I am aware has google said they will index every page of every website. It will index as much as it thinks is appropriate for their users not the webmaster. How many pages will be depend on your site and the contents and not your sitemaps.It may look at each page over time but that does not mean it will make the index.

    If you really have 8 x 50,000 pages of content so 400,000 pages then I bet a hell of a lot of it will be duplicated or autogenerated. Google I would suspect would recognize that and only index a sample of the pages.

    The best way as far as I know of getting a page into the index is to get users viewing it / linking to it etc. i.e. get the page being talked about including social channels so the page is trending and being talked about then the page will I am sure make the index.

    400,000 pages though that users are not interested in I doubt will ever make the cut and may never see the light of day as far as the index is concerned.

    An example, say I had a website all about widgets, and I had 400,000 types of widgets so created a page for each I doubt many would make the index as they are pretty much all the same but just variants.
    On the other hand a site like wikipedia where each page is unique and full of interesting stuff that people want to read with 1,000's of links external and internal then the vast majority of the pages will get indexed.

    So in short your facing this problem the wrong way round. Get your SERPS higher and maybe more pages will then get indexed.
    I understand that Google do not index all the pages but out of 4,00,000 URLs it has indexed only 7000 that is very less I think it should 60% to 80% of urls to be indexed...also, the structure of site is also differ as it's a kind on search and buy type like no category, subcategory in the website and only the product urls and a user can get to that only by searhing through it's search box.

    So, it's not possible for each urls to submit and promote.

    I know there's an issue that google doesn't find access points on the website to crawl it's inner pages through site navigations, categories etc. as well as pages are not linked with each other ecept the CMS pages.

    (actually it's the client demand that they don't want to show their products on the site directly and a use rcan acces the product through search only.

    You can have a look at site at - technologyvs.co.uk and please advise me..it wil be a great help.. thanks in advance!!
  16. #9
  17. Dinosaur
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    5,146
    Rep Power
    7343
    Well there is a major problem straight away.

    Forcing users to use search to get to any of the products will not allow for natural navigation. There is no route to these inner pages without knowing about the sitemap.
    No internal link juice either so the pages ability to rank will also be hindered from any onpage SEO work.

    You / him are not allowing google to do what it does best, act as a end user and look around the site and figure out which are the important pages to the site.
    To be honest I would be quite happy to have 7,000 pages indexed with a site structure like that.

    Comments on this post

    • Will-O-The-Wisp agrees
  18. #10
  19. rod@missionop.com
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2003
    Location
    Palm Beach Gardens FL 33410
    Posts
    16,980
    Rep Power
    0
    In a nutshell - you got the wrong thought process in your head.

    Ranking is a factor of links and links are a factor of editorial content (not 400,000 pages of ad copy as few will link to your ad copy pages), editorial content is a factor of shown expertise, and that is a factor of indexing and indexing factor of crawling... getting Googlebot to crawl more and deeply and more often ... requires the opening condition "ranking".

    Crawling depth and crawling frequency is a factor of PageRank... without it you can't get Googlebot to labor for free for you - seriously if your 400,000 pages lacks "shown expertise"... why is it important for Googlebot to crawl?
    Last edited by fathom; Jun 25th, 2015 at 10:04 AM.

Similar Threads

  1. Expert Advice on Domain Age / Ranking Issue
    By thall89553 in forum Search Engine Optimization
    Replies: 6
    Last Post: Jun 13th, 2010, 08:48 AM
  2. I need an expert SEO advice, help!
    By SEOwarrior in forum SEO Help (General Chat)
    Replies: 7
    Last Post: Sep 18th, 2009, 04:49 AM
  3. Need SEO Expert Advice
    By vuexpress in forum Google Optimization
    Replies: 3
    Last Post: Feb 15th, 2005, 03:16 PM
  4. I really need some Expert advice here....
    By newyorker in forum Google Optimization
    Replies: 14
    Last Post: Jul 8th, 2004, 01:43 PM

IMN logo majestic logo threadwatch logo seochat tools logo