Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0

    How To Test & Analyze??


    Hey guys,

    I've been having a difficult time trying to find information on how to test and analyze the whys & hows of my site in the SERPS. I believe I have all the basics understood, but when something doesn't behave the way I expect, I'm at a loss as to what to do next. When I've done all I know to do, how do I analyze the situation and make corrections and test? Anybody have good resources for something like this?

    Aaron Wall said he hasn't seen it but is planning on doing something like this in a couple of months.

    What do you think?

    A specific example would be a site I currently maintain has about 630 pages but only 255 or so are in the main index of Google (that show up in the site:www.domain.com/*). Back in August/September I made some huge changes to the internal linking structure when I realized some mistakes I made and I expected some kind of result. Google Webmaster Tools shows that I have 6300 internal links when in reality I have 15,000+...that's a huge amount of difference when it comes to internal juice distribution.

    Everything I've done is super clean and I've corrected duplicate content errors, blah blah blah...how do I go about figuring this thing out? I don't necessarily need specific answers to these problems (would be nice :-) ), but at least some kind of checklist, flow chart,reference, whatever in helping me out.

    About 95% of the time I understand what's going on, but when it doesn't work the way I figure it would, I just feel like I'm just stabbing in the dark!

    Thanks!!

    Paul
  2. #2
  3. End game expert
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2007
    Location
    Cebu Philippines
    Posts
    1,272
    Rep Power
    29
    Originally Posted by PaulFerree
    Hey guys,
    Google Webmaster Tools shows that I have 6300 internal links when in reality I have 15,000+...that's a huge amount of difference when it comes to internal juice distribution.
    My two cents:
    a. Check your robots.txt
    b. Check for broken links in your site.
    c. Check for nofollow tags.
    d. Check for no content/under construction pages.
    e. Check for site navigation consistency.
    f. Check for duplicate content.
    g. Check for non-crawlable javascript/flash menu links.
    h. Check that every page should be reachable from at least one static text link.
    i. Submit Google sitemap (XML updated)
    j. Check if the site map is larger than 100 or so links, break the sitemap into another page.
    k. Check if dynamic URL parameters are not too long. (Google technical guidelines)
    l. Check for number of links per page does not exceed 100. (to make sure all links can be followed with priority)

    In addition, check your site in accordance to Google quality and technical guidelines. http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35769

    You can check your site internal link structure (also broken link, sitemaps etc) by downloading this free software:
    http://home.snafu.de/tilman/xenulink.html
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Thanks Emerson,

    All that checks out good...that's what I mean by having done all, how do we diagnose these issues??

    I appreciate your checklist!

    Paul
  6. #4
  7. Contributing User
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2004
    Location
    Malta
    Posts
    1,978
    Rep Power
    73
    Originally Posted by PaulFerree
    Thanks Emerson,

    All that checks out good...that's what I mean by having done all, how do we diagnose these issues??

    I appreciate your checklist!

    Paul
    Check this out as well; http://forums.seochat.com/google-optimization-7/garythescubaguys-seo-checklist-127800.html?p=450791#post450791
  8. #5
  9. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Thanks,

    This is a really good list too: http://www.seomoz.org/article/search-ranking-factors#f3

    But, these are all basic/entry level stuff. I'm more interested in I guess the more advanced stuff like indexing issues. Something that would help answer the question:

    Why does my main index listings seem to be stuck at 250 pages and my site has 670? This is AFTER I've corrected several internal linking issues, etc...

    And another thing like why does a site have a Toolbar PR 0 when the site is completely valid and within guidelines? It seems like the site isn't banned at all. I'm not sure if it's just in between the toolbar update or something else.

    These issues aren't resolved by on page optimization.

    Paul
  10. #6
  11. Contributing User
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    May 2004
    Location
    Malta
    Posts
    1,978
    Rep Power
    73
    Okay lets take a close look now that I understand you know more than the basics...It can be many things, or just one.

    Originally Posted by PaulFerree
    Why does my main index listings seem to be stuck at 250 pages and my site has 670? This is AFTER I've corrected several internal linking issues, etc...
    There are a few potential reasons for this;
    - Duplicate content, or little on-page content may trigger Google to look at the site and see very little uniqueness compared to the rest of your site, so when the robot hits your site and gets so deep, it sees the same content and backs out. Yes its a stretch but I have seen this happen especially when a novice developer builds their entire site a page at a time with no template whatsoever, or they make small changes to the navigation within the template. In a correctly built site most robots will parse the template and crawl the content and see unique content. This allows them to crawl deeper and faster.
    - URL structure such as a stop character on the non-indexed pages like an ampersand
    - External links to an affiliate site or flagged directory (are you an affiliate?)
    - Its a new or somewhat new site and you haven't given it time

    And several other possibilities. Its all guessing without the url.

    I assume you have a robots.txt file in place and an xml site map, and you have a path to it in the robots.txt file right?


    Originally Posted by PaulFerree
    And another thing like why does a site have a Toolbar PR 0 when the site is completely valid and within guidelines? It seems like the site isn't banned at all. I'm not sure if it's just in between the toolbar update or something else.
    -Is the PR bar white or grey?
    -Did you have a PR at one time?
    -When was the site launched?
    -If you bought the domain recently, what was on it before?
    -Has the theme or subject of the site remained the same?
    -Have you tried doing a reverse IP look up to see who else is on your server (if your not on a dedicated server)

    As i mentioned from the start, it can be one or many of these.

    One more thing, have you checked the number of indexed pages across multiple google data centers?

    A little info and we can probably sort this out for you. IM me the url and I'll take a closer look.

    GaryTheScubaGuy

    Comments on this post

    • Emerson agrees : Nicely explained.
  12. #7
  13. End game expert
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2007
    Location
    Cebu Philippines
    Posts
    1,272
    Rep Power
    29
    Originally Posted by PaulFerree
    Thanks,

    Why does my main index listings seem to be stuck at 250 pages and my site has 670?
    Paul
    Take note what Gary said, those are very important points. In addition,Bear in mind that if you have a dynamic site, same page could have different session ID,and could give many URL's with different session ID's but same content. Are you sure that it has 670 unique pages? Or it has 670 pages including session ID's? especially if you are using a sitemap generator that includes session ID>

    It is possible that only 250 pages are unique because Google will filter those with URL's that are duplicated especially with session ID's.You may need to filter the duplicated URL's before submitting a sitemap to Google.

    As an illustration:
    http://www.studentcomputers.co.uk/laptops-c-133.html?osCsid=41cac52e5944588b85eb61c629da993f

    The above is the laptop page with a session ID.

    Again, if different user will click again the laptop page,it will now give another URL:
    http://www.studentcomputers.co.uk/laptops-c-133.html?osCsid=c10016d5c4cb28f67844f2afdc6fc46a

    Looking at the two URL's, they are different but the content is the same. But if you do check the indexed pages in Google by doing an advance search:

    Type: exact phrase : laptop
    return results only from domain: studentcomputers.co.uk
    Where terms occur only at the URL.

    The exact page indexed is:http://www.studentcomputers.co.uk/laptops-c-133.html

    Means only one page is there.
  14. #8
  15. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    You're right...those are good points and I understand the issue with duplicate content due to dynamic pages and such. This isn't the case. These are all unique, individual pages with no session ID in the URL. Back in August, this was a different story and I've corrected all those issues, now this site has been clean as a whistle for almost 4 months now.

    I had a horrible case of anti google guidelines before. I had duplicate index pages, both with an excess of 500+ text links on the page, and every URL contained a token for a SID.

    As I said, these things are completely fixed and I've changed the shopping cart that was used so that no SID is required in the URL.

    But still, I'm having a problem getting these other pages in the main index. I even incorporate nofollow to push as much juice downwards to my product pages as possible (using Halfdeck's PR Bot tool to check the flow is correct).

    It's very frustrating.

    Paul
  16. #9
  17. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Sorry Gary,

    I didn't notice this response. Thank you, these are more like the things I'm looking for.

    Originally Posted by GaryTheScubaGuy
    - Duplicate content, or little on-page content may trigger Google to look at the site and see very little uniqueness compared to the rest of your site, so when the robot hits your site and gets so deep, it sees the same content and backs out. Yes its a stretch but I have seen this happen especially when a novice developer builds their entire site a page at a time with no template whatsoever, or they make small changes to the navigation within the template. In a correctly built site most robots will parse the template and crawl the content and see unique content. This allows them to crawl deeper and faster.
    The content that changes from page to page is basically a product description that varies from 1 to 2 paragraphs. I understand that this could have some small potential for a problem but when I look at 1 page that is indexed vs. another in the supp, it doesn't really add up.

    I would suspect that the engines could determine that my pages are distinct from one another based off of the description text, however small it may be.

    Although, it is something worth noting, but I would probably put it lower on the list.

    Originally Posted by GaryTheScubaGuy
    - URL structure such as a stop character on the non-indexed pages like an ampersand
    Nope, I removed all instances of any kind of special character in my URLs. This was potentially a problem previous to the changes I made in August.

    Originally Posted by GaryTheScubaGuy
    - External links to an affiliate site or flagged directory (are you an affiliate?)
    No external links. Not an affiliate at all.

    Originally Posted by GaryTheScubaGuy
    - Its a new or somewhat new site and you haven't given it time
    The site was created in 2001.

    Originally Posted by GaryTheScubaGuy
    I assume you have a robots.txt file in place and an xml site map, and you have a path to it in the robots.txt file right?
    I do have a robots.txt file containing 2 commands to disallow the robots indexing any page with SID in the URL (which I don't use anymore anyway - no page is referenced dynamically throughout my site, each one is hard linked to the actual page)

    I do have an XML site map and I have registered it with Google Webmaster Central.

    I don't have anything referencing the site map in my robots.txt.


    ***
    In regards to the other site I mentioned:
    ***

    Originally Posted by GaryTheScubaGuy
    -Is the PR bar white or grey?
    White as snow on homepage...other pages are grey...and a few inner pages have PR3

    Originally Posted by GaryTheScubaGuy
    -Did you have a PR at one time?
    I can't recall what the status was before the redesign.

    Originally Posted by GaryTheScubaGuy
    -When was the site launched?
    Site was originally launched in 2004. Redesigned and moved to new host (Lexiconn) September 07.

    Originally Posted by GaryTheScubaGuy
    -Has the theme or subject of the site remained the same?
    It's the same site, just redesigned to use new shopping cart software.

    Originally Posted by GaryTheScubaGuy
    -Have you tried doing a reverse IP look up to see who else is on your server (if your not on a dedicated server)
    No, I don't know how to do this...but it is interesting because I do have another site on the same host using the same software and it's PR0 too.

    Originally Posted by GaryTheScubaGuy
    One more thing, have you checked the number of indexed pages across multiple google data centers?
    I've done it a few times and it seems pretty consistent.

    Originally Posted by GaryTheScubaGuy
    A little info and we can probably sort this out for you. IM me the url and I'll take a closer look.
    Thanks for your time Gary, these were more relevant questions I was looking for. I've sent you a PM with the info.

    I have a thought:

    Although I've fully upgraded my internal linking structure of the site and it looks good technically on paper, one problem is that I'm looking and calculating the site as a whole whereas Google essentially is only looking at or "counting" 250 URLS as good at this point. So the only pages that are really making any kind of difference as far as passing link juice are those in the main index.

    Not only that...even though I've updated the entire site with proper crosslinking to categories and such, there are pages that still may not be updated completely in their index. When I download and look at the internal links table (what they're saying is a representation of my internal linking) I see links that were last found in October but the actual cached date for the pages with those links are much more recent.

    So there seems to be a flow to it all...although I would have figured by 4 months it would've taken into effect. I've noticed that I usually get a big update in my internal link count some time around the 25th of the month, I guess after Google does some heavy crawling near the beginning of the month. So, we'll see how it changes, if at all, this month.

    Here's the shakedown of internal linking changes:

    10-25-07- 4601 (pre revision)
    10-30-07- 6264 (post revision)
    11-26-07- 6708
    Today - 6326 (after page removals and such)

    As I mentioned in my first post my true internal link number is 15,000+

    To make it even more confusing. I've used a test product on the site to see how some various techniques work and I have a page that doesn't show up in a site:www.domain.com/* search (which tells me it's supplemental, or, what used to be supplemental) but now ranks 7th for one of it's key phrases out of a listing of 24,300. So, now what does this tell me? Is supplemental a thing of the past or does Google update it's indexes differently, or in chunks? Is it possible for a listing of site:www.domain.com/* to contain older data than what is used in the main SERPS.....uhoh, brain just fell out, gotta go!

    Thanks for everybody's input, I hope this is helping some other people as well.

    (Sorry for the super long post!)

    Paul
  18. #10
  19. End game expert
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2007
    Location
    Cebu Philippines
    Posts
    1,272
    Rep Power
    29
    Hi Paul,
    I just want to add some advance suggestions to help you to check out.

    a. Assigned crawling priorities in your Google sitemap tag.

    b. 301 redirect all variation of homepage URL (the one that return status 200), you can check it with http://www.ragepank.com/redirect-check/, it will strengthen and concentrate all your link juices/PR to your official homepage URL (one you used as target URL for your link building in the past, the one with the highest link juice) and get more crawling priorities to the homepage which then follows to your deeper pages by internal links.

    c. Optimize file sizes of your website by incorporating file compression to your images and other files, this will lowers the overall sizes of your site and results to faster browser and downloading experience. Overall, this will improve Google bot crawling, since it works by downloading pages from your site. If your site is not optimized in file sizes,chances are the downloading is slow and Google will crawl the site slowly and prioritizes other pages. Remember the importance of bandwidth efficiency. I have tried compressing the files, and it really improves the speed and crawling frequency.

    d. Check your server response speed and compare it to standard, if it is slower or normal.

    About Reverse IP lookup,(the one Gary is talking about) since your site is located in a server (computer), there are also other sites that shared with it. if shared, they have the same class C IP in the notation like 192.01.23, if there are 4 sites sharing on same server, it will have the same class C IP.

    You can check this tool, http://www.domaintools.com/ type the domain name in whois lookup and "reverse ip" info under "domain tools exclusive." It will give data on the number of sites that are shared on the same server.
    Last edited by Emerson; Dec 19th, 2007 at 09:50 AM.
  20. #11
  21. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Originally Posted by Emerson
    a. Assigned crawling priorities in your Google sitemap tag.
    No, I haven't done this. The only priorities I would set would be to the category pages, but the way the link juice is funneled, this should already be apparent to the spider (google visits my index and category pages often, about every 3 days or so). I don't think it would help getting more pages in the index though.

    Originally Posted by Emerson
    b. 301 redirect all variation of homepage URL (the one that return status 200), you can check it with http://www.ragepank.com/redirect-check/, it will strengthen and concentrate all your link juices/PR to your official homepage URL (one you used as target URL for your link building in the past, the one with the highest link juice) and get more crawling priorities to the homepage which then follows to your deeper pages by internal links.
    I should probably look into this a little more, but from what Google is showing me, there is only 1 index page it sees and indexes. So, I think I'm good. But I'll make sure to check and iron clad it.

    Originally Posted by Emerson
    c. Optimize file sizes of your website by incorporating file compression to your images and other files, this will lowers the overall sizes of your site and results to faster browser and downloading experience. Overall, this will improve Google bot crawling, since it works by downloading pages from your site. If your site is not optimized in file sizes,chances are the downloading is slow and Google will crawl the site slowly and prioritizes other pages. Remember the importance of bandwidth efficiency. I have tried compressing the files, and it really improves the speed and crawling frequency.
    That's a good point. Most of my internal product pages are around 25-40k or so...so I believe that's within an acceptable range.

    Originally Posted by Emerson
    d. Check your server response speed and compare it to standard, if it is slower or normal.
    What's the best way to accomplish this?

    Originally Posted by Emerson
    About Reverse IP lookup,(the one Gary is talking about) since your site is located in a server (computer), there are also other sites that shared with it. if shared, they have the same class C IP in the notation like 192.01.23, if there are 4 sites sharing on same server, it will have the same class C IP.

    You can check this tool, http://www.domaintools.com/ type the domain name in whois lookup and "reverse ip" info under "domain tools exclusive." It will give data on the number of sites that are shared on the same server.
    Awesome tip. I checked and the site with the PR0 is on a dedicated server it says.

    Thanks Emerson, I appreciate your time.
    Paul
  22. #12
  23. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Originally Posted by PaulFerree
    To make it even more confusing. I've used a test product on the site to see how some various techniques work and I have a page that doesn't show up in a site:www.domain.com/* search (which tells me it's supplemental, or, what used to be supplemental) but now ranks 7th for one of it's key phrases out of a listing of 24,300. So, now what does this tell me? Is supplemental a thing of the past or does Google update it's indexes differently, or in chunks? Is it possible for a listing of site:www.domain.com/* to contain older data than what is used in the main SERPS.....uhoh, brain just fell out, gotta go!
    Maybe this is the case "Google Officially Drops Supplemental Index" :http://googlewebmastercentral.blogspot.com/2007/12/ultimate-fate-of-supplemental-results.html

    We'll see what happens now.

    Paul
  24. #13
  25. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2006
    Posts
    109
    Rep Power
    9
    another thing you have to consider is google only spiders 4 links deep, so it takes more then 4 clicks to get to the page it won't be indexed. Also if in the past you lost all your pr it will take some time to get it all back again. Loosing pr site wide makes it so you loose all your trusted rank and you are essentially a new site again never mind it's 10 years old.
  26. #14
  27. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    28
    Rep Power
    0
    Originally Posted by neque
    another thing you have to consider is google only spiders 4 links deep, so it takes more then 4 clicks to get to the page it won't be indexed. Also if in the past you lost all your pr it will take some time to get it all back again. Loosing pr site wide makes it so you loose all your trusted rank and you are essentially a new site again never mind it's 10 years old.
    Thank neque. The pages in question are 3rd tier.

    Although, I dont think its accurate to say that Google *ONLY* goes 4 links deep as if it was a technical guideline. Ive seen Google go the distance.

    However, I think its best if we can avoid as many clicks as possible.

    Paul

    Comments on this post

    • Emerson agrees
  28. #15
  29. End game expert
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2007
    Location
    Cebu Philippines
    Posts
    1,272
    Rep Power
    29
    Originally Posted by PaulFerree

    What's the best way to accomplish this?
    You can use the link below (copy and paste) to the browser, I use this one to measure web server speed. It also shows data of web server speeds from other sites to have a comparison.

    http://www.vertain.com/?sst
Page 1 of 2 12 Last
  • Jump to page:

Similar Threads

  1. Replies: 17
    Last Post: Apr 6th, 2010, 07:52 PM
  2. Reciprocal Linking Test
    By tstolber in forum Google Optimization
    Replies: 13
    Last Post: Apr 13th, 2006, 03:31 AM
  3. Old Google Penalty?
    By nnclyn in forum Google Optimization
    Replies: 11
    Last Post: Nov 17th, 2005, 03:30 PM
  4. DC movement
    By OptWizard in forum Google Optimization
    Replies: 34
    Last Post: Mar 19th, 2005, 08:37 PM
  5. The Results Are In! New Test Results!!!
    By hexed in forum Google Optimization
    Replies: 51
    Last Post: May 22nd, 2004, 09:21 PM

IMN logo majestic logo threadwatch logo seochat tools logo