Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17

    PHP pages and depth of spidering info needed


    I run a php generated website. I noticed that although Google seems to index the site one level deep it does not index all the pages and articles stored in the database. So instead of using URL rewriting I wrote a little prog that would convert all database entries into single real static HTML pages which are linked to from an Index page which is also HTML. I now linked to this page from the homepage (hidden as it is only usefule for the spider). Now all these articles are real and there is no spamming intended BUT still google does not seem to index the pages. Any suggestions or ideas how deep google will index PHP generated sites?
    Here is a link to the website http://www.gemfields.com and here is a link to the Static pages for the spider (and hopefully also for the other engines that do not use PHP at all) http://www.gemfields.com/static_html/index.html

    If anyone could help it would really be appreciated.
    Cheers and have fun
    Shedman
    What the Hell is "the dance"? I know I am just a newbie
  2. #2
  3. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17
    The dance is when the hard drives at the Google data centers do the index update twist and shake.

    Cheers,
    theBear
  4. #3
  5. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17
    When did you add the static page set and index before or after the deep crawl?

    Do you have access to your server logs? Did the spider even request those pages?

    It is just a shot in the dark without that information, I see nothing incorect with the links and I had no problem calling up the pages.

    So the when question is probably going to answer the why one.

    Cheers,
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2003
    Location
    Salt Lake City
    Posts
    130
    Rep Power
    17
    [quote]Google seems to index the site one level deep it does not index all the pages and articles stored in the database.

    I have database generated pages. Google does a good job on everything but the calendar. My calendar has a design flaw that lets you search for ever.

    Is this a problem that happened one month? or is it happening every month?

    There is now an entire industry formed around flooding google with data generated sites. So maybe Google is tightening up restrictions. As I recall, Mondera (a gem site) provides data feeds...so you have one in your industry.

    Of course, many of the people playing this data feed game are spoofing directory structures.

    BTW, you might want to add a robots.txt file to the site. The robots.txt file tells robots what parts of the site to avoid.
  8. #5
  9. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17

    static psage creation date


    Thanks guys.
    I have had the static pages since the 10th of March. I noticed that google visited yesterday but my stats say the googlebot only requested 4 pages.
    Alta vista etc of course can't handle the site at all. So for a while I was thinking of using an index.html which redirects to the index.php with links to also the static pages BUT search engine optimisation must stop somewhere. If every one of my visitors has to wait for another 5 seconds (the redirection) to see the homepage just for Alta vista being able to properly index the site there is a problem, so I just forget about altavista.
    AS to Google not indexing more then one level deep seems silly AS a marketer can achieve the same with static page junk pages generation. But the people at google must go one step at the time i suppose.
    Anyway, Thank you very much for your help guys and maybe theBear is right and it is just because the static pages have not been indexed yet.
  10. #6
  11. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17

    Re: static psage creation date


    Originally posted by "shedman"

    Thanks guys.
    I have had the static pages since the 10th of March. I noticed that google visited yesterday but my stats say the googlebot only requested 4 pages.
    Alta vista etc of course can't handle the site at all. So for a while I was thinking of using an index.html which redirects to the index.php with links to also the static pages BUT search engine optimisation must stop somewhere. If every one of my visitors has to wait for another 5 seconds (the redirection) to see the homepage just for Alta vista being able to properly index the site there is a problem, so I just forget about altavista.
    AS to Google not indexing more then one level deep seems silly AS a marketer can achieve the same with static page junk pages generation. But the people at google must go one step at the time i suppose.
    Anyway, Thank you very much for your help guys and maybe theBear is right and it is just because the static pages have not been indexed yet.
    Err... that means that the deep crawl got them and that they will appear after the dance ...

    And Google indexs more than one level. I have over 127,000 pages on one site with many levels some go 8 levels deep. Do not confuse levels with variable in dynamic pages.

    Cheers,
  12. #7
  13. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17
    Hello again, Yes I too have another site where google indexes more then 1 level deep as well, but in PHP generated files tyhey seem to give up after reaching the surface. Possibly because of people spamming the index I suppose.
    When will the changes (new pages) be seen do you think?
    Cheers
  14. #8
  15. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17
    If the deep crawl got them (check your logs) they will appear during the dance as the index gets updated .... we are hoping that will be very soon.

    Cheers,
  16. #9
  17. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2003
    Location
    California
    Posts
    168
    Rep Power
    17
    If I am linked from a php page, will Google see this backlink?
  18. #10
  19. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2003
    Posts
    437
    Rep Power
    17
    can I have a look at that prog you wrote please?

    email me at email@web-designr.com
  20. #11
  21. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17
    Originally posted by "kirby"

    If I am linked from a php page, will Google see this backlink?
    If the page has an embeded <a href ....> </a> in it pointing to your site and the page it is in gets spidered yes, and if it is PR4 and higher it'll show in Google's link query results.

    Note an embeded link is not the only way to actually transfer to your page. But an embeded link is the only thing that counts as far as Google is concerned.

    You have to know what is going on in the script to make a determination, check the code "under the hood" to verify.

    Remember Google has to see a real link that means the page with the link has to be in Google's database.

    Cheers,
  22. #12
  23. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17
    A bit further to the link back that google shows. I am linked from other websites pr4 pages BUT google only seems to show the ones from pr5 sites. Any idea why?
  24. #13
  25. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17
    Originally posted by "shedman"

    A bit further to the link back that google shows. I am linked from other websites pr4 pages BUT google only seems to show the ones from pr5 sites. Any idea why?
    Give us a specific example to look at.

    Cheers,
  26. #14
  27. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17

    specific example


    Backlinks as per google to the site Gemfields.com

    linked from Ozfreebies.com homepage pr5
    also linked from ozfreebies.com subpage which is pr4 (both sow up as backlinks
    Linked from a pr6 ozrefer.com (show up as back link)
    Linked from http://www.cqgemfields.com/Links.htm (pr4 site and page)
    BUT THIS TIME NOT SHOWN AS BACKLINK by google
    I dont know why, the link on cqgemfields has been there a while but google just does not show it.

    We also have a heap of other sites linking to us and I am sure some of them must be above pr4 but they do not show up either.
  28. #15
  29. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    61
    Rep Power
    17

    a few more that are not on the backlinks of google


    quickly had a look and here are a few more that google does not list as backlinks for gemfields.com:
    http://aaakoalagems.netfirms.com/links.htm pr3 homepage BUT this page with link to us is pr4
    http://www.crystalsonline.com/TextLinks.htm pr4
    http://www.kcminerals.com/links.htm the home page is pr5 but this page pr4
    However I do not knoe how old these links are BUT do think they are older then 2 weeks at least
Page 1 of 2 12 Last
  • Jump to page:

IMN logo majestic logo threadwatch logo seochat tools logo