Thread: Lazy Spider.

    #1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2003
    Location
    UK
    Posts
    38
    Rep Power
    17

    Lazy Spider.


    My site has been deep crawled twice in the last 4 days. Each time the crawl has indexed about 16,000 pages (aprox 65% of total pages). Unfortunately it appears to have indexed the same pages on both occassions, ignoring two important areas of the site that have recently (last 6 weeks) been added. Is there anyway or anything I can use to gaurantee Google will deep crawl the whole site.

    Apicius
  2. #2
  3. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2003
    Location
    Maine USA
    Posts
    524
    Rep Power
    17

    Re: Lazy Spider.


    Originally posted by "apicius"

    My site has been deep crawled twice in the last 4 days. Each time the crawl has indexed about 16,000 pages (aprox 65% of total pages). Unfortunately it appears to have indexed the same pages on both occassions, ignoring two important areas of the site that have recently (last 6 weeks) been added. Is there anyway or anything I can use to gaurantee Google will deep crawl the whole site.

    Apicius
    No gaurantee's, Google doesn't always deep crawl an entire site.

    Google's crawler was not on many of the sites I work on early yesterday for several hours so they may have had a problem and did a crawler restart, that may account for the double crawl of content on your site. I haven't checked what the bots collected after they came back.

    Do you have a sitemap? How about really large pages (HTML not images) that may place links outside Googles reach? Seem to recall seeing that beyond a certain point in the page (100K), the spider doesn't go (and since all my pages are a lot smaller than that I can't say for certain).

    Cheers,

    Cheers,
    theBear

Similar Threads

  1. Cool script: Emails you when googlebot hits your domain (*)
    By Angus in forum Web Design, Coding and Programming
    Replies: 87
    Last Post: Jul 12th, 2003, 12:14 AM
  2. What causes a spider to "get stuck" in a site
    By robbielockie in forum Google Optimization
    Replies: 5
    Last Post: Jul 10th, 2003, 01:22 AM
  3. how does google spider flash websites?
    By jimmy-yo in forum Google Optimization
    Replies: 9
    Last Post: Jun 20th, 2003, 11:18 AM
  4. Someone must teach the spider grammaire -or better not
    By avournas in forum Google Optimization
    Replies: 7
    Last Post: Jun 15th, 2003, 06:18 PM

IMN logo majestic logo threadwatch logo seochat tools logo