Google Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat ForumsGoogleGoogle Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old March 19th, 2003, 09:16 AM
apicius apicius is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Jan 2003
Location: UK
Posts: 38 apicius User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 4 m 32 sec
Reputation Power: 6
Lazy Spider.

My site has been deep crawled twice in the last 4 days. Each time the crawl has indexed about 16,000 pages (aprox 65% of total pages). Unfortunately it appears to have indexed the same pages on both occassions, ignoring two important areas of the site that have recently (last 6 weeks) been added. Is there anyway or anything I can use to gaurantee Google will deep crawl the whole site.

Apicius

Reply With Quote
  #2  
Old March 19th, 2003, 09:38 AM
theBear's Avatar
theBear theBear is offline
Contributing User
SEO Chat Novice (500 - 999 posts)
 
Join Date: Mar 2003
Location: Maine USA
Posts: 524 theBear User rank is Private First Class (20 - 50 Reputation Level)theBear User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 25 m 38 sec
Reputation Power: 6
Send a message via AIM to theBear
Re: Lazy Spider.

Quote:
Originally posted by "apicius"

My site has been deep crawled twice in the last 4 days. Each time the crawl has indexed about 16,000 pages (aprox 65% of total pages). Unfortunately it appears to have indexed the same pages on both occassions, ignoring two important areas of the site that have recently (last 6 weeks) been added. Is there anyway or anything I can use to gaurantee Google will deep crawl the whole site.

Apicius


No gaurantee's, Google doesn't always deep crawl an entire site.

Google's crawler was not on many of the sites I work on early yesterday for several hours so they may have had a problem and did a crawler restart, that may account for the double crawl of content on your site. I haven't checked what the bots collected after they came back.

Do you have a sitemap? How about really large pages (HTML not images) that may place links outside Googles reach? Seem to recall seeing that beyond a certain point in the page (100K), the spider doesn't go (and since all my pages are a lot smaller than that I can't say for certain).

Cheers,

Cheers,
__________________
theBear

Reply With Quote
Reply

Viewing: SEO Chat ForumsGoogleGoogle Optimization > Lazy Spider.


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 6 hosted by Hostway
Stay green...Green IT