#1
  1. Professional SEO
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    May 2003
    Location
    Finland
    Posts
    710
    Rep Power
    17

    Question A very long deepcrawl?


    Any comments on googlebot activity described below? This is like one very long (from 11th onwards) deepcrawl rollercoaster ride with ups and downs.

    July Stats for Googlebot:
    day: pages
    3rd: 23 pages
    5th: 6 pages
    6th: 7 pages
    7th: 9 pages
    9th: 2 pages
    10th: 5 pages
    11th: 154 pages
    13th: 448 pages
    14th: 74 pages
    15th: 17 pages
    16th: 154 pages
    17th: 87 pages
    18th: 140 pages
    19th: 66 pages
    20th: 58 pages
    21st: 81 pages
    24th: 175 pages
    Number of distinct pages crawled: 892

    I've never seen a deepcrawl like this:it's so freaking long...

    I'm using a new internal linking technique (called "data islands") on this service so it might be one reason. However; I've never worked with this BIG site (way over 7 000 pages) so it might be partly due to that too. Any comments / experiences ?
  2. #2
  3. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jul 2003
    Location
    Roma, Italy
    Posts
    16
    Rep Power
    0
    Googlebot presences on my site:

    Tue Jul 01st, 2003 0
    Wed Jul 02nd, 2003 0
    Thu Jul 03rd, 2003 3
    Fri Jul 04th, 2003 13
    Sat Jul 05th, 2003 2
    Sun Jul 06th, 2003 2
    Mon Jul 07th, 2003 18
    Tue Jul 08th, 2003 13
    Wed Jul 09th, 2003 13
    Thu Jul 10th, 2003 17
    Fri Jul 11th, 2003 6
    Sat Jul 12th, 2003 11
    Sun Jul 13th, 2003 8
    Mon Jul 14th, 2003 13
    Tue Jul 15th, 2003 32
    Wed Jul 16th, 2003 130
    Thu Jul 17th, 2003 76
    Fri Jul 18th, 2003 207
    Sat Jul 19th, 2003 11
    Sun Jul 20th, 2003 119
    Mon Jul 21st, 2003 34
    Tue Jul 22nd, 2003 267
    Wed Jul 23rd, 2003 283
    Thu Jul 24th, 2003 37
    Fri Jul 25th, 2003 4

    Total, at now, 1,319.
    Note 22nd and 23rd!!
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Location
    London, UK
    Posts
    111
    Rep Power
    17
    i added a link to my new site from an existing PR5 site and 2 days later (today) googlebot is deepcrawling my new site! i didn't expect it to happen that fast!
    Last edited by tim_s; Jul 25th, 2003 at 07:22 AM.
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Location
    London, UK
    Posts
    111
    Rep Power
    17
    Originally posted by tim_s
    i added a link to my new site from an existing PR5 site and 2 days later (today) googlebot is deepcrawling my new site! i didn't expect it to happen that fast!
    omg!!!

    i'm writing this an hour later by the way.

    googlebot has just indexed a page on another my site of mine with the only link coming from a page that it crawled for the 1st time half an hour ago.

    how quick is that!
  8. #5
  9. Contributing User
    SEO Chat Genius (4000 - 4499 posts)

    Join Date
    May 2003
    Location
    Brighton
    Posts
    4,103
    Rep Power
    23
    what is an easy way for me to check how many pages have been crawled???? I have noticed Googlebot not leaving my site...it has been crawling it for almost 2 months and getting over 1,000,000 hits (it's an insanely large site) but hits are hits and pages are pages. any software or is it just manually through the log??

    -PK
  10. #6
  11. Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Apr 2003
    Location
    Weingarten
    Posts
    554
    Rep Power
    17
    I see Googlebot very often in the last time on my Board ( http://board.ranking-konzept.de ) . He goes trhough all posts. Everytime when i look into my Admin-Area i see Googlebot there sometimes with 8`different IP`s at the same time!

    But i`m waiting for the results in the Index - perhabs an update will come soon!

    I hope it!
  12. #7
  13. Professional SEO
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    May 2003
    Location
    Finland
    Posts
    710
    Rep Power
    17
    Originally posted by pk_synths
    what is an easy way for me to check how many pages have been crawled???? any software or is it just manually through the log??

    -PK

    You could use a log analyzer (best freeones are Analog or AWStats). Both of these can crunch log files relatively well and fast.

    If the log file is huge then it might be useful just to dump everything into database (MSSQL, MySQL or even Access all do a decent job) and use SQL for queries (I prefer this, because it gives more freedom).

Similar Threads

  1. query string - how long is too long?
    By lilbit in forum Google Optimization
    Replies: 13
    Last Post: Jun 2nd, 2006, 04:05 AM
  2. New sites, how long in top 10?
    By cityneil in forum Google Optimization
    Replies: 7
    Last Post: Oct 10th, 2003, 11:01 AM
  3. Next time you see the deepcrawl start, don't panic!!!!
    By Nintendo in forum Google Optimization
    Replies: 11
    Last Post: Mar 20th, 2003, 02:08 PM
  4. do deepcrawl / freshbot work together?
    By Homer in forum Google Optimization
    Replies: 1
    Last Post: Feb 12th, 2003, 12:05 PM

IMN logo majestic logo threadwatch logo seochat tools logo