#1
  1. Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2003
    Location
    new york
    Posts
    11
    Rep Power
    0

    Do you undertstand this Googlebot log data?


    Hi

    I need some detective help to understand more about this Googlebot that visted my site often and then disappeared.

    My site was initially indexed on July 5th and then periodically until it stopped after Aug 3. However the last two days the Googlebot came I thought the log looked odd.

    Anyway, below is the log file of Googlebot for the last two days it visted my site, and has not returned since. My suspicion is that the Googlebot did not like something the last times it visted.

    Any thoughts?

    - dylan

    [02/Aug/2003:08:24:43 -0400] "GET /robots.txt HTTP/1.0" 404 204 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    64.68.82.142 - -

    [02/Aug/2003:08:24:46 -0400] "GET / HTTP/1.0" 200 48876 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    195.202.89.107 - -

    [03/Aug/2003:07:35:58 -0400] "GET /robots.txt HTTP/1.0" 404 204 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    64.68.85.10 - -

    [03/Aug/2003:07:35:58 -0400] "GET / HTTP/1.0" 200 48876 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    65.73.152.76 - -
  2. #2
  3. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2003
    Location
    Stuttgart, Germany
    Posts
    4
    Rep Power
    0
    Hi,

    I am not sure about what information you're exactly looking for, but I'll tell you how much I can read from the log files.


    [02/Aug/2003:08:24:43 -0400] "GET /robots.txt HTTP/1.0" 404 204 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    64.68.82.142 - -
    At the mentioned time the bot tried to read the file "robots.txt" and got a 404 error, which means "Not found". I researched on the web for the "204" and found it means "No Content".


    [02/Aug/2003:08:24:46 -0400] "GET / HTTP/1.0" 200 48876 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
    195.202.89.107 - -
    The bot tried to read the file "/" (i.e. your top level index-file). "200" means "OK", so the bot got what he was looking for.


    The other 2 entries look the same to me, just a day later.

    Regards
    Ralph
  4. #3
  5. Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2003
    Location
    new york
    Posts
    11
    Rep Power
    0
    Hi Ralph

    [At the mentioned time the bot tried to read the file "robots.txt" and got a 404 error, which means "Not found". I researched on the web for the "204" and found it means "No Content".]

    > I don't have a robots.txt file on my server, I have robot instructions in my Meta tags. Do you think maybe it is best to have a robots.txt file on your server just to be safe?

    If that Googlebot found a robots.txt file then maybe it would have searched through the site more.

    Regards,

    Dylan
  6. #4
  7. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2003
    Location
    Stuttgart, Germany
    Posts
    4
    Rep Power
    0
    Dylan,

    Coming from the technical side (I am a programmer) I am more a lurker in this forum than a SEO expert. But I have read some posts that suggest it does not hurt to have a robots.txt.

    Maybe you want to check out what Webby wrote about that:
    http://www.searchguild.com/article4.html

    Regards
    Ralph
  8. #5
  9. Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2003
    Location
    new york
    Posts
    11
    Rep Power
    0
    Hi Ralph

    Good article, thanks. I think I'll add a robots.txt file to be safe although I'm not too concerned about keeping the bad spiders out. I'm only interested to invite and be a good host to the friendly spiders. I wish I could leave some cookies and milk in my robots file for them.

    Best regards,

    Dylan
  10. #6
  11. Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jan 2003
    Location
    Madrid, Spain
    Posts
    1,382
    Rep Power
    18
    Originally posted by New Yorker
    I wish I could leave some cookies and milk in my robots file for them.
    Put a nice sitemap on your homepage for them - they'll gobble it up

    Gringo.

Similar Threads

  1. Google Data Center Data - A Wee-bit
    By Surf-Dude in forum Google Optimization
    Replies: 0
    Last Post: Dec 4th, 2003, 07:27 PM
  2. Urgent Help with Googlebot
    By mariobox in forum Google Optimization
    Replies: 11
    Last Post: Sep 18th, 2003, 11:30 AM
  3. why googlebot dont like our section pages?
    By clif in forum Google Optimization
    Replies: 8
    Last Post: Jul 30th, 2003, 03:49 PM
  4. Nice Googlebot DOS Attack!!
    By Nintendo in forum Google Optimization
    Replies: 9
    Last Post: Mar 17th, 2003, 07:15 PM
  5. Large Variences in search term for 8 data centers
    By RicKimbell in forum Google Optimization
    Replies: 4
    Last Post: Mar 5th, 2003, 10:13 AM

IMN logo majestic logo threadwatch logo seochat tools logo