Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Aug 2003
    Location
    Idaho USA
    Posts
    158
    Rep Power
    16

    Has anyone found a script to track Googlebot through a website?


    I have not found a easy way to follow Googlebot through a site and discover what pages it looks at the most. I like to know which pages GoogleBot has been scanning on a site. If I had this information (the pages being hit most often) those pages would get special treatment. Does anyone know of a script I can install on my site that will track what the BOTS are looking at in detail!
  2. #2
  3. No Profile Picture
    Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Detroit
    Posts
    540
    Rep Power
    16
    There are scripts, unfortunately i cant give you any directions to free ones, we have many we have coded internally and i cant give those out, but i can mention another method if you dont mind doing some extra work until you find the code or find a coder that can do it for you.

    Take an excel spreadsheet and make a few tabs for different spread sheets and make 12 months, and average 29 days per each spread sheet.. so 29 columns representing days.

    Go through your raw log file every 24 hour period until the next day. Search for "Googlebot" and look which .html/.php/.jsp whichever pages you have it is visiting and in each day mark it down in order so you can make a tree of which files by # are viewed more per month, so you could even right list all your file names and put numbers in each day how many times googlebot hit it.. so you would have a 29 day long spreed sheet (columns) with "X" amount of file names on the right per each day.. this graph should give you the info you need to tweak your users to the proper page that you have noticed google see more and which brought you more traffic..

    Hope this helps.
  4. #3
  5. Head Honcho
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Chicago, IL
    Posts
    556
    Rep Power
    16
    I wrote one for one of my clients. Are you running PHP? I should have a free version done soon (read: Jan 1st 2004) that you can install, but it is PHP/MySQL. Let me know if you want to be a Beta tester and I can get an early release candidate out to you much sooner.
  6. #4
  7. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Aug 2003
    Location
    Idaho USA
    Posts
    158
    Rep Power
    16
    I am looking for a tried and true script, not really interested in being a Beta tester -- thanks though.
  8. #5
  9. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Aug 2003
    Location
    Idaho USA
    Posts
    158
    Rep Power
    16
    Has anyone used Darrin Ward's SpyderTrax ?
  10. #6
  11. Head Honcho
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Chicago, IL
    Posts
    556
    Rep Power
    16
    First of all, you should know that in the software industry "Beta" doesn't necessarily mean "unfinished". I just sold a license to a client for a nice pre-paid yearly sum, and they didn't seem to mind the fact that we had JUST CODED IT the night before (aka "Beta").

    If we waited until the script was "perfect" before we released it, it would be 2006 before it came out. I guarantee that most of the scripts you will find out there, even GoogleStats and SpyderTrax, didn't go through a beta phase because the authors wanted to get to market ASAP.

    For those of you that might be interested in Beta testing my script, send me a PM.

    Here is what the script also includes (aside from Google tracking):

    (1) afterBurner caching of your pages - 5x avg. speed increase delivering your HTML content to the user;

    (2) dynamic mirroring of your webpages through a MySQL database - if your server is down and doesn't respond for 5 seconds, the script serves a day-old cache to the user instead;

    (3) keyword hunting - when users surf your site, the script looks at each page and determines which keywords each page is optimized for, and then checks Google for EACH PAGE to see if each page has been indexed yet;

    (4) real-time stats - a toolbar at the bottom of every page on your site (that you can turn on and off) shows you (a) if that particular page has been scanned by Google yet; (b) how many times Google has been to that page; (c) what keyword Google ranked that page for, and what rank; (d) how many visitors came from Google to that page; (e) the most common keyword on that page.

    (5) Code Compress - compress your HTML output up to 40% by ZIPping it up before it is sent to the browser, and then unzipping it when it finally gets to the user. This feature is seperate from afterburnerCache, but they do work together.

    (6) Step Spy - find the path that Google takes through your website on each visit.

    (7) Quick Submit - the program tracks every page on your site to make sure it is indexed in Google, and if it has never been indexed it will submit it that very instant.

    Like I said, if anyone wants to try it out send me a PM and I'll add you to the list. We originally wrote it for a client but we are working on a release candidate for public use, and we'd like some of the people here to try it out before we actually let it go out. But it will take two or three weeks to get out the RC1 to the beta testers, so please be patient! ;-)
    Last edited by Manga27; Oct 13th, 2003 at 02:25 PM.
  12. #7
  13. iTamer
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Apr 2003
    Location
    Auckland, New Zealand
    Posts
    170
    Rep Power
    17
    Manga that sounds awesome.

    On a slightly smaller scale I should have my Botspotter service complete soon. Take a look at http://www.pcpropertymanager.com/botspotter/home.php

    I've already got it running in house and just have to get it ready for the public.

    I was having problems with the SourceForge version and decided to simplify it.
  14. #8
  15. Head Honcho
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2003
    Location
    Chicago, IL
    Posts
    556
    Rep Power
    16
    Sarah,

    I notice you became the project admin of BotSpotter and then took away the project files. Do you have the original unfinished project files that were on SourceForge? I'd be interested in taking a look.

    Maybe you want to work together on this? We seem to be developing similar services.

    Send me a PM. ;-)

    Take care,

    ... Christopher
  16. #9
  17. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2003
    Posts
    61
    Rep Power
    16
    You should check out “spider tracks” it runs on CGI. I'm not sure if the author still supports it. I’ve used it and found it simple to install and it worked for what I wanted.

    http://www.rainbo.net/pub/spiderTrac...derTracks.html
  18. #10
  19. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2003
    Location
    Israel
    Posts
    411
    Rep Power
    16

    script to track google crawl


    yep, there is a free one
    check out http://www.crawler-alert.com
  20. #11
  21. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Location
    Leeds, England
    Posts
    487
    Rep Power
    16
    Ive tried crawler alert, ran the test and didnt receive an email BTW I used the correct email address!
  22. #12
  23. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jul 2003
    Posts
    135
    Rep Power
    16
    Wow Manga, that sounds pretty slick... I've got a much simpler version that I've made available free (PHP/MySQL)

    http://www.marketing-seo.com/software-tools.php

    It tracks unique hits on all your pages as well as every hit by Googlebot and Slurp... I'm working on a version with more advanced features, but still nothing like what Manga's got there ;)
  24. #13
  25. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    80
    Rep Power
    17
    When you say PHP/MySQL does that mean the site has to be in PHP running on an SQL database or does the host just have to support them? My site is an ASP with SEO's htm pages running on an Access database. I've also signed up to crawler-alert.com and did everything right but haven't received any mails yet? I presume it only mails on deep crawls which is why i haven't had any mails and which would be more what I'm looking for since googlebot visits every day. Great service for free! Looking forward to getting that mail!
  26. #14
  27. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jul 2003
    Posts
    135
    Rep Power
    16
    Yeah, my script needs you to be running PHP pages, and have access to a MySQL database (you can use a different type of database for the rest of your site if you like).

    Basically, there is a PHP script that needs to be included on each page you want to track, and the data it records is stored in a MySQL database.

    You can see a live demo of what the script does here:

    http://directory.bonustracker.com/webstats/
  28. #15
  29. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2003
    Posts
    80
    Rep Power
    17
    Could i put the php script on our htm pages which are the seo'd ones so really all we would want to track anyway and link it up to a SQL database just for it, our forum is linked up to a SQL database at the moment so setting up another would be no problem?
Page 1 of 2 12 Last
  • Jump to page:

Similar Threads

  1. Reality & Truth - Competative Keywords -v- Real Life Searches
    By -search-engines-web in forum Google Optimization
    Replies: 2
    Last Post: Dec 27th, 2003, 05:25 PM
  2. Spiderview script file not found
    By Cor in forum Web Design, Coding and Programming
    Replies: 0
    Last Post: Aug 15th, 2003, 07:53 AM
  3. Cool script: Emails you when googlebot hits your domain (*)
    By Angus in forum Web Design, Coding and Programming
    Replies: 87
    Last Post: Jul 12th, 2003, 12:14 AM
  4. Googlebot php script = .txt log?
    By Nintendo in forum Google Optimization
    Replies: 9
    Last Post: Feb 23rd, 2003, 11:56 AM

IMN logo majestic logo threadwatch logo seochat tools logo