bot tracking software

Hi what software do you all use to see what pages googlebot has spidered? Im new to this SEO stuff and would like to know what pages googlebot is visiting.

Thankyou

I don't use any software .. I just download my log files and scan them for the spiders I'm tracking!!

It's much faster this way!

Darrin do you have a list with all spider adresses?

Nope .. In the case of Google I do a search for ".googlebot.com" (the remote host) in my log file, then I keep my eye on the requested URL and keep hitting F3 (which is find next).. then I can see all the pages Google has requested. I do the sale for Inktomi except I replace ".googlebot.com" for "slurp@intomi.com" (the user agent).

As I've said previously in this thread:
http://www.google-dance.com/chat/viewtopic.php?t=49&highlight=grep

You can use this command if you can login to your server via SSH / Telnet and know the location of your logfile(s) to email you all records of Googlebot:

I use this a LOT, especially if the log file is over about 30 megs i.e. would take me more than about 1 minute to download!!

May I just say, that is sweet... why didn't I think of that! Using it now though

Amy

Darrin where exactly do you put this code .....in very simple instructions and I dont understand the part about logging into your server and the other SSH and the Tel thingy lol...I am a novice at this and I dont want to mess with something I do not understand.....thanks in advance ops: and God Bless Our Astraunauts and their families so sad...so very, very sad

if you have access to your server (telnet or ssh)

you can run the commands:

look at live googlebot hits to your site
tail -f /home/virtual/path/to/site/access_log | grep googlebot

look at last 10 hits to your site
tail /home/virtual/path/to/site/access_log | grep googlebot

look at all googlebot hits to your site
less /home/virtual/path/to/site/access_log | grep googlebot

[/b]

Darrin when i run that command i get:
grep : access_log: No such file or directory.

Thankyou

Look at johnny's post. You need to either specify the path to the access_log or be in the directory with it in at the time.

I cant seem to find where my access_log is located. Can anyone help me find where it is? i need the path.

Thankyou

At the command line as root type:-

locate access_log

That'll tell you where any files called access log are.

Hi thanks for the relpy.

I was trying that before and was getting:
warning: locate: could not open database: /var/lib/slocate/slocate.db: Permission denied.

Ive tryed contacting my host with no reply.

Thankyou

make sure you are logged in as root

to log in as root (superuser) type in: su
then you should be asked your root pw, enter it and hit enter.

then try to see if it will work

As I said, you need to be root. However, if you have a virtual account or a managed hosting account on a dedicated server then you may not get the root password from them.

In which case you need to tell them that you want to be able to get full access to your access_logs and they may set up something different for you.

I don't know your circumstances, but they're almost certainly different to mine (I manage my own servers).

Good Luck.

Amy

For those of you without the knowledge or direct access to your server, consider a tracking software program, but be careful that you don't use one that is not "interactive". Webtrends log Analyzer will tell you how many spiders visits there were and how many pages they went to, but it's basically a hard copy report. I use Nettracker, which allows you to drill down into the spider visits and find out all of the information that is in the log, such as which pages, in what order, the length of time spent, etc. While all this information is in the logs, it might be easier to understand in this format.

J-Net