Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Spiders
Reload this Page

How did the spiders find me?

Discuss How did the spiders find me? in the Search Engine Spiders forum on SEO Chat.
How did the spiders find me? Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
Speeding Shipping, Improving Compliance – Read the ScanCode Systems Case Study
  #1  
Old May 21st, 2007, 05:02 PM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
How did the spiders find me?
I have a one page site with no outgoing or incoming links from it or to it. It is a static page that I just put up on my host about 3 weeks ago to see how it looked.

My url is not and has not been written out on any other site, forum, chat room, etc.

How the heck did they index my site??????
Reply With Quote
  #2  
Old May 21st, 2007, 05:11 PM
Le GoogelGuRu Le GoogelGuRu is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2007
Posts: 2 Le GoogelGuRu User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 m 29 sec
Reputation Power: 0
Are you entirely sure your site isn't in any backlink anywhere on the web? :confused:
Reply With Quote
  #3  
Old May 21st, 2007, 07:26 PM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
absolutely positive. When searching for links to my site on Google - no links are found. However, when I type in the name of my site, I find its indexed.
Last edited by BillyA : May 21st, 2007 at 07:29 PM.
Reply With Quote
  #4  
Old May 21st, 2007, 07:43 PM
SEO_AM's Avatar
SEO_AM SEO_AM is offline
B afraid.. B very afraid!
SEO Chat God 1st Plane (5500 - 5999 posts)
  
Join Date: Mar 2004
Location: Land of enchantment... deserts of the Southwest
Posts: 5,876 SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level)SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level)SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level)SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level)SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level)SEO_AM User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 2 Months 2 Weeks 3 Days 22 h 40 m 5 sec
Reputation Power: 47
No big mystery. Google is not showing your link(s). If you did not submit manually to Google, try a link command on Yahoo and/or MSN. I guarantee, if you are truly indexed, you will find one or more backlinks. You will not find all backlinks using Google.
Reply With Quote
  #5  
Old May 21st, 2007, 08:12 PM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
I've used this to check:

http://www.prsearch.biz/inbx.php

and have used the link: function in Google and Yahoo to try to find any links. There are none.

However, when I type in the name of my site, the page is indexed. With no links in and no links out, how does Google even know the site is there?

The reason I bring it up is that its a download page for sensitive files and I don't want it indexed. I now know I have to include the meta tag robot command to stop the indexing, but since there were no links in or out, I never thought it would even get indexed.

The only thing I can think of - could it be my hosting company in some way?
Last edited by BillyA : May 21st, 2007 at 08:22 PM.
Reply With Quote
  #6  
Old May 21st, 2007, 08:26 PM
ClickyB's Avatar
ClickyB ClickyB is offline
Scru u guys I'm goin home
Click here for more information.
  
Join Date: Mar 2004
Location: Gloucester (South West UK).
Posts: 3,624 ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level)ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level)ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level)ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level)ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level)ClickyB User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 2 Months 3 Weeks 1 Day 11 h 38 m 47 sec
Reputation Power: 44
Personally I agree with SEO_AM.

I have a feeling that Google can sometimes spider a server and log any domains without .htaccess or robots.txt exclusion... but I wouldn't expect them to index a site unless they found links to it somewhere.

It could be that your host/server has indexed pages (stats etc) which include links to all domains or it could be something like the domain name was previously used etc.

Quite honestly - now you know you're indexed - I don't see any point in trying to second guess the reason

Be aware that "the link: command" in Google is no use at all in determining whether a site has backlinks or not... it shows only a random sample of links - thus may show you "0 of 3" !
Comments on this post
googler agrees!
__________________
ClickyB
"The quality of the visitor is more important than the volume".. Egol 22nd Feb 2008
New to SEO? Start Here: SEO FAQ
Forum Rules & Posting Guidelines
Last edited by ClickyB : May 21st, 2007 at 08:28 PM.
Reply With Quote
  #7  
Old May 21st, 2007, 08:32 PM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
Yeah, thanks for the responses....

Just a little peeved as its cost me a domain name.
Reply With Quote
  #8  
Old May 21st, 2007, 10:04 PM
googler's Avatar
googler googler is offline
Cool Dude
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Aug 2003
Location: Vancouver, Washington, U.S.A.
Posts: 1,389 googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 16 h 30 m 59 sec
Reputation Power: 13
Send a message via AIM to googler Send a message via Google Talk to googler Send a message via XFire to googler
It happens a lot to me. When I buy a domain name which is usually through godaddy or network solutions and then put a hosting account on it it gets spidered, with no links. Never really thought about it but it could have to do with being on shared Ip addresses or from google looking at whois records. or server domain lists
Comments on this post
SEO_AM agrees: Yep... Somebody somewhere is pointing at the site.
Reply With Quote
  #9  
Old May 22nd, 2007, 04:37 AM
ChrisWade ChrisWade is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2007
Posts: 10 ChrisWade User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 5 h 19 m 59 sec
Reputation Power: 0
Another reason for the Indexing could be the hosting service you used. When you registered the domain it could have been listed on the site which would cause a backlink that you can not find out about.

The only way to prevent this from happening from what I have been able to find out is by using the robots.tx and including your site as an excluded page.
Comments on this post
googler agrees: Robots.txt is the simplest and prefferred solution to this problem
Reply With Quote
  #10  
Old May 22nd, 2007, 07:44 AM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
For a one page site do I need to have a robot.tx file and list the page to be excluded? Or will the meta tag instruction on that page be fine?
Reply With Quote
  #11  
Old May 22nd, 2007, 07:53 AM
pro_seo's Avatar
pro_seo pro_seo is online now
Moderator
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2006
Location: I N D I A
Posts: 2,604 pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 21 h 10 m 35 sec
Reputation Power: 30
Send a message via AIM to pro_seo Send a message via MSN to pro_seo Send a message via Yahoo to pro_seo Send a message via Google Talk to pro_seo Send a message via Skype to pro_seo
Quote:
Originally Posted by BillyA
For a one page site do I need to have a robot.tx file and list the page to be excluded? Or will the meta tag instruction on that page be fine?


robots.txt is the best option always.

Coz though chances are very less, robots meta tag can still be overlooked by the SE spiders.

So robots.txt is the best exclusion tool you have
__________________

Social Mobile Messaging
SEO FAQs - You might find your answer here.
SEOchat Forum Rules - Read Before You Post

**Do what you feel in your heart to be right- for you'll be criticized anyway. You'll be damned if you do, and damned if you don't.**
Reply With Quote
  #12  
Old May 22nd, 2007, 11:55 AM
googler's Avatar
googler googler is offline
Cool Dude
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Aug 2003
Location: Vancouver, Washington, U.S.A.
Posts: 1,389 googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level)googler User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 16 h 30 m 59 sec
Reputation Power: 13
Send a message via AIM to googler Send a message via Google Talk to googler Send a message via XFire to googler
User-agent: *
Disallow: /


Thats it save it as robots.txt and upload it to the root of your webserver. Some rouge bots ignore robots.txt though so if the info is very sensitive you may want to lock the folder and put a User Name and Password on it
Reply With Quote
  #13  
Old May 22nd, 2007, 06:41 PM
BillyA BillyA is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2006
Posts: 30 BillyA User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 13 h 31 m 45 sec
Reputation Power: 2
Thanks everyone! You guys have been a great help!!
Reply With Quote
  #14  
Old September 20th, 2007, 03:53 AM
gibi gibi is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Sep 2007
Posts: 18 gibi User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 h 18 m 17 sec
Reputation Power: 0
i had the same problem
real hidden webpage ...
and the google found it from my google toolbar ...
Reply With Quote
  #15  
Old April 1st, 2008, 04:00 AM
nasskobar nasskobar is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Mar 2008
Posts: 4 nasskobar User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 h 6 m 45 sec
Reputation Power: 0
One more concern though
Okay, so I used robot.txt to have my website crawled. What is interesting is that I find some of the of the robots.txt file has folders that are usually not shown or found like the /admin or /adminonline or /superuser in the "disallow" part.

So, why is it that users do use these folder names here so openly? How can I prevent any body from just seeing it?

Its so easy you see, if I type a website with /robot.txt it would simply show me the contents.
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > How did the spiders find me?

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread: