Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Spiders
Reload this Page

Robots.txt

Discuss Robots.txt in the Search Engine Spiders forum on SEO Chat.
Robots.txt Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
Stay one step ahead of the competition. Evaluate and give feedback on some of the hottest web development tools on the market today. Make your opinion heard! Click Here
  #1  
Old October 3rd, 2007, 06:53 AM
SquirrelJ SquirrelJ is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2007
Posts: 157 SquirrelJ User rank is Lance Corporal (50 - 100 Reputation Level)SquirrelJ User rank is Lance Corporal (50 - 100 Reputation Level)SquirrelJ User rank is Lance Corporal (50 - 100 Reputation Level) 
Time spent in forums: 3 Days 10 h 49 m 35 sec
Reputation Power: 2
Robots.txt
Would the following be a correct way to produce a robots.txt. I am not interested in hiding any urls from the spiders

Code:
User-agent: *
Disallow:

Sitemap: <http://www.domain.com/sitemap.xml>
Reply With Quote
  #2  
Old October 3rd, 2007, 06:57 AM
JagNet's Avatar
JagNet JagNet is offline
Smoke me a kipper...
Click here for more information
 
Join Date: Aug 2007
Posts: 1,458 JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level)JagNet User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 3 Weeks 6 Days 8 h 25 m 4 sec
Reputation Power: 23
If you don't need to block any robots from any areas of the site, just include the sitemap reference (without the <>):

Code:
Sitemap: http://www.domain.com/sitemap.xml
Reply With Quote
  #3  
Old October 20th, 2007, 05:21 PM
pondlife pondlife is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Feb 2007
Location: UK
Posts: 49 pondlife User rank is Private First Class (20 - 50 Reputation Level)pondlife User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 4 h 46 m 2 sec
Reputation Power: 2
Quote:
Originally Posted by JagNet
If you don't need to block any robots from any areas of the site, just include the sitemap reference (without the <>):

Code:
Sitemap: http://www.domain.com/sitemap.xml


I've not done much work with my robots file because I can't see why you'd want to block any part of your site to spiders... Am I being thick here? Could someone explain this to me please?
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Robots.txt

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

 Free IT White Papers!
 
Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.
 
Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.
 
Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.
 
Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five &quot;checkpoints&quot; for any implementation plan that will help make IP communications a successful proposition.
 
Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 4 hosted by Hostway.