HTML Coding
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Other > HTML Coding
Reload this Page

robot.txt file

Discuss robot.txt file in the HTML Coding forum on SEO Chat.
robot.txt file HTML Coding forum discussing anything related to HTML coding, such as using proper HTML code, optimizing code to be spidered easily, creating layers and tables. Make pages compatible for different web browsers like IE, Firefox, and Opera.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old April 21st, 2003, 11:42 PM
vishal's Avatar
vishal vishal is offline
Junior Member
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2003
Location: noida
Posts: 28 vishal User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
Send a message via Yahoo to vishal
robot.txt file
what is robot.txt file and how it benefit for SEO .....

explain in detail
Reply With Quote
  #2  
Old April 22nd, 2003, 07:09 AM
wm_asia wm_asia is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Mar 2003
Posts: 319 wm_asia User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 6 m 39 sec
Reputation Power: 6
Hi,

Because there are many questions already asked before it could be wise to check out previous postings about robots.txt

The quality of this board would go down if the same questions would be repeated over and over again with the same answers given.

So please use the search function....and type robots.txt , sure
you will find the information to answer your question.

regards, hans
Reply With Quote
  #3  
Old April 22nd, 2003, 07:23 AM
Pooner Pooner is offline
Junior Member
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2003
Posts: 23 Pooner User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 m 40 sec
Reputation Power: 0
Quite right - heard of google?
Yes, and also take the first 3 words of your question and...

http://www.google.com/search?q=what+is+robot.txt&sourceid=mozilla-search&start=0&start=0
Reply With Quote
  #4  
Old April 22nd, 2003, 09:20 AM
rustybrick's Avatar
rustybrick rustybrick is offline
Contributing User
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2003
Location: New York, USA
Posts: 2,642 rustybrick User rank is Private First Class (20 - 50 Reputation Level)rustybrick User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 2 Days 17 h 40 m 42 sec
Reputation Power: 8
What is the benefit - if you do not want to disallow any of your pages?

I don't currently use a robot.txt file b/c I don't care if it crawls everything.

Things the spiders wont get to are password protected.
Reply With Quote
  #5  
Old April 22nd, 2003, 11:21 AM
sensovision's Avatar
sensovision sensovision is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Mar 2003
Location: Ukraine
Posts: 87 sensovision User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 6
Send a message via ICQ to sensovision Send a message via AIM to sensovision Send a message via Yahoo to sensovision
Quote:
Originally posted by "rustybrick"

What is the benefit - if you do not want to disallow any of your pages?

I don't currently use a robot.txt file b/c I don't care if it crawls everything.

Things the spiders wont get to are password protected.

there are some reasons to not allow spiders to visit some pages, it could be e.g. if you don't want visitors to find page in SE results but want only home page to be visible in SE, or maybe page contain something which couldn't be good for spider e.g. if you're using anti-SPAM scripts like WPoison and in order to not let legitimate spiders to be trapped in bogus pages you should disallow this pages to be crawled so crawlers which obey robot instructions will not visit them(SPAM Bots probably ignore this rules anyway and would be trapped...). Also you can do this in order to simply save bandwidth(of course it's pretty drastic measure ).
__________________
Denis
Webmaster Key - Discussion Forums
Pharos Search - A Human Edited Directory
Reply With Quote
  #6  
Old April 22nd, 2003, 11:55 AM
rustybrick's Avatar
rustybrick rustybrick is offline
Contributing User
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2003
Location: New York, USA
Posts: 2,642 rustybrick User rank is Private First Class (20 - 50 Reputation Level)rustybrick User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 2 Days 17 h 40 m 42 sec
Reputation Power: 8
Quote:
there are some reasons to not allow spiders to visit some pages, it could be e.g. if you don't want visitors to find page in SE results but want only home page to be visible in SE, or maybe page contain something which couldn't be good for spider e.g. if you're using anti-SPAM scripts like WPoison and in order to not let legitimate spiders to be trapped in bogus pages you should disallow this pages to be crawled so crawlers which obey robot instructions will not visit them(SPAM Bots probably ignore this rules anyway and would be trapped...). Also you can do this in order to simply save bandwidth(of course it's pretty drastic measure).


But if I want it to spider every page of mine then why have it?

Do I need to disallow a .css or .js page? They wouldn't get anything from it anyway. Let's talk about a fairly small >20 page site with fairly static content.
Reply With Quote
  #7  
Old April 22nd, 2003, 12:22 PM
tonyfelice's Avatar
tonyfelice tonyfelice is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jan 2003
Location: PA
Posts: 214 tonyfelice User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 6
Send a message via ICQ to tonyfelice Send a message via AIM to tonyfelice Send a message via Yahoo to tonyfelice
Rustybrick

Why not just have a robots.txt file that allows all?

User-agent: *
Disallow:

That will acomplish the same thing and the robots will actually find a file on your server when they go looking for it.
__________________
Soli Deo Gloria,
Tony
Reply With Quote
  #8  
Old April 22nd, 2003, 12:39 PM
mr_tim mr_tim is offline
Junior Member
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2003
Posts: 8 mr_tim User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
yep - having one that allows all just prevents a 404 error which is worthwhile i suppose. gotta keep those spiders happy!
Reply With Quote
  #9  
Old April 22nd, 2003, 12:45 PM
rustybrick's Avatar
rustybrick rustybrick is offline
Contributing User
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2003
Location: New York, USA
Posts: 2,642 rustybrick User rank is Private First Class (20 - 50 Reputation Level)rustybrick User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 2 Days 17 h 40 m 42 sec
Reputation Power: 8
Quote:
Originally posted by "mr_tim"

yep - having one that allows all just prevents a 404 error which is worthwhile i suppose. gotta keep those spiders happy!


I have a 404 page. Are search engines happier if you have a robot.txt file that allows everything? Whats the point.

Anyone know what is the point for a robot.txt file for a site that wants to be fully crawled?
Reply With Quote
  #10  
Old April 22nd, 2003, 12:59 PM
wm_asia wm_asia is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Mar 2003
Posts: 319 wm_asia User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 6 m 39 sec
Reputation Power: 6
Quote:
Originally posted by "rustybrick"

What is the benefit - if you do not want to disallow any of your pages?

I don't currently use a robot.txt file b/c I don't care if it crawls everything.

Things the spiders wont get to are password protected.


Even IF a search engine should index all....it's wise to have a robots.txt
file in the root of your website.

First to prevent a 404 error (not only one but many since SE are looking for this file first before indexing your website) ....and to see how many spiders visits are coming to your website, since normal visitors will not look for it !!!

Your stats will show it.....!!!

regards, hans
Reply With Quote
  #11  
Old April 22nd, 2003, 01:24 PM
rustybrick's Avatar
rustybrick rustybrick is offline
Contributing User
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2003
Location: New York, USA
Posts: 2,642 rustybrick User rank is Private First Class (20 - 50 Reputation Level)rustybrick User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 2 Days 17 h 40 m 42 sec
Reputation Power: 8
thanks
Reply With Quote
Reply

Viewing: SEO Chat Forums > Other > HTML Coding > robot.txt file

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 2 hosted by Hostway. 
Stay green...Green IT