Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat ForumsSearch Engine StrategiesSearch Engine Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old January 14th, 2005, 04:13 AM
thall89553 thall89553 is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Aug 2004
Posts: 277 thall89553 User rank is Private First Class (20 - 50 Reputation Level)thall89553 User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 20 h 17 m 15 sec
Reputation Power: 5
Please check my robots.txt

User-agent: *
Disallow: /includes/
Disallow: /database/
Disallow: /scripts/
Disallow: /images/
Disallow: /gallery/
Disallow: /panoramas/

I am curious, do I need the "/" at the end of each of these? Otherwise, is everything else ok?




Reply With Quote
  #2  
Old January 14th, 2005, 04:20 AM
Wit's Avatar
Wit Wit is offline
http://tinyurl.com/cz56g
SEO Chat God 2nd Plane (6000 - 6499 posts)
 
Join Date: Sep 2004
Location: D0RDRECHT NL
Posts: 6,065 Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 2 Months 6 Days 10 h 52 m 26 sec
Reputation Power: 18
Your robots.txt is fine. The trailing slash tells the bot you only want to hide these directories (and everything in it). If you removed those slashes ("/"), any FILES starting with /includes or /database etc. would be hidden too.

For example:
/gallery-of-widgets.html would be hidden as well.

But in your robots.txt those filenames are NOT hidden, only the directories.
I presume that's what you intended, so it's perfect.
__________________
...please help me w/ the real Redscowl Bluesingsky...how2 check backlinks...now postin' @ SEO Refugee ...
·`)~ LOL now that I finally have a paypal account, I'm charging 19,- for SEO advice via PM. Seriously...

Reply With Quote
  #3  
Old January 14th, 2005, 09:09 AM
thall89553 thall89553 is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
 
Join Date: Aug 2004
Posts: 277 thall89553 User rank is Private First Class (20 - 50 Reputation Level)thall89553 User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 20 h 17 m 15 sec
Reputation Power: 5
Hey a follow up please, I saw this today for Google Disallow: /*? (or something like that). If I leave that out, they will index pages whose url's contain variables, no?

Reply With Quote
  #4  
Old January 14th, 2005, 09:44 AM
Wit's Avatar
Wit Wit is offline
http://tinyurl.com/cz56g
SEO Chat God 2nd Plane (6000 - 6499 posts)
 
Join Date: Sep 2004
Location: D0RDRECHT NL
Posts: 6,065 Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level)Wit User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 2 Months 6 Days 10 h 52 m 26 sec
Reputation Power: 18
Yes, but you should in fact leave everything out, and only disallow spidering of pages/dirs that you really want to "hide". Note that you can't really hide the pages, sometimes they're read anyway, just not indexed. And sometimes they are indexed anyway (accidentally or on purpose).

Googlebot can read wildcards in the url strings (to some extent). You're right that this particular phrase would tell g-bot to leave pages with variables and search queries alone... I guess you only want that if you have a proper mod_rewrite in place, and only want the "cleaned-up" urls to appear on the serps.

Reply With Quote
Reply

Viewing: SEO Chat ForumsSearch Engine StrategiesSearch Engine Optimization > Please check my robots.txt


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump



 Free IT White Papers!
 
How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.

Request Your Free Technology Downloads!
 
Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.

Request Your Free Technology Downloads!
 
Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.

Request Your Free Technology Downloads!
 
Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.

Request Your Free Technology Downloads!
 
Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.

Request Your Free Technology Downloads!
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2009 by Developer Shed. All rights reserved. DS Cluster 3 hosted by Hostway
Stay green...Green IT