Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
User Name:
Password:
Remember me
Go Back   SEO Chat ForumsSearch Engine StrategiesSearch Engine Spiders

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old November 29th, 2007, 01:46 PM
Swirleigh's Avatar
Swirleigh Swirleigh is offline
SEO Gawdess
SEO Chat Novice (500 - 999 posts)
 
Join Date: Jul 2006
Location: San Francisco Bay Area
Posts: 586 Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 9 h 45 m 59 sec
Reputation Power: 10
Mod_rewrite & robots.txt

I wish to disallow indexing of a couple directories of a website that is using mode-rewrite.

Do I list the "real" directory folder names or the mode-rewriten names in the robots.txt for exclusion?

I am thinking the mode-rewrite names, but I want to make sure.

Reply With Quote
  #2  
Old November 29th, 2007, 03:03 PM
dzine's Avatar
dzine dzine is offline
Vergruizer: Vot tebe khuy
SEO Chat Regular (2000 - 2499 posts)
 
Join Date: Oct 2005
Location: in a life preserver @ seorefugee
Posts: 2,188 dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Month 1 Week 3 Days 18 h 27 m 50 sec
Reputation Power: 58
That depends.

If you still have links - either internal or inbound - pointing to the old URLs then you have to block those as well.

For starters, why not try a site:yoursite.com search on google and yahoo to see if they still have some of the "old" urls in the index
__________________
Oooh, my 8000th ...Love my host...Check your gender ...

Reply With Quote
  #3  
Old November 30th, 2007, 12:18 PM
Swirleigh's Avatar
Swirleigh Swirleigh is offline
SEO Gawdess
SEO Chat Novice (500 - 999 posts)
 
Join Date: Jul 2006
Location: San Francisco Bay Area
Posts: 586 Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 9 h 45 m 59 sec
Reputation Power: 10
The site is under developement. There is no "old" urls.

I am kind of new to mode-rewrite... and usually want all sections of site listed, but we are creating a section of site that users can only see after filling in a form. That section I do not want listed in search results. So... that section I want to disallow in robots.txt.

Question being... to disallow the mode-rewrite version name directories (/topic/detail.html) of pages, or... to disallow the actual folders in my ftp list for those pages (/after-form/subject123id549087723ddsw/)?

Reply With Quote
  #4  
Old November 30th, 2007, 12:45 PM
dzine's Avatar
dzine dzine is offline
Vergruizer: Vot tebe khuy
SEO Chat Regular (2000 - 2499 posts)
 
Join Date: Oct 2005
Location: in a life preserver @ seorefugee
Posts: 2,188 dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level)dzine User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Month 1 Week 3 Days 18 h 27 m 50 sec
Reputation Power: 58
Just disallow the url that you use on your pages

Reply With Quote
  #5  
Old November 30th, 2007, 12:45 PM
JagNet's Avatar
JagNet JagNet is offline
Smoke me a kipper...
SEO Chat Regular (2000 - 2499 posts)
 
Join Date: Aug 2007
Posts: 2,487 JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level)JagNet User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 4 Days 3 h 37 m 22 sec
Reputation Power: 63
Use the rewritten version (that's the version that Google sees).
__________________
... I'll be back for breakfast
New to SEO? SEOChat SEO FAQs
Forum Rules and Posting Guidelines
URL canonicalization code solutions
Vigorously pursuing the floccinaucinihilipilification of cheap SEO tricks

Reply With Quote
  #6  
Old November 30th, 2007, 12:49 PM
Swirleigh's Avatar
Swirleigh Swirleigh is offline
SEO Gawdess
SEO Chat Novice (500 - 999 posts)
 
Join Date: Jul 2006
Location: San Francisco Bay Area
Posts: 586 Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level)Swirleigh User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 9 h 45 m 59 sec
Reputation Power: 10
Thanks!

Reply With Quote
Reply

Viewing: SEO Chat ForumsSearch Engine StrategiesSearch Engine Spiders > Mod_rewrite & robots.txt


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump



 Free IT White Papers!
 
How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.

Request Your Free Technology Downloads!
 
Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.

Request Your Free Technology Downloads!
 
Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.

Request Your Free Technology Downloads!
 
Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.

Request Your Free Technology Downloads!
 
Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.

Request Your Free Technology Downloads!
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2009 by Developer Shed. All rights reserved. DS Cluster 3 hosted by Hostway
Stay green...Green IT