Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Spiders
Reload this Page

Robot.txt and Dynamic Pages...

Discuss Robot.txt and Dynamic Pages... in the Search Engine Spiders forum on SEO Chat.
Robot.txt and Dynamic Pages... Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
Application developers can seamlessly integrate the Advantage Database install with their application install. Learn the best practices used when setting up silent installs with this seminar.
  #1  
Old November 6th, 2007, 02:51 PM
luthervondude luthervondude is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Pittsburgh
Posts: 10 luthervondude User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 32 m 27 sec
Reputation Power: 0
Send a message via Google Talk to luthervondude
MySpace
Robot.txt and Dynamic Pages...
I have a 301 redirect set up on

/index.php?page=signup&sid=eaace6f74f82579cdeeae49120ba5629&key=y6YdmCZrUFxyaiiGBbiGdHHjTooSQeae

It is redirecting fine but the variable information is staying on the end of the URL and Yahoo and MSN are indexing it twice.

Can I safely create a robots.txt file to disallow

/index.php?page=signup

There are other redirects based on the "sid" variable so I want to disallow all links that have "page=signup".
Reply With Quote
  #2  
Old November 6th, 2007, 05:50 PM
dzine's Avatar
dzine dzine is offline
Vergruizer: Vot tebe khuy
SEO Chat Intermediate (1500 - 1999 posts)
  
Join Date: Oct 2005
Location: in a life preserver @ seorefugee
Posts: 1,841 dzine User rank is Sergeant (500 - 2000 Reputation Level)dzine User rank is Sergeant (500 - 2000 Reputation Level)dzine User rank is Sergeant (500 - 2000 Reputation Level)dzine User rank is Sergeant (500 - 2000 Reputation Level)dzine User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Month 4 Days 14 h 41 m
Reputation Power: 20
Yes you can.
Reply With Quote
  #3  
Old November 7th, 2007, 02:21 PM
luthervondude luthervondude is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Pittsburgh
Posts: 10 luthervondude User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 32 m 27 sec
Reputation Power: 0
Send a message via Google Talk to luthervondude
MySpace
Sweet...
That is perfect... so would this work?

Disallow: /index.php?page=signup*
Reply With Quote
  #4  
Old November 7th, 2007, 02:41 PM
Jean-Luc Jean-Luc is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2004
Location: Brussels, Belgium
Posts: 352 Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 5 Days 12 h 42 m 17 sec
Reputation Power: 6
Do not put a * at the end. You should use:
Code:
Disallow: /index.php?page=signup

This means that robots should not crawl URL's starting with /index.php?page=signup.

Jean-Luc
__________________
AWStats Support : add-on's, extra sections, forum, installation assistance
Get AWStats without the trouble of installing it
Checking redirects is now as easy as 1 2 3, even if you are not a HTTP-header guru !
Reply With Quote
  #5  
Old November 7th, 2007, 02:43 PM
luthervondude luthervondude is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Pittsburgh
Posts: 10 luthervondude User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 32 m 27 sec
Reputation Power: 0
Send a message via Google Talk to luthervondude
MySpace
Right... but there may also be a variable at the end...

/index.php?page=signup&a-bunch-o-crap-i-don't-want-indexed

So the * should eliminate it all, correct?
Reply With Quote
  #6  
Old November 7th, 2007, 02:53 PM
Jean-Luc Jean-Luc is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Dec 2004
Location: Brussels, Belgium
Posts: 352 Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level)Jean-Luc User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 5 Days 12 h 42 m 17 sec
Reputation Power: 6
The * as a special character is not part of the robots.txt standard.
Quote:
Originally Posted by luthervondude
Right... but there may also be a variable at the end...
That's why I wrote: "robots should not crawl URL's starting with /index.php?page=signup."

In other words, the line I wrote disallows access to:
- /index.php?page=signup
- /index.php?page=signup&sid=acdefgh...xyz
- /index.php?page=signup_followed_by_any_characters

Jean-Luc
Reply With Quote
  #7  
Old November 7th, 2007, 02:58 PM
luthervondude luthervondude is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Pittsburgh
Posts: 10 luthervondude User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 32 m 27 sec
Reputation Power: 0
Send a message via Google Talk to luthervondude
MySpace
Perfect. Thanks Jean-Luc
Reply With Quote
  #8  
Old November 9th, 2007, 03:12 PM
Dart Shop's Avatar
Dart Shop Dart Shop is offline
Dart Shop
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Sydney Australia
Posts: 111 Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 1 h 55 m 42 sec
Reputation Power: 2
Sorry
Last edited by Dart SHop : November 9th, 2007 at 03:17 PM. Reason: Unintentional post, meant to be new thread.
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Robot.txt and Dynamic Pages...

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

 Free IT White Papers!
 
Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.
 
Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.
 
Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.
 
Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five "checkpoints" for any implementation plan that will help make IP communications a successful proposition.
 
Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 3 hosted by Hostway.