Search Engine Spiders
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Spiders
Reload this Page

Robot.txt restriction advice

Discuss Robot.txt restriction advice in the Search Engine Spiders forum on SEO Chat.
Robot.txt restriction advice Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Closed Thread
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
A high performance database engine using optimized data access for all development environments including Delphi, Visual Studio .NET, Visual Basic, Visual FoxPro. and more. Learn More
  #1  
Old November 9th, 2007, 03:20 PM
Dart Shop's Avatar
Dart Shop Dart Shop is offline
Dart Shop
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Sydney Australia
Posts: 111 Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 1 h 55 m 42 sec
Reputation Power: 2
Robot.txt restriction advice
I was exchanging a link with a person and he advised me my Resources Page cant be indexed. Checking my robots.txt file i can see my 5 static pages i have, are restricted from being crawled or indexed via dissallow of pages.php

I actually want these pages to be indexed and seen by the spiders etc, but i dont know if i should take this line out of the robots.txt files, so that it can be crawled.

Can somebody pls advise if this would be restricted for a reason, security issue etc and if allowing spiders to crawl pages.php is ok. Im just concerned at changing something that I have little to no knowledge about.

I need to know any security repurcussions etc in SIMPLE terms.

Many thanks.
Reply With Quote
  #2  
Old November 10th, 2007, 12:35 AM
pro_seo's Avatar
pro_seo pro_seo is offline
Moderator
SEO Chat Frequenter (2500 - 2999 posts)
  
Join Date: Apr 2006
Location: I N D I A
Posts: 2,604 pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level)pro_seo User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 1 Month 2 Weeks 21 h 10 m 35 sec
Reputation Power: 30
Send a message via AIM to pro_seo Send a message via MSN to pro_seo Send a message via Yahoo to pro_seo Send a message via Google Talk to pro_seo Send a message via Skype to pro_seo
Can we have a look at your site or robots.txt file content ?
__________________

Social Mobile Messaging
SEO FAQs - You might find your answer here.
SEOchat Forum Rules - Read Before You Post

**Do what you feel in your heart to be right- for you'll be criticized anyway. You'll be damned if you do, and damned if you don't.**
Reply With Quote
  #3  
Old November 10th, 2007, 12:41 AM
Dart Shop's Avatar
Dart Shop Dart Shop is offline
Dart Shop
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Oct 2007
Location: Sydney Australia
Posts: 111 Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level)Dart Shop User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 1 Day 1 h 55 m 42 sec
Reputation Power: 2
Thanks for replying, I really should have updated this question.

I actually found the problem after some guidance and all is now well.

I appreciate you repying though. Thanks.


****ALL FIXED NOW****
Reply With Quote
Closed Thread

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Robot.txt restriction advice

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

 Free IT White Papers!
 
Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.
 
Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.
 
Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.
 
Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five "checkpoints" for any implementation plan that will help make IP communications a successful proposition.
 
Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway.