Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engine Strategies > Search Engine Optimization
Reload this Page

Spider should not crawl my website!!What can I Do?

Discuss Spider should not crawl my website!!What can I Do? in the Search Engine Optimization forum on SEO Chat.
Spider should not crawl my website!!What can I Do? Search Engine Optimization forum discussing general tips and tricks to optimize your website for the search engines as a white hat or black hat. Find ideas to fine tune your website for top rankings using keywords, meta tags, SEF, linkbacks, and our

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old March 22nd, 2007, 07:13 AM
seokutty seokutty is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Mar 2007
Posts: 4 seokutty User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 8 m 3 sec
Reputation Power: 0
Spider should not crawl my website!!What can I Do?
I have website "A".I have another website "B"but its contents and all are same as Website "A".I know it will be spamming.But the spider should not crawl website "B". what I can do for this? Can I go through Robot.txt?

Please help me friends
Reply With Quote
  #2  
Old March 22nd, 2007, 07:17 AM
pro_seo's Avatar
pro_seo pro_seo is offline
Moderator
SEO Chat Loyal (3000 - 3499 posts)
  
Join Date: Apr 2006
Location: I N D I A
Posts: 3,266 pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level)pro_seo User rank is Second Lieutenant (5000 - 10000 Reputation Level) 
Time spent in forums: 2 Months 1 Week 3 Days 19 h 33 m 22 sec
Reputation Power: 87
Send a message via ICQ to pro_seo Send a message via AIM to pro_seo Send a message via MSN to pro_seo Send a message via Yahoo to pro_seo Send a message via Google Talk to pro_seo Send a message via Skype to pro_seo
Quote:
Originally Posted by seokutty
I have website "A".I have another website "B"but its contents and all are same as Website "A".I know it will be spamming.But the spider should not crawl website "B". what I can do for this? Can I go through Robot.txt?

Please help me friends


Hi,

Welcome to seochat.

yes, robots.txt is the way to go.

make your robots.txt file as

User-agent: *
Disallow: /

This will not let any spider to crawl any page of your site.
__________________

SEO FAQs - You might find your answer here.
SEOchat Forum Rules - Read Before You Post

**Do what you feel in your heart to be right- for you'll be criticized anyway. You'll be damned if you do, and damned if you don't.**
Reply With Quote
  #3  
Old March 22nd, 2007, 10:14 AM
-mj-'s Avatar
-mj- -mj- is offline
Customized?
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Jan 2007
Location: Budapest
Posts: 1,083 -mj- User rank is Sergeant (500 - 2000 Reputation Level)-mj- User rank is Sergeant (500 - 2000 Reputation Level)-mj- User rank is Sergeant (500 - 2000 Reputation Level)-mj- User rank is Sergeant (500 - 2000 Reputation Level)-mj- User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 5 Days 18 h 22 m 45 sec
Reputation Power: 15
Send a message via Google Talk to -mj-
google.com/support/webmasters/bin/answer.py?answer=35301&topic=8459
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Optimization > Spider should not crawl my website!!What can I Do?

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump



 Free IT White Papers!
 
How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.
Request Your Free Technology Downloads!
 
Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.
Request Your Free Technology Downloads!
 
Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.
Request Your Free Technology Downloads!
 
Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.
Request Your Free Technology Downloads!
 
Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.
Request Your Free Technology Downloads!
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2009 by Developer Shed. All rights reserved. DS Cluster 1 hosted by Hostway. 
Stay green...Green IT