Standard Robots.txt file

SEO Chat Forums Sponsor:

November 20th, 2008, 04:49 PM

tstolber

Contributing User

Join Date: Jul 2004

Location: Bedfordshire

Posts: 2,922

Time spent in forums: 2 Weeks 1 Day 9 h 46 m 45 sec
Reputation Power: 38

Send a message via Google Talk to tstolber

Standard Robots.txt file

Hi Guys

For a while I have been trying to standardise the robots.txt file I use. The only "rogue" bots that I have found are listed in the file I am using as my standard file below.

For a while I just used a blank file but as I charge for it as part of an initial set up package I thought I should really develop the list a bit.

Does anyone else use a standardised robots.txt file or is the consensus to just use a blank file unless a particular bot becomes a problem?

Thanks

Code:

User-agent: Microsoft URL Control 6.00.8862 
Disallow: /

User-agent: Microsoft URL Control 5.01.4511
Disallow: /

User-agent: Microsoft URL Control 6.00.8169
Disallow: /

User-agent: Microsoft URL Control
Disallow: /

User-agent: URL Control
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: TurnitinBot/2.0
Disallow: /

User-agent: Roverbot
Disallow: /

__________________
SEO Tutorials for Beginners, SEO News, SEO Testing

IKROH SEO for UK Search Engine Optimisation call 01908 379938

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Optimization > Standard Robots.txt file

« Previous Thread | Next Thread »

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox

Forum Jump

Free IT White Papers!

How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.
Request Your Free Technology Downloads!

Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.
Request Your Free Technology Downloads!

Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.
Request Your Free Technology Downloads!

Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.
Request Your Free Technology Downloads!

Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.
Request Your Free Technology Downloads!

More Free IT White Papers!