Robots.txt Help - SEO Chat

Search Engine Spiders

User Name:

Password:

Remember me

SEO Chat Forums Sponsor:

Anyone looking for a way to modernize legacy data or easily migrate to a more cost-effective database without sacrificing functionality will benefit from this seminar. View the Intro to Advantage Database Server now!

January 22nd, 2008, 11:50 AM

walkdiggler

Registered User

Join Date: Apr 2007

Posts: 5

Time spent in forums: 2 h 39 m 18 sec
Reputation Power: 0

Send a message via Google Talk to walkdiggler

Robots.txt Help

We develop affiliate sites as subfolders on our site. Example: our site is example.com...our affiliate site will be example.com/affiliate.

Our site is replicated 100% for the affiliate, which means we have some duplicate content issues. If we Disallow the /affiliate/ directory, then do we lose all of the link credit coming to all of the /affiliate pages?

Our site shows 50K+ links...many of them coming from affiliate.com and pointing at example.com/affiliate, all of which example.com gets credit for. If we disallow /affiliate/, then we are afraid that all of the affiliate.com links pointing at example.com/affiliate will not be counted toward our total.

What do you all think? Any suggestions?

January 22nd, 2008, 12:30 PM

pro_seo

Moderator

Join Date: Apr 2006

Location: I N D I A

Posts: 2,604

Time spent in forums: 1 Month 2 Weeks 21 h 10 m 35 sec
Reputation Power: 30

Send a message via Google Talk to pro_seo

Pages blocked through robots.txt can accrue PR and LJ.

Quote:

Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results.

Source

However, your best option would be to put <meta name="robots" content="noindex,follow"> on those pages.

This will allow Google to follow and credit the links, yet keep those pages out of the index.

__________________

Social Mobile Messaging
SEO FAQs - You might find your answer here.
SEOchat Forum Rules - Read Before You Post

**Do what you feel in your heart to be right- for you'll be criticized anyway. You'll be damned if you do, and damned if you don't.**

January 22nd, 2008, 01:31 PM

walkdiggler

Registered User

Join Date: Apr 2007

Posts: 5

Time spent in forums: 2 h 39 m 18 sec
Reputation Power: 0

Thanks for the info. Is there any chance that I could be penalized for using the "noindex" tag? I found the following comment about "noindex":

"When not used for legitimate purposes, this tag can be dangerous because it can put you at risk for penalization by most, if not all search engines. This is because you can use a noindex tag to hide pages with multiple links that you don't want visitors to see but that you do want all search engines to index."

Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > Robots.txt Help

« Previous Thread | Next Thread »

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox

Forum Jump

Free IT White Papers!

Accelerating Trading Partner Performance
One in five. That's how many partner transactions have at least one error. That is an amazing statistic, particularly given the extraordinary leaps in innovation across the global supply chain during the past two decades. Download this white paper to learn more.

Competing on Analytics
This Tech Analysis is designed to help identify characteristics shared by analytics competitors, and includes information about 32 organizations that have made a commitment to quantitative, fact-based analysis.

Cost Effective Scaling with Virtualization and Coyote Point Systems
An overview of the industry trend toward virtualization, how server consolidation has increased the importance of application uptime and the steps being taken to integrate load balancing technology with virtualized servers.

Five Checkpoints to Implementing IP Telephony
Implementation planning for IP PBX software and IP telephony has become vital as businesses replace discontinued legacy PBX phone systems. This informative whitepaper outlines five "checkpoints" for any implementation plan that will help make IP communications a successful proposition.

Hosted Email Security: Staying Ahead of New Threats
In the last two years, email has become a fierce battleground between the nefarious forces of spam and malware, and the heroes of messaging protection. The spam volumes increased alarmingly every month, bringing clever new forms of phishing and virus propagation attacks.

More Free IT White Papers!