NO FOLLOW

I am trying to keep SE bots out of some of my pages. What's the best way to do this? I have tried using
<META NAME="robots" content="noindex,nofollow">

Freshbot came through one of my sites and completely ignored it.

I've read that Googlebot ignores meta tags. Is there another way? I'd like to avoid putting it in the robots.txt file.

why do you not want to put a robots.txt file

kapsat

I use a single directory. Plus, I've had problems with robots.txt files in the past and don't like putting information in a robots.txt file because it gives information to people I don't want to give information to.

Yeah, I worry about that too. robots.txt screams: "This is where the good stuff is." I only use it for pages I want the search engines to ignore. I simply hope that no one ever puts a link to the admin screens for the site.

As for NO FOLLOWS, I suspect that just tells the robots not to put the links from the page in their list of pages to follows. I suspect that they would end up reading the links if there are other pages with links to your inner pages.

So if you had a format: index -> inner pages.

Putting NO FOLLOW on the index page will not prevent the inner pages from being indexed if there are other pages on the web pointed to your innies.

So the NO FOLLOW applies to the links on the page and NOT the page itself?

I really don't see how

User-agent: *
Disallow: /

is giving any information away.

1. no one actually gets your robots.txt unless they type in the exact location e.g. www.domain.com/robots.txt

2. even so -- what security, privacy, copyright, and/or intellectual property issue are of concern with:

User-agent: *
Disallow: /

Am I missing something?

Also .htaccess is another alternative

I beleive the best way is to create a new folder and drop all those files in there and use a .htacces or a login module so that only the intended audience can access the files there.

Hope this helps ;)

fathom, thanks for the second post. I don't want to block robots from every page, just the duplicates.

simple-biz, thanks. I think that may be the solution. I was hoping for something simpler (i.e. something that wouldn't require changing code on many of my pages).

That noindex tells the bot to not index that page. That means it will remove that page from the index. Don't think that is what you really want.

Cheers,

Yes. I have several sites that have identical pages due to overlap. I want the correct pages to be indexed on the right sites.

So if Goolgebot visits a page with the metatag on it, it won't index that page? And this won't affect any of the pages linked to from that page if they are linked to from another page such as the home page?