#1
  1. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2007
    Location
    Manchester NH
    Posts
    365
    Rep Power
    14

    Difference btw rel="nofollow" & putting urls in robots.txt


    I've been going through all my sites and using nofollow's all over the place, but I've been wondering lately what the difference is between using the rel="nofollow" and putting the url's in the robots.txt file?

    If the url is in the robots.txt file will it still receive PR juice?
  2. #2
  3. SEO Gawdess
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2006
    Location
    San Francisco Bay Area
    Posts
    711
    Rep Power
    137
    Blocking the urls from indexing in your robots.txt prevents crawlers from visiting them at all, or listing them.

    Ref=nofollow passes no link juice, but the urls may get followed and indexed anyway. Yahoo follows them.

    Danny Sulivan says "Google WILL follow nofollow links in the sense that if someone else links to a page without using nofollow, then Google might find the page that way. Nofollow on one link to a page doesn't somehow cause ALL links to be off limits. "

    Comments on this post

    • CommonDavid agrees : Perfect! Thanks for the excellent response.
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2007
    Location
    Manchester NH
    Posts
    365
    Rep Power
    14
    Originally Posted by Swirleigh
    Blocking the urls from indexing in your robots.txt prevents crawlers from visiting them at all, or listing them.

    Ref=nofollow passes no link juice, but the urls may get followed and indexed anyway. Yahoo follows them.

    Danny Sulivan says "Google WILL follow nofollow links in the sense that if someone else links to a page without using nofollow, then Google might find the page that way. Nofollow on one link to a page doesn't somehow cause ALL links to be off limits. "
    I was just thinking about this, and it's an important thing to use a nofollow, but not to block the url with the robots.txt if you've got pertitiant info on the page, and/or if you've got sitelinks or plan on getting them,

    ex: you might not want to waste your PR on the "Driving Directions" page, but you still want to see that sitelink show up in your listing ;-)

    Thanks again
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2005
    Posts
    76
    Rep Power
    13
    Originally Posted by CommonDavid
    I was just thinking about this, and it's an important thing to use a nofollow, but not to block the url with the robots.txt if you've got pertitiant info on the page, and/or if you've got sitelinks or plan on getting them,

    ex: you might not want to waste your PR on the "Driving Directions" page, but you still want to see that sitelink show up in your listing ;-)

    Thanks again
    how do you block google following an outgoing link in your robots file?

    Can someone show me the code?
    Last edited by Sniffer1; Mar 5th, 2008 at 03:53 PM.
  8. #5
  9. SEO Gawdess
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2006
    Location
    San Francisco Bay Area
    Posts
    711
    Rep Power
    137
    No way to disallow indexing or following of an outbound link in your robots.txt file. Your robots.txt list files or sections of your site you want the robots/search engine crawlers to stay out of and not index.

    User-agent: *
    Disallow: /real-estate-articles/


    Wow, wouldn't that be great if you could disallow outgoing links?! List all your competitors links and disallow indexing of them. heehee
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2007
    Location
    Manchester NH
    Posts
    365
    Rep Power
    14
    The robots.txt is really just a way to make it so the spider doesn't index admin pages, and in one of my personal instances: pages that let a user add a review. But the more I think about it, the more I realize that that's all you should really put in there, and then just use the nofollow for everything else, the way that seochat has done here.
  12. #7
  13. SEO Gawdess
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Jul 2006
    Location
    San Francisco Bay Area
    Posts
    711
    Rep Power
    137
    Not just admin pages. How about member's only areas? What good is it to charge a fee, or to only display content after user becomes a lead, if they can view the same content viewing Google's cache? There are many area's you might want to disallow indexing for.

    cgi bin
    drafts
    private
    duplicate printable versions
    iframes content, that have no navigation
    members only

Similar Threads

  1. Replies: 12
    Last Post: Nov 23rd, 2006, 10:00 AM
  2. excluding urls w/ a specific variable with robots.txt?
    By smallbiz in forum Google Optimization
    Replies: 5
    Last Post: Aug 30th, 2005, 06:21 PM

IMN logo majestic logo threadwatch logo seochat tools logo