#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Mar 2011
    Location
    Delhi
    Posts
    5
    Rep Power
    0

    Question Exclude pages from search engines using robots.txt vs meta robots tag


    Hello,,

    Under what circumstances should I look to exclude pages from search engines using robots.txt vs meta robots tag?
  2. #2
  3. No Profile Picture
    Contributing User
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Jan 2008
    Posts
    1,834
    Rep Power
    426
    If a page is already cached, you can't get rid of it with robots.txt you have to use the no index nofollow meta tag (I think). They are quite similar, just use what works best for you.
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2011
    Posts
    306
    Rep Power
    21
    You need to use robots.txt when you do not want some content on your website to get crawled.

    You can use robots.txt files in another very interesting way.
    If you place robots.txt files with your sitemap (according to Sitemap protocol), all the major search engines can discover your Sitemap automatically without submission.

Similar Threads

  1. Are Meta Search Engines popular or Unpopular?
    By myvoice in forum Other Engines
    Replies: 8
    Last Post: Jul 19th, 2013, 06:08 PM
  2. Replies: 0
    Last Post: Oct 7th, 2006, 04:38 PM
  3. Need N/A SE Rank explained
    By Ramses357 in forum Google Optimization
    Replies: 21
    Last Post: Jul 12th, 2005, 09:06 PM
  4. Vs. Sheer Link Mass...How to win?
    By Insomnia in forum Google Optimization
    Replies: 2
    Last Post: Mar 26th, 2004, 12:33 AM

IMN logo majestic logo threadwatch logo seochat tools logo