#1
  1. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2013
    Location
    Kolkata,India
    Posts
    56
    Rep Power
    12

    Robots.txt to block a particular url


    Suppose I want to block a particular url with robots.txt



    https://members.abc.com

    what will be the format

    I have written this:
    -------------------
    User-agent: *
    Disallow: /members.abc.com
    ----------------------------------------

    am i right?
    Last edited by dzine; Feb 16th, 2017 at 08:06 AM.
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,366
    Rep Power
    3208
    That's not a single URL... it's a complete domain you're blocking, with everything in it.
  4. #3
  5. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2013
    Location
    Kolkata,India
    Posts
    56
    Rep Power
    12
    Originally Posted by dzine
    That's not a single URL... it's a complete domain you're blocking, with everything in it.
    So can u please give me an example how to block only a single url with robots.txt?
  6. #4
  7. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2017
    Posts
    6
    Rep Power
    0
    Try copying the entire URL and paste it into where you put the domain URL at. Would that work?
  8. #5
  9. Entrepreneur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Oct 2007
    Location
    New Delhi India
    Posts
    3,275
    Rep Power
    967
    Originally Posted by smkghosh
    So can u please give me an example how to block only a single url with robots.txt?
    User-agent: *
    Disallow: /~joe/junk.html
    Disallow: /~joe/foo.html
    Disallow: /~joe/bar.html

    Read more about this The Web Robots Pages
    You do your business I do mine because you are you and I am I If we meet it is nice.
  10. #6
  11. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,366
    Rep Power
    3208
    For the record, I wouldn't even TRY to block a single URL using robots.txt
    I'd use the "robots" meta tag on that particular webpage instead.

    Comments on this post

    • Prof.stan agrees : you always different than others
  12. #7
  13. Entrepreneur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Oct 2007
    Location
    New Delhi India
    Posts
    3,275
    Rep Power
    967
    Dzine most of the time we have to use "robots" meta tag but still robots.txt used for different purpose.
    You can get more about from https://developers.google.com/webmas...index/docs/faq
    You do your business I do mine because you are you and I am I If we meet it is nice.
  14. #8
  15. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2013
    Location
    Kolkata,India
    Posts
    56
    Rep Power
    12
    Hello All,
    Thanks all for your response.
    So basically if i want to block this url https://members.abc.com
    does the below works?
    ------------------

    User-agent: *
    Disallow: /members.abc.com
    ----------------------------------------
    Last edited by smkghosh; Feb 17th, 2017 at 12:38 AM.
  16. #9
  17. Entrepreneur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Oct 2007
    Location
    New Delhi India
    Posts
    3,275
    Rep Power
    967
    following code in your htaccess file

    RewriteEngine on
    RewriteCond %{HTTP_HOST} ^members.abc.com$
    RewriteRule ^robotx\.txt$ robots-members.txt

    Then add:
    User-agent: *
    Disallow: /

    to the robots-members.txt file

    That should work

    Comments on this post

    • Will-O-The-Wisp agrees
    You do your business I do mine because you are you and I am I If we meet it is nice.

Similar Threads

  1. Robots.txt Block all vs block specific
    By ThomasHarvey in forum Search Engine Optimization
    Replies: 5
    Last Post: Feb 9th, 2016, 10:36 AM
  2. Block some pages in robots.txt
    By scape in forum Search Engine Optimization
    Replies: 10
    Last Post: Aug 7th, 2015, 08:32 AM
  3. To Block or Not To Block (from Robots.txt) -ECOMMERCE
    By bkseo in forum Google Optimization
    Replies: 2
    Last Post: Nov 16th, 2010, 12:21 PM
  4. Should i block www in robots if i want to use non www
    By pocketp in forum Google Optimization
    Replies: 3
    Last Post: Jul 23rd, 2009, 10:26 AM
  5. can u block robots accessing https pages with robots.txt?
    By Loco007 in forum Google Optimization
    Replies: 7
    Last Post: Feb 27th, 2009, 03:36 PM

IMN logo majestic logo threadwatch logo seochat tools logo