Thread: Robots.txt

    #1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jul 2010
    Posts
    12
    Rep Power
    0

    Robots.txt


    I have restricted some pages from robots.txt, still they are being crawling and showing errors. Can you please suggest me how to restrict them.

    I have used disallow: /pagename.php
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,774
    Rep Power
    1797
    Is that all you have in the robots.txt file?
  4. #3
  5. No Profile Picture

    Join Date
    Jul 2013
    Posts
    23
    Rep Power
    0
    Originally Posted by ShailajaSheela
    I have restricted some pages from robots.txt, still they are being crawling and showing errors. Can you please suggest me how to restrict them.

    I have used disallow: /pagename.php
    when did you uploaded the robots file.....
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2013
    Posts
    63
    Rep Power
    2
    Hi ShailajaSheela,
    Robots.txt files can be kinda tricky sometimes. You have to get the formatting just right. Can you give us a little more information? What is in your current robots.txt file? What are disallow code have you implemented and what crawlers have you specified? What is an example URL that you are trying to disallow?
  8. #5
  9. No Profile Picture

    Join Date
    Jul 2013
    Posts
    23
    Rep Power
    0
    You have to use a proper format in the robots file. After uploading a robots file with proper format give some time for bots to crawl your site, or try fetching your site in webmaster tools. this should rectify your problem. If the results of your pages still shows in the search engines, try removing those url's in the webmaster tools (google index ----> remove url's).
  10. #6
  11. No Profile Picture
    Banned
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2013
    Posts
    15
    Rep Power
    0
    At first you check your robot.txt file by a xml expert, because there may be have some error. After that you upload the file and take a time for Google crawler. I thin your problem will be solve.
  12. #7
  13. No Profile Picture
    Banned
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Oct 2013
    Posts
    4
    Rep Power
    0
    Hi...
    Upload all the files in a separate sub directory that you don’t want robots to visit. Make that directory un listed on the web. List only the directory name in the /robots.txt. This will not let an ill-willed robot to traverse that directory unless you or someone else puts a direct link on the web to one of your files, and then it's not /robots.txt fault.

Similar Threads

  1. Replies: 2
    Last Post: Jul 6th, 2011, 07:01 AM
  2. Study Results: Search Engines, Meta Robots Tag and Robots.txt
    By SEO Chat in forum SEO Chat Articles
    Replies: 1
    Last Post: Jan 13th, 2010, 12:47 PM
  3. How Search Engines Handle the Meta Robots Tag and Robots.txt
    By SEO Chat in forum SEO Chat Articles
    Replies: 0
    Last Post: Nov 23rd, 2009, 08:00 AM
  4. can u block robots accessing https pages with robots.txt?
    By Loco007 in forum Google Optimization
    Replies: 7
    Last Post: Feb 27th, 2009, 03:36 PM

IMN logo majestic logo threadwatch logo seochat tools logo