#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Dec 2016
    Posts
    37
    Rep Power
    2

    Robots Txt Issue


    I have not added any robots txt in my webmaster but still when I submit my sitemap and test it. It says that
    Url blocked by robots.txt.
    Sitemap contains urls which are blocked by robots.txt.

    I am using Yoast SEO free plugin and not that technical kind of SEO.

    Is there anyone who can help me with this?
  2. #2
  3. Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Location
    United Kingdom
    Posts
    1,306
    Rep Power
    1205
    Have you checked yourdomain.com/robots.txt?
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Oct 2018
    Posts
    9
    Rep Power
    0
    it may possible you might block the whole website by mistake. you can check in server www.yourdoamian.com/robots.txt you can watch here which URL is blocked.
    you can get a basic idea about robots.txt
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Dec 2016
    Posts
    37
    Rep Power
    2
    Originally Posted by hiren_popat
    it may possible you might block the whole website by mistake. you can check in server www.yourdoamian.com/robots.txt you can watch here which URL is blocked.
    you can get a basic idea about robots.txt
    Hi thanks for the help.

    Interesting part is, i have not used or submitted any robots.txt file anywhere but still it says it is blocked.

    Moreover half of my website urls are indexd while half of them are not. And this for the first time happening with me.
  8. #5
  9. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Dec 2016
    Posts
    37
    Rep Power
    2
    Yes I did and as mentioned, I have not added any robots.txt file anywhere. Because I know it's not that important though until, we restrict search engines not to crawl. And I have already added the only urls i want to crawl in search engine.
  10. #6
  11. No Profile Picture
    Moderator
    SEO Chat Super Hero (2500 - 2999 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    2,566
    Rep Power
    3194
    Did you create a robots.txt file on your website?

    Now before you say you didn't, pay attention.

    It doesn't matter if you submitted the robots.txt file to be indexed, Google will find it and honor it.

    You are using the Yoast Plugin which tells me you are using WordPress. Between the two it is most likely you have a robots.txt file that was automatically created for you. On top of that, WordPress can be told to discourage search engines...
    Go to your dashboard, click Settings > Reading
    If you tic that Search Engine Visibility box,(see graphic below) guess what, you get your problem.



    Now to address your statement...
    I have not added any robots.txt file anywhere. Because I know it's not that important though until, we restrict search engines not to crawl. And I have already added the only urls i want to crawl in search engine.
    This is backwards.. you should be letting all search engines search the whole site, and restrict just the pages you do not want to index.

    Also using robots.txt is not one of the proper and correct ways to stop indexing a page.
    Last edited by KnowOneSpecial; Nov 2nd, 2018 at 08:45 PM.
    If you have never failed in your life, you have never achieved anything Noteworthy !

Similar Threads

  1. Meta description/robots.txt issue
    By icicle in forum New User SEO Questions and Answers
    Replies: 3
    Last Post: Jan 13th, 2017, 07:44 AM
  2. Robots.txt Disallow Issue
    By HappyJackJunior in forum Search Engine Optimization
    Replies: 2
    Last Post: May 16th, 2016, 07:32 AM
  3. URL blocked by Robots.txt Issue in WMT, Pleas Help!!!
    By ansrikanth in forum Search Engine Optimization
    Replies: 2
    Last Post: Nov 15th, 2012, 06:14 AM
  4. Robots.txt issue
    By imot3k in forum Google Optimization
    Replies: 12
    Last Post: Feb 12th, 2010, 09:26 AM
  5. Robots.txt issue : help needed
    By jjaall99 in forum Google Optimization
    Replies: 1
    Last Post: Jul 25th, 2006, 09:19 AM

IMN logo majestic logo threadwatch logo seochat tools logo