Great community. Great ideas.

Welcome to SEOChat, a community dedicated to helping beginners and professionals alike in improving their Search Engine Optimization knowledge.  Sign up today to gain access to the combined insight of tens of thousands of members.

Thread: URL blocked by Robots.txt Issue in WMT, Pleas Help!!!

Results 1 to 3 of 3
Share This Thread →
  1. #1
    ansrikanth is offline Registered User SEO Chat Explorer (0 - 99 posts)
    Join Date
    Nov 2012
    Posts
    10
    Rep Power
    0

    URL blocked by Robots.txt Issue in WMT, Pleas Help!!!

    Hello,
    My website - http://infworm.com
    Robots.txt - Http://infworm.com/robots.txt
    SEO Plugin - Yoast SEO Plugin
    Sitemap - http://infworm.com/sitemap_index.xml

    I have recently recovered from a problem where my website was accidently put into noindex mode. I have fixed it. Now, I have re-submitted my Sitemap to Google, generated by Yoast plugin. Google Indexes my site weekly and after my site went to index,follow mode; The number of indexed pages in google started increasing slowly, until recently, Google is showing just 2 pages indexed, and with 586 Warnings telling: "Sitemap contains urls which are blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."


    Example:
    Sitemap: infworm.com/post-sitemap.xml
    Value: http://infworm.com/wp-content/uploads/2012/01/Apple-iPad-Vs-Amazon-Kindle-Fire.jpg

    Sitemap: infworm.com/post-sitemap.xml
    Value: http://infworm.com/wp-content/uploads/2012/01/AS-Social-media.png

    In My XML Sitemaps settings in Yoast, I have not excluded any post types in my sitemaps (like images,categories, tags, etc). But I have in Titles and Meta settings - Noindex categories, tags and formats.

    Please Help me with the Google WMT sitemaps issue!!!

  2. #2
    zico82's Avatar
    zico82 is offline Contributing User SEO Chat Discoverer (100 - 499 posts)
    Join Date
    Feb 2005
    Location
    England
    Posts
    198
    Rep Power
    199
    Quote Originally Posted by ansrikanth
    Hello,
    My website - http://infworm.com
    Robots.txt - Http://infworm.com/robots.txt
    SEO Plugin - Yoast SEO Plugin
    Sitemap - http://infworm.com/sitemap_index.xml

    I have recently recovered from a problem where my website was accidently put into noindex mode. I have fixed it. Now, I have re-submitted my Sitemap to Google, generated by Yoast plugin. Google Indexes my site weekly and after my site went to index,follow mode; The number of indexed pages in google started increasing slowly, until recently, Google is showing just 2 pages indexed, and with 586 Warnings telling: "Sitemap contains urls which are blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."


    Example:
    Sitemap: infworm.com/post-sitemap.xml
    Value: http://infworm.com/wp-content/uploads/2012/01/Apple-iPad-Vs-Amazon-Kindle-Fire.jpg

    Sitemap: infworm.com/post-sitemap.xml
    Value: http://infworm.com/wp-content/uploads/2012/01/AS-Social-media.png

    In My XML Sitemaps settings in Yoast, I have not excluded any post types in my sitemaps (like images,categories, tags, etc). But I have in Titles and Meta settings - Noindex categories, tags and formats.

    Please Help me with the Google WMT sitemaps issue!!!
    The reason those URLs are blocked is exactly the reason Webmaster tools is telling you. You have blocked the /wp-content/ directory so anything in there should be blocked.
    Last edited by zico82; Nov 15th, 2012 at 05:13 AM.

  3. #3
    ansrikanth is offline Registered User SEO Chat Explorer (0 - 99 posts)
    Join Date
    Nov 2012
    Posts
    10
    Rep Power
    0
    Quote Originally Posted by zico82
    The reason those URLs are blocked is exactly the reason Webmaster tools is telling you. You have blocked the /wp-content/ directory so anything in there is blocked. Remove that line from your robots.txt and they will no longer be blocked.
    Thanks for the quick reply!!
    I have removed the /content/ disallow line from my robots.txt and re-submitted my sitemap.
    Now Google WMT has crawled it and is showing more warnings than the previous time.

    Example:
    Sitemap: infworm.com/page-sitemap.xml
    Value: http://infworm.com/wp-content/uploads/2012/01/FACEBOOK-COVER.gif

    What am I still doing wrong related to sitemap_index.xml and robots.txt? I am not getting it!!!

Share This Thread →

Become Part of This Conversation

Join NowFor Free!

Similar Threads

  1. Is www.ripfast.com banned?
    By guna in forum Google Optimization
    Replies: 21
    Last Post: Jan 29th, 2007, 03:40 AM
  2. Robots.txt issue : help needed
    By jjaall99 in forum Google Optimization
    Replies: 1
    Last Post: Jul 25th, 2006, 08:19 AM

SEO Chat Advertisers and Affiliates