- Total Members: 263,815
- Threads: 454,051
- Posts: 1,062,567
Great community. Great ideas.
Welcome to SEOChat, a community dedicated to helping beginners and professionals alike in improving their Search Engine Optimization knowledge. Sign up today to gain access to the combined insight of tens of thousands of members.
-
Nov 15th, 2012, 04:48 AM
#1
URL blocked by Robots.txt Issue in WMT, Pleas Help!!!
Hello,
My website - http://infworm.com
Robots.txt - Http://infworm.com/robots.txt
SEO Plugin - Yoast SEO Plugin
Sitemap - http://infworm.com/sitemap_index.xml
I have recently recovered from a problem where my website was accidently put into noindex mode. I have fixed it. Now, I have re-submitted my Sitemap to Google, generated by Yoast plugin. Google Indexes my site weekly and after my site went to index,follow mode; The number of indexed pages in google started increasing slowly, until recently, Google is showing just 2 pages indexed, and with 586 Warnings telling: "Sitemap contains urls which are blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."
Example:
Sitemap: infworm.com/post-sitemap.xml
Value: http://infworm.com/wp-content/uploads/2012/01/Apple-iPad-Vs-Amazon-Kindle-Fire.jpg
Sitemap: infworm.com/post-sitemap.xml
Value: http://infworm.com/wp-content/uploads/2012/01/AS-Social-media.png
In My XML Sitemaps settings in Yoast, I have not excluded any post types in my sitemaps (like images,categories, tags, etc). But I have in Titles and Meta settings - Noindex categories, tags and formats.
Please Help me with the Google WMT sitemaps issue!!!
-
Nov 15th, 2012, 05:06 AM
#2

Originally Posted by
ansrikanth
Hello,
My website - http://infworm.com
Robots.txt - Http://infworm.com/robots.txt
SEO Plugin - Yoast SEO Plugin
Sitemap - http://infworm.com/sitemap_index.xml
I have recently recovered from a problem where my website was accidently put into noindex mode. I have fixed it. Now, I have re-submitted my Sitemap to Google, generated by Yoast plugin. Google Indexes my site weekly and after my site went to index,follow mode; The number of indexed pages in google started increasing slowly, until recently, Google is showing just 2 pages indexed, and with 586 Warnings telling: "Sitemap contains urls which are blocked by robots.txt." and "Sitemap contains urls which are blocked by robots.txt."
Example:
Sitemap: infworm.com/post-sitemap.xml
Value: http://infworm.com/wp-content/uploads/2012/01/Apple-iPad-Vs-Amazon-Kindle-Fire.jpg
Sitemap: infworm.com/post-sitemap.xml
Value: http://infworm.com/wp-content/uploads/2012/01/AS-Social-media.png
In My XML Sitemaps settings in Yoast, I have not excluded any post types in my sitemaps (like images,categories, tags, etc). But I have in Titles and Meta settings - Noindex categories, tags and formats.
Please Help me with the Google WMT sitemaps issue!!!
The reason those URLs are blocked is exactly the reason Webmaster tools is telling you. You have blocked the /wp-content/ directory so anything in there should be blocked.
Last edited by zico82; Nov 15th, 2012 at 05:13 AM.
-
Nov 15th, 2012, 05:14 AM
#3

Originally Posted by
zico82
The reason those URLs are blocked is exactly the reason Webmaster tools is telling you. You have blocked the /wp-content/ directory so anything in there is blocked. Remove that line from your robots.txt and they will no longer be blocked.
Thanks for the quick reply!!
I have removed the /content/ disallow line from my robots.txt and re-submitted my sitemap.
Now Google WMT has crawled it and is showing more warnings than the previous time.
Example:
Sitemap: infworm.com/page-sitemap.xml
Value: http://infworm.com/wp-content/uploads/2012/01/FACEBOOK-COVER.gif
What am I still doing wrong related to sitemap_index.xml and robots.txt? I am not getting it!!!
Similar Threads
-
By guna in forum Google Optimization
Replies: 21
Last Post: Jan 29th, 2007, 03:40 AM
-
By jjaall99 in forum Google Optimization
Replies: 1
Last Post: Jul 25th, 2006, 08:19 AM