#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Oct 2011
    Posts
    2
    Rep Power
    0

    Thumbs down Robots.txt allowing everything, Google says 4,200 pages blocked?


    We just recently launched a new shopping cart website (using Magento). When we launched the website on 9/27 the robots.txt file was changed from disallowing everything to the following:

    User-agent: *
    Disallow:

    I have no other robots.txt files anywhere on the website, I did a search. Ever since we launched then I have been checking the webmaster page and more and more pages were being blocked. Up until 5 days ago (10/5) they were being blocked daily. No more are being added now, but they still show as being blocked. Here's an example of a page it says is blocked:

    (guess I can't add a link, I dunno, maybe PM me if you want the link?)

    When I run google's robots analyzer it says that link is accessible and not blocked, yet there it is on my list of blocked by robots.txt links. Everything I know about robots.txt is failing me here. What am I missing?

    Thanks for any help at all!

    Matt
  2. #2
  3. Contributing User
    SEO Chat Super Genius (4500 - 4999 posts)

    Join Date
    May 2007
    Posts
    4,515
    Rep Power
    1985
    Check the source code of some blocked pages and look for robots meta tags.

    You may have your Magento setting blocking search engines, which would be done using the tag, not the file.

    Comments on this post

    • joshz agrees : which is common if you've checked the developmental optoin on a CMS.
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Oct 2011
    Posts
    2
    Rep Power
    0
    Thank you very much for the suggestions.

    I've checked for this as well and unfortunately I don't believe that's the case. The admin has a dropdown called "Default Robots" and it's set to "INDEX, FOLLOW". On the pages that Google thinks are blocked, the meta tag also says "INDEX, FOLLOW".

    Is there anything else that can make Google think it's being blocked?

    Thanks again for your help,

    Matt
  6. #4
  7. Contributing User
    SEO Chat Super Genius (4500 - 4999 posts)

    Join Date
    May 2007
    Posts
    4,515
    Rep Power
    1985
    got a url? just break it up - you won't be able to PM it to anyone yet

Similar Threads

  1. Tricking Google Search Results Using Doorway Pages!!!
    By Jamesraz in forum Google Optimization
    Replies: 12
    Last Post: Jun 14th, 2006, 04:22 AM
  2. Tricking Google Search Results Using Doorway Pages!!!
    By Jamesraz in forum Search Engine Optimization
    Replies: 4
    Last Post: Jun 13th, 2006, 05:04 AM
  3. too much content too quickly?
    By deano6410 in forum Google Optimization
    Replies: 5
    Last Post: Aug 11th, 2005, 08:21 PM
  4. Letter To Google About Non-Relevant Results
    By Sweat in forum Google Optimization
    Replies: 17
    Last Post: Feb 5th, 2005, 02:12 PM
  5. GoogleBot FAQ
    By darkroom in forum Google Optimization
    Replies: 6
    Last Post: Mar 7th, 2003, 02:39 AM

IMN logo majestic logo threadwatch logo seochat tools logo