#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    19
    Rep Power
    0

    Small Handful Of Disallowed Robots.Txt Pages Indexed


    I have 1000's of pages in a second level directory which I have disallowed that makes price calculations using dynamic URLS.

    It is a second level directory disallow. This has worked until recently.

    Here is an example URL e.g: www.phuky.com/this_whole_directory_is_disallowed/etc/etc/etc/?tx_cats%2C%20

    KnowOneSpecial replied that the URL structure should have been written correctly in this recent post

    My question is: if a higher level directory is disallowed how come the lower one has been indexed (bad URL writing or not).

    Thanks in advance for any input.
  2. #2
  3. Super Moderator
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2004
    Location
    Gloucester (South West UK).
    Posts
    6,483
    Rep Power
    3367
    Disallowing a page in robots.txt doesn't prevent it from being indexed, it just prevents google from crawling.
    If you want to prevent indexing, forget robots.txt* and add a robots meta "noindex" instead.

    *(Literally forget robots.txt or you will prevent the crawler from seeing the robots.txt directive)!

    Comments on this post

    • Ann Smarty agrees
    • dzine agrees : Yup
    ClickyB
    "The quality of the visitor is more important than the volume..." (Egol 22nd Feb 2008)
    [New to SEO/SeoChat?] [Canonical Problems?] [Forum Rules & Posting Guidelines]
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    19
    Rep Power
    0
    Thanks everyone.

Similar Threads

  1. Big mistake with Robots.txt - disallowed by accident
    By johnjohn in forum SEO Help (General Chat)
    Replies: 9
    Last Post: Nov 5th, 2012, 03:49 PM
  2. Replies: 1
    Last Post: May 7th, 2008, 01:13 AM
  3. How long for disallowed robots.txt pages to be dropped
    By DrQuincy in forum Google Optimization
    Replies: 10
    Last Post: Apr 7th, 2008, 12:56 PM
  4. Has anyone disallowed robots.txt?
    By l3vi in forum Search Engine Optimization
    Replies: 3
    Last Post: Oct 3rd, 2006, 12:39 AM
  5. PR for Disallowed Pages
    By mama323 in forum Google Optimization
    Replies: 3
    Last Post: Sep 27th, 2003, 08:17 PM

IMN logo majestic logo threadwatch logo seochat tools logo