#1
  1. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2018
    Posts
    2
    Rep Power
    0

    Question Google Indexing my pages which are blocked by robots.txt


    Hi there,

    Google is indexing my pages which are blocked by robots.txt. I don't want google to index those pages so I have blocked them already.

    Does anyone have idea about how can I remove them from google? Because lots of irrelevant and unoptimised pages can bring my keyword ranking down and also they are not serving any value to users.

    Any idea how to remove them?
  2. #2
  3. No Profile Picture
    Moderator
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    3,041
    Rep Power
    3634
    Originally Posted by james99
    Google is indexing my pages which are blocked by robots.txt. I don't want google to index those pages so I have blocked them already.
    That is because you are doing it wrong.

    Robots.txt was never designed to stop indexing of pages from search engines. It was designed to stop bots from crawling pages, but not indexing pages, there is a difference. This is a miss conception that a lot of folks fall victim to.

    Use either robots meta tag or x-robots header tags to stop indexing by search engines. You also have to remove the block in robots.txt so Google can crawl the page to recognize the no index directives. Then sit back and wait for Google to crawl the pages again.

    What you have to understand is that if any of your pages or any page on the internet has a link to a page that you do not want indexed, Google when it sees that link will crawl that page, and if you don't use a no index directive on the page using either of the methods I mentioned above the page gets indexed.
    Last edited by KnowOneSpecial; Nov 7th, 2018 at 06:57 PM.
    If you have never failed in your life, you have never achieved anything Noteworthy !

Similar Threads

  1. Robots.txt allowing everything, Google says 4,200 pages blocked?
    By number1superguy in forum Google Optimization
    Replies: 3
    Last Post: Oct 11th, 2011, 08:39 AM
  2. Replies: 1
    Last Post: Sep 21st, 2009, 09:38 AM
  3. Google and Noindex blocked by robots.txt
    By roseberry in forum Google Optimization
    Replies: 0
    Last Post: Oct 30th, 2008, 03:26 PM
  4. Google indexing few pages / not resecting "robots.txt"
    By webguy0822 in forum Google Optimization
    Replies: 5
    Last Post: Mar 20th, 2007, 03:51 AM
  5. Strange Google indexing and robots.txt question
    By adsthegenius in forum Google Optimization
    Replies: 1
    Last Post: Dec 2nd, 2005, 04:46 PM

IMN logo majestic logo threadwatch logo seochat tools logo