#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2017
    Posts
    79
    Rep Power
    3

    Need only tens of pages to be indexed out of hundreds: Robots.txt is Okay for Google?


    Hi all,

    We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. But we are not sure if this is the right approach as Google been suggesting to depend mostly on "noindex" than robots.txt. Please suggest whether we can proceed with robots.txt file.

    Thanks
  2. #2
  3. Retired from SEO
    SEO Chat High Scholar (3500 - 3999 posts)

    Join Date
    Oct 2007
    Location
    India
    Posts
    3,701
    Rep Power
    1742
    Why you are not using 301 redirect from old pages to new pages? And if possible then remove or make it in the draft if you are not getting any traffic from old pages
    You do your business I do mine because you are you and I am I If we meet it is nice.
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jul 2018
    Posts
    112
    Rep Power
    0
    @Prof.stan
    There are no old pages and new pages.
    Probably you have misread the post.
    He is not moving to another domain.
  6. #4
  7. No Profile Picture
    Moderator
    SEO Chat Super Hero (2500 - 2999 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    2,898
    Rep Power
    3508
    Let me clarify Prof.stan post for you ....
    He is proposing that the OP redirect the un-needed pages, (read that as the old pages), to the 50 pages that the OP wants to keep in the index, (read that as the new pages).

    The language barrier can be tough to decipher sometimes. I often times miss-understand a post here.

    Originally Posted by satish.seo
    We 2 sub domains with hundreds of pages where we need only 50 pages to get indexed which are important. Unfortunately the CMS of these sub domains is very old and not supporting "noindex" tag to be deployed on page level. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed. So we are planning to block the entire sites from robots.txt and allow the 50 pages needed.
    But to address the OP.

    I have numerous times told, informed folks that using Robots.Txt to stop Google from indexing is not the correct way to do things. Sure you can block Google from those pages, but if there is a link anywhere on the net, the page will promptly show up again. As is the case with these sites, I suspect and expect there are backlinks and or navigation links to these pages, which will immediately defeat the use of robots.txt to do this.

    In this situation, I would use header tags to facilitate the no indexing of these pages.

    You would implement this via htaccess on a linux server or web.config on a IIS server. This will allow you to specify the pages to inject the "noindex" and if so inclined "nofollow" tags on the site. This solves your cms issues of not being able to put the tags on the page.

    Comments on this post

    • Prof.stan agrees
    Last edited by KnowOneSpecial; Feb 15th, 2019 at 10:17 AM. Reason: Spelling correction
    If you have never failed in your life, you have never achieved anything Noteworthy !

Similar Threads

  1. Google Indexing my pages which are blocked by robots.txt
    By james99 in forum New User SEO Questions and Answers
    Replies: 1
    Last Post: Nov 7th, 2018, 06:54 PM
  2. Small Handful Of Disallowed Robots.Txt Pages Indexed
    By phunky in forum Google Optimization
    Replies: 2
    Last Post: Jun 15th, 2017, 09:12 AM
  3. Robots.txt allowing everything, Google says 4,200 pages blocked?
    By number1superguy in forum Google Optimization
    Replies: 3
    Last Post: Oct 11th, 2011, 08:39 AM
  4. 3 yr old site, 15,000 product pages, only hundreds indexed
    By seomonopolist in forum Google Optimization
    Replies: 7
    Last Post: Nov 20th, 2009, 11:05 AM
  5. Replies: 1
    Last Post: May 7th, 2008, 01:13 AM

IMN logo majestic logo threadwatch logo seochat tools logo