Thread: Search result

    #1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2016
    Posts
    7
    Rep Power
    0

    Question Search result


    When I type exact URL of my dev site in google it will shown in 1st position in google. I blocked my dev site in Robot.txt but still it shown in search result when I type exact URL of my dev site.
    The below text I got in search result.


    https://devsitename.com/
    A description for this result is not available because of this site's robots.txt
    Learn more

    Please help me on this.
  2. #2
  3. Dinosaur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,196
    Rep Power
    5816
    Well thats correct, its not showing any information about the site because of robots.txt.


    You should not really use robots.txt to block the site being indexed
    your need to add the following to your head section of your page

    <meta name="robots" content="noindex,nofollow">
    and remove the entry in robots.txt

    Now google know about the site though its going to be quite a while before it drops it.
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2016
    Posts
    7
    Rep Power
    0
    Hi Chedders

    Thank you for your reply.

    We already added <meta name="robots" content="noindex,nofollow"> in our header section.
    And also blocked in robots.txt using below code
    User-agent: * Disallow: /
  6. #4
  7. SeoRaptor
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2016
    Location
    France, Saint-Etienne
    Posts
    688
    Rep Power
    1222
    We already added <meta name="robots" content="noindex,nofollow"> in our header section.
    And also blocked in robots.txt using below code
    User-agent: * Disallow: /
    That should be enough but sometimes bots do not really follow the rules... If this was the case for your dev site, you could also block the dev website directory on your server by protecting the folder with a password in .htaccess, very effective but the downside is that you might need to reenter you password from time to time while working on your website.
    Owner of Bennetonable - "My opinions are my own - Feel free to disagree & think above the fold."
    Need a quick analysis > Free SEO grader Beta
  8. #5
  9. Dinosaur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,196
    Rep Power
    5816
    A lot will depend when it was added, I would guess it was added after the dev site was live and google nipped in and indexed it prior to you adding the instruction.

    Its not a major deal though, I would expect it wont rank for anything other than the domain name.

    Comments on this post

    • Pierre Benneton agrees
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  10. #6
  11. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2016
    Posts
    7
    Rep Power
    0
    Originally Posted by Pierre Benneton
    That should be enough but sometimes bots do not really follow the rules... If this was the case for your dev site, you could also block the dev website directory on your server by protecting the folder with a password in .htaccess, very effective but the downside is that you might need to reenter you password from time to time while working on your website.
    Thank you for your reply.
  12. #7
  13. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2016
    Posts
    7
    Rep Power
    0
    Thank you for your reply.

    I exactly don't know when the code NOFOLLOW and NOINDEX was added in header section but I'm sure it was added more that 2years before. The only thing I wondering is it create any issues or not.
  14. #8
  15. Dinosaur
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,196
    Rep Power
    5816
    It wont create any issues, it just means people are able to find it but if they already know the domain name to search for it then they are already there.
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  16. #9
  17. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2016
    Posts
    7
    Rep Power
    0
    Thank you...

Similar Threads

  1. how to Follow search engine guidelines to rank high in search result?
    By hoanghong in forum Search Engine Optimization
    Replies: 2
    Last Post: Mar 28th, 2013, 12:54 AM
  2. How can I completely remove a search result from a Google search?
    By SusieMack in forum Google Optimization
    Replies: 1
    Last Post: Nov 22nd, 2009, 01:48 AM
  3. Should I stop Search Engines Indexing my Search Result Pages?
    By michaelphipps in forum Search Engine Optimization
    Replies: 0
    Last Post: Jul 21st, 2008, 01:17 AM
  4. Yahoo Directory Search Info Showing on Search Result
    By rhwd2003 in forum BING/Yahoo Search Optimization
    Replies: 1
    Last Post: Feb 2nd, 2007, 05:17 PM
  5. when search engines list new website in their search result?
    By jeeseo in forum Google Optimization
    Replies: 3
    Last Post: Nov 14th, 2005, 09:58 AM

IMN logo majestic logo threadwatch logo seochat tools logo