#1
  1. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2010
    Posts
    128
    Rep Power
    57

    Site map robot no index


    Kind of complicated but try:

    My manual site map was coming up for results, and personally, I don't like competitors seeing my site map. So someone here suggest I do a robot no index, so I did that. But now I'm wondering, does Google then not crawl my site map anymore?


    I put this is a new months ago. The goal, was that the site map would not show up in search results, but that Google would still crawl the page.

    <html><head><meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
    <title>site map</title>
    <meta name="robots" content="noindex">
    </head>
  2. #2
  3. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Dec 2014
    Location
    Barcelona
    Posts
    104
    Rep Power
    77
    You have done the contrary of what you want because if you add the attribute "noindex" Google will not index it but the sitemap still in the server and anyone can download it

    If you do not want people to find your sitemap give it a name that nobody knows, you can add numbers as well so it will be impossible for users to know how is called the URL of the sitemap

    Comments on this post

    • Hikin Mike agrees : A sitemap doesn't have to be called 'sitemap'.
    • kevs agrees
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2010
    Posts
    128
    Rep Power
    57
    Thanks guys, boy I got bad advice.

    Can I ask Google to not have the page come up in search results, but yes to indexing it, and crawling?

    The problem is my full name, firstlast, is in url... www.firstlast.com/ sitemap This has been name for 20 years. So because of that the dang this seems to pop up in random search results, which is the last thing I want. It's a noisy ugly page..

    But it would be better to name it someone.. what 4545Code? Some weird/ odd like that? The issues also on the site map are lots of keyword phrases so that with my name in url is why it comes up in long tail searches.
  6. #4
  7. No Profile Picture
    Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    1,972
    Rep Power
    2995
    So let me make sure I understand you...

    You are stating that your domain name contains your first and last name, is this correct ?

    Lets assume that your first name is john and your last name is doe, using this info what is your exact url ?
  8. #5
  9. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2012
    Posts
    97
    Rep Power
    135
    I think the OP is referring to HTML sitemap (manual) and not XML. In that case, it may have been made not for bot indexing purposes but for user experience. Therefore it can very well be set to noindex.

    I think the misunderstanding is on this:

    Originally Posted by kevs
    Can I ask Google to not have the page come up in search results, but yes to indexing it, and crawling?
    If you want it indexed, then it will come up in search results because it was indexed/crawled. If you don't want it to come up on search results, that is when you noindex it.

    Originally Posted by kevs
    It's a noisy ugly page.
    I think that is what the OP needs to address above anything else.
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2010
    Posts
    128
    Rep Power
    57
    Is this hopeless? I want Google to index the html page, crawl if fully, but not have it come up in search results. If you can help me. thanks!
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2012
    Posts
    97
    Rep Power
    135
    Originally Posted by kevs
    Is this hopeless? I want Google to index the html page, crawl if fully, but not have it come up in search results. If you can help me. thanks!
    If that is what you want to achieve, I think it is hopeless because I believe it's impossible and doesn't make sense. That's like:

    Oh you all come here, you whose job is to show my goods to the world (you were inviting Google to index your page). But wait, I don't want you to show my goods to the world (you don't want to come up in search results).

    Why would you invite them then?

    You already figured out the problem by saying that your manual sitemap is a "noisy ugly page". Focus on that instead of trying to hide something that is noisy and ugly.

    Comments on this post

    • KnowOneSpecial agrees : Good advice
    • Doodled agrees
  14. #8
  15. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2010
    Posts
    128
    Rep Power
    57
    This page is and old clunky sitemap, noisy... not well designed, so I don't want clients to see it.
    The url has my name and there are search terms on the page, hence, it comes up randomly on searches.
    But I want it working. I want it indexed and crawled.
    I just don't want it appearing as a search result.
    Can this be achieved? Anyone else? Thanks.
  16. #9
  17. No Profile Picture
    Moderator
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    1,972
    Rep Power
    2995
    Kevs I sent you a PM about this ..

    I just want to add...

    If you go to a doctor and complain that you hurt.

    The doc will ask you, Where do you hurt?

    You reply, well doc I don't want to say, because it is embarrassing.

    The doc says, well if you won't tell me where you hurt specifically then I can't diagnose your problems. If you are not allergic to aspirin, take two with water and maybe the pain will go away in 20 minutes , but if you tell me where you hurt we can definitely remove the pain with proper diagnosis and procedures .

    So with that said....

    Originally Posted by kevs
    This page is and old clunky sitemap, noisy... not well designed, so I don't want clients to see it.
    Step 1
    Redesign the page so it isn't clunky, noisy, and something you can be proud of.

    Originally Posted by kevs
    The url has my name and there are search terms on the page, hence, it comes up randomly on searches
    Ok, so you need to remove the file.

    Step 2
    Make the current url return a 410 status code. This will tell Google the page is gone, never coming back, and remove it from the Google index all within 24 hours after Google detects the 410 status code.

    Step 3
    Now create a new sitemap file with a filename you can be proud to show and not be embarrassed by. I highly recommend you use the following name >>> sitemap.html if it is a normal page, meaning like any other page on your site.

    Originally Posted by kevs
    But I want it working. I want it indexed and crawled.
    I just don't want it appearing as a search result.
    Sorry kevs, if Google Indexes it, Crawls it, then it is going to appear in the index and people can see it.

    Now kevs there may be a another solution but without seeing the issue eyes on, this is the best I can recommend you do !

    Your problem is probably one that can be resolved within 10 to 15 minutes if we could physically see the page. It would also eliminate playing 20 questions over and over.

    Comments on this post

    • kevs agrees
  18. #10
  19. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2010
    Posts
    128
    Rep Power
    57
    Thanks again Knowone, replied.

Similar Threads

  1. Original site had index.html. New site has index.php. Problem?
    By Cataclysmic in forum Google Optimization
    Replies: 5
    Last Post: Nov 16th, 2010, 05:45 PM
  2. What is best robot txt coding to insure spiders index all pages?
    By EandA in forum Search Engine Optimization
    Replies: 8
    Last Post: Dec 17th, 2008, 04:10 PM
  3. Replies: 0
    Last Post: Apr 29th, 2008, 04:03 AM
  4. Robot no index
    By Glenn Kilpatric in forum Web Design, Coding and Programming
    Replies: 5
    Last Post: Feb 3rd, 2007, 03:39 AM
  5. Robot index-follow - twist ...??
    By JoeBob in forum Google Optimization
    Replies: 2
    Last Post: Jan 23rd, 2005, 02:12 AM

IMN logo majestic logo threadwatch logo seochat tools logo