#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2012
    Posts
    10
    Rep Power
    0

    Noindex an entire folder?


    I have had a problem with spammers linking to a /js/ folder on my site. They had hacked my site and injected lots of pages. They then sent thousands of links to pages in that folder. I've removed all the pages. Can I instruct Google to noindex the whole folder? Even though the pages are 404, since there are so many links pointing to the pages, they are still in the Google Index.
  2. #2
  3. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2013
    Location
    UK
    Posts
    118
    Rep Power
    64
    Add the following to Robots.txt:

    User-agent: Googlebot (*if not already added*)
    Disallow: /foldername/

    Should solve your problem.

    You could also add 'Options -Indexes' with no quotes and leave an empty line after (just press return) in your .htaccess file, this will return an 'access forbidden' and prevent anyone else browsing the folder or linking to files they shouldn't and will help to sort your Google index problem after a little while by removing it from the index also.

    If you have removed the pages already then don't sweat it too much as they will be de-indexed naturally over time but keep an eye on the crawl errors in Webmaster tools and if you haven't already, check the security issues tab. Just to be on the safe side but it should not affect it, check manual actions as well, you never know.
  4. #3
  5. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,802
    Rep Power
    1841
    That said: adding this to your robots.txt file won't magically de-index these pages.

    You may have to pull some Google Webmaster Tools stunts to pull that off.
  6. #4
  7. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2013
    Location
    UK
    Posts
    118
    Rep Power
    64
    Originally Posted by dzine
    That said: adding this to your robots.txt file won't magically de-index these pages.

    You may have to pull some Google Webmaster Tools stunts to pull that off.
    Didn't mention it would magically de-index pages nor would it completely solve the problem, but it can help. How about giving some advice to the OP by explaining your webmaster stunts instead of being critical of everyone else every time you post. Some moderator you are.
    Last edited by Grizzler; May 11th, 2014 at 09:02 AM.
  8. #5
  9. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2012
    Posts
    10
    Rep Power
    0
    Originally Posted by Grizzler
    Didn't mention it would magically de-index pages nor would it completely solve the problem, but it can help. How about giving some advice to the OP by explaining your webmaster stunts instead of being critical of everyone else every time you post. Some moderator you are.
    I followed Grizzler's advice. I don't have any manual webspam actions I need to take according to Google. Those pages were all 404'ed on my site. I assumed Google would de-index them anyway. Now, since Google won't crawl the folder they're in, will it not even see that they're 404? Will it still think they're online? There are a lot of spam links pointing to now non-existing URLs in that folder, but Google won't remove the URLs from their index because there are tons of spam links pointing at them.
  10. #6
  11. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2013
    Location
    UK
    Posts
    118
    Rep Power
    64
    If the folder cannot be crawled or returns 404's then the pages will disappear over time. If your really worried, and if you have the info of the urls, remove the indexed urls by creating a remove url request in Webmaster Tools.

    One last one is to password protect the folder via your end.

    Hope this helps.

    Comments on this post

    • dzine agrees : Well possible that THAT will do the trick...
  12. #7
  13. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,802
    Rep Power
    1841
    Originally Posted by Grizzler
    Didn't mention it would magically de-index pages nor would it completely solve the problem, but it can help. How about giving some advice to the OP by explaining your webmaster stunts instead of being critical of everyone else every time you post. Some moderator you are.
    Like I said "That said", meaning that your answer holds value but that there is more to it.

    Moreover, I have already explained maybe dozens of times that robots.txt doesn't work with old/indexed pages. Especially not if other pages (be they blackhat or not) link to them. So while I felt that your statement -- and I quote -- "Should solve your problem." was a bit premature, I was hoping for someone else to chime in and to save me from sounding like a broken record

    Normally I would advise people to add a "robots" meta tag, but since this appears to be a folder stuffed with javascript rather than html, I fear this may not be an option. In which case I suspect that - on top of your suggestion of adding a robots.txt file - one should use GWT in some way or other and have Google actively drop the offending page(s). I don't know exactly how that is done, since I usually don't bother: yet another reason why I was hoping for someone else to chime in after I was keeping the thread 'warm' so to speak. Which is one of my minor tasks as a moderator here.

    P.S.: please know that, as a moderator, my main duty is to cull up to dozens of spam posts each day
  14. #8
  15. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2013
    Location
    UK
    Posts
    118
    Rep Power
    64
    Thanks for sharing, didn't hurt did it. And keep on culling those spam posts

Similar Threads

  1. NoIndex, NoFollow
    By sawyer in forum Search Engine Optimization
    Replies: 10
    Last Post: Jan 9th, 2012, 09:53 PM
  2. Noindex,follow or noindex,nofollow?
    By malhyp in forum Google Optimization
    Replies: 3
    Last Post: Sep 5th, 2007, 01:15 AM
  3. Are /folder/ and /folder counted as different pages?
    By jozomannen in forum Google Optimization
    Replies: 6
    Last Post: Sep 18th, 2005, 07:46 AM

IMN logo majestic logo threadwatch logo seochat tools logo