#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2017
    Posts
    95
    Rep Power
    1

    How to force google to recrawl my website


    I made a mistake with the robots.txt and the noindex meta tags and google crawled and indexed about 1000 pages of thin content that I was trying to hide. Now I have correctly set the noindex tag, I did allow google to crawl it (through robots.txt) and I am waiting for it to happen.

    Is there a way to speed it up? Should I request DEINDEX of those pages one by one? Should I fetch them in WMT and request reindexing?

    Here is a sample page that google reports it is indexed, but I now have the noindex tag:
    learn-greek-online.com/ask-greek/register?to=
  2. #2
  3. Administrator
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2014
    Posts
    213
    Rep Power
    584
    GWT is the way to attract Gbot, but it should find you quickly if your site is "interesting"
    Make sure you have a sitemap it can find.
    You really cannot speed it up more than that.
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2017
    Posts
    149
    Rep Power
    374
    I had a similar thing happen...I accidentally blocked a javascript that was needed to render my product pages....didn't notice after a couple weeks, and soon noticed my ranks and indexed pages were all dropping simultaneously....such a bonehead move...

    In any case...i removed the block, and it took about a month for the pages to be re-indexed, and about 2 months to regain my rankings...it's a slow process unfortunately. Good news for you, I think it's quicker to de-index than it is to re-index pages, based on my experience.
  6. #4
  7. Administrator
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2014
    Posts
    213
    Rep Power
    584
    Yes, de-indexing is relatively fast, especially if you want to force it.
  8. #5
  9. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Nov 2017
    Posts
    95
    Rep Power
    1
    Originally Posted by engine
    Yes, de-indexing is relatively fast, especially if you want to force it.
    I have about 950 pages that I want them recrawled and deindexing. This is my problem.
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2017
    Posts
    149
    Rep Power
    374
    You can't ask Google to recrawl and deindex at the same time....Not sure what you are trying to do? If there are certain pages you don't want indexed, make sure there is a no index tag in the HTML of each page. Most CMS have some sort of check box for that.

    If they were blocked with a no index tag and/or robots.txt file, and now you want them indexed, remove the tag and the disallow. Also make sure you have an updated sitemap that only includes the pages you want indexed, and upload that to Search Console.

    Comments on this post

    • KnowOneSpecial : Not true
    Last edited by Digital Marketeer; Aug 27th, 2018 at 09:35 AM.
  12. #7
  13. No Profile Picture
    Moderator
    SEO Chat Hero (2000 - 2499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    2,401
    Rep Power
    3129
    Originally Posted by Digital Marketeer
    You can't ask Google to recrawl and deindex at the same time.
    Actually you can.....

    1. Do not block the page in robots.txt
    2. Specify the page to be "NoIndexed" by using either robots meta tag no index or X-Robots-Tag: noindex.
    The latter is sent via the header.
    3. Have Google re-crawl the page to recognize the no index request via the x-Robots-tag or robots meta tag

    By the way, the above is Google's recommended solution.

    Just for the record...



    Google also states...

    Comments on this post

    • Digital Marketeer agrees
  14. #8
  15. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2017
    Posts
    149
    Rep Power
    374
    I poorly worded my response - I took it as him wanting to index and de-index at the same time, so I wasn't fully understanding his intent. Your reply is accurate.
  16. #9
  17. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2018
    Posts
    26
    Rep Power
    0
    You have done everything right, now you just need to wait.

Similar Threads

  1. How to force search engines to recrawl my website
    By andyellen in forum New User SEO Questions and Answers
    Replies: 8
    Last Post: Apr 18th, 2015, 06:06 PM
  2. Avg Time from ReCrawl to ReIndex?
    By consultant in forum Google Optimization
    Replies: 9
    Last Post: Aug 2nd, 2011, 12:03 PM
  3. Replies: 6
    Last Post: Apr 7th, 2009, 03:59 PM
  4. Time required - crawl vs recrawl?
    By cavemanlawyer15 in forum Google Optimization
    Replies: 4
    Last Post: Jun 29th, 2007, 01:09 AM
  5. Cyber attacks force Estonian bank to close website (AFP)
    By RSS_News_User in forum Technology News
    Replies: 0
    Last Post: May 15th, 2007, 01:03 PM

IMN logo majestic logo threadwatch logo seochat tools logo