#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2017
    Posts
    8
    Rep Power
    0

    How to stop crawling particular webpage?


    Hello guys,
    Did anyone know about how to stop crawling the particular webpage of a website which has already been crawled by the googlebot. We have tried from robots.txt but still it is crawling. We also temporary remove particular pages from webmaster, but did not got the permanent solution. If anyone knows how to stop crawling it permanently please suggest me.
    Last edited by ClickyB; Jun 3rd, 2017 at 12:14 PM.
  2. #2
  3. Super Moderator
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Mar 2004
    Location
    Gloucester (South West UK).
    Posts
    6,483
    Rep Power
    3367
    Hi kalyanit,

    Are you sure you want to stop crawling, (or do you actually want to prevent indexing)?

    Googlebot normally obeys robot.txt directives, (although stopping them from crawling it will not prevent them from indexing it).

    Give us the contents of your robots.txt file if you want assurance that it's correctly formatted (or give us the page url).
    ClickyB
    "The quality of the visitor is more important than the volume..." (Egol 22nd Feb 2008)
    [New to SEO/SeoChat?] [Canonical Problems?] [Forum Rules & Posting Guidelines]
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2017
    Posts
    8
    Rep Power
    0
    Hello sir,
    I want to prevent indexing and also stop crawling the page and image too. Below is robots.txt format. Please help me to solve this issue.

    User-Agent: *
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
  6. #4
  7. Just a Photographer
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    May 2011
    Location
    Atwater, CA
    Posts
    787
    Rep Power
    1429
    Your `robots.txt` looks fine.

    If you want to prevent indexing, go to your WordPress Dashboard > Settings > Reading > Search Engine Visibility > click "discourage search engines from indexing this site".
    - Mike

    Jack of all trades, master of none.
  8. #5
  9. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2017
    Posts
    8
    Rep Power
    0
    Hello,
    Thanks, but I don't want to prevent indexing the whole site...only particular pages or images.
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2017
    Posts
    25
    Rep Power
    0
    Originally Posted by kalyanit
    Hello guys,
    Did anyone know about how to stop crawling the particular webpage of a website which has already been crawled by the googlebot. We have tried from robots.txt but still it is crawling. We also temporary remove particular pages from webmaster, but did not got the permanent solution. If anyone knows how to stop crawling it permanently please suggest me.
    If you have a few directories like /cgi-bin/, /private/, and /tmp/

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /private/
    Disallow: /tmp/
    Disallow all search engines from particular files:

    If you have a files like contactus.htm, index.htm, and store.htm

    User-agent: *
    Disallow: /contactus.htm
    Disallow: /index.htm
    Disallow: /store.htm
  12. #7
  13. Just a Photographer
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    May 2011
    Location
    Atwater, CA
    Posts
    787
    Rep Power
    1429
    If you are using one of the SEO plugins (like Yoast), you can set a page to 'noindex'.
    - Mike

    Jack of all trades, master of none.
  14. #8
  15. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2017
    Posts
    8
    Rep Power
    0
    Thankyou guys for helping me. Your answers really work out for me.

Similar Threads

  1. Stop crawler from crawling and indexing iframe & img tags
    By SureNot in forum Search Engine Optimization
    Replies: 0
    Last Post: Jan 16th, 2014, 09:44 AM
  2. Why stop a search engine crawling a directory?
    By Jonagik in forum Search Engine Optimization
    Replies: 2
    Last Post: Nov 9th, 2007, 03:54 AM
  3. Google stop crawling my site?
    By jlbest in forum Google Optimization
    Replies: 4
    Last Post: Jan 31st, 2005, 01:28 AM
  4. Does Y! stop crawling when updating
    By raz in forum BING/Yahoo Search Optimization
    Replies: 1
    Last Post: Jul 29th, 2004, 11:17 PM
  5. Do Divs stop crawling?
    By cityneil in forum Google Optimization
    Replies: 4
    Last Post: May 18th, 2004, 01:23 AM

IMN logo majestic logo threadwatch logo seochat tools logo