Robots.txt - SEO Chat

SEO Chat Forums Sponsor:

May 19th, 2009, 09:30 AM

mauro21pl

Registered User

Join Date: May 2009

Posts: 5

Time spent in forums: 1 h 29 m 43 sec
Reputation Power: 0

Robots.txt

OK, so I want to remove some pages from my site. Looks like google crawled the site , I think. In a Google Webmaster tools > Tools > Analyze robots.txt I can see thart status is 200, whih is good, and las downloaded time is 19 hours ago. The content of the robots.txt file is excutly the same that I have in my file and it seem to be perfectly fine. When I test against my pages in the provided box in Google Webmaster tools > Tools > Analyze robots.txt it seems to be woking just like I would love to. However the pages are still there. How come? Is there any limit of time that I have to wait until the pages will be removed.

thanks

May 19th, 2009, 09:56 AM

dzine

DIYSEO

Join Date: Oct 2005

Location: sharing a room with my ego

Posts: 3,618

Time spent in forums: 2 Months 2 Weeks 4 Days 6 h 8 m 17 sec
Reputation Power: 734

Robots.txt is NOT gonna remove pages. It only tells the bots not to spider the pages again.

In fact it is best to change (=remove or empty) the robots.txt file, after you have added the meta tag <meta name="robots" content="noindex, noarchive" /> to the pages you want to get rid of. Then wait.

__________________

...Oi, DIY your SEO ...Love my hosting company...

May 19th, 2009, 10:21 AM

mauro21pl

Registered User

Join Date: May 2009

Posts: 5

Time spent in forums: 1 h 29 m 43 sec
Reputation Power: 0

are you really sure of that? Look at the google article. I can provide a link since I am new on Forum but it is under

Google Help › Webmasters/Site owners Help › Using Webmaster Tools › Site configuration › Removing your entire website using a robots.txt file

Plus I read some more reagrding removing pages using robots.txt

May 19th, 2009, 12:27 PM

dzine

DIYSEO

Join Date: Oct 2005

Location: sharing a room with my ego

Posts: 3,618

Time spent in forums: 2 Months 2 Weeks 4 Days 6 h 8 m 17 sec
Reputation Power: 734

Removing pages using the Google tool is only temporary.

AND if those pages have links pointing to them, they'll only return.

So yes

I'm sure

May 19th, 2009, 12:34 PM

mauro21pl

Registered User

Join Date: May 2009

Posts: 5

Time spent in forums: 1 h 29 m 43 sec
Reputation Power: 0

I did not mantion that I want to use Google tool to remove the page temporily. I just provided a link to a web page that explains that you actually can use robots.txt to remove web pages from your site.
And you are right that it will remove web pages only temporily.
The last my option may be to use the tool mention by you and the same time design appriopriate robots.txt, so it would prevent the removed sites from indexing. I think that will work perfectly fine.
But still I would like to know if it is possible to remove the sites from google index by using robots.txt. Opinions vary. I guess I will have to test it on my own.
Thanks