How to handle URL changes of pages?

Well, sad but inevitable! We had to change URLs for quite a few of our sub-pages, which is for practibility and optimisation reasons in the long run .

While still quite a few of the old URLs get indexed, the spidering of the new
URLs goes slowly, naturally. We like to keep our old URLs live for the current traffic coming from google now, so are unsure about how to deal with the duplicate content that arises from this situation. I would be happy, if you could comment on the following suggestions:

- Leave both URLs for google to spider. Google will sooner or later notice
that the old pages are not linked up anymore and disregard them (not good, is it? ).

- Any form of 301, 302 re-direct or any other re-direct of the 300'er range
for the old pages. Are there differences in how google handles them (too many re- directs are risky, arent they?)?

- Serve a 404 on the old pages, but still display their content (google would scrap the pages, or not?).

- Put a meta robots tag with values noindex, nofollow on the old pages for google to disregard them rather sooner then later. It is difficult to do robots.txt with our pages, but google says it supports the meta html tag (favourable solution? ).

- Or, what about a simple noindex, but still allow google to follow the links? All links are accessible from the new pages as well (I guess that might salvage a bit of PR, while some of the new pages are not getting spidered yet).

Any other suggestions, I appreciate your help?

I would serve an "intelligent" 404, asking human surfers to click a certain link to go to the new page - or maybe the index page - and perhaps also putting a META refresh to do just that automatically after 10 seconds or so. This will keep your human visitors - the bots will just spider the new pages if they're linked to correctly from the index and other sections of the web.

Gringo.

Many thanks, for your reply, Gringo.

You wouldn't recommend a 404 return code together with the content of the old pages, why is that.

Do you reckon google would still parse the page, even with a 404? Or ... would you just say that is too uncertain to assume, and therefore not worth taking the risk?

Which technique would you use to return a 404 together with the content ?

To be honest, in terms of Google I wouldn't worry too much about the duplicate content. If there are no external links to these pages, Google will simply index the new pages within a month or so and drop the old ones.

My advice advice above was aimed more at not losing the human visitors - to return a customizable 404 (if your host allows this) to redirect surfers to the index page.

Gringo.

I had the same problem, I started out .html, then I was .shtml, then .asp, now .php... my partner is a programmer -what can I say.

But I still get hits from really old pages, now it's logical to redirect the index.html to index.php, etc.

But I have a ton of misc. odd pages, like a logo design I had in a portfolio section, there are too many to keep track of.

What we did was make a specialized 404, for a lot of other reasons, like speed of loading, etc I have a top nav section and a footer section that are server include files. In other words, One file has navigation and it is used on all pages. We made a custom 404, which shows up with all the current nav info. So, if you go to my site by some odd link, like /vnetlogo.html; you end up on a page with easy nav access to the current content.

I agree that I am less worried about 404's as the new pages get picked up, I am more concerned with helping users find content on my site.

I should add that Inktomi doesn't seem to drop pages, they keep slurping the same old dead links for years!

hope this helps,

-cris

Wow that is weird about inktomi slurping the same pages. What i would do is just use a meta redirect. Ive only had success with them in the past. But wouldnt a 302 redirect work well also?

-Josh

I'm just ignoring them, I only redirect the important core pages and my custom 404 does the rest.

-cris

Well, starting point is that you have your URLs controlled by ONE script/application only. For example, you have a page with a querystring (you can rewrite the querystring, if you wanna have fun, but for google it doesnt matter anymore).

I forgot to mention that these are the URLs I am talking about. I admit that for an arbitrary amount of all sorts of URLs it probhably is to tedious to make them all return content still.

In one application, though, I am able to virtually "catch" the querystring "error" in ASP by going:

If Request.Querystring("myVar") <> "" Then
Response.Status = "404 NOT_FOUND"
End If

Everything else you might wanna do with myVar in your application still goes, i.e. returning content. I am sure there are equivalent commands in other web languages, or probably, you can write out such an http status header instead.

[/quote]

Id leave the old page where it is, add a javascript redirect to the new page which will get 90% + of surfers as well as a "This page has moved click here" link so Google can crawl and find the new page.

I do the same with javascript but deleting the title and content of the old page. This way the page isn't spidered and I can delete the old pages in 2/3 months.