Forcing The Spider To Go Deep

[Glenn Kilpatric]

Im in the middle of a large clean up (301) of : Old, Thin or Trashy Articles on my site. Same date back a lot of years and probably the spider doesnt see them very often.

Is there a way to force the spider to take a deep look at my site, or is it just a matter of waiting for it to eventually take another look for these articles.

[joshz]

A sitemap would help.

Creating a strong menu structure would help, too.

Are you using a CMS? They have pretty good archiving and structuring options.

[KernelPanic]

The amount of pages of your site that Google will index is related to your PR. Build quality inbound links to your interior pages and you will get them all crawled

[Glenn Kilpatric]

Cheers guys, I think the majority do get crawled, its just the site is quite large and and some of the posts havent changed over the years so im certain the spider wont go to some of them very often at all, and certainly the most important ones do get crwled. Sadly I had a lot of poor quality pages. Some were news related that are no longer relevent some were thin pages with little quality. Im deleting them and redirecting to category homepage or other relevent page with better content.

And yes Im running cms, well its wordpress, I do have yoast sitemap plugin, but Ive never been sure if it actually works. I really need to learn how to get the best out of it. One step at a time though as Im concentrating on getting the rubbish off the site. Im aiming at fewer higher quality pages.

[realityhack]

Generally speaking the more changes Googlebot sees the more it will look. Which makes sense because if you completely re-did your site or the domain was sold it would need a fresh index of everything. If you are talking mostly 301s it might not care because it doesn't effect users much. If a lot of pages actually changed, especially completely, google will tend to start poking around to see what else changed.

You can submit a site map but in the end you are mostly looking at waiting for Google to do it's thing.