|
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
| ||||||||||||||||||||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
#1
|
||||
|
||||
|
do deepcrawl / freshbot work together?
I'm looking over yesterday's google visits to my site, and I see mostly freshbot visiting the entryway pages to the new sections I had hoped the deepcrawl would visit. do the two work in tandem, with freshbot tipping deepcrawl off to areas to check out on its next visit? There are 1,000s of new pages I am hoping to have deepcrawl visit, but so far, freshbot has only visited them, and it hasn't done a very thorough job of indexing the directories (follows only 1 link to new page, then stops).
I've done away with SIDs, and straightened out the php URLs, so everything should be good to go. |
|
#2
|
||||
|
||||
|
looking at the logs more closely, I can see that freshbot arrived at the new directory first, catalogued each link (in no particular order) over a period of an hour, then left. Five hours later, deepcrawl came by and went to the exact same pages (again, in no particular order - no relation to the way the links appear on the page, or to the order freshbot visited) and disappeared. maybe the two form a tag team of sorts.
|
![]() |
| Viewing: SEO Chat Forums > Google > Google Optimization > do deepcrawl / freshbot work together? |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|
|