|
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
|
|
A high performance database engine using optimized data access for all development environments including Delphi, Visual Studio .NET, Visual Basic, Visual FoxPro. and more. Learn More |
|
#1
|
|||
|
|||
|
How crawler crawls a website
Hi friends
Merely tell me that my bandwith for the month september of my website is showing 700mb in logfile analyser i.e advanced statistics of a of xyz website so i want to know that is it possible that merely crawling my website by xyz website crawler or spider or robot results in 700mb bandwith or these pages are visited by the actual person who comes to buy our product from xyz website. Thanks in advance |
|
#2
|
||||
|
||||
|
It depends on the construction of your website:
If you have a .07mb image 10,000 surfers would do it! If you have 1,400,000 x 50kb pages one full crawl would do it! It seems a little excessive though ![]()
__________________
ClickyB "The quality of the visitor is more important than the volume".. Egol 22nd Feb 2008
New to SEO? Start Here: SEO FAQForum Rules & Posting Guidelines |
|
#3
|
|||
|
|||
|
It's a sound practice to review your visitor logs and see where the requests are coming from (as opposed to "general" stats provided by hosts).
In addition, not every crawler or visitor is beneficial to your website (s) and appropiate actions should ensue as a result of these detrimental crawls and/or visitors. |
|
#4
|
||||
|
||||
|
If its just visitors, you could reduce this by making sure revisited pages are cached... this can make alot of difference.
official Spiders aren’t going to download images... just the link to the image and name and alt... They only look at the text of the page, how many pages do you have indexed? There are nasty malicious spiders that may do this, you may want to think about blocking these types. Have a look at the ip addresses from apache/hosts log info, then look these ips up and see if any are malicious. |
![]() |
| Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Spiders > How crawler crawls a website |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|
|
|
|
|