MSN Search Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
 
User Name:
Password:
Remember me
Go Back   SEO Chat Forums > Search Engines > MSN Search Optimization
Reload this Page

HELP! MSN Bot Eating Up ALL my bandwidth!!!

Discuss HELP! MSN Bot Eating Up ALL my bandwidth!!! in the MSN Search Optimization forum on SEO Chat.
HELP! MSN Bot Eating Up ALL my bandwidth!!! MSN Search Optimization forum discussing the Microsoft search engine for MSN and Live.com. Find tips to help you get high rankings in Microsoft's search. Discuss what factors are most important to its algorithm.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread SEO Chat Forums Sponsor:
  #1  
Old May 16th, 2004, 11:58 AM
e1superman e1superman is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2004
Posts: 2 e1superman User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
Exclamation HELP! MSN Bot Eating Up ALL my bandwidth!!!
MSN Bot Statistics for the month of May (up to May 15th -- only 15 days of stats!!!)

msnbot64047.search.msn.com:
bandwidth: 7625158 Kbytes (7.6GB)
files: 166806
hits: 167027

This is 29.4% of my sites bandwidth this month just from MSN Bot! Yesterday alone it used 1.3 GB bandwidth!

By comparison, Googles bot(s) have only used about 1.4GB total bandwidth this month.

How can I stop MSN from killing me! At this rate itwill use 15 GB this month! Just from being indexed!

HELP!!!!
Reply With Quote
  #2  
Old May 16th, 2004, 12:22 PM
Xia Xia is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 206 Xia User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 5
Send a message via MSN to Xia
If you want drastic measures just block it from your site in robots.txt

User-agent: MSNbot
Disallow: /

Or you can block it from specific directories. (recommended)
Reply With Quote
  #3  
Old May 16th, 2004, 12:27 PM
e1superman e1superman is offline
Registered User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2004
Posts: 2 e1superman User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
Is this much bandwidth worth it? I have a forums and I think it is indexing every thread! I am not sure on the payoff here. I will say that I am lucky that my major keywords are #1 on msn, google, alexa, yahoo, hotbot, lycos, and many more. Still, this is a major pain!
Reply With Quote
  #4  
Old May 16th, 2004, 03:13 PM
insurance quote insurance quote is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2004
Posts: 139 insurance quote User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 Day 7 h 9 m 34 sec
Reputation Power: 5
block them from getting to any big files or i wonder if you set the revisit date in the meta to like 7 days would do anything.
Reply With Quote
  #5  
Old May 16th, 2004, 04:15 PM
sorvoja's Avatar
sorvoja sorvoja is offline
Contributing User
SEO Chat Intermediate (1500 - 1999 posts)
  
Join Date: Apr 2004
Posts: 1,547 sorvoja User rank is Sergeant (500 - 2000 Reputation Level)sorvoja User rank is Sergeant (500 - 2000 Reputation Level)sorvoja User rank is Sergeant (500 - 2000 Reputation Level)sorvoja User rank is Sergeant (500 - 2000 Reputation Level)sorvoja User rank is Sergeant (500 - 2000 Reputation Level) 
Time spent in forums: 1 Week 6 Days 1 h 11 m 41 sec
Reputation Power: 11
Hi,

You could disallow msnbot all but your most important pages with robots.txt
If it can not go deep, then it can not use all that bandwidth.

I see the same thing. Msnbot grabbing every page, on every site, every day.
Reply With Quote
  #6  
Old May 28th, 2004, 07:00 PM
vprp vprp is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2004
Posts: 143 vprp User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 7 m 43 sec
Reputation Power: 5
i have a forum so i block bots from most areas of my site. there are so many pages such as registration, replies, new posts, printable pages. If you disallow them, you will save a lot of bandwidth. Keep bots to pages with content you would like people to find on search engines such as your home page, threads, archives and forum indexes.
Reply With Quote
  #7  
Old May 28th, 2004, 07:01 PM
vprp vprp is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2004
Posts: 143 vprp User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 7 m 43 sec
Reputation Power: 5
Think of how many members you have, too. Each member has a member profile but they don't have to be spidered. Most pages on the forum don't have to be spidered. I would go to the home page for whatever forum software you use and ask for a list of pages and directories to block from robots. Chances are, someone has already done so and has posted about it.
Reply With Quote
  #8  
Old May 28th, 2004, 10:38 PM
sraloff's Avatar
sraloff sraloff is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Jul 2003
Location: Sioux City, IA
Posts: 121 sraloff User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 3 h 37 m 47 sec
Reputation Power: 6
Send a message via MSN to sraloff Send a message via Yahoo to sraloff
You need to spring for a dedicated server with better bandwidth....
__________________
--------------------------------------------------
"You miss 100% of the shots you don't take"
--------------------------------------------------
Side 2, Inc. | City Info | City Shopping Directory | Sioux City Church
Reply With Quote
  #9  
Old May 28th, 2004, 11:10 PM
vprp vprp is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Apr 2004
Posts: 143 vprp User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 h 7 m 43 sec
Reputation Power: 5
Well, if MSNBot is taking up 29.4% of your sites monthly bandwidth in just one day, you need to control where bots are allowed to spider in robots.txt.
Reply With Quote
  #10  
Old June 6th, 2004, 12:27 AM
disgust's Avatar
disgust disgust is offline
Contributing User
SEO Chat Novice (500 - 999 posts)
  
Join Date: Jun 2004
Posts: 772 disgust User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 5
how can msn even afford to do this, though?

that's just insane, using that much bandwidth..
__________________
Get Relevant Links! Buy Links from the TextLinkBrokers!
Building Quality SEO Blog - Editors Wanted | Free SEO Tools
Reply With Quote
  #11  
Old June 6th, 2004, 01:22 AM
happyallday happyallday is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: May 2004
Posts: 338 happyallday User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 7 h 53 m 54 sec
Reputation Power: 5
in my log, msn visits very often, but I just don't know why it loves your site so much and eat so much bandwidth, I think this is not bad at all
Reply With Quote
  #12  
Old June 6th, 2004, 03:36 AM
Ricky Ricky is offline
Contributing User
SEO Chat Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 378 Ricky User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 Day 6 h 37 sec
Reputation Power: 5
Quote:
Originally Posted by happyallday
in my log, msn visits very often, but I just don't know why it loves your site so much and eat so much bandwidth, I think this is not bad at all



If u have forum simply allow to crawl topics.. nothing else.. no profile.. no search page and hope u will be fine. .;)
__________________
Believe in Yourself !

Linux Forums|Indian Forums
Shayari|Bodybuilding
Reply With Quote
  #13  
Old June 7th, 2004, 08:58 AM
gabs's Avatar
gabs gabs is offline
;-)
SEO Chat Beginner (1000 - 1499 posts)
  
Join Date: Mar 2004
Location: London
Posts: 1,325 gabs User rank is Corporal (100 - 500 Reputation Level)gabs User rank is Corporal (100 - 500 Reputation Level)gabs User rank is Corporal (100 - 500 Reputation Level)gabs User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 2 Weeks 3 Days 19 h 57 m 18 sec
Reputation Power: 7
how about blocking the /images folder ?

might help...
__________________
-->>Photo gallery <<-- Mortgage link exchanges welcome PM me today UK links directory - add your UK site TODAY!

philips flat screen tvs

flat screen tvs
I want an ipod
www.seohome.co.uk
Reply With Quote
Reply

Viewing: SEO Chat Forums > Search Engines > MSN Search Optimization > HELP! MSN Bot Eating Up ALL my bandwidth!!!

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support |