Need some help related to page and video loading time

Hello,

I am getting a web portal developed in which there are gonna be lots of videos ( ofcourse in flv format ) which can vary anywhere between 1 MB - 500 MB.

Now, when we are testing our application with the videos we noticed that the videos are taking forever to load, for example 4 Min video took more than 15 mins to completely buffer on 2 Mbps line...ahhh thats awful.

I read somewhere that the loading of pages and videos of a website has a lot to do with the kind of server its being hosted on and also the traffic of that website.

Well, my site is being hosted on a dedicated server and currently I am the only one accessing the website so for obvious reasons the load time has to be fast.I know that there is something which went wrong with the site but I am unable to figure that out.

I am also mentioning the features of the server I got just in case if you think the server isnt good might let me know :

Intel Xeon 3060 2.4 GHz Dual Core Processor
Windows Server 2003 - Standard R2 - Unauthenticated - 32 bit
5 IPs
100 Mbps Uplink
2000 GB Bandwidth
2 GB RAM
SATA
500 GB IDE/SATA HDD

Please advise me on how can I reduce the load time of the videos.If you know any links or articles or sites which you think might be helpful to me then please do share it with me.

Really hoping that I can get some sort of help here.

Any kind of help is greatly appreciated.

-Nikhil