Okie Dokie,

I'm in the process of setting up a very very large scale community web site.
It'll be very similar to faceparty, and we're hoping to have about 2,000,000 members in time, and every member will have their own profile.

Now, here's the bit i'm a bit worried about..

Each member will have an "interests" part of their profiles, and all the "interests" on their profile will be links to a page which lists all the members with the same interest listed. Exactly how livejournal user profiles work.

The site is expected to have literally hundreds of MILLIONS of pages in total. many of which will be very similar (i.e. user profiles, similar-interest pages..etc..)

Obviously google/msn/whoever isnt going to spider all the millions of pages, but would it be advisable to put a robots.txt file in anway, to make sure the site doesnt get banned?

or should i just let it do it's work?