

Ask YC: Scalability advice for web apps - Frocer

I am currently hosting my web app on Media Temple's Dedicated Virtual servers.  It has been working great, but since we are pretty early, we aren't getting crazy amount of traffic either.  Our web app is basically a social network, and there's a matchmaking algorithm which could potentially become a bottleneck (think Match.com or eHarmony's matching system).<p>Given that, and as we are growing about 25% every month.  I am just curious on what kind of scaling issue we should be worried about?  Are there any best practices in both back-end / database design to accommodate future scaling issues?<p>I read database performance is usually the bottleneck for most web apps -- and since our matching algorithm does hit the DB pretty hard, what kind of things I could do to improve DB performance and scale it accordingly (besides the obvious, e.g. caching and index).<p>Thanks for any advice!
======
yan
I think you just might have a legit candidate for a MapReduce-like algorithm
here since your work seems to be a great candidate for parallel work. Check
the Hadoop article that was posted here not too long ago.

It will let you scale the work over many nodes in a fairly linear fashion.

You might not even have to go that route if you properly shard your data among
your computers. If you store user profiles based on locality or the like, the
work is guaranteed to be local. Also, you can try segmenting your users based
on general factors that group them (a course version of your matching
algorithm) to preserve locality.

------
umangjaipuria
a) Find out how to partition your data.

b) Can you pre-process some of the queries and make the data available to the
algorithm in a BDb or some such?

c) Can you make your heavier components into independent services?

d) You know what they say about premature optimization... :)

HTH.

------
mg1313
Well, I guess this website can answer to your question:
www.highscalability.com

Hope this helps.

