Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Ask YC: Scalability advice for web apps
7 points by Frocer on Oct 8, 2008 | hide | past | favorite | 3 comments
I am currently hosting my web app on Media Temple's Dedicated Virtual servers. It has been working great, but since we are pretty early, we aren't getting crazy amount of traffic either. Our web app is basically a social network, and there's a matchmaking algorithm which could potentially become a bottleneck (think Match.com or eHarmony's matching system).

Given that, and as we are growing about 25% every month. I am just curious on what kind of scaling issue we should be worried about? Are there any best practices in both back-end / database design to accommodate future scaling issues?

I read database performance is usually the bottleneck for most web apps -- and since our matching algorithm does hit the DB pretty hard, what kind of things I could do to improve DB performance and scale it accordingly (besides the obvious, e.g. caching and index).

Thanks for any advice!




I think you just might have a legit candidate for a MapReduce-like algorithm here since your work seems to be a great candidate for parallel work. Check the Hadoop article that was posted here not too long ago.

It will let you scale the work over many nodes in a fairly linear fashion.

You might not even have to go that route if you properly shard your data among your computers. If you store user profiles based on locality or the like, the work is guaranteed to be local. Also, you can try segmenting your users based on general factors that group them (a course version of your matching algorithm) to preserve locality.


a) Find out how to partition your data.

b) Can you pre-process some of the queries and make the data available to the algorithm in a BDb or some such?

c) Can you make your heavier components into independent services?

d) You know what they say about premature optimization... :)

HTH.


Well, I guess this website can answer to your question: www.highscalability.com

Hope this helps.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: