

Can someone explain database scaling issues for web? - zxcvvcxz

I don't really understand the technical challenges behind scaling, when people talk about using NoSQL or Redis.<p>I mean at the end of the day aren't you still trying to lookup (or insert or delete) into a very, very, very large list? Is one technique to "flatten" out your entire user database into a tree, say by location and then further subdivide so you just go down the tree to choose which smaller DB to access?<p>Those are my initial naive thoughts on the matter; could someone help me understand some of the non-trivialities to database scaling?<p>Thanks
======
andrejewski
Nowadays instead of having the sub-systems you are referring to for storing
data, many scaled databases use NoSQL and Redis for databasing and caching.
The caching is probably the more scalable of the two because caching at any
level has to do with usage and not all database data is used. Caching has
allowed the most used data to be readily available. The limits of scalability
may only require the system you described, but bigger and more complex systems
like Twitter and Pinterest have been documented using database scaling via
caching with things similar to Redis and Memcache. Keywords for finding more
information are "cache/caching".

Problems with the sub-system of databasing are too many transactions (I/O) and
the speed of getting the data and acting on it is very inefficient.

If you want to talk to me about it in more detail, message me on Twitter:
@compooter or get my email at mayorpear.com

~~~
zxcvvcxz
Here's a question: where is all this caching taking place? I understand
caching in the context of processors but not so much what you're describing.

------
jeffbcross
With MongoDB, there's a concept called Sharding, that divides a collection
based on a property in the collection the developer selects.

<http://www.mongodb.org/display/DOCS/Sharding+Introduction>

Each shard can be either a single node, or can be divided among nodes in a
Replica Set for distributed reads.

<http://www.mongodb.org/display/DOCS/Replica+Sets>

