Ok, just did some math. In order to run Twitter using just Redis 1.001 without to use any new feature that may allow for memory savings, and guessing that Twitter currently contains 4,000,000,000 of tweets (assuming they save the full history for all the users, and that the recent 32 bit overflow means they rougly have 4 billion messages), 30 128GB Linux Boxes are needed.
Is this crazy? I don't know honestly as I don't know how many servers they may be using currently for the DB backend.
Btw the whole point is, IMHO, that many times to take the full dataset on Redis is not needed. For instance in twitter only recent messages are accessed frequently, together with user data, so it's probably a good idea to take only the latest N messages of every user in Redis (with background jobs moving old messages on disk incrementally), and take all the rest on MySQL or other on disk solution suitable to access stuff by-id.
So when you want to get a message from Redis, and from time to time get a NULL accessing to message:<id> you can run the same query against MySQL to get the data. That's something like this:
m = redis.get("message:"+id)
m = getMessageFromMySQL(id)
Also note that Redis supports expires in keys. So old messages got from MySQL can be set as expiring keys in order to avoid that a message that got linked in some front page will stress the MySQL too much.
This is just to give a feeling about scaling a pretty big service using Redis as main DB without caching layers.
So saying on the home page that "Redis can do [sharding] like any other key-value DB, basically it's up to the client library" is inaccurate. Distributed key-oriented databases like cassandra, voldemort, dynomite, riak handle all of that so it's totally invisible to your app, including (at least in Cassandra's case, and I think dynomite) adding nodes to the cluster.
it's really a matter of design. I like the idea that the Redis servers are dummy, and it's up to the client logic to handle sharding. For instance the Ruby client supports this feature in a way mostly transparent to the client.
In traditional databases sharding is hard not because they are not good at it form the point of view of "feature set" (like in Redis VS Cassandra), but because the data model itself is not right for working with data split across different servers. If you use an SQL DB just with tables accessed by IDs and without queries more complex than lookups by primary key, then sharing starts to become simpler.
Even if Redis will ever get server-side sharding, I'll code another process that handles this issue instead to put the logic inside Redis itself.
Btw how is it possible to build something really horizontally scalable without to use client-level sharding?
What you want is to have N web server and M databases, without any single-dispatch-node. At least this is how I'm used to thing at it.
Without any kind of client help I guess there is some kind of master node handling the dispatching of requests. Maybe I missed the point, please give me some hint.
Update 6: Some interesting changes from Twitter's Evan
Weaver: everything in RAM now, database is a backup; peaks
at 300 tweets/second; every tweet followed by average 126
people; vector cache of tweet IDs; row cache; fragment
cache; page cache; keep separate caches; GC makes Ruby
optimization resistant so went with Scala; Thrift and HTTP
are used internally; 100s internal requests for every
external request; rewrote MQ but kept interface the same;
3 queues are used to load balance requests; extensive A/B
testing for backwards capability; switched to C memcached
client for speed; optimize critical path; faster to get the
cached results from the network memory than recompute them locally.
It covers a bit more than KV stores - it's basically all of the non-relational data stores.
If anyone wants access to update it, let me know.
License: BSD (new version)
Expansion: not sure about the meainig
SLOC: 9262 including all the libs, no external dependence
Persistence: snapshotting on disk
Client protocol: custom ASCII
Data model: data structures (Strings, Lists, Sets)
Docs: can't rate it myself
Community: It's more known inside the Ruby community AFAIK
Yes, Lisp stores with C API :-D
AllegroCache, Elephant and ManarDB.
AllegroCache can store persistent maps and sets, so you could use it as a KV store, but that's hardly taking advantage of its power.
I have to say that I'm only considering free, open source solutions in that spreadsheet.
I have brewed up my own thing as well. Shoot me an email will ya? or I can get yours off of the .uk site in your profile.
Would love to see some kind of benchmark on that table though. I know it's hard to compare, but maybe add some comments to the results?
The kind of comparison needed in my opinion is a document explaining the different data model of the different projects, and of course also more general information like speed, test coverage, user base, and so on.
Any feedback is really welcomed, for instance, do you know of better data structures to do the work? I'm using a red black tree and an hash table. The pastie above documents the specification more or less. Thanks
Note that with this setup you don't need N additional memcached servers, so all the $ can be spent in RAM for the DBs.
Im my personal experience with high traffic web services what I discovered is that anyway when the on-disk dataset starts to get bigger than the available RAM performances are horrible anyway. Also note that when there is a lot of data to Index, the right thing to do can be the following:
take all the metadata on Redis, for fast access. Take the "bulk" data on MySQL, just an incremental ID + Blob field. Perform queries against both.
It is absolutely possible to add some kind of aging algorithm to swap not recently used keys on disk to Redis in order to allow for bigger than RAM datasets, but most people dealing with high traffic sites told me "don't do it, it's useless because we have access patterns that are more or less evenly distributed among the key space".
This guy found most of the bugs with huge datasets, and thanks to his work Redis 1.01 was tested in extreme conditions.
Keep up the good work. Some of us know what we're doing and we're using Redis as a cheap reliable message queue :-)
grazie per tutto!
The versatility is my main reason for loving it. From working on large a dataset without waiting for it to finish loading (…as mr. Willison explained), to storing object properties in a webapp-environment, to simple persistent object queues, etc..
It's a very nice tool to have in the box.
For instance Redis edge (the latest git version) is able to save a lot of RAM (more or less 25%) encoding in a particular way objects that can be represented as integers internally, but there is still a lot of work to do in this regard.
This was the only one I could find:
Unfortunately my spoken English is even worse than my written one, so go figure what can happen if I try to make Redis videos... :)
Edit: another video by Bob Ippolito: http://blip.tv/file/1949416/