I'd like to see some recommendations on similar articles for other related tech (couch, mongo, hadoop, etc) that did the same for others.
Digging a little deeper, you can think of it as a faster, extremely powerful version of Memcached.
The "movement" is about polyglot persistence and not leaving RDBMS completely. Pull pain points out into something that's a better fit. Rinse and repeat.
If you're using ActiveRecord, if the save to redis fails, raise an exception and it'll rollback the transaction.
Note that you'll always want a way to resync the redis copy of the data with what's in your database.
Why is that? Isn't Redis persistent?
If you're just using redis for write-through caching/memoizatoin and there's some failure, you still have the answer in hand and can return it to the user, you just lose the benefit of the speedup from the cache.
Its better to use it as the write-cache for complex datasets with the database being the backup.
What Redis appears to do great is that it is much more than just a key-value store with CRUD operations. You can model sets, lists, queues, counters, and do complex operations on these in-memory values.
It looks to me like it is best thought of as providing an interface to data-structures that are well optimized and persist across requests.
We just added it to our stack for caching and storing sessions.
It's blazing fast !
We're now trying to use it for different other purposes; autocompletion, counting and ab testing.
Resque is for background jobs (with many add-ons for locking, scheduling, retries, etc.), and redis-store is a drop-in store for Rack::Session, Rack::Cache and Rails.cache. Easy and super fast.
Yes, you've replaced a "select * from comments order by created_at limit 10" with a "select * from comments where id in (list_of_ids_from_redis)".
Wouldn't you cache the comment models in a top-10 list?
The database can read the last 10 via the index directly and they would all most likely be on the same index page. Assuming any sort of normal caching, this would be at most 1-3 random reads and most likely none since I presume that the created_at index is generally be written in ascending order.
Once that step is done, it is essentially identical to your IN statement you did.
MySQL doesn't support descending indexes, so for a large class of problems you will have to scan the entire index to find the last 10 items, especially when sorting by items in two directions. This is really slow when you have one hundred million entries (a huge events table). Looking up by primary key is very very fast in MySQL with InnoDB. If you profile this query you can see MySQL spends most of its time figuring out the IDs, and almost no time reading them back to you. Using Redis in this manner is very memory efficent, easy to update, and gets you 95%+ of the potential performance gains. It means we don't have to keep Redis up to date with edits or changes, because the PKs are set in stone.
The same composite index in MySQL can be used to do ASC-ASC and DESC-DESC sorts but cannot be used to perform an ASC-DESC sort.
I have a console app that's backed by Redis (in much the same manner as described in this post), but I save my sessions to h5 when I switch between datasets. That means I need to combine the Redis data with my app data and export -- I do this using two separate h5 files, with with the appropriate links.
It would be nice (for me anyway) if I could do a Redis-native save, and move the resulting file. That would also improve my startup times when I reverse the process.
But, while h5 is nice for My data, I can't say it would be any good for generic Redis data...
Really great tool for the belt.
I'm not sure of the typical game server stack though.
Redis is suitable for persistence, so where it would really be appropriate would be storing things like player profiles and other misc data that is not tied to a specific server. In fact, that's exactly what I'm using it for.
You might finally be able to break the model that current large games like WoW use - heavily sharded with all persistent data stored in Oracle. Redis should allow you to shard less and use SQL less, resulting in a much better experience, especially when parts of the virtual world get crowded.
I'm curious: what you think (or know?) they are doing that is causing the database to be a limiting factor?
If Redis can help them scale from the small shard to larger shards that support more players in an area, it might be interesting.
More apropos: would you really want one?