
How to take advantage of Redis just adding it to your stack - antirez
http://antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html
======
bretthopper
I've read about Redis before and heard how companies are using it, but never
completely understood it's purpose. After reading this I can actually say I
understand Redis now and how it's useful. Amazing that after hearing so much
about it all it took was a relatively simple article.

~~~
jarin
On a basic level, you can think of it as just a faster version of Memcached.

Digging a little deeper, you can think of it as a faster, extremely powerful
version of Memcached.

~~~
eliben
I'm not too familiar with memcached, but isn't one of Redis's strong points is
the data structures it provides out of the box, and not simple lookups?

~~~
code_duck
Yes, that's the advantage. Redis is a memory-based DB like memcache, but is
more than just a simple key-value store.

------
bretthoerner
We use a ton of Redis, but I think the main takeaway from this article applies
to all "NoSQL databases".

The "movement" is about polyglot persistence and not leaving RDBMS completely.
Pull pain points out into something that's a better fit. Rinse and repeat.

~~~
stephth
I'm a little concerned with the added complexity of decoupling the datastore
into two different systems that are relied on for application logic. What are
good strategies to maintain consistency between the two, in the event of a
failure? In the first example, what if the SQL transaction succeeds but the
redis one fails? Would you rollback the SQL transaction?

~~~
edanm
It's a big problem, at least for our application (chattybar.com - a chat
plugin). We have 2 copies of every chat message sent, one in the DB (for
persistency) and one in Redis for quick retrieval. Keeping them synced has a
lot of edge cases, especially since multithreading means that adding things to
the DB and then to Redis doesn't necessarily guarantee they'll end up in the
same order.

~~~
dolinsky
If all you're doing is keeping a copy of objects in a memory store for quick
retrieval you might want to implement a read-through caching pattern using
Memcached (or Redis but I'd turn any persistence off). Most chat clients don't
allow for editing of posts, so you could just use Redis as your persistent
storage using lists and RPUSH.

------
true_religion
What I get from this is that Redis is so powerful that its best to not use it
as a simple read-cache where the database is still the cannonical source.

Its better to use it as the write-cache for complex datasets with the database
being the backup.

~~~
Periodic
Right. I believe we already have plenty of read-caches out there. Our web
servers can implement them, we have Varnish and MemCache and ~100 others.

What Redis appears to do great is that it is much more than just a key-value
store with CRUD operations. You can model sets, lists, queues, counters, and
do complex operations on these in-memory values.

It looks to me like it is best thought of as providing an interface to data-
structures that are well optimized and persist across requests.

------
mickeyben
Very good article: there is some good exemples on how you could use Redis.

We just added it to our stack for caching and storing sessions.

It's blazing fast !

We're now trying to use it for different other purposes; autocompletion,
counting and ab testing.

------
jarin
Resque and redis-store are like auto-adds for almost any Rails project I work
on these days.

Resque is for background jobs (with many add-ons for locking, scheduling,
retries, etc.), and redis-store is a drop-in store for Rack::Session,
Rack::Cache and Rails.cache. Easy and super fast.

------
randito
In your first example, you use redis to cache the id's of the latests
comments, with a fallback to SQL in order to populate the list. However, you
still need to call the DB to load the comments. I don't see the gain here.

Yes, you've replaced a "select * from comments order by created_at limit 10"
with a "select * from comments where id in (list_of_ids_from_redis)".

Wouldn't you cache the comment models in a top-10 list?

~~~
antirez
Often the DB will show unacceptable time to reply to the ORDER BY stuff but
will fetch comments by single ID without problems. But when that second part
is a problem as well it is a good idea to use Redis as a "vulgar" cache for
items as well, so that the recent stuff are probably into a Redis hash and you
can fetch everything with a single pipelined MULTI/EXEC call (and fetch all
the items returned as nils from the DB).

~~~
rbranson
A composite index will work wonders for this situation. EXPLAIN is your
friend.

------
rch
Does anyone know if hdf5 would be an acceptable optional replacement for the
current Redis disk format?

I have a console app that's backed by Redis (in much the same manner as
described in this post), but I save my sessions to h5 when I switch between
datasets. That means I need to combine the Redis data with my app data and
export -- I do this using two separate h5 files, with with the appropriate
links.

It would be nice (for me anyway) if I could do a Redis-native save, and move
the resulting file. That would also improve my startup times when I reverse
the process.

But, while h5 is nice for My data, I can't say it would be any good for
generic Redis data...

thoughts?

------
datadon
These little fixes are how I got into Redis and a month or so later, it's a
primary data store (with disk based fall back) and I find myself doing 99% of
aggregation and temporary storage operations with it.

Really great tool for the belt.

------
Joakal
Would it help game servers to run it with Redis? Like say, a FPS or RTS
server.

I'm not sure of the typical game server stack though.

~~~
reitzensteinm
Usually the game server would keep everything in the game instance in memory,
for an FPS or RTS. As great as Redis is, it's way more cumbersome than having
everything in your data structures ready to go.

Redis is suitable for persistence, so where it would really be appropriate
would be storing things like player profiles and other misc data that is not
tied to a specific server. In fact, that's exactly what I'm using it for.

~~~
illumin8
I realize that for most FPS or RTS titles, Redis might not be a perfect fit
because the dataset is small for each map and the number of players never
exceeds some preset limit like 64, but I think Redis might really shine as a
means of storing persistent world data - for example MMOs, without having to
shard heavily. Most modern MMOs use heavy sharding and can really only support
a couple hundred players in the same area or shard at the same time. With
Redis, you might be able to handle many thousands of players in the same area
without sharding. You don't need sub-millisecond response time - as long as
you can do most operations in a few milliseconds, things will be fine, since
most players are on ~100 millisecond WAN connections anyway.

You might finally be able to break the model that current large games like WoW
use - heavily sharded with all persistent data stored in Oracle. Redis should
allow you to shard less and use SQL less, resulting in a much better
experience, especially when parts of the virtual world get crowded.

~~~
sbov
I would be extremely surprised if WoW has issues when lots of people are
crammed in a small space due to their Oracle backend. More likely it is all
the calculations that the serverside is performing. I know a few people who
work on other MMOs and they keep _everything_ in memory, even accounts for
logged off users.

I'm curious: what you think (or know?) they are doing that is causing the
database to be a limiting factor?

~~~
illumin8
I didn't mean to imply that the bottleneck was due to Oracle. I was saying the
exact same thing you are saying - it's due to memory or CPU limitations in
their application server, which forces them to heavily shard.

If Redis can help them scale from the small shard to larger shards that
support more players in an area, it might be interesting.

------
geuis
What are some inexpensive cloud options to run redis for large data sets up to
say a gb or so?

~~~
jarin
"The Cloud" isn't always the right solution for everything. If you're looking
to go inexpensive, you can put it on a 2 GB Linode VPS on a ramen budget.

~~~
code_duck
The confusing thing is some people refer to VPSs as 'cloud', which sort of
makes sense when you compare them to EC2. EC2 seems more 'cloudy' because...
um, we don't know what kind of hardware it's running on? I don't know. I was
trying to explain what 'cloud' means to my tech-savvy, but non-industry
brother and was puzzled by the issue of whether Linode is a 'cloud' service or
not.

~~~
jarin
I guess it sort of could be, but I'd consider a "cloud" service something
where you either a) can spool up or shut down servers dynamically from an
image without any additional configuration (ala EC2 or Heroku), or b) a
commodity service like Amazon S3 or MongoHQ where you pay by usage and they
handle resource allocation.

~~~
code_duck
I see scriptability and automation through an API as important to 'cloud'
classification too. Good point about the automatic resource allocation, I
think that's actually the key. On Linode you're paying for a segment of a
specific machine, more like a dedicated server than EC2.

------
pg_bot
This is a great example of how to promote adoption of a new technology. More
companies should pay attention to how their product can be used rather than
what their product is.

------
ww520
Anyway to use Redis in App Engine setting? Or does Google have similar service
in App Engine?

~~~
StavrosK
No and no. They only offer Memcache.

------
bravery
"just adding it to your stack" is the best conclusion that i have seen so far.
Great article!

------
va_coder
I'm looking forward to the book

