
Redis cluster, no longer vaporware - untothebreach
http://antirez.com/news/79
======
ryanpetrich
Can't wait to see it get the Jepsen treatment.

~~~
jrallison
It won't do well, but at least Antirez is up front about the shortcomings.

The article itself states that Redis Cluster will lose acknowledged writes
during partitions (no P), will be eventually consistent (no C), and won't be
available during partitions (no A).

~~~
swartkrans
Seems like given the above problems you would only want to use this with non-
critical data of which you didn't care a great deal about its integrity. This
doesn't describe a lot of use cases.

It's not too hard to get memcached running for cluster of machines, nor is it
really all that difficult to get started with cassandra for basic
applications, although I haven't done it. I've heard ReThinkDB wants to be a
serious contender and that they've tried to solve these problems, but I
haven't read about how great it is nor have I heard anyone use it in
production so I don't know where it stands.

~~~
seiji
_This doesn 't describe a lot of use cases._

Do you have any idea how many people run master/slave mysql? That's async and
will _definitely_ give you data drift over time. There's an entire suite of
Percona tools to detect and repair broken mysql datasets because mysql breaks
data so often due to replication problems.

The world is much more async and much less consistent than anybody realizes
(cash machines are even async), but everything still works pretty much okay.
The last line of defense for solving irreconcilable consistency problems is
customer service.

~~~
jrallison
async != an incorrect system. Cash machines _are_ a good example of an
asynchronous, eventually consistent system that works. As is any well built
eventually consistent data store.

Redis/Mysql has it's problems, as does all databases. However, as a
practitioner it's important I know the shortcomings of each, so that I can
know exactly what a certain database is offering in terms of safety and
whether it's a good fit for my use-case. Antirez being open about them is a
huge +1 in my book.

Data drift or loss of acknowledged writes isn't correct for a wide range of
use-cases. If you know the limits, you can choose the incorrect system, and
deal with the fallout when that "rare" event does happen.

~~~
seiji
_async != an incorrect system._

Very correct! Most of the mysql problems derive from it being statement
replication and not log shipping. A proper postgres replication setup will be
less prone to having data inconstancies.

------
peterwwillis
@antirez, what would you say is the most useful lesson you've learned
throughout this process?

~~~
antirez
Argument with people just with technical arguments, don't get into Twitter
fights, and at the end of the day, use the feedbacks as a learning process,
but do what you believe makes sense. So a social lesson, more than
technological.

~~~
legedemon
Thanks for this lesson! As much as I love using Redis, this learning will help
me way more than Redis has done.

------
chrisan
Anyone know how AWS clusters Redis?

I had alwas just assumed Redis released this a while ago and that is why AWS
started to support Redis along with Memcache in their clustered ElastiCache
product

[http://aws.amazon.com/about-aws/whats-
new/2013/09/04/amazon-...](http://aws.amazon.com/about-aws/whats-
new/2013/09/04/amazon-elasticache-for-redis/)

~~~
mfenniak
I have very little evidence to support this, but they probably use twemproxy.
twemproxy is a proxy for redis that automatically manages routing commands to
multiple redis servers efficiently.

They definitely don't use redis cluster, because redis cluster requires
explicit client-side support. That support is only available in a limited
number of clients today.

~~~
seiji
Nope, they don't (and neither should anybody else... twemproxy is weird).

~~~
mfenniak
Oh, weird in what way? I was considering using it for horizontal redis scaling
recently, under the impression that redis cluster was further away from
release than it seems to be.

~~~
seiji
The code is just strange. To support Redis, it has a 2,500 line Redis adapter.
It has special cases for each Redis command because it doesn't extract key
position information from Redis directly.

Plus, despite the name and where it's hosted, Twitter never used it in
production for Redis. Redis support was just added "because we can."

It probably works for very direct use cases, but it's one of those things that
could probably bite you very severely under edge cases, and you'd be left on
your own to figure out what happened.

~~~
zamio
pinterest, snapshot, vine, wanelo use the twemproxy in prod for redis

~~~
seiji
Well, companies also use ejabberd, but that doesn't make ejabberd a valid
piece of software either.

Then there's the massive problem of "everyone" using MongoDB and we know how
much pain _that_ sucker causes. Playing popularity contests with software
isn't useful. People install software either because of social reasons ("I
don't know how anything works, but I hear everybody is using X, so I'm going
to use X too!") or because they need the functionality ("redis is AMAZING at
30 different things!").

If install base at well known companies were a valid measure, we'd all be
using Windows laptops while talking into our Kin and squirting our Zunes all
over each other.

------
brianpgordon
> From the point of view of distributed databases, Redis Cluster provides a
> limited amount of availability during partitions, and a weak form of
> consistency. Basically it is neither a CP nor an AP system. In other words,
> Redis Cluster does not achieve the theoretical limits of what is possible
> with distributed systems, in order to gain certain real world properties.

@antirez Could you elaborate on what these properties are and your thinking
behind why they're more important than AP?

------
itamarhaber
Congratulations @antirez and the Redis community - this is a key step forward
for this amazingly useful database!

------
thorway-redis
Your communication with your users is wonderful. I'm grateful for it, and
appreciate how much effort goes into it.

Thank you.

------
alphadevx
Single-master Redis and replication overhead of current Redis architecture not
ideal for getting Redis beyond the "it's a great cache" use case, so I'm been
looking forward to this.

------
merlin83
I created a Fig environment in case anyone is interested in giving it a spin.

[https://github.com/merlin83/rediscluster-
docker](https://github.com/merlin83/rediscluster-docker)

------
joshbaptiste
Wonder is Antirez is considering starting a commercial Redis alternative
providing commercial support a la Nginx.

~~~
antirez
The world if full of possibilities... for now to be honest I focused _a lot_
in the technical / community side, and the sponsorship model helped a lot with
that. However for 10 years before starting this project I was self-employed
and started two companies. So yes, doing companies is generally tempting for
me, just in the case of Redis so far it was more an evolution of an hacking
session, lasting 6 years.

------
jpgvm
I am going to be honest, I am somewhat disappointed.

I was hoping for a CP system with sharding + replication.

Without which we can't have things like multi-key atomic operations etc...

Will wait and see though, maybe things are better than they seem.

~~~
antirez
I'm pretty sure that if you are a Redis user, whatever you hate or like the
current Redis Cluster design, what you _never_ wanted, is a CP system. CP
systems need agreement at every query so the latency and OPS/sec figures are
not suitable for Redis-alike use cases. Actually with an existing CP system
you can easily build the same Redis data structures since CP systems are
linearizable so you can have a CP shell that internally runs a Redis kernel.
Imagine the CP system is Raft, you just consider Redis your internal state
machine, and every operation that is committed is applied to Redis just
sending it the command.

~~~
jpgvm
That is actually exactly what I wanted. :)

I have done something similar with Zookeeper recently (through I wrote it in
Ruby so it's terribly slow).

You would be surprised how little CP stores are actually available. There is
an abundancy of AP stores but very very few useful CP stores, especially ones
with shared-nothing architecture.

You don't need for every query to traverse the consensus algorithm. It's
sufficient to use the consensus algorithm to agree on the master shards for
each chunk of data and have sync replication.

Considering Redis is designed for 100% in-memory workloads if you are
operating on a low latency network this is more than fine. Especially because
this behavior would only be required for writes, reads could be served from
either the master or a sync replica (and in theory if you had configurable
read consistency a possibly out of date async replica).

I guess what I am sad about is just the lack of CP stores that are useful
right now, I probably just need to suck it up and start writing my own.

------
abluecloud
>stopped and stopped

