
Don't Settle for Eventual Consistency - mhausenblas
http://queue.acm.org/detail.cfm?id=2610533
======
rdtsc
Don't settle ... unless you can.

Some business processes/problems can handle eventual consistency. Banking is
the classic (and perhaps un-expected) example. You can overdraw your account,
there are processes to go back and solve conflicts. Some are automatic, some
are manual some are legal.

Bitcoin at the blockchain level is eventually consistent, just wait 10 minutes
or whatever the current time it (I don't use it personally just know about it)
and then you can be fairly sure of the validity of the transaction. There are
built-in incentive to assure histories will converge. But, wallets kept at
some exchange should _not_ be eventually consistent. Should _not_ be able to
take 100x more than your wallet holds and send it to someone else. There is no
regulatory, automatic of any other kind of framework to revert transactions
that went through.

There is crdt (a commutative replicated data type) research. So these are data
types that an always solve inconsistencies should they arise and instead of
diverging they auto-converge, in face of conflicts. Think set union operation
or max() function.

These kind of trade-off will percolate up through your data layer into your
business problem. For some cases you'd want to pick one, for some pick
another.

~~~
exabrial
Yah but banking sucks as an eventual consistency problem. Sure there are
processes to go back and solve conflicts, but they suck too.

Part of the reason we can't transfer money instantly between two accounts in
the USA is due to sorting out eventual consistency. Federal guidelines on
transfer intentionally make it a slow process so the manual processes can
catch up.

~~~
Terr_
> Part of the reason we can't transfer money instantly between two accounts in
> the USA is due to sorting out eventual consistency.

Yeah, but that's because __everything __is a case of eventual consistency.
Causality itself is limited to the speed of light, and "instantly" is
impossible. The only question is whether you want to block/wait, or gloss over
it with eventual-tricks.

Just look at online FPS games! Even with some of the best consumer-grade
communication links and high expectations for each node, it's _impossible_ to
provide actual "instant" behavior, and all modern games contain huge reams of
code dedicated to maintaining an _eventually_ -consistent environment.

------
sargun
So, I've actually been thinking a lot about this problemset. I think that
polyglot databases are the future. Most applications can deal with eventual
consistency, or at least bounded stale reads [see: most interactions with
Twitter, Facebook, etc..].

I think a hybrid consistency model will end up becoming the way we end up
going, but not without some changes to the working definition of eventual
consistency.

I think that eventually consistent systems that we'll see in the future will
at a minimum have atomic, consistent (as in ACID), durable transactions. The
big thing you're missing is isolation, and by that, you're also missing
serializability.
([http://www.vldb.org/pvldb/vol7/p181-bailis.pdf](http://www.vldb.org/pvldb/vol7/p181-bailis.pdf))

Bounded staleness consistency will probably make most people happy -- as in
"Give me a consistent snapshot of the database as of 1 second ago," or "Give
me a consistent snapshot of the database as of this logical time" (assuming
the database consumer has some sort of logical clock you're passing back and
forth -- In essence, MVCC with some exposure of the timestamp. (better
explanations:
[http://pages.cs.wisc.edu/~cs739-1/papers/consistencybaseball...](http://pages.cs.wisc.edu/~cs739-1/papers/consistencybaseball.pdf))

In addition to this, part of the problems that were outlined in the paper
talked about multi-datacenter issues -- a lot of issues with banking-like apps
can be solved with escrowing For example, each datacenter has 10% of the
account's balance, and you can use consistent commits in the datacenter. If an
interactions effects more than 10% of the account's value, it should occur
across datacenters, and make a consistent commit across datacenters. (More
info: [http://mdcc.cs.berkeley.edu/](http://mdcc.cs.berkeley.edu/))

Anyways...the future is bright, but we need someone to throw a ridiculous
amount of money, and time at it.

~~~
rdtsc
Basho's Riak DB is becoming such a hybrid. It is an traditionally an
eventually consistent database, but they are adding strong consistency in the
new release. [I don't work and am not affiliated with them, just follow them
on Github].

And by hybrid in this case I mean some part to keyspace can be tagged as
consistent while others remain as before.

~~~
sargun
Yep. I've done a lot of work with Basho's Riak already, but there is still a
lot of work to be done.

------
jude-
A more recent version of the COPS system is described here:
[http://www.cs.princeton.edu/~wlloyd/papers/eiger-
nsdi13.pdf](http://www.cs.princeton.edu/~wlloyd/papers/eiger-nsdi13.pdf)

Github for the more recent system:
[https://github.com/wlloyd/eiger](https://github.com/wlloyd/eiger)

You can find the first author's posters and talks on the subject of scalable
casaully-consistent storage here:
[http://www.cs.princeton.edu/~wlloyd/research.html](http://www.cs.princeton.edu/~wlloyd/research.html)

------
toolslive
Really, you don't want to drop problems like consistency, transactional
reasoning, data loss, ... on the head of an application developer. They should
be able to have a simple, idealized view on a database or key value store;
otherwise you risk their sanity.

~~~
vidarh
I'd argue that if you _don 't_ make application developers consider these
things, _then_ you risk their sanity when they have to deal with the fallout.

They need to understand the tradeoffs, and be able to decide which guarantees
are important when.

~~~
kstrauser
Exactly this. I wrote a blog (at [https://www.crittercism.com/blog/scaling-
with-eventual-consi...](https://www.crittercism.com/blog/scaling-with-
eventual-consistency/)) about why my company designs for eventually consistent
stores from the beginning. There are a lot of use cases - and the number is
growing as companies deploy large, scaleable systems - where a consistent
store just isn't practical or even needed.

You can't just use an eventually consistent store as an ACID system and expect
it to work, and it's almost always a Really Bad Idea to try to implement ACID
on top of EC. Understand how your database works and design to its strengths
instead of trying to pretend that its weaknesses don't exist.

------
Nacraile
Am I missing something, or is this just a rehash of lamport timestamps, which
were first described in 1978, and are part of any reasonable undergrad
distributed systems course?

~~~
notacoward
It's a pretty good rehash, but yeah, it's a rehash of Distributed Systems 101.
In particular, the idea that changes can be propagated eventually but still in
order is almost as old as computers, but don't say it too loud. For some
reason that seems to bring out the knives in the dark.

------
terranstyler
Very well written and informative once you distinguish "causal" from "casual".

------
david927
I think this is an eloquent response to this:
[http://www.businessinsider.com/the-future-of-the-
blockchain-...](http://www.businessinsider.com/the-future-of-the-
blockchain-2014-4)

------
tomp
Another take at this problem is HyperDex [1], which provides strong
consistency [2] and multi-key transactions [3].

[1] [http://hyperdex.org/](http://hyperdex.org/)

[2]
[http://hyperdex.org/papers/hyperdex.pdf](http://hyperdex.org/papers/hyperdex.pdf)

[3] [http://hyperdex.org/papers/warp.pdf](http://hyperdex.org/papers/warp.pdf)

------
exabrial
Eventual consistency violates the principle of least surprise.

"Oh you hit the save button? Yah, we'll get to that."

~~~
Nacraile
Eventually consistent systems aren't built because eventual consistency is
considered a "good" property. Eventually consistent systems are built because
eventual consistency is the least-bad option in a fundamental trade-off.

When a distributed system is partitioned, it is impossible to execute all
operations on a shared resource serializably. So, the system must either fail
some requests, or fail to be serializable.

You can have your principle of least surprise, or you can have a system that
is capable of serving traffic even after some drunk sailor drags an anchor
through a submarine cable. You can't have both.

------
mad44
Here is a summary of their SOSP 11 paper.
[http://muratbuffalo.blogspot.com/2012/09/dont-settle-for-
eve...](http://muratbuffalo.blogspot.com/2012/09/dont-settle-for-eventual-
scalable.html)

------
notacoward
The title is misleading link-bait (eventual does not necessarily imply
unordered) but otherwise looks like a pretty good discussion of issues many of
us have to deal with.

------
toong
The page is timing out. Or is that the 'eventual consistency' pun intended ?

~~~
roeme
Not timing out here, Pun would suck, and here's a cached version for you:
[http://webcache.googleusercontent.com/search?q=cache:KNB7FI3...](http://webcache.googleusercontent.com/search?q=cache:KNB7FI3WkdwJ:queue.acm.org/detail.cfm%3Fid%3D2610533+&cd=1&hl=en&ct=clnk)

