
A generalised solution to distributed consensus - based2
https://blog.acolyer.org/2019/03/08/a-generalised-solution-to-distributed-consensus/
======
karl_gluck
Is anyone else surprised at the relatively mild reaction to this result here
on Hacker News? Only a couple dozen comments, and many are addressing other
topics.

Can we take a second to appreciate that this result will rapidly become the
first thing taught in every single distributed systems class? And if this
holds as a generalization of trustful distributed consensus as a field, then
she has defined its Turing Machine equivalent. And it is even remarkably easy
to understand! That is wild!!

Props to the authors. Personally, this is going on the top of my “to memorize”
list.

~~~
brighton36
I'm very skeptical that many people will want this. Data misrepresentation is
rarely (never?) due to a fault in the persistence medium. Its usually a
political reason, or human error that perverts consensus.

~~~
atombender
This has nothing to do with "data misrepresentation". It's about distributed
consensus. It's not even really about storage, but about multiple parties
agreeing on a single truth.

~~~
eru
Even just something as simple as which of several replicas is supposed to be
the master. Not some great external truth about the world.

------
sagichmal
Great paper with an incredibly important result. It will take years to shake
out in industry and implementations, but should dramatically improve
situations where strong consistency is important.

~~~
pathseeker
Can you explain how? In many cases when a "generalized result" comes out there
was already a generalized result that covered all but obscure corner cases.

~~~
DannyBee
This is the first paper i've seen to generalize consensus while only using
immutable state.

That is, you can describe PaxOS, Raft, etc using the generalization given
here.

The result is fairly understandable.

(Of course, as mentioned, Raft is trivial in her framework because there is
only one thing deciding values, Raft is spending it's time in the leader
election part to find that one thing)

~~~
DennisP
Also, from the abstract, "the surprising result of this analysis is a
substantial weakening to the quorum requirements of these widely studied
algorithms."

------
0xb100db1ade
Would someone ELI5?

~~~
bch
I can’t, but I can say:

1) Heidi Howard’s papers are very fun to read. She’s done lots of work I’ve
enjoyed esp wrt Raft 2) She touches on two things in this story’s paper that I
first read in another paper of hers that are a joy to realize for two reasons
a) She made a practical improvement in the already simple Raft protocol b)
quorum != (obvious) majority - in (eg) Raft she identifies that quorum
actually spans two states, which are

* leader election and

* “regular” mode with a leader and requisite # followers

Read/watch on and be amazed[0][1]

[0] [http://hh360.user.srcf.net/blog/2016/08/majority-
agreement-i...](http://hh360.user.srcf.net/blog/2016/08/majority-agreement-is-
not-necessary/)

[1] [https://youtu.be/gYkueS5sKqo](https://youtu.be/gYkueS5sKqo)

------
grogers
Really cool result and a very readable paper. I'm struggling to figure out how
this would be applicable to iterated consensus (i.e. logs). It seems like the
extension examples are focused on reducing the message round trips for
specific cases that don't matter when applied to muti-paxos or raft.

------
ccleve
I have a lot of questions about this paper. Parts of it aren't clear to me.

Is anyone aware of a forum where consensus algorithms are discussed? A mailing
list, subreddit, or website?

I found a Raft mailing list a while ago, but I'm looking for a place for a
more general discussion, where real experts hang out.

~~~
macintux
For questions specifically about this paper, you might have luck with Adrian
Colyer's take on it: [https://blog.acolyer.org/2019/03/08/a-generalised-
solution-t...](https://blog.acolyer.org/2019/03/08/a-generalised-solution-to-
distributed-consensus/)

Or Howard's blog post: [https://hh360.user.srcf.net/blog/2019/02/towards-an-
intuitiv...](https://hh360.user.srcf.net/blog/2019/02/towards-an-intuitive-
high-performance-consensus-algorithm/)

Twitter used to be a useful place for general distsys discussions, but I
haven't been around for a few years now so I'm not certain how much still
takes place.

This tweet, for example, I would have expected to see more people chime in on.

[https://twitter.com/adriancolyer/status/1103935905948028928](https://twitter.com/adriancolyer/status/1103935905948028928)

There's a distributed systems theory Google Group, but it appears to have
completely died off:

[https://groups.google.com/forum/#!forum/distsys-
discuss](https://groups.google.com/forum/#!forum/distsys-discuss)

To inquire what avenues for theoretical discussion are currently active, I
would suggest reaching out on Twitter to Chris Meiklejohn (cmeik), Michael
Bernstein (mrb_bk) or Kyle Kingsbury (aphyr).

------
anonymousDan
My understanding is the cost of all these weaker quorums is reduced fault
tolerance. Is that correct?

~~~
DblPlusUngood
Yes. In theory, maybe it's useful to have more flexibility to trade-off
durability for performance (smaller quorum size hopefully reduces deciding
latency [unless your small quorum contains the straggler!]) for specific kinds
of data in the same replica group.

In practice though, it seems easier just to run separate instances of Paxos.

------
jrootabega
Forgive the tangent, but could quantum entanglement one day provide consensus
with trivial (if expensive) complexity? E.g., nodes maintain a supply of
preentangled particles, observing them at predetermined timestamps, and using
that known shared randomness to make decisions?

Edit: ok after that sat there for a while I realized that generating shared
pseudorandom bits would be just as effective.

~~~
karl_gluck
If you abstract quantum-entangled particles to a communication channel with
zero latency, you still have the consensus problem—you just run into the edge
cases less frequently. So it would help, but consensus is still a fundamental
challenge with parallel processes.

~~~
krastanov
Given the handwaviness I might just be misunderstanding what you mean by "zero
latency", so I just wanted to clarify: entangled particles do NOT enable FTL
communication. Sorry if I am saying something you know, but others might
misread your comment.

------
thaumaturgy
As an alternative to Paxos-related consensus algorithms, some projects may
want to take a look at the Raft consensus protocol:
[https://www.brianstorti.com/raft/](https://www.brianstorti.com/raft/)

Whereas Paxos tries to maintain a fully distributed consensus state, Raft
instead proposes a protocol for automatically electing a leader node from a
group of follower nodes, and coordinating the replication of the write log
through that leader node.

I haven't had the pleasure of experimenting with either one though and would
love to hear from people who have.

~~~
killjoywashere
Isn't raft the default for Bitcoin?

~~~
narrator
No. Bitcoin uses its own protocol. Raft could not be used for a secure
cryptocurrency bevause Raft does not handle byzantine faults[1]. It assumes
nodes are not run by malicious actors and messages sent are unaltered in
transit.

[1]
[https://en.m.wikipedia.org/wiki/Byzantine_fault](https://en.m.wikipedia.org/wiki/Byzantine_fault)

~~~
hugofirth
True. Though in tradeoff it has massively increased throughput. I'm always
frustrated when I see blockchains recommended as a solution for a consensus
problem in _anything_ but a zero-trust scenario

~~~
parasubvert
Even in a zero trust scenario, you need a majority of nodes to be good faith,
non-colluding actors. Even anarchy breaks down in the face of unequal power.

------
thecoinrepublic
The client maintains a decision table with one entry for each quorum of each
register set.

