
Stellar Consensus Protocol: Proof and Code - joyce
https://www.stellar.org/blog/stellar-consensus-protocol-proof-code/
======
leothekim
I'm finding the graphic novel explaining federated consensus to be really
entertaining: [https://www.stellar.org/stories/adventures-in-galactic-
conse...](https://www.stellar.org/stories/adventures-in-galactic-consensus-
chapter-1/)

~~~
joyce
Joyce from Stellar here. Thanks! As we were working on the white paper, we
realized how difficult it was to explain complex concepts like federated
Byzantine agreement.

We know it’s part of our jobs to make these ideas understandable. That way
more people can join the dialogue and think of ways this infrastructure can be
used to build services for their communities, which may be really far from the
nearest computer science program. So we decided to add a lighter approach in
hopes of making it fun for people to learn.

~~~
leothekim
Hi Joyce, thanks for chiming in! I'm glad Stellar is committed to elucidating
the ideas behind the technology, and this is a thoughtful and creative
approach. Reading the graphic novel first helped me understand the idea behind
quorum slices while reading the paper. Can't wait to see more of this!

------
ef4
I'm excited for the ideas here and have been following Stellar.

But I'm hugely disappointed to see that they went with C and C++ for their new
core codebase. This is the kind of code that needs strong safety, security,
and correctness guarantees, and here in 2015 we have several mature languages
with better safety & correctness guarantees.

C# and Java are both mature and mainstream, and either would have been a sane
choice. Go is slightly less mature but also a safe and conservative choice.

(I personally love where Rust is going too, but I could excuse people for not
choosing it yet due to immaturit.)

~~~
jude-
If a program is formally proven to be correct, it doesn't matter what language
it's written in.

~~~
jroesch
Language choice can be important. If you do the common thing and build a
distinct model of your program and prove it correct your guarantees don't hold
about the actual implementation. You need a second method to ensure
equivalence between your model and implementation, this obviously requires
much more work and can vary greatly based on the tools and programming
languages involved.

~~~
jude-
Yes--choose the language and development process that makes it easiest to
reason about the desired properties of the program.

Example: seL4 is written in C (as in, _written by hand by fallible humans_
\--not compiled down from a higher-level language [1]). But, the C is written
in such a way that it is feasible to prove its equivalence to a Haskell
prototype which had previously been proven correct.

Yes, language choice can be important. But, the degree of its importance
depends on what you're building with it and how you're using it.

[1] [http://www.nicta.com.au/pub-
download/full/7371](http://www.nicta.com.au/pub-download/full/7371)

------
GhotiFish
I'm getting very frustrated by this presentation, it's all "This is so
amazing, it satisfies so many criteria, there's a big problem with financial
institutions"

I can only read those lines so much before I get the feeling of being
whitewashed. How does it work? Where is the data?

I'm reading the white paper now, but I felt compelled to post this comment
after I read through yet another 10 paragraphs of exactly what I described
above.

Something that takes on distributed consensus is a fantastically interesting
project, this is so frustrating!!!

~~~
jude-
Did you read the whitepaper? It's much more technical. Link:
[https://www.stellar.org/papers/stellar-consensus-
protocol.pd...](https://www.stellar.org/papers/stellar-consensus-protocol.pdf)

------
kanzure
Commentary and review from IRC: [https://botbot.me/freenode/bitcoin-
wizards/msg/36135395/](https://botbot.me/freenode/bitcoin-
wizards/msg/36135395/)

------
lucian1900
Interesting that Graydon Hoare [1], Rust's (initial) creator is one of the
core developers.

1\. [https://github.com/graydon](https://github.com/graydon)

~~~
neilk
Graydon also created monotone, which was a big influence on git.

[http://www.monotone.ca/monotone.pdf](http://www.monotone.ca/monotone.pdf)

[https://en.wikipedia.org/wiki/Monotone_%28software%29#Monoto...](https://en.wikipedia.org/wiki/Monotone_%28software%29#Monotone_as_Git_inspiration)

~~~
LukeHoersten
I think Monotone was an alternative to BitKeeper and BitKeeper was the
inspiration of Git and Mercurial. It even seems to suggest that in the link
you posted.

~~~
pgeorgi
Linus played with Monotone before he started git (and it shows)

He is even credited in the changelog:
[http://lwn.net/Articles/131744/](http://lwn.net/Articles/131744/)

------
nullc
"It is the responsibility of each node v to ensure Q(v) does not violate
quorum intersection".

::Sigh:: This sounds like it does not even speak to one of the major
fundamental issues of their approach; which I pointed out in 2013
([https://bitcointalk.org/index.php?topic=144471.msg1548672#ms...](https://bitcointalk.org/index.php?topic=144471.msg1548672#msg1548672))
and appeared to play a critical role in Stellar's spontaneously faulting, and
has been avoided in ripple by using effective centralized admission to the
consensus in the system to ensure a uniform topology.

The (generalized) ripple "as-advertised"* consensus model can only be safe if
the participants trust is sufficiently overlapping. In spite of requests by
myself and several others (E.g. Andrew Miller) Ripple never formalized the
topology requirement, much less how users are to go about achieving it. This
paper goes forward in formalizing it, but still provides no guidance on
achieving it; and absent that the only reliably way I know to achieve it is to
have a central authority dictate trust. (*Ripple, as-deployed, centrally
administers the "trust"; and Stellar faulted when it failed to do so and
switched to a fully centralized approach (at least temporarily))

Consider a trivial example of two fully meshed subgraphs of 100 nodes each
with an overlap of a single node. Assuming that each nodes behavior is
tolerant to at least one ill behaved node, then both of the subgroups can come
to a consensus (achieving at least 99 out of 100) about mutually exclusive
state, and this can happen spontaneously without any attacker. More
complicated partitioning-- ones involving many nodes in the min-cut, or more
than two partitions-- are possible, to avoid it there must be 'sufficient'
overlap.

Deciding on what the 'trust' topology must be to achieve safety requires non-
local (and presumably private) information about what everyone else in the
network trusts. The required minimum set of additional edges to make any
particular natural trust topology into a safe one may have no relationship to
whom anyone actually finds trustworthy in the real world. As far as I can tell
no mechanism is proposed to establish a safe topology; just "the
responsibility of each node".

To me that sounds a lot like saying "It is the responsibility of each node to
not connect to any faulty nodes." Its a very strong assumption.

Separately, this system proposes a kind of consensus which is weaker with
respect to blocking than e.g. Bitcoins. This is perhaps made most obvious by
the point at the end about being unable to use the consensus to safely
arbitrate global parameters (like system settings or version upgrades),
something we do regularly in Bitcoin. It isn't clear to me why the authors
believe that the system is fit for cryptocurrency use when it cannot guarantee
eventual agreement about _all_ of the state. In Bitcoin the transaction
'light-cone' from coins splitting and merging appears to grow exponentially
for most coins, so a failure to reach consensus on one transaction would
eventually block most transactions. It's not clear to me if all participants
could reliably detect stuck statuses and avoid dependance on them (if they
could, why cant consensus continue). I'll need to read more carefully to
understand this point.

~~~
mazieres
You are correct that safety requires overlapping quorums. However, the trust
decisions are public, as this is what allows participants to discover quorums.
The scenario you describe of two groups of 100 participants overlapping at one
node might or might not be a problem. The most likely cause of such a topology
is a Sybil attack, in which an attacker with one seat at the table gloms an
extra 99 nodes onto the system that nobody trusts. The attackers' 100 nodes
might of course diverge if they are so configured, but nobody will care.

A priori, we cannot definitively answer what kind of topology will emerge. But
there is certainly precedent for building a robust network out of pairwise
relationships, namely inter-domain routing on the Internet.

~~~
jude-
Hello Dr. Mazieres, thank you for making the Stellar whitepaper available!

While inter-domain route agreements show precedent for building a robust
network out of pairwise relationships, might this particular class of
agreement also work because it happens to be a "small world" where failures
are obvious? That is, the set of major AS operators is small enough that all
the major players know one another (since ICANN) maintains a centralized list
of who owns which AS number), and bogus route announcements are easy for an AS
operator to detect since they coincides with floods of angry tech support
calls asking why www.foo.com no longer loads (or loads www.bar.com instead).
By contrast, it seems that Stellar is geared towards environments with neither
of these properties--large worlds with hard-to-notice failure modes.

I ask because I'd love to hear your thoughts on how to select quorum slices
when considering the political and economic incentives that might influence
which node operators I choose to trust. Specifically, do you foresee the
emergence of a small set of big-player node operators that application
developers almost universally (and blindly) select for their programs' quorum
sets, like how web browsers and OEMs regard CA operators today? How can
Stellar help users do better than blindly trusting a small set of operators?
I'm assuming that the fact that big-player node operators must nevertheless
externalize the same slot values in order to enjoy liveliness makes it easy
for the application to automatically detect any equivocation? If so, how would
nodes be deployed to resist DDoS attacks that try to break the vast majority
of usres' quorum sets? I'm getting the impression that there's a missing
precondition here that for a large-scale Stellar deployment to be successful,
there must be a very diverse set of quorum sets.

Thanks again!

~~~
mazieres
So first, a lot of problems with Internet routing do not really affect SCP's
safety--for example bogus route announcements. Part of the reason is that SCP
depends on transitive reachability. And of course part of the reason is that
SCP is built on top of the Internet, so can already assume a basically running
network underneath.

I can't predict the future, but I do think it is likely that a bunch of de
facto important players will emerge and have fairly complete pairwise
dependencies. One reason is that Stellar follows a gateway model, where
counterparties issue credits. So for example, down the line people might want
to store their money as Chase or Citibank USD credits. So people will already
have some notion that some participants are trustworthy enough to hold their
USD deposits, and these institutions will emerge as important. If I'm a
Citibank customer and you send me money, I obviously won't consider the
payment complete until Citibank says it is. And of course Citibank is likely
to want to depend on a bunch of institutions they do business with, so even
just one bank should give me good transitive reachability.

But the nice thing about safety is that you can reduce any individual party's
trust by depending on more people. So for example Stellar will run a validator
node for the foreseeable future, and their incentives are different from
Citibank's. To gain more trust in the system, I might want to wait for both
Stellar and Citibank to agree before considering any payment fully settled.

~~~
jsprogrammer
Why would I ever want to depend on Citibank?

~~~
rfugger
You obviously wouldn't. But lots of people depend on them already because lots
of other people do and they don't want to spend any more time thinking about
it than they absolutely have to. If you can convince them to change, more
power to you.

------
davewasmer
I find this kind of stuff fascinating, but lack the CS and/or mathematics
background to understand the discussion beyond the basics. I think I grasp the
concepts outlined in the graphic novel linked elsewhere in these comments, but
the whitepaper is too deep for me.

Any pointers for someone looking to gain an amateur understanding of this, or
is this a topic of sufficient complexity that it precludes an amateur
understanding?

~~~
andrewstellar
Here's an overview that attempts to explain it in a less CS/math way and a
more general and approachable way: [https://medium.com/a-stellar-journey/on-
worldwide-consensus-...](https://medium.com/a-stellar-journey/on-worldwide-
consensus-359e9eb3e949)

------
Rhapso
I'm still reading the paper, but I'm not seeing any discussion of a sybil or
eclipse attack defense.

~~~
spolu
(disclaimer: I had early access to the white paper for review)

Sybil attacks are not really directly applicable here since each node in the
system picks its own quorum slices (basically the set of nodes that it
trusts). There is no notion of global reputation and nodes do not need to know
every other nodes to participate. Looking at the definition of quorum
intersection[0] section 4.1 should give you a sense of the conditions that are
required on the choice of quorum slices for the network to function properly
(quorum intersection ensures safety)

The proof exposed in the paper guarantees safety and liveness for the network
provided a certain number of reasonable conditions are held true. What that
means is that an attacker cannot force on intact nodes (definition p14)
invalid transactions nor prevent the network from making progress.

That being said, (at least in the version I reviewed) there is no guarantee
provided with respect to ensuring that all valid transaction will eventually
make it into the network. Indeed a set of highly trusted nodes (present in a
lot of quorum slices) could attempt to preempt a specific set of transactions
X (originated by edge nodes) by opportunistically broadcasting valid
transaction set V_i for each successive ledger entry i that explicitly do not
include the targeted set of transactions X. Under raw SCP as described in the
paper and for certain topologies this preemption could be real and this is the
closest I can think of a Sybil attack. It's important to note that we still
have liveness and safety in that case.

I believe the same kind of attacks to be plausible with the Bitcoin network
and I know protection mechanisms against it are currently being evaluated by
David, Jed and the rest of the team. I will let them share their progress when
they think it's right. I also hope they will correct me if I stated anything
inaccurate here!

[0] [https://www.stellar.org/papers/stellar-consensus-
protocol.pd...](https://www.stellar.org/papers/stellar-consensus-protocol.pdf)

~~~
nullc
> I believe the same kind of attacks to be plausible with the Bitcoin network

This isn't anyone elses understanding. Can you suggest a mechanism by which it
would be possible for a minority conspiracy to perpetually exclude a
transaction in Bitcoin?

~~~
spolu
Well in bitcoin, of course, trust would map to computing power.

~~~
nullc
In Bitcoin we can make a pretty concrete statement about computing power that
one can reason about; the blocking attacker will not be successful without a
majority of it.

Whats the similar statement for 'trust' which is sufficient for security?
Obviously "attacker is partitioned from the network" is sufficient, but not
very plausable. I'm sure there is a better statement possible, but its not
clear to me what it is.

~~~
andrewstellar
Why wouldn't this be plausible? Let's say one day China had enough of Bitcoin,
and used their essentially limitless resources to gain enough hashing power at
will, to block transactions or rewrite them or what have you. Entirely
plausible with Bitcoin (in this case, China doesn't care about the coin reward
and therefore is not a "rational attacker" as the popular game theoretic model
of Bitcoin security presupposes).

Now let's look at the Stellar model in this same situation. We've got a bunch
of large company nodes that are probably Gateways (for the sake of argument
say JCB, Wells Fargo, Barclays, and Bank of Brazil). We've got a ton of other
nodes that belong to research universities, and then we have a bunch of "non-
profit" or hobbyist or whistle blower nodes. There's a nice graph topology
between all of these. Then one day China comes along and decides its had
enough. How does it attack the network in this case? By hacking enough
organizations to take control of their nodes? Seems a bit more unlikely than
it gaining 51% of hashing power on the Bitcoin network...

~~~
vbuterin
> Let's say one day China had enough of Bitcoin, and used their essentially
> limitless resources to gain enough hashing power at will, to block
> transactions or rewrite them or what have you. Entirely plausible with
> Bitcoin

That's the Maginot Line attack, at Tim Swanson calls it. The more realistic
attack is that China just hacks into five data centers and serves a warrant to
another ten. An interesting property of the PoW incentive structure is that
there is actually fairly little incentive to protect oneself against hacks, so
I would not be surprised if it was fairly easy.

> By hacking enough organizations to take control of their nodes?

The key point in Stellar consensus is that even if enough nodes are hacked,
then users can just stop trusting them and switch to other nodes, and so the
network would "route around" the damage. With Bitcoin PoW, there's no way to
exclude an attacker from participating; you have to accept their work just as
much as everyone else's.

~~~
andrewstellar
Maginot Line attack, I like that. And yep, that's basically the point I've
been trying to make in my posts. IMO Bitcoin isn't "trustless" \- you need to
implicitly trust those with hashing power aren't colluding to screw you.

------
yodsanklai
Are they talking about computer verified proofs? I wonder, are researchers
able to prove the correctness of distributed algorithms the same way they
would prove sequential algorithms (for instance, using some type of Hoare
logic and sat solver/ proof assistant).

~~~
djb_hackernews
You may be interested in this[0] paper outlining AWS use of TLA+ to formally
prove its systems. Also previous discussion here[1].

[0] [http://research.microsoft.com/en-
us/um/people/lamport/tla/fo...](http://research.microsoft.com/en-
us/um/people/lamport/tla/formal-methods-amazon.pdf)

[1]
[https://news.ycombinator.com/item?id=8096185](https://news.ycombinator.com/item?id=8096185)

------
olh
Is this protocol isomorphic to bitshare's Delegated Proof of Stake (DPOS) [1]?
Seems to have the same qualities.

*[1] [https://bitshares.org/delegates](https://bitshares.org/delegates)

~~~
mazieres
Bitshares is more democratic than decentralized. Basically people vote their
stakes to elect 100 nodes that ensure consensus, but everyone knows who the
100 nodes are. By contrast, the FBA trust model is completely independent of
coin holdings, and just depends on pairwise relationships between the
validator nodes. The resulting quorum structure is unlikely to look like a
single fully-connected group.

------
foobarqux
Where should we first see Stellar deployed in a major way?

~~~
tlrobinson
Stripe?

([https://stripe.com/blog/stellar](https://stripe.com/blog/stellar))

~~~
foobarqux
What significant application of Stellar do they have?

------
oskarer
It looks very promising, but I was unable to find the answer to this simple
question: How do I get my money from my bank account into the Stellar network?

~~~
themusicgod1
> How do I get my money from my bank account into the Stellar network?

In addition to using gateways, as diyang suggests, there is another way, but
you first need two things

1) Someone on the network who you trust. This may be your bank, but it could
also be something else. I'm not going to tell you who or what you should
trust, and to what extent, that is a decision that should be always in your
hands

2) A path between the entity you trust and an entity that either you or your
bank has access to.

Until #2 exists, you can do what the poster below just said: use a gateway.
This is not recommended, as they are probably going to track you and it may
not be always possible to deal with them in a humane way.

So really it depends on what your bank is, whether it makes sense to draw
money from your bank into another form/service that is compatible with a
service that stellar can talk to. What bank? What country? These things are
going to matter on the global scale.

------
zenincognito
Click View Source on Homepage. Easter egg ?

------
lectrick
How do they do proof-of-work?

~~~
wmf
The whole point of Ripple/Stellar is that it doesn't use proof of work. They
have an alternative that trades off trust for resource consumption.

