
Bitcoin and the Byzantine Generals Problem - rmason
http://nonchalantrepreneur.com/post/70130104170/bitcoin-and-the-byzantine-generals-problem
======
eof
To the uninitiated, bitcoin solves the byzantine general's problem by having
each "general" work on a mathematical problem that is known to take a certain
average amount of time; and, when they solve the problem pass their solution
onto the other generals who will then incorporate the answer to the previous
problem into a new problem.

The "consensus" is intrinsically linked to the "math problem" so that the
generals will always "trust" the chain-of-answers which is the longest; as it
would be impractical / impossible for an attacker to counterfeit the long-
chain-of-answers.

Bitcoin uses sha256(sha256( x )) < `target` as its "math problem" where X
contains the a hash of the previous "consensus" and new transactions which
should become part of the new "consensus". `target` is adjusted over time

~~~
tomp
Bitcoin doesn't really solve the Byzantine General's Problem, as demonstrated
by the March 2012 network split [1], which was later resolved by consensus
between users (lead developers and biggest miners) using communication
channels outside of the Bitcoin network.

[1]:
[http://en.wikipedia.org/wiki/History_of_Bitcoin#The_fork_of_...](http://en.wikipedia.org/wiki/History_of_Bitcoin#The_fork_of_March_2013)

~~~
DennisP
A cursory look at your link reveals that it was caused by an incompatibility
between different versions of the bitcoin client; the older client wouldn't
accept blocks as large as the later one. People downgraded to the older
version, and then "The network reached consensus and continued to operate as
normal a few hours after the split."

So I guess you could say it only solves the problem when everybody's software
implements the same protocol.

~~~
goldenkey
So we could say, the newest version of the bitcoin software correctly solves
the Byzantine General problem, though older versions faltered.

And this could be the case ad infinitum if the protocol needs to be changed
due to newly discovered flaws.

In summary, unless there are no bugs in your p2p software, it's very hard to
solve the Byzantine General problem without outside consensus.

~~~
chrisseaton
I still don't understand how it solves it. I get that you can agree a time,
but how does it solve the problem that each general needs to be certain that
all the other generals are also certain? I may have seen a completed block
chain for the attack and time, but how do I know all the other generals have
also seen it? I know they signed part of it, but how do I know the completed
chain then got back to them? If it didn't they're not going to attack and my
army will be destroyed.

~~~
SkyMarshal
The time delay introduced by the proof of work is the key to all your
questions.

Imagine 10 generals in the network all trying to agree on a time of attack.
Each simultaneously sends 9 couriers to the other generals suggesting an
attack time. That's 90 couriers with 10 different messages all pinging around
the system simultaneously. Each general receives 9 messages at roughly the
same time, and either has to choose one to sign and rebroadcast, or cheat and
sign multiple and rebroadcast them. There are too many options available at
any given time, and odds of reaching consensus aren't good.

However, by introducing a time delay you slow the rate of message passing
enough to be manageable - now's there's only 9 couriers and 1 message pinging
around the system at the same time. All nodes in the network work the same
problem, but only one will find a solution first and broadcast it, and with
the time delay there's enough time for that solution to disseminate through
most of the network before another node discovers another solution and
broadcasts it. By the time the second and subsequent solutions are found, the
the first has disseminated to enough nodes that they've already incorporated
it into the next proof-of-work, and reject subsequent solutions to the prior
PoW.

~~~
chrisseaton
I still don't see the solution - doesn't any solution have to solve the
problem of messages being lost?

Say that after some time a block that is long enough is made by some general A
and is distributed to generals B and C. General D doesn't get it though - his
messenger is killed in transit. He helped make an earlier block, but has never
seen the fully completed block.

How do the other generals know that he hasn't got the final block that is long
enough?

Or is it just the case that a majority of generals know when to attack? I
thought it had to be all of them, but maybe that's the two-general problem and
Byzantine-general's is an easier problem. I was pretty sure there were
multiple good proofs of the impossibility of a solution.

~~~
tomp
The basic idea is (based on Satoshi's explanation linked from a nearby
comment) that each general _i_ proposes some time _T_i_ , where _T_i - now > 2
hours_ and starts solving the problem using the value _T_i_. If it finds a
block, it broadcasts it. After some time (e.g. after 2 hours if each block is
expected to be mined in 10 minutes) there is an overwhelming probability that
all generals have synchronized and are working on the problem with the same
initial value _T_n_ for some _n_ (they synchronize by always working on the
longest chain).

The key point is that after 2 hours, all of the generals can _independently_
assess, by examining the previously mined blocks in the chain they are working
on, how much CPU was spent working on the solution, and can "see" how many
nodes are in the network, and hence can see if all the nodes have worked on
this solution (if yes, they all know of the arranged time of attack).

This doesn't fully solve the problem (one general could be rouge, or one might
be killed just before the attack, ...) but it at least raises the chances :)

~~~
chrisseaton
Well if it doesn't fully solve the problem, does it solve it at all?

That's the thing about BG; you can keep increasing the chances, but if your
requirement is that you must be certain that all the other generals will
attack, and at the same time, then we know of no solution. We also have more
than one good proof that this is impossible.

~~~
tomp
Right, but the assumptions of the BG problem are also a bit harsh for the real
world; network exhibits latency and/or splits, which are followed by joins. If
two servers cannot communicate (ever again), then you have bigger problems
than simply "synchronizing attacks". For the usual problems of varying latency
and some dropped messages, the proof-of-work, coupled with a cryptographic
authentication, will suffice for eventual consistency.

------
throwawayforhn
I would like to warn those who don't know that the author of this post has
vested interest in bitcoin. It doesn't mean anything per se, but you might
want to take those articles with a grain of salt.

So this is how it goes. a16z invests in Coinbase, so cdixon posts supportive
posts regarding bitcoin.

We can now safely expect more and more HN readers to buy bitcoins because of
the fear of missing the bitcoin train, and bitcoin detractors will soon look
like iPhone detractors in 2007. That means that no matter what the value of
bitcoin is, you should buy some, because the whole SV is soon going to be on
it.

For the fist time on HN, a significant part of what hits the front page are
posts about an asset that you can buy simply, and will likely make you a
millionaire in a couple of years without creating any value. This is as great
as it is sad. Enjoy it.

Make us long time holders rich.

~~~
cdixon
Yes, our investment was very publicly disclosed on my blog, Coinbase's blog,
and multiple news sites. Also, I was posting positive things on my blog about
Bitcoin long before I (or a16z) had a financial interest.

I am interested in Bitcoin as a new payment system / economic protocol for the
Internet. I don't think people should buy Bitcoins for speculative purposes.
On the other hand I'd love to see more developers build things on top of the
Bitcoin protocol. My job is to invest in new technologies and I believe
Bitcoin is one of the most interesting new technologies in the past 20 years.

~~~
greyman
> I don't think people should buy Bitcoins for speculative purposes.

Why not, if you suppose that its price will go up?

------
Rhapso
We have noticed, and It lets you do a lot a previously impossible things in
decentralized computing. Give folks a few more months to polish proof of
concepts.

~~~
cdixon
We (a16z) would love to see more innovation along these lines and would be
excited to try to help.

~~~
patrickk
Some radical ideas: [http://startupboy.com/2013/11/07/bitcoin-the-internet-of-
mon...](http://startupboy.com/2013/11/07/bitcoin-the-internet-of-money/)

------
gnaritas
Bitcoin mining is wasteful compared to some of the alternatives, solving
useful hard problems as proof of work (Primecoin) or using a proof of stake
system to remove the need for relying so heavily on energy wasting mining
(Peercoin).

~~~
lukifer
From what I understand as a layman, it is overly generous to call the prime
number sequences found by PrimeCoin "useful".

I do think that a hybrid proof-of-stake system is probably superior and more
efficient. But bear in mind that capitalism involves lots of waste, from high-
frequency trading, to plastic doodads that wind up in the trash, to the vast
majority of advertising and marketing.

I'm absolutely in favor of a Star Trek / Buckminster Fuller neo-communism, if
it can be achieved. In the meantime, we have to fail forward and innovate as
best we can, and I think proof-of-work cryptocoins are far less wasteful in
toto than the current global banking system. (What do you think it took in
terms of energy and waste products to manufacture and distribute the dollars
in your wallet?)

Note also that intentional wastefulness shows up frequently in nature: costly
signals are honest signals.
[http://en.wikipedia.org/wiki/Signalling_theory#Costly_signal...](http://en.wikipedia.org/wiki/Signalling_theory#Costly_signalling_and_Fisherian_diploid_dynamics)

~~~
gnaritas
> I think proof-of-work cryptocoins are far less wasteful in toto than the
> current global banking system.

Maybe, maybe not, I don't think anyone's crunched enough numbers to make a
reasonable measure of that yet.

> (What do you think it took in terms of energy and waste products to
> manufacture and distribute the dollars in your wallet?)

I doubt either of us know.

I'm a Bitcoin supporter, but one has to think there are useful hard problems
to solve rather than just banging out SHA-256 hashes over and over. If that's
true at all, then Bitcoin is wasteful.

~~~
lukifer
> one has to think there are useful hard problems to solve rather than just
> banging out SHA-256 hashes over and over.

I sincerely hope that a truly useful proof-of-work can be discovered. It would
be a very big deal.

The problem is, the work has to be reliably verifiable, and easier to verify
than just doing the work over again, which is why hashes are a good fit.

~~~
reginaldjcooper
Unless I am mistaken, you can verify arbitrary computations[0].

[0] [http://arxiv.org/abs/1105.2003](http://arxiv.org/abs/1105.2003)

~~~
lukifer
The abstract says that "a practical general-purpose protocol for verifiable
computation may be significantly closer to reality than previously realized".
This should be interesting though, thanks for sharing!

~~~
reginaldjcooper
You have a fair point, but I claim for a proof-of-work computation it might be
good for it to be a bit unpractical. (Depending on the parameters, this paper
is on my reading list but I haven't gone through it yet.) I think it's
incredibly interesting also, in any case I hope you enjoy the paper :)

------
reillyse
"Before the Bitcoin protocol was invented, most computer scientists thought a
system like Bitcoin was impossible because of a famous problem in computer
science called the Byzantine Generals Problem." ... wait what?

~~~
andybak
Maybe you should explain specifically what you find difficult to understand in
that quoted sentence?

~~~
reillyse
To elaborate a little more, because there seems to be tons of discussion about
this when there really shouldn't be.

Here is a probabilistic solution to the problem.

1) Each General sends n messages to all the other generals. 2) After a time
period y all the generals count their messages and decide how to act.

3) This is not a "solution" to the original problem. If we lose sufficient
messages we may "attack" at the wrong time. However it is a probabilistic
solution because as n & y increase we are less likely to make the wrong
decision.

There are lots of known probabilistic solutions and while I'm a fan of the
Bitcoin Protocol the existence of another doesn't change anything for computer
scientists.

------
001sky
This section on the "costs" of mining bitcoin is well raised and relevant>Here
is the passage:

 _< One thing I haven’t seen emphasized, however, is the extent to which the
whole concept of having to “mine” Bitcoins by expending real resources amounts
to a drastic retrogression — a retrogression that Adam Smith would have
scorned.>_

This he calls out as completely misplaced:

 _How much does the existing banking /payment infrastructure cost? One
reasonable measure are the fees charged. Standard online payment fees are
2.5%, not including the added costs fraud (chargebacks plus transactions
blocked out of fear of fraud)._

And he's right. but The real cost of running a market is not, however a bid-
ask spread. And he gets at the point, but its not clear, here:

 _Bitcoin payment fees are close to zero and fraud is impossible since Bitcoin
is a bearer instrument._

The [true costs] of running a market are thos that instill [trust] in the
market system. That is, what is commonly called "transaction costs" in
economics. But these are not literal costs, which tend to be rent-extraction
wherein the transaction is merely instrumental to effect a scaling biz
model.[1] The true transaction cost of "effective honesty" are to be found in
"governance costs", that is...the cost of lawyers. And thus more generally,
and indirectly, the primary purpose of government (eg schooling, police,
courts, national defense). So, it is worth putting in context the "cost" of
mining bitcoins here. The "innovation" that is provided is provided also at
this seperate level of abstraction, far away from the "overhead" style
transaction costs in a literal definition. And to the authors point, these are
both measurable and _large_ ; such an innovation thus actualy saves wated
resources that would otherwise be deployed (think of all the energy spent on
anti-spam and anti fraud by CCs...that 2.X is ~mostly profits tho).

In any event, interesting topic and interesting post. And I think he intuits
the right answer, but the exact words put forth sort of murky the point abit,
IMHO.

[1] eg 7% of an IPO to a Bank, X% to your real estate broker, 1/8 of a point
in a pre-decimalized stock market, 2.x% on a credit or paypal transaction.).

~~~
jackgavigan
Bitcoin payment fees may be close to zero at the moment but that's only
because miners get paid with newly-minted bitcoins.

As I write this, there are 12,130,075 bitcoins in existence[1]. Over the
coming year, approximately 1,314,000 bitcoins will be "minted" (25 new
bitcoins every ten minutes). If Bitcoin were a real currency, that would
equate to an inflation rate in excess of 10%. So, in effect, every bitcoin
owner would be paying 10% of their Bitcoin wealth for "free" payments, whether
or not they actually make/receive any payments or not.

[1]: [http://blockexplorer.com/q/totalbc](http://blockexplorer.com/q/totalbc)

~~~
Gurkenglas
1\. Those 10% are per year, not per transaction. 2\. This kind of inflation is
one of many factors that is already incorporated into the market value of
Bitcoins.

------
synchronise
These are exactly the sorts of reasons why the more energy efficient Proof of
Stake (PoS) was envisioned and implemented into several cryptocurrencies, like
Peercoin and Novacoin.

------
buluzhai
here is a post talk about it
[http://expectedpayoff.com/blog/2013/03/22/bitcoin-and-the-
by...](http://expectedpayoff.com/blog/2013/03/22/bitcoin-and-the-byzantine-
generals-problem/)

------
Buge
You can't measure the bitcoin infrastructure cost by the fees the miners
charge. Miners are mostly paid with newly generated coins. Credit card
companies do no have the luxury to create new money, so of course they will
charge higher fees.

~~~
cdixon
I agree it's imprecise. But I'd argue the de facto fees in the existing system
are much higher than 2.5%. The biggest cost are all the transactions that
should happen but don't. Most merchants/payment companies block payments from
dozens if not hundreds of countries due to fears of fraud.

~~~
comex
That's only indirectly related to Bitcoin's use of proof of work, though.
While Bitcoin's overall design has advantages and disadvantages w.r.t.
security, I don't know why an irreversible version of the current system
wouldn't similarly be able to reduce fees without needing to "waste" CPU power
computing hashes.

------
akandiah
Is there a good explanation on how bitcoin addresses the problem out there? I
haven't come across a simple explanation that validates the approach (in my
mind at least).

~~~
meowface
This doesn't answer your question specifically, but:

For the Bitcoin network to properly operate, it requires at least 51% of the
computing power of the network to be "good", well-behaving nodes. That 51%
means 51% in terms of block-mining. So, computing power in this case means
"ability to find plaintext that results in a certain double-SHA-256 digest".

Therefore, Bitcoin has not come across a completely rock-solid solution (as
with many cryptographic protocols), because a determined attacker or group of
attackers could theoretically achieve control of the network with sufficient
computing power. At this stage though, it's difficult enough to basically be
considered infeasible, unless a global superpower tried to tackle it.

See more here: [http://bitcoin.stackexchange.com/questions/658/what-can-
an-a...](http://bitcoin.stackexchange.com/questions/658/what-can-an-attacker-
with-51-of-hash-power-do)

~~~
kolev
I also wonder what impact Quantum Computing would have on Bitcoin. I'm pretty
sure NSA uses it already.

~~~
habitue
There are algorithms out there that are resistant to QC attacks:
[http://en.wikipedia.org/wiki/Quantum_digital_signature](http://en.wikipedia.org/wiki/Quantum_digital_signature)

~~~
kolev
Yup - algorithms not used by Bitcoin. :)

~~~
warfangle
Yet they could be, if a consensus was reached among miners to implement the
new algorithm :)

------
swswsw
At first glance, mining appears to use a lot of energy. But it has a very
important property:

    
    
      mining is "fair" in money generation.  
    

And being fair may be much more important than the energy consumed.

~~~
wmf
Depends how you define fair, I guess. Many people complain about the effects
of a fixed block reward with ever-increasing difficulty.

