
Not a protocol fault: MtGox and transaction malleability - oleganza
http://blog.oleganza.com/post/76213549017/mtgox-and-malleable-transactions
======
patio11
"It's a protocol bug!" "It's not a protocol bug!" is sort of a moot point.

Bitcoin is a protocol in the sense that the IE6 rendering engine is a
protocol: yes, rigorously examining what the client monoculture does and does
not emit does allow you to describe some set-of-rules. Trying to create
another Bitcoin client which attempts to agree with that set of rules is,
ahem, _fraught_. You have to be bug-for-bug compatible with the Satoshi client
implementation, not with the Satoshi client's declared behavior, design
intent, or whatever constellation of PDFs and Wiki posts that the Bitcoin
community anoints as "the protocol."

~~~
oleganza
Satoshi never had any "whitepaper spec" which proclaims "intended" behaviour.
The protocol is whatever 100% of users agree with. It started with one
implementation, today we have many of them. And we already had the protocol
changed in a hard way ("hardfork") at least 4 times:

1\. In 2009 when they fixed OP_RETURN and OP_CODESEPARATOR bug allowing to
spend anyone's coins.

2\. In 2010 when integer overflow created billions of BTC in one tx.

3\. In 2012 when "OP_HASH160 <hash> OP_EQUAL" script was redefined to be
interpreted as a hash of a script (see BIP16, Pay-to-Script-Hash aka "P2SH").

4\. In 2013 when v0.8/v0.7 difference in database engines caused half of the
network mine parallel blockchain due to obscure limit on number of file
handles. The fix which was gradually introduced after reverting "incompatible"
(although, longer) chain, was changing the protocol in a hard way.

The takeway from this: there's no parallel universe where a clearly defined
spec in English is how all software operates to produce _exactly_ the same
blockchain. The most precise description would be a computer code itself
running on the same hardware everyone uses. Anyone coming up with a "spec" of
Bitcoin will only have a "description", not a "prescription".

~~~
patio11
_In 2013 when v0.8 /v0.7 difference in database engines caused half of the
network mine parallel blockchain due to obscure limit on number of file
handles._

Can I just highlight, for generic web developers who might not parse this
sentence, that this is like saying "One of the ways we make sure all clients
agree on HTTP is by making sure that all clients and servers stay on the same
point release of Oracle DB because when they didn't for a few minutes the
entire Internet died in fire. We put the fire out by reverting the Oracle DB
version on a few big sites, telling everyone else to downgrade as well, and
posting on an out of band channel that if you were doing anything important
with the Internet you might want to stop for a few hours because there were
going to be two separate and very mutually incompatible Internets and one of
them was going to suddenly vanish a couple hours later and it sure would suck
if your mail/web browsing/Dropbox/etc went into that one."

Seriously. That actually happened.

~~~
weavejester
That's not really what happened. The majority of Bitcoin clients need to agree
whether or not to accept a block. In 0.7, a limitation of the database library
was causing a very large block to be rejected, whereas in 0.8 the block was
correctly accepted. This caused a blockchain split, with the 0.7 clients going
one way, and the 0.8 clients going the other.

This sort of problem doesn't really occur on the web, so trying to draw an
analogy to HTTP isn't going to help people understand the issue.

~~~
patio11
The point of the post is to make web developers say "But wait, Patrick, if
HTTP actually worked like that that would be _batshit insane_. I mean,
separation of concerns. Having to coordinate patch schedules worldwide across
billions of clients. Being to-death-do-us-part committed to every library
choice made by the first HTTP client. Wait, there could never be a second HTTP
client, because you'd never be sure it worked warts-and-all like the first
one. You'd probably have to do something like extract the One True HTTP Client
into a wrapper program and, I don't know, call out to it from your web
browser, so that changes in web browser chrome weren't tightly coupled to the
One True HTTP client. Developers on the One True HTTP Client would probably
refuse to help people who made competing HTTP clients, because they'd be
building _potential existential threats to the Internet_ , and they'd probably
gloat when those other developers suffered misfortune.

Please tell me, Patrick, that you're exaggerating and that Bitcoin doesn't
work like this."

Guess what?

~~~
weavejester
HTTP isn't trying to maintain a consistent, decentralised database. Bitcoin
is. HTTP doesn't have a problem maintaining data consistency because that's
outside the scope of the protocol.

You're suggesting the Bitcoin protocol is badly designed because it's had
problems that HTTP hasn't had. But this is like saying aeroplanes are poorly
designed compared to cars because the latter don't fall out of the sky when
their engines die. Planes have to deal with problems cars don't, not because
of poor design, but because of the nature of what they're made to do.

Bitcoin concerns itself with providing data integrity across a distributed
system. The only way of doing this, without a centralised server, is to ensure
that all the clients are performing the same calculations. If it turns out a
significant proportion of clients are performing the _wrong_ calculation, then
you have a problem. This isn't a problem inherent to Bitcoin; it's a problem
with _any_ distributed system trying to maintain a consistent database.

~~~
drewblaisdell
Patrick is a very smart guy, which is part of the reason I (like you) am so
confused as to why he thinks HTTP makes for a good analogy.

~~~
yeahbutbut
I'm pretty sure that by HTTP client he meant rendering engines. And in this
analogy the data integrity is "whatever crap I have to send to the client to
get the expected output on all client machines". So it's somewhat fair to say
that if you have wildly divergent interpretations of HTML rendering then you
have a data integrity problem, since you can't guarantee that the same data
will be interpreted the same way by all clients. None of that has to do with
HTTP the protocol though... so I'm confused as to why HTTP the protocol was
mentioned instead of HTML and rendering engines.

------
wfunction
Utter nonsense:

> If you need a quick answer: there’s no bug in the Bitcoin itself.

> Is it a design issue in Bitcoin to allow slight changes in the transactions?
> Yes, probably is.

~~~
kubiiii
A design issue is not a bug is it? This design issue seems to have lead to the
bug in Mt Gox implementation though.

~~~
wfunction
That's like saying "it's only illegal if you get caught".

------
DennisP
Interesting comment from the lead dev of Ethereum (a new cryptocurrency):

> aside from cryptocurrencies, there really is no other situation where the
> fact that you can take a valid signature and turn it into another valid
> signature with a different hash is a significant problem, and yet here it’s
> fatal.

[http://blog.ethereum.org/2014/02/09/why-not-just-use-x-an-
in...](http://blog.ethereum.org/2014/02/09/why-not-just-use-x-an-instructive-
example-from-bitcoin/)

------
donpdonp
"Or, MtGox themselves see that they’ve been watching for transaction for too
long and could automatically re-send another transaction"

There is no amount of time to wait until a bitcoin transaction is considered
dead. Once its been communicated over the network, one has to assume it can be
redeemed (incorporated into the blockchain) anywhere from never to a thousand
years from now. Checking for txid even without malleable transactions is not
helpful.

If a bitcoin payment has been issued, and a customer wishes it to be re-
issued, the (only?) safe approach is to move the entire address contents to a
new address, wait for that to hit the blockchain, and reissue the payment from
the new address.

When different customer's balances are mixed in the same address, things get
more difficult, and seeing as how its customary to pull balances from multiple
addresses to satisfy the inputs a single transaction, I think mtgox has a
claim to complexity beyond their ability to fix without help from the protocol
itself.

As far as being affected by malleable transactions, an exchange should have a
value on their 'books' for each customer and a value in the 'bank'/wallet for
each customer. By keeping track of each set of addresses for each customer,
like bitcoind does in its wallet, the two can be audited with each new block
on the chain.

------
joosters
How on earth can this 'fix' scale?

Let's imagine a future world where all monetary transactions go through
Bitcoin. So the only way for me to find out if my transaction took place is to
search the list of _every_ worldly transaction to spot mine?

~~~
StavrosK
Don't you already have to do that? You have to download the entire blockchain
to see if a transaction happened. The fix is just to look for a different
attribute on the data you're already looking at.

~~~
jnbiche
Precisely. If you're offering any type of merchant services (and Gox does),
then you _already_ have to monitor all incoming transactions. All this means
is that you have to keep a small list of the addresses of people with pending
withdrawals, along with the corresponding Gox outputs, and then remove those
addresses from the list as each block rolls in (at however many confirmations
they determine they need).

------
Kenji
"MtGox remembers a hash of that transaction (unique fingerprint of its
contents)"

Technically, there might be collisions with hashes, they're in no way
unique... Though the probability is rather low.

Anyways, this is a nice explanation of the real issue.

~~~
oleganza
For all practical purposes it's unique. Otherwise, Bitcoin is broken and
useless.

------
n1c
> they instead should watch if the address X (specified by user) got amount N
> (specified by user) from outputs Y, Z and W (used by MtGox).

This doesn't seem ideal.

~~~
jnbiche
Why? In order to be running an exchange like Gox, you have to be monitoring
arbitrary addresses anyway.

It's a simple solution to a simple problem, and it's pretty telling that Gox
had this problem to begin with (but doesn't surprise me at all).

~~~
rout39574
Because if this mode of transfer ever becomes more than a toy, then there will
be massive overloading of those values.

Can you come up with circumstances in which someone might choose to make
successive transactions of the same value? Especially if that might confuse
the issue?

You want to be able to tell if _this_ transaction completed, not if, say "A
transaction that really looks like this one" completed.

~~~
SEMW
No. "successive transactions of the same value [between the same people]" will
not have the same transaction inputs. A 'transaction input' in bitcoin isn't
the originator's pubkey, it's the output of a previous transaction. A second
transaction which literally had the same inputs as a first would be a
doublespend attempt.

------
angryasian
I've been through the other thread of another article declaring mt.gox is
wrong, but I still don't see the solution.

What is the recommended solution by bitcoin implementers to verify a
transaction succeeded, with transaction malleability existing ?

~~~
greyman
I am not bitcoin implementer, but I believe you need to maintain the list of
pending withdrawal addresses, and monitor that the transactions with mtgox
outputs sent to those addresses received at least 6 confirmations.

~~~
angryasian
Where would those 6 confirmations come from ?

~~~
greyman
From the network. Or, at least as I understand it, Transaction Malleability
basically means, that the "final" transaction can have different txid, so you
shouldn't rely on it. But you can check if the transaction went through the
same way as the bitcoin-qt does it, which is what BTC developer Gregory
Maxwell recommends here:

[http://sourceforge.net/mailarchive/message.php?msg_id=319565...](http://sourceforge.net/mailarchive/message.php?msg_id=31956576)

------
ck2
On a side note, if you believe in BTC, I cannot remember the last time you
could buy it for less than $600 like today.

~~~
gst
You mean like 2 months ago?

~~~
ck2
Had to look at the exchange charts - I guess there was a two day dip around
December 18th but it quickly corrected.

~~~
ErsatzVerkehr
"corrected"

I love this teleological market-speak.

~~~
ck2
Ha, yeah I watch NBR before the PBS Newshour some days so unfortunately I've
picked up on some of the terminology. I jokingly refer to NBR as PBS's dark
little corner.

------
bjourne
That explanation is so simple it just can't be right. :) Usually if two
objects are equal their hashes should be identical too. That's the whole point
of hashing since otherwise O(1) lookups aren't possible. Apparently the
bitcoin protocol breaks that expectation since two semantically equal
transactions can have different hashes.

~~~
makomk
The main and most important use of transaction hashes in Bitcoin is to ensure
that, when blocks are mined, the transactions contained within those blocks
are locked in and any modification of those transactions can be detected and
rejected. Transaction malleability is meant to change the hash of the
transaction, because that ensures no-one can mutate transactions after they're
in a block. If you really need to track transactions in the way MtGox is
doing, you should construct your own transaction ID that hashes everything
_except_ the scriptSigs (which are malleable).

~~~
bjourne
> Transaction malleability is meant to change the hash of the transaction,
> because that ensures no-one can mutate transactions after they're in a
> block.

Huh?

> If you really need to track transactions in the way MtGox is doing, you
> should construct your own transaction ID that hashes everything except the
> scriptSigs (which are malleable).

Yeah. Except the Bitcoin network also apparently produces its own hashes of
transactions and those hashes include mutable data leading to undesirable
situations like hash(txn1) != hash(txn2) even though txn1 == txn2. It's like
hashing people by their name and age instead of name and date of birth. Next
year you won't find them in your table because their age is different.

------
baby
Can someone link to a better explanation? I'm not really satisfied by this
blog post.

~~~
wcoenen
This is a good start:
[https://en.bitcoin.it/wiki/Transaction_Malleability](https://en.bitcoin.it/wiki/Transaction_Malleability)

------
tempodox
What Mount?

~~~
donpdonp
Mount Gox is a malapropism of the acronym MTGOX (which is expanded in other
comments on this thread)

