Hacker News new | past | comments | ask | show | jobs | submit login

Ok, here's what happened and what it means. Bitcoin is built on a chain of blocks, each of which contains a set of transactions, the hash of the previous block, and a cryptographic proof-of-work which takes a lot of computing power to construct. Whichever block-chain is highest (that is, has the most total computing power invested in it) is considered valid, and has more blocks added to it by miners. People "mine" by taking outstanding transactions, writing them into a block together with the hash of the current highest-numbered block, performing cryptographic proof-of-work and publishing their new block.

A block was mined (by the Slush mining pool) which is accepted as valid by 0.8 clients, but rejected by 0.7 clients. More than half of the mining power was on 0.8, so the longest chain includes this block - but the 0.7 clients reject it, and have built a side-chain. The developers have asked all miners and mining pools to switch from 0.8 to 0.7, so that the 0.7 chain will grow to be longer than the 0.8 chain; once this happens, all Bitcoin clients will agree that it is the real one. In the meantime, however, merchants running Bitcoin 0.8 may be targeted by double-spend attacks: if they receive money in a transaction that exists only in the 0.8 blockchain, the same money is spent in a different way on the 0.7 blockchain so the transaction can't just be copied over, and they act on the receipt of money by sending goods before the situation is resolved, then they could lose money. This is why MtGox has suspended Bitcoin deposits. (This can only happen if the coins were sent by someone who is using malicious, nonstandard software; transactions made by honest users will be copied to the 0.7 fork and mined without any issue.)

The height of the two chains can be monitored at https://coinbase.com/network/blocks . Currently the 0.7 chain is 13 blocks behind, which (since the normal mining rate is 6 blocks per hour) sets a lower bound of about two hours on resolution - assuming everyone switches their mining power immediately - but realistically it will probably be more like 6-12 hours.

So at least (13 blocks x 25btc * $40/btc =) $13,000 in block rewards miners thought they had -- and possibly more -- will be reassigned into the favored chain's winners.

This is part of why it takes much longer for mining rewards to be confirmed than other payments.

Since most people mine as part of pools these days, it probably doesn't matter too much. The average rate for the pool won't be affected too much (though I suppose pools which have been on the 0.8 block chain will hurt a little bit). There may be a one or two individuals who have mined a block and lose out by this, but if so, they were rolling the dice anyhow by mining outside of a pool; you can't expect a consistent income by mining over a span of 13 blocks.

Most pools pay out as soon as the block is mined, or at most one or two blocks later, and pay for any blocks that are later reorganized out of the chain from their own pockets. I fully expect to see several pools fail and take some or all of their miner's funds with them as a result of this incident.

Most pools don't distribute funds to their users until 120 confirmations have been reached so none of the coin that they mined will show up in their users' accounts. The only way this could cause problems is if the pool's maintainer is somehow still mining on the 0.8 branch which would not happen as the 0.7 branch blockchain has surpassed it in length.

If you are solo-mining and you found a block on the 0.8 branch of the fork, the standard client prevents spending newly minted coin with less than 120 confirmations. It will be the same as what happens when you get any other orphaned block, the funds will just disappear.

bzzt wrong. Most pools payout only after 100-120 blocks after the block they have mined is built upon by other miners (they go via an "estimate" then "unconfirmed" stage first). Additionally, the biggest mining pools either 1) only pay out their slave-miners after these confirms or 2) factor this in to their PPLNS slaves as a mining pool cost. Either way, you can bet they have enough in reserves to cover a small number of orphaned blocks. There will be no pools that fail after this event and I am prepared to bet on this.

"The developers have asked all miners and mining pools to switch..."

In the grand scheme of things, how much control over bitcoin do the developers have? My understanding of bitcoin is that there is no central authority controlling it like a central bank, but this statement almost implies that the developers have some kind of control, or influence at least, over bitcoin.

They have no actual control over what the miners do, but they have a lot of respect in the community, and most miners are likely to listen to their advice.

It's in the miners best interest to work for the bitcoin economy as a whole since their coins value depend on a coherent response to these types of black swan events.

Its in the banker's best interest to work for the US Economy as a whole, since their money and investments depend on the valid repayment of subprime mortgages.

I don't want to start an argument here, but that's not true; the banks' (i.e., shareholders') money depended on the US Economy. The bankers' money (salaries and bonuses) got paid even after the crisis.

The bankers were playing with other people's money, not their own.

The finance industry went through a huge contraction during the crisis. It was not smooth sailing on some sweet government money.

Sure, they fired a lot of employees, but I wasn't exactly talking about the branch managers or salaried analysts when I said bankers. The Lehman Brothers executives still paid themselves $100M three days before the bankrupcy, and the others weren't exactly on ramen noodles either.

Nevertheless we the people paid for the risk with none of the upside.

Perhaps. But they would have made even more money if the financial crisis never happened. We've got a case where executives make bad decisions for short-term personal gain.

This assumption that people always work for the long-term gain, even for themselves, was proven wrong in the financial crisis. The economy is made up of humans, and humans make mistakes.

Later, you bring up the point of $100 million paid to the Lehman Brothers executives. Do you know how much they got every year ? These guys are paid $50 million / year or $30 million / year PLUS stock options... if they wanted to make more money, they should have kept Lehman Brothers operating for just 3 more years.

The rules governing bitcoin are maintained by consensus, and the bitcoin developers have a large sway on this consensus.

If the majority of people decided they didnt like the current rules (eg all the people complaining about bitcoin deflation) they could very well change them by simple agreement.

It's not really consensus, it's rule by whichever coalition can get 51% of miners.

Not really. For example if 51% (or even 99.99999999999%) of miners decided they wanted to go back to 50 BTC block reward (or even a new 5,000 BTC block reward) that would simply create an incompatible fork. A fork which would most likely be promptly rejected by the consensus of users, merchants, exchanges, and service providers.

One can't simply force a change to the rules. All you can do is make a hard fork in the network. If nobody adopts your hard fork well you can keep mining worthless coins but people can continue to use/mine the existing fork.

Bitcoin is highly resistant to change (almost to a fault). It is a common myth that the person who controls the majority of hashing power can change the rules.

A (malicious) person who controls a majority of the hashing power can change the rules. Central to bitcoin is the idea that the longest chain is the valid one. If you control 51% of the network, you can create a chain that looks however you want it to look, and then publish that chain. Because your chain is longer than the correct one, it will be accepted, and your changes will be accepted. Using this, you can modify the chain to be what it would look like according to your rules.

Of course, this is only possible becuase your chain looks as if it was following the rules. If you have a non malicious computational majority, than any change in the rules would make an incompatible block-chain and a hardfork.

So how hard would it be to get 51% control of the network?

The software is the central authority. As far as I have been able to tell there is no decent specification, like an RFC, that you could use to build other client implementations from so you are all at the mercy of the bugs in a single piece of software.

"The software is the central authority."

The collective "authority" decides how the bugs are to be addressed. Bitcoin is not fully autonomous and without social guidance.

The developers have influence in that they are relied upon to a massive extent to keep bitcoin running.

As far as I can tell this sort of centralisation of influence is a common emergent property in most anarchic human systems.

Luckily if something particularly egregious occurs in this case, the community can easily fork the codebase and start a rival blockchain.

"Homesteading the Noosphere" by Eric Raymond covers exactly this topic. He compares the open source software ownership to Lockean land titles. With Bitcoin, it's interesting since the project itself is a system of value and transactions.


Would they need to start a rival block chain? Or do they just need 51% of the current community to run their clients to take over the current block chain? (as suggested by a comment above)

Technically speaking, getting 51% of the community to switch is still creating a rival block chain. You don't even need 51%. As was the case in this incident, if you have incompatible nodes, the block chain would fork itself naturally.

Do we know why the block was only accepted by 0.8 clients? Were 0.8 clients wrong in accepting it, or were 0.7 clients wrong in rejecting it?

0.8 clients were wrong to accept it, because doing so caused a failure of consensus.

Unfortunately the limit that was hit (which isn't quite as simple as 'size') was implicit in BDB and how Bitcoin prior to 0.8 used BDB. So no one was aware of it.

Extensive testing during 0.8 turned up several consensus breaking issues that were created during the database switch out, and very large blocks were tested... but the particular kind of very large block required to reveal this difference in behavior was not.

0.8 and 0.7 have different databases (LevelDB and BDB respectively). The 0.7 database doesn't accept some high-volume blocks under certain conditions (not 100% sure what those conditions are, that's for the devs to fix now). As a result, the chains forked when an 0.8 node mined a block 0.7 could not handle.

Everyone followed proper behaviors, it's just a compatibility problem. So miners should resort to 0.7 until 0.8 can prevent a fork like this from happening again.

The block exceeded a size limit in the Berkeley DB storage that the client uses to store the blockchain. By the spec, 0.7 is wrong to reject it.


They do not need everyone to stop using 0.8, just a majority (so that the 0.7 chain becomes the strongest).

You would need to get everyone to stop using 0.7 (or much more than a majority, or as soon as another block of this kind resurfaces the clients would be useless again).

0.8 accepts blocks that 0.7 rejects, 0.8 does not reject any blocks that 0.7 accepts. So 0.7 is more compatible going forward until the problem is solved with 0.9 etc.

There's a bug in 0.7. That bug means 0.7 clients reject what is, by spec, a valid block, while 0.8 clients accept it. So they can coexist for a while, until a block comes along that causes 0.7 to reject it.

Even once that happens, if there are more 0.7 clients than 0.8 clients, they will keep mining, eventually generating a longer chain, and the 0.8 clients will start accepting that longer chain. It's only once 0.8 has more mining power behind it that you get a split; now the 0.8 clients will keep forming a longer chain, and the 0.7 clients will be left behind, as they still see that chain as invalid (due to the invalid block), so they only see the 0.7 chain.

So, while the 0.8 clients were not "wrong" in accepting the bad block, it's pretty hard to fix the problem by convincing everyone to update to 0.8; some people may just not be able to or want to update. But it is easier to fix the problem by getting just enough people to downgrade to 0.7; as long as more than 50% of the mining power is on 0.7, that block chain will be the longest and both 0.7 and 0.8 will work fine with it.

Another way to look at it is to ignore the spec, and just look at the reality of implementation. According to the reality, 0.8 introduced a non-backwards-compatible feature allowing it to accept new kinds o blocks. In terms of protocols, this is sometimes known as "embrace and extend"; support the open protocol that everyone else supports, but add extra incompatible features. While this probably wasn't intended, the effect was that 0.8 miners got an advantage on mining new blocks; they would accept block mined by 0.7 miners, thus keeping ahead of the chain, but once they managed to mine an incompatible block, they would now be a block ahead of the 0.7 miners.

Now, what does this mean in terms of transaction security? Most likely nothing, unless there were any double spends, where the same coins were spent in one way on one chain and another way on the other. If there were double spends, you could look at both block chains and find them; given how quickly this was caught, it's pretty likely you could track down where the double spends went, and it's unlikely that the double spends have changed hands more than once given how short this split is. But there are other factors that make the double spends unlikely. Most transactions will propagate throughout the network pretty quickly, to miners on both sides of the block chain split, meaning that both block chains are likely to record the same transactions. You would have to notice this split, and target transactions at each side of the split so that miners on each side would pick up separate transactions, without the person that you are sending the coins to (and who is presumably sending you something else in return) noticing, all within the span of a few hours.

Now, given that this has happened once, I would be surprised if someone doesn't write a bot that noticed this kind of split and do just that, to see if they can get any double transactions accepted. To defend against this, anyone who cares that much (such as any large Bitcoin merchants or payment processors) can do the converse, monitoring for any chain splits and following both sides to watch for double payments, treating those as fraud and returning (or keeping) them without actually delivering the goods in question.

Good luck tracking down all the double-spends, because there were an awful lot of them: http://blockchain.info/double-spends Most of the ones I've looked at so far are Satoshi Dice bets, but there may be more substantial double-spending hiding amongst the list.

Turns out someone did a $10,000 dollar double-spend which - as far as I can tell - no one spotted until they pointed it out.

Now 8 blocks behind.

great comment. update - they have resumed deposits.

I don't understand why 0.7 gets the preference though, considering that it's the version with the DB bug in it.

Because it won't generate blocks that 0.8 won't accept, and 0.8 does generate blocks that 0.7 won't accept.

This is a quick fix to get the system back on its feet while they figure out the longer term strategic answer.

I don't understand this. If 0.8 fixed the bug that 0.7 contains, shouldn't encourage everyone to upgrade instead of downgrade? And it's fine that the longest blockchain is on 0.8; that is the latest version. If you're not patching your software, I am not so surprised that you're vulnerable (in this case to double-spending attacks).

A second thing that I don't understand is why 0.8 people are vulnerable to double-spending attacks. It wasn't said that 0.7 was safe, but it was said that 0.8 was vulnerable. But with two separate blockchains, wouldn't both be vulnerable? Or at least 0.7 because they don't accept 0.8's blockchain?

If they pushed for people upgrading to 0.8 then everyone not upgrading would have their own blockchain. By getting more than 50% of the users to use 0.7, the rest of the 0.8 clients will discover that everyone else uses the 0.7 chain and thus replay all of the transactions on that chain.

Upgrading: Those who doesn't upgrade end up with their own chain.

Downgrading: Those who doesn't downgrade will automatically use the other chain when it becomes dominant.

> But with two separate blockchains, wouldn't both be vulnerable?

It's only the chain that doesn't become the main chain that is vulnerable. Because they're pushing for 0.7 to be the main chain you don't have to worry about double spending there.

The choice was made for backwards compatibility. People are still running version 0.3-0.7 of bitcoind and up to this point that was not a problem. If we decided the 0.8 fork was canonical we'd lose a big chunk of the userbase.

Who is "they"? Is there a central authority in charge of BTC? I thought part of the advantage of it was that there wasn't.

"They" is the dev team for the main client.

They don't (can't) rule with an iron fist, but they have a lot of good will with the community, so most will listen to them.

EDIT- The good/bad news is that since most mining (and thus transaction verification) has consolidated into large pools of machines, only a few people have to agree with "them". So in a sense, there is a central committee of people who run the large mining pools that does have a fair amount of power.

Yes, you could say that ultimately the miners rule which blockchain grows, but have strong reasons to defer to the judgement core developers.

Even though the pool operators have large discretion, the computational power that makes up their pools can be moved elsewhere... a little like a transferable proxy. Most smaller miners probably trust the pools who they've grown confident in... but a large enough controversy or campaign could conceivably shift the mining-power away from pool operators who make unpopular decisions.

Yes. A large enough controversy could also threaten to devalue Bitcoin drastically. No one wants to store assets in a currency which could potentially lose the faith of all of its merchants, vendors when it forks and no one can agree on which fork is the 'correct' one. It's in everyone's best interest to do whatever it takes to get back to a single long blockchain.

So what you're saying is that there isn't a central authority like the Bank of England but there is a cartel like OPEC?

It's easy to fall prey to the idea that the ideal is that there is no authorities anywhere, but that's not the goal. The point is that you can choose your authority freely, and if enough people decide a given authority is no longer worthwhile, they can choose a new one freely.

Open source works the same way. Of course there are authorities for a given project who can choose what the final release is, but if they get too full of themselves or something it's easy enough to bypass them. It's happened quite frequently, so it's not an idle threat, either. You can't prevent authority from developing, that's wouldn't even be a good thing, but you can prevent it from abusing its current position of authority to lock in future authority.

More like customers of amazon, Google and apple. If suddenly what they were offering sucked, the 'little guys' would put their resources elsewhere.

Yeah, cause access to compilers really is a scarce resource.

Trust is the limiting factor.

Exactly, no-one is forcing the miners to do this. The 0.8 miners could decide to keep that branch alive, and if it stays longest, then the "central committee" would be SOL and just have to accept it.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact