"It's like all these bitcoin sites are running a bug bounty without even realizing it.."
Isn't ACID the solution to these problems for like 30 years?
Yes, but this is bitcoin, which is so totally unlike anything that existed before, that such outdated notions as ACID aren't valid anymore. We need to start with a clean slate and reinvent all the wheels all over again.
No it's not. It's just sloppy code and/or a misunderstanding of database transactions and isolation levels. If you use an ACID compliant database like PostgreSQL it's pretty straightforward to write proper code that would not have this issue.
The main concept to understand is that you need to lock all the rows that you'll be touching before you use them. In most case this method works fine with the default read-committed isolation level.
Code that queries your database in multiple round trips and then compares the values for validity on the app side will have this problem. Anything you read could have been modified by the time you "act on" the information you read. Heck even stored procedures running on the DB itself will have this problem if run in the default read-committed isolation level.
The high level solution is to use SELECT ... FOR UPDATE and lock everything that you'll be touching before you update anything.
> The solution is to implement double bookkeeping and bank reconciliation, but it's tedious and complex to implement.
This is a business requirement for any accounting/financial software but it's not a technical one. Having an audit trail, double entry accounting, and recon are business functions. You'd be crazy to design a system that didn't include them but it's not strictly required (again from a technical sense, not a business one!).
Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
I'm thinking that they had both issues. If the system let you submit multiple withdraw requests (totalling more than your balance) then they definitely weren't checking at request submission before queuing them for later processing.
> Obviously you can't have a SQL transaction that spans multiple applications, so SQL transactions wouldn't really work in this case. It was just a straight forward mistake: orders could be placed before previous orders were finished executing.
XA transactions allow you to have transactions across multiple services. The most common use case is a database and a message queue. It's nowhere near as simple as dealing with a single transactional resource but it's not that uncommon in the finance world. I've used them quite a bit and they're really convenient. Getting it right from a dev-ops perspective is a bit pretty tricky though as you need to really understand how the transaction manager itself works and make sure it actually runs.
> Transactions wouldn't actually work with bitcoin in this case anyway. eg, transaction 1 checks balance, deducts amount, does BTC transfer, and then commits. Transaction 2 does the same, but the commit would fail as transaction 1 has already modified it. Transaction 2 can't actually roll back though - the bitcoin transaction cannot be rolled back.
Yes you can't have XA transactions with bitcoin as it doesn't include any concept of rollback. What you can do though is guarantee that a transaction is only run once and re-submit it if it failed to send. If you can uniquely identify a bitcoin transaction then you can guarantee that you don't send it out more than once. Just add in a significant delay (couple hours? a day?) before any retries to ensure the transaction has been merged into the block chain.
: And not by just using the transaction hash because we all know what can happen with that: https://en.bitcoin.it/wiki/Transaction_Malleability
You'd use one transaction to indicate you were beginning a transfer, putting that amount in a hold state, attempt the bitcoin transfer, then a transaction to indicate the bitcoin transfer was successful or not.
If there is an error in the bitcoin transfer or after, and the second transaction is never run, the funds are still in hold whether transferred or not and can be manually corrected by checking the system logs.
Still, this doesn't inherently solve the problem. That is, you would still need to be aware of transaction isolation levels and ensure that you're locking rows properly as you conduct each transaction.
Continuing on the OMG, bitcoin makes proper bookkeeping obsolete(!)- bzz, NOT!!. Theme.
You certainly can use transactions for bitcoin orders, you just have to use the logic that many other business uses.
That is, a user has an account that they can place orders from. The only action they can take is turning their funds into an order. Correctly coded transactions assure that they can only purchase orders of up to the value in their account. Order processing is independent from this order entry system. Orders may succeed and become bitcoins or dollars and then be transferred back to the accounts or the order could fail and the dollars could be returned appropriately as well. This order processing is queued, could involve a series of messages, etc. I only have a degree in econ but I recall stock exchanges work basically this way (you place orders, money deducted immediately but you have to wait to see if your order actually worked, sorry).
Essentially, the final result of orders shouldn't be the point where the funds are debited from the users' accounts, the initially order placement should be where the debiting happens.
Isn't it a quick fix to add an ACID-friendly clause to the UPDATE statement, e.g. some pseuosql like:
UPDATE balances SET balance = (balance - amount) WHERE user_id = input_user_id AND (balance - amount > 0)
You should have a ledger.
"INSERT INTO Ledger (Account, Date, Amount) VALUES (PersonToDebit, Now, -withdrawl)"
And then someone's "balance" is the sum of their ledger.
If multiple withdrawls go through, the account could still go negative if you don't check at the time of the withdrawl, but worse case you still end up with negative balances rather than flat out losing track of the balance because you have a balance that you UPDATE.
With the regular banking system or even with stock market, some trades and transactions can be reversed in case of errors and mistakes. So just having a ledger works, some bad updates can be reversed. Not so with bitcoin.
So when dealing with someone's account at a place like this an ACID compliant balance update is also needed.
Amateurs have jumped in and started developing banking software that deals with hundreds of millions of dollars -- so this is not surprising.
But at the same time it is kind of nice. It is sort of a Darwinian if you wish, the weakest and unlucky ones fail and get robbed the competent and/or lucky get to survive.
I was arguing against the grandparent's claim of ACID race conditions; my intention wasn't to advocate in favour of a "balance" column.
Edit: that said, I wouldn't mind hearing the pros or cons of maintaining a convenience (e.g. page display) "running balance" that is updated from the ledger after every transaction.
If you want to keep one for optimization purposes (i.e. you are reading more than writing and don't want to constantly sum the ledger table), then that's probably outside the scope of the parent's point.
"What accounts for the consistency of bank data is not the consistency of its databases but the fact that everything is written down twice and sorted out later using a permanent and unalterable record that is reconciled later."
But isn't that talking about "traditional" financial institutions that (in my experience) have staggeringly complex (messy) computing environments just by the nature of how these businesses grow (e.g. lots of acquisitions).
Surely for a brand new service such a Bitcoin exchange there isn't that excuse?
That is why you have ledgers. Being able to go back and "replay" step by step how you got where you are makes all the difference when you run into stupid bugs. And you will run into stupid bugs.
That you're dealing with a brand new service that has not been battle-tested for years is yet another reason why you want a ledger and proper reconciliation rather than relying on a running balance.
To further clarify, an ACID database, used correctly, are a drop-in solution to problems like that of someone attempting multiple simultaneous account withdrawals.
And, of course, "you can still run into these problems using an ACID database" (lol), yes, all you have to do is not understand transactions and not use them.
* At T=1, transaction 1 will read the current balance
* At T=2, transaction 2 will read the current balance
* At T=3, transaction 1 will update the current balance
* At T=4, transaction 2 is in a race condition situation, since the update of transaction 1 'leaks' into transaction 2, but the last read balance in transaction 2 still is the 'old' balance.
In other words: this is an area where a lot of people introduce race condition bugs without being aware of it.
And also risk-avoiding schemas (in the general provisions of the business, not only on the system running it)
Well, if someone could publish best practice as to how not get hacked, that would be great: it seems VCs and advocates are unhappy to see those scaring the seven billion people still to be convinced this is worth their attention. No points in having regulations, voluntary or state-enforced, if you have no idea what to check.
These are earnest questions, not statements.
It is entirely possible and I would argue acceptable that regulatory authorities require KYC compliance for these gateway entities, but that only affects people moving national currency in and out of the system. It's entirely possible that people will trade and do business in USDcoin without cashing out directly through the regulated banking gateway. This is a reality that regulators will have to get used to.
I'm not sure about this one. Regulators have a tendency to "motivate" others to "get used to" things, not the other way around.
With E-gold years ago, the proprietors were criminally indicted for activity that took place on their network.
With your architecture, if the Feds deemed that their was laundering taking place or the exchange was otherwise being used to facilitate illicit activity, then I can easily envision them expanding the "responsible parties" definition beyond gateway operators to any node in the exchange. This, of course, would mean that anyone participating would be liable for activity that routed through them, thereby having a chilling effect on participating peers. So, if their goal is to shut down the exchange, they can effectively do so via this implicit threat of criminal prosecution.
If any of this sounds far-fetched, you really have to review the E-gold case  to see just how much the regulatory regime will contort itself to keep financial transactions under its constant watch.
I can't believe programmers are alive today who don't know what a race condition is. In the future, when parallel processing is more the norm than the exception, race conditions will be the first thing we'll have to think about, not the last.
I'm pretty sure it's been more than that in the last few years...