
Eric Brewer on Why Banks Are BASE Not ACID – Availability Is Revenue (2013) - mpweiher
http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html
======
brandur
Interesting article, but it's really talking about ATM networks rather than
the broader "banks". A bank's core systems could still hugely benefit from
having an ACID compliant database even if any individual ATM node could still
be allowed to operate independent for some amount of time to maximize
availability.

Consistency is a feature whose importance is regularly downplayed by hawkers
of NoSQL databases. I've seen firsthand what happens when a NoSQL database
scales out and it's not pretty — a significant number of people end up
spending a significant amount of their time trying to figure out how to
maintain data integrity across failures in the system, even the kind of minor
failures that are occurring every day. A lot of the time that involves going
in and cleaning up data _by hand_. There is no general solution for this
problem, and it'll be a persistent operational tax that dogs you as long as
your NoSQL database stays in use.

~~~
brockers
No the article uses ATMs as an example but the general accounting principals
apply regardless. Consider paper checks or IOUs. Humanity literally has
thousands of years doing eventual financial consistency while ACID compliance
has only existed for the last 50 or so.

I'm not saying ACID isn't good (or that NoSQL is a panacea) but we have long
had exceedingly successful financial systems without ACID.

~~~
xkarga00
Define "successful financial systems"

~~~
brockers
Every bank that existed before 1960 starting with ancient Greek temple
depositories all the way through the first "official" banks of 12th century
Venice.

------
PaulHoule
It is easy to laugh at Bitcoin but with Bitcoin you can send $15 from New
Jersey to France immediately but the fee to do a bank transfer is many times
that.

I wrote a check for $130,000 dollars that bounced once because the teller
deposited the check I got for selling a house in somebody else's account.
Mistakes happen, but the bank is able to figure it out and (hopefully) make it
right - that is why the system is built to tolerate slop.

~~~
Nursie
Odd, I transferred tens of thousands between the UK and Australia for not much
more than those 15 dollars.

~~~
aianus
No, they likely just hid it in the forex spread. No bank is doing $20,000
retail cross-currency wire transfers for a $15 flat fee.

~~~
Nursie
So what?

I transferred cash across the globe for peanuts. The underlying mechanism
doesn't really matter to me.

~~~
aianus
No, you're just not aware of how much you paid.

------
fauigerzigerk
This is a very selective view of banking. In financial reporting you have a
lot of situations where you need a consistent point-in-time snapshot of your
data or you might be in very big trouble.

It's true that clearing of payments and trades is usually a multi-step process
that isn't done in one huge distributed transaction. But implementing each
stage of that process without ACID transactions seems like a daunting task to
me.

So I think Eric Brewer's remarks lack a bit of nuance and detail, but he's not
entirely wrong of course.

~~~
reqres
Could you be specific. In what particular areas do you need a consistent
point-in-time snapshot?

Having worked in finance myself, as part of a team that procured and replaced
an international core banking system with a state-of-the-art, multi-hundred
million dollar successor; we've never had a strong consistency requirement.
Much weaker forms consistency sufficed for their needs and the requirements of
regulators.

Also what big trouble do you mean? I do not know of any financial regulators
that expect banks to have a totally consistent point-in-time view of their
balance sheet. Furthermore the Basel Accords and similar banking supervision
rules do not mandate such a requirement.

I be very much grateful if you could provide me with this information, as I
can pass it to my former colleagues who will be quite surprised to discover
this requirement.

~~~
BoorishBears
That last paragraph implies you're less interested in finding out and more
interested in insisting those requirements don't exist because you worked on a
project that overlaps with the incredibly broad term "financial reporting".
The term is so broad I'd almost say your anecdotal experience is more likely
to be irrelevant than not.

~~~
reqres
A core banking system captures and stores all the transactions and business a
bank performs. Without this a bank would not be able to function let alone
produce a meaningful financial report.

~~~
BoorishBears
In what way does that mean you'd be able to disprove the statement:

"In financial reporting you have a lot of situations where you need a
consistent point-in-time snapshot of your data or you might be in very big
trouble."

You're trying to sell your anecdotal experience rather than explain your
point.

------
sfifs
I find quite often in India that whem networks are down, ATMs go down
displaying a network failure error message on screen. I've never heard of
anyone being able to overdraw an account. Credit cards also need authorization
both through PIN on terminal and bank authorization over the networks (which
do fail from time to time). Over the Internet, two factor authentication is
pretty much mandatory especially for credit cards.

Therefore this whole Banks don't value consistency argument feels very fishy
to my experience. Is this a uniquely American thing like lack of chip and pin?

~~~
carlob
I don't know about India, but I know that in many western European countries
overdraft fees such as those that are common in the US would be considered
above the maximum legal interest rate, because of that the bank has no big
incentive in letting you withdraw money you don't have. Consider also how much
more widespread chip and pin cards are on this side of the pond, and you're
starting to make me suspect that the wider adoption of chip and pin might have
been helped by usury laws.

------
hackuser
1234 days ago:

[https://news.ycombinator.com/item?id=5642010](https://news.ycombinator.com/item?id=5642010)

------
droro
The conclusions of this article have far more implications for the future of
blockchain technologies than for the future of databases.

Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID
qualities above all else- including availability and responsiveness. Does it
make sense that completing a $1 transaction would require the same amount of
resources as a $100,000 transaction? The traditional banking system has the
flexibility to expend fewer resources on the $1 transaction, but Bitcoin does
not- by design.

Interestingly, I think it is the inconvenience of the traditional banking
system that gives it this flexibility. If financial transactions were
something that could be done instantaneously, by a computer, without having to
provide personal information to set up an account, then stealing money in
100,000 $1 transactions might be easier than stealing $100,000 in one big
heist. Add a little inconvenience to each transaction and this problem goes
away.

Unless blockchain transactions are extremely cheap, it will limit their
competitiveness with the traditional banking system for this reason.

~~~
amscanne
> Bitcoin and Ethereum and their ilk have, out of necessity, valued ACID
> qualities above all else

They definitely don't have ACID properties.

The blockchain gives you something like eventual consistency, but it's
probabilistic.

You have atomicity within a single block, but given the acceptance of the
block is only eventually consistent, I'm not sure it's meaningful.

------
marknadal
This is such an important article for people to read, especially people so
bent on strong consistency. I've done talks on this for our database
(eventually consistent, [http://gun.js.org/](http://gun.js.org/) ), and people
always ask about banking. Yes banks should use strongly consistent approaches,
but they don't - user experience trumps perfect academics. And having Eric
Brewer say this is fantastic evidence.

~~~
lotyrin
What do you do when your reservation app gives two people the same table?

What do you do when your auction app thinks there are two winning bidders?

It's not about "perfect academics" it's about risk management. How do you
prevent or manage inconsistencies.

~~~
marknadal
These are great questions and I hope I can convince you with some simple
answers.

We cannot get this problem if we treat the table as its own thing. So for
instance, rather than having a "reservation" that has an "assignedTable"
column with a string of "2A", if instead we have a table that has a property
of "7pm" with a value of "Mr & Mrs Smith" then this value cannot be a string
munged with the other couple.

This easily applies to many other areas, like finance. Rather than having a
"purchase" record that has a "debit" amount on it, make the software model the
real world of currency. Instead you have a "dollar" record with a unique
serial number, it has a "owner" property of "lotyrin". No matter how many
times we concurrently write to this property, the end result must necessarily
be 1 value.

This is the way humans have been thinking for thousands of years. And guess
what, I sat down with a large airline and had to warn them "we're not Strongly
Consistent" and they laughed at me saying "you realize we've been booking seat
reservations before there was internet, before you were born, and before there
was cheap telephony. Seat reservation has never been strongly consistent - we
used to have hundreds of travel agents booking seats and it would take 2 weeks
before we would hear about it."

Again, I want to emphasize this is exactly the reason that perfect academics
doesn't always help - it is a solution to a problem in isolation, devoid of
its real world context. I am in no way saying the academics aren't useful or
interesting, but I am saying that even Eric Brewer himself recognizes where
academics aren't helpful.

But to stress your point, let's take your example further. Rather than
assuming the restaurant actually has a credit card authorization server that
is the source of authority, let us pretend it is more of a bitcoin crowd
hosted reservation service. Now despite our "Mr & Mrs Smith" value being
straightforward, we wind up having a replication problem. One server has it as
that value, and another peer-to-peer server has it as "Mr & Mrs Lionheart".
What happens?

Well, under our database (like an Open Source Firebase, see the link in my
original comment), this is why realtime updates and push notifications are so
handy. Even if 2 couples reserve the same table on different peer servers at
roughly the same time (such that when the peer servers checked availability
there was no conflict), when the new update saves it gets pushed out to all
the peers and runs through a conflict resolution algorithm locally. So all
peers sync based off of push, not an outdated pull.

In fact, we made a great explainer video for this type of situation, see
[http://gun.js.org/explainers/conflict_resolution/lexical_sor...](http://gun.js.org/explainers/conflict_resolution/lexical_sorting.html)
(might change the URL naming scheme in future, so careful of a broken link).
But I'm going to again assume you aren't talking about "what practically
happens" but are intentionally trying to push for the extreme edge case bugs.
If so, let us add that the peers are offline while these updates happen
(because yes, this is something we support, being offline-first) so therefore
cannot sync with each other fast enough for the 2 couples to realize one of
them got ousted.

We are now only left with two scenarios: A) Because when the app tries to save
the data it gets an error that the push notification failed due to the network
being down, we display a warning to the user that the "Reservation is pending,
proceed at your own risk or try again later". Or B) Because we are being
academic, we yes go ahead and make the reservation (unaware of the conflict).
How do we address it? Well...

Finally, both the Lionhearts and Smiths show up at the same time. Initially
they argue both asserting "I have the reservation" until a hostess steps in.
The hostess apologizes and says "We are terribly sorry, this is our mistake
and let us fix it for you." They then offer:

A) Would one of you like to sit at the open table over here, that somebody
canceled last minute on?

B) If (A) is not available, then here are free drinks* for all of you, but who
would like a 20% discount* off their next dinner? Next open table should be in
about 45 minutes?

*Total loss is 75cents on sugar water and $10 of discount, but profit GAIN is $50 (because had the reservation app said "booked out" the 2nd couple would have never turned into a customer), leaving the restaurant at a net of +$39.25 at the end of the night.

Again, let us make it more extreme, both couples are uncompromising, saying
"We cannot do that, we have a show at 9pm", then there are only 2 remaining
situations:

A) You flip a coin and 1 couple leaves angry and never comes back to use the
20% discount because they are so angry the coin did not choose them. (Note,
the restaurant still made a profit filling out the seat though) OR

B) You basically flip a coin and 1 couple leaves bummed, but enjoying their
free drink and looking forward to coming back another time with a 20%
discount. (Note, restaurant still makes a positive profit)

The moral of the story is that you want to create Win-Win options for
everybody. Why? Because as you say, it is all about risk management and the
probabilities that Murphy's Law is going to strike so many times (multi-
continuously-offline-peers-that-do-not-error-with-above-average-unsatisfiable-
angry-humans) such that you get an angry customer with a free drink and a
coupon is extraordinarily unlikely. And if you have not already calculated for
ever losing 75cents on sugar water plus a discount, then something is wrong
with your risk management department - not with your algorithm.

With that, I'll leave the bidding app to your imagination. The trick question
for finding simple solution is: What would humans do before the telephone?

I hope this was a pleasure to read, and I want to thank you for calling me out
by asking the glaring question. If you have any concerns or counter arguments,
please let me know. I would be happy to hear them. If you like what you see
here though, check out the rest of our work on the Open Source database.
Cheers!

------
brokencode
While interesting, this is not satisfying as a general argument for non-ACID
databases. Incrementing and decrementing are about the simplest possible
operations, and even the article admitted that these are commutative, so the
order doesn't matter. It is indeed much easier to work with operations that
can be ordered arbitrarily, but I think that most real-world problems are
sequential.

------
danjoc
(2013)

It would be interesting to know if BASE at ATMs had anything to do with

[http://www.theatlantic.com/international/archive/2016/05/jap...](http://www.theatlantic.com/international/archive/2016/05/japan-
atm-theft/483902/)

Coordinated attack over two hours? A little DDOS + BASE perhaps?

------
erikb
I could only read a third of the article before starting to crinch too much.
If this was on purpose and I missed the same conclusion I am presenting here,
sorry.

The article sounds like "this of course is the smart way, make money first,
worry about details later". But if you think about it this is very, very
wrong. First, if any system will fail drastically at some point. You wait long
enough and something unexpected happens. Second, if this consistency tolerance
becomes widely known people will try, and at some point succeed, to cheat the
system big time.

If any of these two scenarios happens, the whole system breaks. And then, what
do you do? You can only get so many bail outs from the tax payers. The smart
way, of course, is to not be too greedy and stick close to a relatively
consistent scenario.

