Hacker News new | comments | show | ask | jobs | submit login

My opinion is exactly opposite: Consistently is overvalued.

Requiring consistency in distributed system generally leads to designs that reduces availability.

Which is one of the reasons that bank transactions generally do not rely on transactional updates against your bank. "Low level" operations as part of settlement may us transactions, but the bank system is "designed" (more like it has grown by accretion) to function almost entirely by settlement and reconciliation rather than holding onto any notion of consistency.

The real world rarely involves having a consistent view of anything. We often design software with consistency guarantees that are pointless because the guarantees can only hold until the data has been output, and are often obsolete before the user has even seen it.

That's not to say that there are no places where consistency matters, but often it matters because of thoughtless designs elsewhere that ends up demanding unnecessary locks and killing throughput, failing if connectivity to some canonical data store happens to be unavailable etc.

The places where we can't design systems to function without consistently guarantees are few and far between.




As you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases.

Also, the banking system does have the system of compensating transactions (e.g. returned checks) when constraints are violated, as well as an extensive system of auditing to maintain consistency.

Do most distributed systems have the ability to detect inconsistency? How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done.

I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC.

Come to think of it, there was also something that used to happen frequently with Amazon video, though I haven't seen it happen in a while. When I would resume a video, it gave me two choices of times to resume, one based on the device and one based on the server. The odd thing was that I had only ever used one device to watch the video but it was always the server-stored time that was right. Just a bug, I guess, but that does indicate the kind of inconsistencies that have to be resolved somehow.


> Do most distributed systems have the ability to detect inconsistency?

They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x".

For operations that require you move money from one account to another, you need to guarantee that both or none steps gets durably recorded, and you may (or may not, depending on your requirements) insist on local ordering, but even there global consistency is unnecessary.

> How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done.

You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput.

On failure, your system needs to be able to determine which operations have been applied, and which have not, and resume processing of operations that have not.

> I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC.

This type of thing is often down to copping out on guarantees when a system includes multiple data stores, and is often fixed by applying a local transactional event log used to trigger jobs that are guaranteed to get processed. E.g. consider the Reddit example - if you want to be able to effortlessly shard, you might denormalize and store data about the comment both tied to a thread, and tid to a user. If these are in two different databases, you need some form of distributed transactional system.

A simple way of achieving that is to write a transactional log locally on each system (as long as operations are design to not need coordination) that ensures that both "attach post to thread" and "attach post to user page" will get logged successfully before giving a success response to the end user. You can do that by using a commit marker in your log. Then you "just" need a job processor that marks jobs done when all operations in a transaction has completed (and that can handle re-running the same ones).

But this is something people often fail to do, because it's easier to just write out the code for each step sequentially and deal with the rare instances where a server dies in the middle of processing a request.

So the basic pattern for ditching global consistency is to write transaction logs locally, representing each set of operations you need to apply, and put in place a mechanism to ensure each one is applied no matter what. Then use that "everywhere" including in the many places where you should've had guarantees like that, but don't because developers have been lazy.


> You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput.

You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. People often forget this: the user is sort of the unstated extra party in a distributed system. A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really work for me that the application is free to reorder my actions.

At some point your desire to have HA via a weakly consistent system conflicts with intuition and usability of an application.


> You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered.

The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations.

There are exceptions, naturally, but the point is not that you should be able to do this "in general", but that expecting global consistency is a relatively rare exception.

If I post a comment here, I expect my comment to be visible when I return to the page. But there is no reason for me to notice if there is a strict global ordering to when posted comments first appears in the comment page. If someone presses submit on their post 100ms before me, but the comment first appears to me the next time I reload, while it appeared the opposite way for the other poster, it is extremely unlikely to matter.

> A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really for me that the application is free to reorder my actions.

This is a design issue. Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements.

As long as you do that, it makes no difference to you or anyone else if the setting and post was replicated globally in the same order or not.


> Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements.

I think you're misunderstanding what the problem is here. I want global consistency (really I want two party consistency) to prevent the said user from witnessing the unkind thing I posted about them. It's necessary that my own actions appear in their original order to me, but also to anyone else who is plausibly affected by their reordering.

The full example also involves posting on a mutual friend's wall and having that post hidden by unfriending. You could route all requests to fetch posts to the owning poster's pinned set of servers, but this would be extremely expensive. Sharding solutions also weaken availability guarantees, because you necessarily shrink the set of nodes able to service a request.


> I think you're misunderstanding what the problem is here. I want global consistency (really I want two party consistency) to prevent the said user from witnessing the unkind thing I posted about them.

You don't want global consistency and you're saying it yourself.

What you want is to be presented with a view that is consistent with the actions you've done. [it's really basics101 for user experience.]

The internals of the system don't matter to you. It only matter to us, as the builders. We can make it so that it is internally 'inconsistent'[0] yet it always presents a 'consistent'[0] view to the user.

[0] I don't think the word consistent describes well the situation. Nothing is ever strictly perfectly consistent.


You seem to be assuming that I'm a UX designer. I'm not, I'm a backend engineer. Just a note though: I wouldn't say something like "The internals of the system don't matter to you. It only matter to us, as the builders." to a UX designer, since that's super condescending.

Sequential consistency is basically the most lax model you can get away with in this case (and have it generalize to other similar operations). This is a very strong consistency model and weakening linearizability to this basically provides no advantage for availability. There is a proof that anything stronger than causal consistency cannot provide "total availability", which I'd argue is somewhat mythical anyways (except when you're colocated with where data is being stored).

My point here is that this is clearly a case where many of the strongest ordering guarantees are required to get the desired behavior, and subsequently availability must be sacrificed.


I was making the distinction on being the user VS being the builder of the system. The user doesn't need to know or understand the internals.

> consistency... consistency... consistency

What if I am not aiming for consistency in the first place?

We both understand that "total availability" is mythical. So why not target partial availability in the first place?


I'm not saying you should not have availability. On the contrary, many strongly consistent algorithms actually have very good availability characteristics. The whole point of being distributed is often to increase availability. Algorithms like raft and multi-paxos can provide availability so long as a majority of nodes remain reachable to the client, which I think is pretty acceptable in many cases. This is sort of why the availability vs consistency dichotomy is a false dilemma. Sure, having linearizability means you cannot have total availability and vice versa, but generally you can have very good availability and preserve strongly consistent semantics. What I disagree with is that trading strong consistency to achieve total availability is useful in many domains. To be fair, there are some, and for write-heavy workloads that tend to materialize a readable view in the future or for things like immutable caches, it can be extremely useful to weaken consistency, I just have often found that for the bulk of business logic, weaker consistency semantics make unacceptable tradeoffs to users.


> It's necessary that my own actions appear in their original order to me, but also to anyone else who is plausibly affected by their reordering.

No, it's necessary that the effects appear in the right order, and in the example case what matters is that the privacy setting is correctly applied to the subsequent post and stored with it.

> The full example also involves posting on a mutual friend's wall and having that post hidden by unfriending

If that is a requirement - and it's not clear it would be a good user experience, as it would break conversations (e.g. any reply; and what do you do about quoted parts of a mssage) and withdraw conversation elements that have already been put out there -, but in any case this does not generally require global consistency. It does require prompt updates.


Do you really need global consistency for that? I think you can get by with user-consistent ordering as long as your data model's right.

It sounds like you're saying that the friend-state for a given post are part of the permissions for the post, and therefore part of the post itself. In which case, when a user posts, I'd include the version number of their friend-state in the post's permissions. Then on view attempts, I make sure that the friend-state I'm working with is at least as new as the friend-state referred to by the post.


There are definitely solutions for handling this with consistency semantics that are something weaker than linearizable consistency. But they usually become expensive in other ways and loose a lot of the availability benefits to recover the consistency properties you do need. Most of the models we're even talking about here are something in the realm of sequential or causal consistency, which are still considered strong consistency models (and are usually implemented on top of linearizable algorithms that allow for weakening invariants that protect against witnessing inconsistency. spanner is an example of this).

Your friend-state solution only protects the visibility of the post if the friend-state between the two users has not changed more recently than both the post and your version.

I.E.

Friend state 0 is we're friends.

Friend state 1 is we're not friends.

Friend state 2 is we're friends.

Friend state 3 is we're not friends.

I have friend state 2, and the post is at state 1. I accept my version incorrectly and am able to view the post, despite the fact that we're no longer friends.


I'm seeing a lot of comments that are basically saying, "we don't need no stinking transactions because we can mostly reinvent them in the app." And the point of the article was, yeah, or you could pick a solution that already solves this.


Was that the point of the article? I didn't read it that way. E.g.: "In conclusion, if you use micro-services or NoSQL databases, do your homework! Choose appropriate boundaries for your systems."


You are right. The article says to understand your choice, it doesn't say WHAT to choose.

Pretty sure the subtext is that developers are making their lives harder than they need to by not understanding the choice they are making.


Do you perhaps have a different example that might make the problem more apparent here? Because if I post something you don't want to see, give you permission to see it, and then take it away again, I can live with you seeing it. I probably won't even know if you saw it because there was some sort of system glitch or if you just happened to look during friend state 2.


> I want two party consistency

aka snapshot consistency.

(Thoroughly enjoyed your "the user is sort of the unstated extra party in a distributed system", btw.)


Correct me if I'm wrong, but I feel like bank transactions aren't always commutative. Consider:

balance = 0 balance += 100 balance -= 50

You can't swap the order of addition and subtraction, because that would mean withdrawing from an empty account, and afaik you can't do that. Is there something I'm missing?


You can execute a withdrawal operation on an empty bank account. E.g. a credit card or a loan account are both accounts that are allowed to go negative, but regular accounts with overdrafts also do, and regular accounts without overdrafts are also often allowed to go negative (but often with prohibitive fees and restrictions attached).

It is entirely up to the bank whether or not it will be allowed to complete, but regardless the response from the bank the end result of the operations will be correct, but it may be different (it will be different if the bank will reject an operation in one order, but accept all of them when presented in a different order).

You are right that they are therefore technically not strictly commutative in the sense the outcome may differ in edge cases.

But the point is that what will not happen, is that you will get incorrect results. E.g. money won't "disappear" or "appear" spontaneously.


There are a couple ways you could do this, though according to jerf in a comment below, banks often don't implement transactions as commutative.

Most commonly things like this are done by somehow reifying the operation as data and attaching some information to determine ordering. One way would be to attach a timestamp to bank transactions. By itself, that would be enough to ensure bank transactions commute (but without bounds on clock synchronization, wouldn't guarantee that you see those operations in the order you issue them). You could also require a single consistent read to an account to get a state version number and attach an incremented version of that in your transaction. Ties in both cases could be resolved by account id of the person initiating the transaction.


I don't have much to add to your excellent post; I just wanted to underline this:

> Log events durably first, and then act on the events.

Nothing has changed my view of how to build systems more than the log-then-act paradigm. So many problems go away when I start from that perspective.

I think it took me so long to see it because so many common tools work against it. But those tools were built in an era when most systems had "the database", a single repository of everything. That paradigm is very slowly dying, and I look forward to seeing what tools we get when the distributed-systems paradigm becomes dominant.


Exactly, not least because it is fantastic for debugging to be able to roll a copy of a sub-system back and re-process operations and watch exactly what breaks instead of trying to figure things out from a broken after-the-fact system state.


Does commutative mean something different in CS than in math? In math, commutative means that the order of operands doesn't matter.

For example: addition is commutative, because 1+2 = 2+1

Subtraction, however, isn't.

Since a transfer is basically a transfer from one account to the other, it couldn't possibly be commutative in a mathematical sense.

I suspect there might be a better term for the property you are trying to name.

"Atomic" seems adequate to me...


(TL;DR - this is a concept that computer science borrowed from math)

In math (say abstract algebra / group theory), the operations on your database form a group under composition. Given two operations f,g (say f is "subtract $10" and g is "subtract $20"), the composition of those operations could be written "f•g" (depending on notation this can mean g is done first or f is done first, typically the former convention is used).

(So, instead of numbers and plus (+) in your example, we're using functions and composition (•) as our group.)

This group, of operations under composition, is said to be commutative if f•g = g•f for all f and g. (That is for all combinations of your possible operations.)

For addition/subtraction operations you're good to go.

If you can consider operations on a hash table being things like 'set a value of 10 for key "a"' the order that the operations are executed in do matter, and the group of those operations, under composition, is not commutative.

The subject of CRDT's in computer science is an attempt to create data types whose operations are commutative.

One reason to want do deal with commutative operations is that you don't have to figure out the order in which the operations officially occurred.


A bank transfer is rarely atomic unless within the same bank (and even then it depends how centralised their operations are). A transfer decomposes into operations on two separate accounts, which may involve additional steps (e.g. the sending bank will record having received a request to transfer it, record it's taken money out of the source account, record settlement, record the transfer has completed, etc.)

When talking about the operations being commutative, we're generally talking about each individual account, though it may apply wider as well.

In any case, the difference is that we are not talking about the elements of a given operation. You are right that if you see the operations as being "balance = balance + x" or "balance = balance - y", then the subtraction in the latter is not commutative.

But the operations seen as indivisible units are commutative.

Consider the operations instead as a list, such as e.g. "add(x); sub(y)" (in reality, of course, we'd record each transaction into a ledger, and store a lot more information about each).

See that list as an expression with ";" as the operator. This is commutative - the operands to ";" can be reordered at will. That ability to reorder individual operations is what we're talking about in this context, not the details of each operation.


My (admittedly limited, so far) experience in the financial industry has indicated that, while it is the norm is to use settlement and reconciliation in situations where making ACID-style guarantees is impossible, ACID-style guarantees are most emphatically _not_ foregone when it isn't necessary to do so.

One big reason for that is that the settlement and reconciliation process is easier to manage when the parties involved can at least be confident that their own view of the situation is internally consistent. Gremlin-hunting is a lot less expensive if you can minimize the number of places for gremlins to hide.


The bank example really needs to die, its not reflective of what people often use databases for.

1. Bank transactions are easy to express as a join-semilattice[1], which means its intrinsically easy to build a convergent system of bank accounts in a distributed environment. Many problems are not easily expressible as these and usually trade off an unacceptable amount of space to formulate as a join-semilattice and still work with complicated types of transactions.

2. A bank transaction has a small upper bound on the types of inconsistency that can occur: at worst you can make a couple overdrafts.

3. A bank transaction has a trivial method for detecting and handling an inconsistency. Detection is easy because its just a check that the transaction you're applying doesn't leave the account with a negative balance. Handling is easy, because the bank can just levy an overdraft charge on an account holder (and since they have the ability to collect debts on overdrawn accounts, this easily works for them). Complicated transactions often don't have a trivial method of detecting inconsistency and handling inconsistency is rarely as simple as being able to punish the user. In fact corrective action (and likewise, inconsistency) is often non-intuitive to users because we expect causality in our interaction with the world.

> The real world rarely involves having a consistent view of anything

I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world.

We expect most applications to behave the same way, because the distributed nature of them is often hidden from us. They present themselves as a singular facade, and thus we typically expect our actions to be reflected immediately and in the order we issue them.

Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2].

[1]: https://en.wikipedia.org/wiki/Semilattice

[2]: http://research.google.com/archive/spanner.html


"Bank transactions are easy to express as a join-semilattice[1],"

That may be true for a given set of transactions in a very abstract way, but that has some serious practical problems. First, like it or not, banks have a history of making transactions non-associative and non-commutative; there are many reported cases from the wild of banks deliberately ordering transactions in a hostile manner so that the user's bank account drops below the fee threshold briefly even though there is also a way to order the transactions so that doesn't happen. So as a developer, telling your bank management that you are going to use a join-semilattice (suitably translated into their terms) is also making a policy decision, a mathematically strong one, about how to handle transactions that they are liable to disagree with.

There is also the problem of variable time of transmission of transactions meaning that you may still need to have some slop in the policies, in a way that is not going to be friendly to the lattice.

The nice thing about join semilattices and the similar mathematical structures is that you get some really nice properties out of them. Their major downside, from a developer point of view, is that typically those properties are very, very, very fragile; if you allow even a slight deviation into the system the whole thing falls apart. (Intuitively, this is because those slight deviations can typically be "spread" into the rest of the system; once you have one non-commutative operation in your join-semilattice it is often possible to use that operation in other previously commutative operations to produce non-commutative operations... it "poisons" the entire structure.)

I think you're badly underestimating the difficulty of the requirements for anything like a real bank system.


Apologies, you're right, I've never implemented bank accounting in the real world, I'm only talking about it in the sense of an account balance as a "toy problem", where its often used to discuss the virtues of weak consistency by being able to use overdraft charges.

Whether or not its actually implemented with something like CRDTs in the real world isn't really my point, my point is that its a cherry-picked example of a system that clearly does reorder operations and has weaker-than-usual concerns about witnessing inconsistency.


> There is no perception of reordering my intentions in the real world.

Never tried to type something by thinking only of the word (rather than specifically the sequence of key strokes) only to have your fingers execute the presses in slightly the wrong order -- giving you something like "teh" instead of "the"? I have, a lot actually, and if I'm in a hurry or distracted, it can even pass by unnoticed, leaving me with the impression I typed the correct word while having actually executed the wrong sequence. (Hurry or distracted I think would be the most analogous to systems which don't take the time to register aggregate feedback and verify correct execution after the fact.)

Humans have at least one experience of trying to execute a distributed sequence of actions and having the output be mangled by the actual ordering events so common they invented a word for it -- the "typo".

It's insanely easy to explain to people that sometimes really complicated systems make a typo when they're keeping their records, because it's such a shared, common experience.


It's a cool metaphor, and I'm sure there are occasions where its easy to pass this off as innocuous. But inconsistent semantics can produce blemishes and bugs that can be difficult to anticipate and can very negatively impact user experience if you aren't careful. The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).

To be clear: there's an economic tradeoff to make here that isn't identical at all scales. At a certain point, lost revenue due to user dropoff from availability is more expensive than users who stop using a site due to a frustrating experience (and sometimes that scale itself is the compelling feature able to stave off a user from giving up on your app completely). But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself. Hell, Spanner is the foundation for Google's ad business, if that's not living proof that you can preserve strong consistency semantics and keep your system sufficiently available, I'm not sure what is.


> The typo explanation is acceptable when I just see some comment I posted being reordered after a page refresh. It's less so if I get locked out of my account because my password change hasn't fully replicated to all servers (and even less so if my consistency model doesn't converge).

The typo model is merely meant to explain why it happened. You'd be no more happy if you were locked out of a door because a locksmith fat-fingered the code on setting a pin. Not all typos are harmless, and that wasn't the point of my analogy.

Helping people understand why these technology mistakes happen -- through intuitive models like typos -- helps us discuss when mistakes are and aren't acceptable, and how we're going to address the cases where they're not. It simply gives me a way to discuss it with non-technical people: under what situations would a person doing the computer's job making a typo be catastrophic instead of merely annoying?

Most managers or clients can understand that question, at an intuitive level.

> But in most cases, I think its advisable to start with stronger consistency semantics and only weaken them when a need presents itself.

Completely agree! Unless you have a specific reason it's necessary for you, you likely don't have to actually address the inconsistency directly and are better off letting the database handle that for you.


Oh apologies, I misunderstood what you were saying! Yes, agreed, this is a good method for communicating what's occurring in weakly consistent systems to stakeholders. Definitely going to use this in the future.


Do you (or anyone) know of a better example than the bank account? I wouldn't be against changing it.

I wanted to communicate through a fairly succinct and understandable example that there are potentially high stakes, and I wanted to show a 'NoSQL' example rather than a micro-services one. It was the first one that came to mind.


Apologies, I didn't mean it as a critique of your post. What I meant was that its often used as a strawman example of why you don't need consistency[1]. Bank transactions are extremely simple compared to the majority of things we build on top of databases.

[1]: http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on...


Also, banks have very low consistency requirements.

Almost every transaction is of the form: 1. estimate authorization; 2. present request for settlement; 3. settle during batch processing; 4. any additional resolution steps (eg, returned checks)

The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, < 10 sec) it won't affect the usage patterns of most accounts and b) the bank only need quantify the risk to their system (and be less than the cost of improving the system), rather than have it be risk-free. Risk management techniques like preventing rapid, high-value transactions on top of a basic estimated value like "available balance" is fine for the estimation used in authorization phase.

The actual settlement is run during a special batch transaction during hours where they can't receive new transactions, and thus is a special case of it being really easy to implement consistency of the outcome, and it's acceptable if the settlement has some number of accounts end up negative or requiring additional steps (such as denying the transaction to the presenting bank), as long as the number isn't too high, eg, the estimator in step 1 is right most of the time.

Basically, the only requirement the banks have is that a) they can estimate how inconsistent or wrong their system is until their batch settlement where they resolve the state and b) the number of accounts which will be in error when settled remains low. Then they just hire people to sort out some of the mess and eat some amount of losses as easier than trying to make the system perfect.

The fact is that banks make better use of what databases are and aren't than most designs, because they're very clear about when and where they want consistency (weekdays, 6am Eastern in the US) and the (non-zero!) rate at which the system can be in error when it resolves the state. Any application that can be clear about those can get good usage of databases.


> The estimator used in 1 need not be fully consistent because a) as long as the periods of inconsistency are low (eg, < 10 sec) it won't affect the usage patterns of most accounts

A lot of transactions that need to be reconciled happens without involving 1. e.g. checks and offline card transactions. Even ATMs can often continue to dispense cash when their network is offline, though there arguable 1 applies in the form of maximum withdrawal limits.

But in general I agree with you - their system is built around having accepted, calculated and accounted for the risks rather than try to mitigate every possible risk. Which makes sense because a large proportion of a banks business is to understand and manage risk, and some risks are just happens cheaper to accept than to mitigate via technology or business processes (and the ones that aren't, or aren't any more, are where you'll see banks push new technology, such as making more settlements happens faster and fully electronically)


> I strongly disagree with this. Most of our interaction with the world is perceived consistently. When I touch an object, the sensation of the object and the force I exert upon the object is immediate and clearly corresponds to my actions. There is no perception of reordering my intentions in the real world.

You misunderstand me, and I appreciate that I should have been clearer: A single observer in a single location that is both the sole observer and creator of event will have a consistent view. But this isn't a very interesting scenario - the type of interactions that matter to us are operations that requires synchronisation between at last two different entities separated in space.

Consider if you have two identical objects, on opposite sides of the world, that represents the same thing to you and another person. When you move one, the other should move identically.

Suddenly consistency requires synchronisation limited by the time it takes to propagate signals between the two locations to ensure only one side is allowed to move at once.

This is the cost of trying to enforce consistency. If you lose your network, suddenly both or at least one (if you have a consistent way of determining who is in charge) are unable to at least modify, and possibly read the state of the system, depending on how strict consistency you want to enforce. In this specific case, it might be necessary.

But most cases rather involves multiple parties doing mostly their own things, but occasionally needing to read or write information that will affect other parties.

E.g. consider a forum system. It requires only a bare minimum amount of consistency. You don't want it to get badly out of sync, so that e.g. there's suddenly a flood of delayed comments arriving, but a bit works just fine. We have plenty of evidence in the form of mailing lists and Usenet, which both can run just fine over store-and-forward networks.

Yet modern forum systems are often designed with the expectation of using database transactions to keep a globally consistent view. For what? If the forum is busy, the view will regularly alrady be outdated by the time the page has updated on the users screens.

> Furthermore, this availability vs consistency dichotomy is an overstatement of Brewer's theorem. For instance, linearizable algorithms like Raft and Multi-Paxos DO provide high availability (they'll remain available so long as a majority of nodes in the system operate correctly). In fact, Google's most recent database was developed to be highly available and strongly consistent [2].

They'll remain available to the portion of nodes that can reach a majority partition. In the case of a network partition that is not at all guaranteed, and often not likely. These solutions are fantastic when you do need consistency, in that they will ensure you retain as much availability as possible, but they do not at all guarantee full availability.

Spanner is interesting in this context because it does some great optimization of throughput, by relying on minimising time drift: If you know your clocks are precise enough, you can reduce synchronisation needed by being able to order operations globally if timestamps are further apart than your maximum clock drift, which helps a lot. But it still requires communication.

In instances where you can safely forgo consistency, or forgo consistency within certain parameters, you can ensure full read/write availability globally (even in minority partitions) even in the face of total disintegration of your entire network. But more importantly, it entirely removes the need to limit throughput based on replication and synchronisation speeds. This is the reason to consider carefully when you actually need to enforce consistency.


> You misunderstand me, and I appreciate that I should have been clearer: A single observer in a single location that is both the sole observer and creator of event will have a consistent view. But this isn't a very interesting scenario - the type of interactions that matter to us are operations that requires synchronisation between at last two different entities separated in space.

My point in bringing that up is that applications don't present the appearance of appearance of entities separated in space, that's why weak consistency is unintuitive. We expect local realism from nearby objects the same as we expect behavior in an application that exhibits effects from our immediate actions.

> E.g. consider a forum system. It requires only a bare minimum amount of consistency. You don't want it to get badly out of sync, so that e.g. there's suddenly a flood of delayed comments arriving, but a bit works just fine. We have plenty of evidence in the form of mailing lists and Usenet, which both can run just fine over store-and-forward networks.

Most threaded or log-like things are examples that fare well with weak consistency (because they're semi-lattices with lax concerns about witnessing inconsistency). I'm not saying that there aren't examples where its a tradeoff worth making, I'm saying that many applications are built on non-trivial business logic that coordinates several kinds of updates where it is tricky to handle reordering.

> They'll remain available to the portion of nodes that can reach a majority partition. In the case of a network partition that is not at all guaranteed, and often not likely. These solutions are fantastic when you do need consistency, in that they will ensure you retain as much availability as possible, but they do not at all guarantee full availability.

Weakly consistent databases can't guarantee full availability either when a client is partitioned from them. In fact there's no way to guarantee full availability of internet services to an arbitrary client. I think the importance of network partitions is somewhat overstated by HA advocates anyways. They're convenient in distributed systems literature because they can stand-in for other types of failures, but link failures (and other networking failures, but link hardware failures are disproportionately responsible for networking outages[1]) happen at an order of magnitude less than disk failures.

> In instances where you can safely forgo consistency, or forgo consistency within certain parameters, you can ensure full read/write availability globally (even in minority partitions) even in the face of total disintegration of your entire network. But more importantly, it entirely removes the need to limit throughput based on replication and synchronisation speeds. This is the reason to consider carefully when you actually need to enforce consistency.

I'm all for giving careful consideration to your model. And I agree, there are several cases where weakening consistency makes sense, but I think they are fairly infrequent. Weakening consistency semantics per operation to the fullest extent can add undue complexity to your application by requiring some hydra combination of data stores, or come at a large cost by requiring you to roll your own solution to recover some properties like atomic transactions. You called this "laziness" earlier, but I think this is actually pragmatism.

[1]: http://research.microsoft.com/en-us/um/people/navendu/papers...


>designs that reduces availability.

It's a slimy abuse of language to call a service "available" when it's giving incorrect output and corrupting its persistent state.

I can write a program with lots of nines that is blazingly faster than its competitors if its behavior is allowed to be incorrect.


A lot of the time there is no "correct" output.

E.g. a search result from a search engine is outdated the moment it has been generated - new pages come into existence all the time, and besides the index is unlikely to be up todate.

What matters is whether "close enough" is sufficient, and a large proportion of time it is: E.g. for the search engine example don't care if I get every single page that matched my query right at the moment I pressed submit. I care that I find something that meets my expectations.

Getting "good enough" results and getting them at a reasonable time beats not getting results or waiting ages because the system is insisting on maintaining global consistency.

Yes, there are cases where "good enough" means you need global consistency, but they are far fewer than people tend to think.


> I can write a program with lots of nines that is blazingly faster than its competitors if its behavior is allowed to be incorrect.

Congratulations! You just understood distributed systems, or put more simply the real world of app development.

Most of the programs in the real world ARE allowed to be incorrect at times.

Corollary: Most of the programs in the real world do NOT need to be correct every single time.

These statements should be on top of system design and requirement gathering classes.


Banks are forced to be eventually consistent because of their distributed nature. If it was practical to have a big central ACID database I'm sure that's what they'd use. There is an extra layer of complexity with distributed systems.

That said, if the problem has been thought about, I'm happy. My frustration is with people not understanding the trade-offs.


The problem is that a big central ACID database enforces strict limits on throughput limited at best by the the speed of light (in reality it's worse).

Enforcing a globally consistent views basically requires coordination. Coordination requires that a signal traverses a distance and is acted on no faster than the instruction throughput of a single CPU core.

When the sources of operations are geographically spread out, you will be limited based on the fastest communication possible between those locations in addition to the processing latency.

Ultimately this means that there are scaling limits to throughput of a fully serialised system that causes real problems all the time.

Sometimes a single global truth is still fast enough. Very often it isn't, when e.g. your users are spread out globally. That's why it's important to consider avoiding consistency when it is not necessary.


I guess whether it's overvalued or undervalued depends on whom we're talking about. Your point is well taken, but I think the real problem is that inexperienced programmers want the best of both worlds. In other words, they design systems that require consistency to function correctly because it's easier, but then they do not actually guarantee said consistency, because hey, schemaless is so much faster than boring old SQL.

The way I like to think about it (and mind you, I'm an early-stage kind of guy, so I rarely work on huge systems) is that consistency guarantees give you a lot of leverage, and while it's almost unattainable in the real world, in the computing world we have extremely powerful CPUs and networks that allow communication between any two points on the globe in less than a second. There is frankly a huge number of applications that can be built on an ACID database and never have to worry about the offline or distributed use cases that would require building a proper distributed system.


The bank example is terrible.

The inconsistency is an absolute PITA for the consumer. The delays are are a terrible, terrible user experience, and the consumer has to spend more money in the end.

It's extremely expensive, both in money the bank has to spend on auditing and in the legal framework that has to be put in place.

The bank example is the perfect argument for consistency, not against.


It's a perfect argument for shorter transaction processing times. The bank inconsistency is there because historically it was impossible to guarantee consistency without making the processing times far worse. E.g. want to make a payment via check? Sorry, need to send a courier to the branch holding your account to verify the funds are there, and then afterwards send a courier to inform of the updated account state before another payment would be allowed.

But the shorter you make the processing time, the less benefit you get from introducing consistency requirements, to the point where the thing that is important to address is the remaining cases where processing transactions take time, not the consistency (at least as long as banks also don't take the piss by taking advantage of the lax consistency to charge unwarranted overdraft fees).


This!

There is no single "truth".

Under the unfortunate consequences of the CAP theorem and the speed of light, consistency grinds to a halt as the number of observers increases. Time itself would have to stand still in the limit if at each time step every particle in the universe had to agree on the state of the system.

What we need to preserve is mostly strong "causal consistency", which means that from a single observers perspective, the system "appears consistent".

This article is really good and isn't obscured by NoSQL vs SQL arguments.

http://www.grahamlea.com/2016/08/distributed-transactions-mi...


For the specific case of distributed datastores, by threshold for considering any NoSQL/NewSQL is scaled writes and consistent reads. Designing a system based on consistent reads is much simpler without quirks that need to be explained to end-users.


I think you are saying that consistency matters but a system is more robust with settlement and reconciliation where consistency is like a luxury that could optimize the really robust method


In most cases what matters is very localised consistency - e.g. I want to see the comments I post on a comment page right away - while global consistency matters much less, as long as the system keeps converging (it may or may not be consistent at any given point in time, but the older a change is, the more likely that it appears the same everywhere, basically).


absolute need for consistency is rare, unless deal with anything remotely similar to "financial transactions".

Thankfully the author's premise of "NoSql == inconsistency" is wrong in at least one case. Google Cloud Datastore offers ACID compliance. I use that to great effect (I run a subscription SaaS with usage quotas)


There's a big difference between being availability being predictable & unpredictable - it's called consistency.

Picking a pattern, and sticking to it is fundamental to systems flow.

Yes, have "randomness" is important, but trying to cultivate chaos from utter disorder is a something at the very least historically speaking people are not good at collectively.

Chaos functions best when it's on the edge of a system, not embedded in it.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: