Entirely agree with you, eventual consistency and quorums protocols all have their drawbacks. Difficult of developing against that (and the natural reaction of just "handwaving" the consistency model rather than using optimistic locking with a vector clock at the identified synchronization points) is the main one. I have, however, also seen the situation where developers would fail to understand how to correctly deal with availability and build ad-hoc, poorly thought-out HA layers on top of non-HA systems. The typical memcache + MySQL deployment is the prime example of that: there's neither the high availability, nor an understandable consistency/cache coherence model.
> Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes).
Dynamic repartitioning is also what HBase/BigTable do ("tablet splitting"). The MVCC approach is interesting way around the unavailability window, by the way.
> By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent.
Yes, I did not mean that Megastore is eventually consistent. It is not, it's ACID compliant and uses Paxos. Real question: was this not stated clearly in my comment (I'd like to edit to clarify if it is)?
Megastore, however, is an example of a "NoSQL" system and I meant to use it as proof that not all "NoSQL" systems give up consistency: ACID is orthogonal to the query language.
Quite frankly I find the whole query language debate (and "NoSQL" marketing gimmick) to be quite pointless, the distributed systems and systems programming aspects are a lot more interesting to me.
> Yes, I did not mean that Megastore is eventually consistent. It is not, it's ACID compliant and uses Paxos. Real question: was this not stated clearly in my comment (I'd like to edit to clarify if it is)?
I was interpreting the nod to Megastore in the context of the first sentence, which seemed pretty strongly about eventual consistency. Nothing to worry about, though - your followup comments are very clear!
> Megastore, however, is an example of a "NoSQL" system
Looking at Google's recent papers, they actually seem to be moving more in a SQL-ish direction. Dremel, in particular, appears to be a variant of SQL that allows easier access to nested and/or repeated columns. I thought Megastore was in a similar vein, but at that moment I can't seem to find the link that made me believe that (other than seeing that it uses strongly typed schemas).
Regardless of the benefits of a SQL-like language as an interface, however, I think you have a good point: the underlying architectural decisions about things like consistency or stability are very interesting, but often obscured.
> Note that his "right answer" for performance (not availability, so it's a bit of an aside from your comment) is to dynamically repartition, which is what Clustrix does. Note that we can actually do this type of repartitioning on just the "hot" data, using MVCC to avoid any downtime (unlike Mongo, which blocks all writes).
Dynamic repartitioning is also what HBase/BigTable do ("tablet splitting"). The MVCC approach is interesting way around the unavailability window, by the way.
> By the way, the Google Megastore project I believe you're referring to is not actually eventually consistent in the same way that, say, Cassandra is. Megastore is fully ACID within an entity group, meaning that lowering eventually consistency down to the per-node level was not the solution that Google went with. BigTable is also not eventually consistent.
Yes, I did not mean that Megastore is eventually consistent. It is not, it's ACID compliant and uses Paxos. Real question: was this not stated clearly in my comment (I'd like to edit to clarify if it is)?
Megastore, however, is an example of a "NoSQL" system and I meant to use it as proof that not all "NoSQL" systems give up consistency: ACID is orthogonal to the query language.
Quite frankly I find the whole query language debate (and "NoSQL" marketing gimmick) to be quite pointless, the distributed systems and systems programming aspects are a lot more interesting to me.