
Pick strong consistency whenever possible - rbanffy
https://cloudplatform.googleblog.com/2018/01/why-you-should-pick-strong-consistency-whenever-possible.html
======
iamdave
Slightly off topic but articles like this make me weep as I look at my
infrastructure and think about all the best practices that we by all means
should be following, but aren't because of a senior engineer stuck in the
1990's and is seemingly not only paralyzed by analysis (and bikeshedding), but
seems to subscribe to the notion as a way of living, and management who defers
to him at every turn despite all evidence of how it holds us back (not to
mention directly contributing to outages). I'd give my first born child for
immutable servers.

 _Consistency in database systems refers to the requirement that any given
database transaction must change affected data only in allowed ways. Any data
written to the database must be valid according to all defined rules_

I would _LOVE_ for this to be a thing. Just as soon as I can convince people
that maybe storing my.cnf in VCS is a better idea than updating it in prod.
Live.

Sighs.

~~~
mikekchar
Not really related to your main point, but speaking as a greying "old guy",
one of the things that's really hard is to gracefully allow the next
generation to contribute. I still vividly remember starting out and being
incredulous about some of the stupid ideas the senior guys on the team had. I
was proven right quite a lot of the time too. It's easy to build that inner
narrative of the hero righting wrongs.

But then fast forward to the back end of your career and you've got a kind of
weird dilemma. The vast majority of ridiculous things that the young guys try
to push is absolute crap. They read about it in a blog post somewhere and
don't have nearly enough experience to understand the nuances. Or they have
yet another stupid f-ing framework that they just _know_ is going to
revolutionise the industry. Yeah... been there, done that, my closet is
overflowing with T-shirts, thank you very much.

You sit there thinking, "I spent 20 or 30 years fighting my way through this
crap and when I finally have a good handle on it, these new guys are just
going to throw it all away". But... Dammit. You absolutely know that _one_ of
them has a great idea and you're not quite sure which one of them has it.
Because they _all_ think they have it sorted (even when none of them do). They
feel they are completely sure that "this is the best way and only a moron
couldn't see it". But you know that it's always more subtle than it appears.
If you let them all run with their ideas, then 90% (99%???) will crash hard...
Let's face it -- just like you did when you had that stupid idea that you were
absolutely certain was "the way forward".

Anyway, I belabour the point. From my vantage point, it's pretty difficult to
let go. Some people just can't do it. In my position now, I have a _lot_ more
sympathy for the "old guard" than I did when I was younger. To be honest, I
think it is technically impossible for me to have less sympathy than I did --
and I think this is pretty common in the industry. Probably both sides of the
equation are in a similar situation -- neither side feels comfortable letting
the other side make decisions.

Just talking out of the hole in my head right now, but I think the main thing
is that programmers generally like to program. They like to have freedom to
try approaches that they think will be successful. As much as possible, it's
probably best to allow your teammates, whoever they are, to have the freedom
that they need. At the same time you should expect to have your own freedom.
Getting the balance is not easy and sometimes you have to change jobs in order
to get into a team where you are allowed the freedom to do your best. In my
experience, teams that take this seriously are the best teams to work on (even
if they sometimes do stupid things).

~~~
bluesnowmonkey
This is something I think about a lot nowadays. You have to try and fail a
_lot_ to develop an intuition for good design. How do you give people the room
to experiment without paying for all the mistakes? It is tremendously
expensive to produce a senior developer yet vital to do so because junior
developers mostly produce crap. We could only assign them projects off the
critical path, but we can't really afford to have anyone working on anything
but the critical path. Plus it's a lot to ask of someone's ego to say, here
work on things that don't matter for 5 or 10 years and we'll talk.

~~~
mikekchar
I was randomly watching Youtube the other day and there was an ad for Herbie
Hancock's Masterwork class (ha ha -- not a referral, no idea if it's any
good). Anyway, in the ad he mentioned that he was playing with Miles Davis one
time and that he made what was technically a mistake. Miles listened to it,
picked it up and made it right. Suddenly I realised: "That's it! That's what I
want to do." But, of course, it's much easier said than done. I won't pretend
that I'm able to do it yet :-)

------
jacobparker
Related to the overall topic but not Spanner specifically: it's interesting to
compare the consistency models of S3, GCS and similar offerings:

* S3: [https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction...](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)

* GCS: [https://cloud.google.com/storage/docs/consistency](https://cloud.google.com/storage/docs/consistency)

* Azure: exercise left to the reader

The best usages of S3 I've seen are designed to only use the one major form of
strong-consistency that S3 guarantees: if you write a key that has never been
written to or read from, and then do a read, it will always work. Sticking to
this guarantee will avoid nasty difficult/impossible-to-diagnose bugs. To take
advantage of this you do things like:

A. Use unpredictable keys (e.g. prefix with a GUID) to guarantee that you
don't get EC behaviour, communicate those keys to other services/users after
the upload has succeeded B. Store immutable/content-addressable data. Use new
keys for new versions of the data (don't ever over-write a key.) If necessary,
use an external (strongly-consistent) index to map identifiers to S3 keys. C.
Avoid "upload if not exists" in favour of just uploading (e.g. in the case of
a possible failure just reupload, don't check if the key exists on S3. Uploads
are atomic.)

GCS offers a far greater amount of strong-consistency and this allows makes it
much more flexible.

The key to scalability/performance/simplicity of services like S3 is lack of
transactions between keys, not eventual consistency. It's informative to start
from Paxos and figure out how you'd build an S3 and why it might exhibit EC
behaviour like S3 (hint: a caching front-end) and question what value that
really brings vs. the extreme decrease in usability (consider that a caching
front-end doesn't give you scale-out in the way that lack of cross-key
transactions does.)

~~~
k__
How to get rid of old data if you (basically) treat S3 as append-new-files-
only-store?

~~~
jacobparker
That really depends on the application, who has access to the keys, how they
learn about the keys etc. There isn't a one size fits all solution and that's
kinda why eventually consistency is a pain in the ass.

The "I want strong consistency but have to use S3 because its the cheapest way
to store stuff and its great at serving reads and just otherwise great" option
is to maintain an external index that maps between persistent ids (e.g. file
paths) and S3 keys. We use this for hosting user content at my company. Our
index is in MS SQL of all things ;)

The MVP option is to not to delete stuff. :) This can work well in some
scenarios (e.g. you want to keep all versions around anyway) but be careful to
not dig a hole that's hard to get out of.

A more esoteric option is lifecycle based cleanup. This can work well for
things like a build artifact cache (where an expired key isn't a big deal
because you can regenerate it.)

Some of this is discussed in this blog post by Netflix:
[https://medium.com/netflix-techblog/s3mper-consistency-in-
th...](https://medium.com/netflix-techblog/s3mper-consistency-in-the-
cloud-b6a1076aa4f8) (caveat: s3mper is dead(?) and there is an AWS-native
solution for HDFS now.)

~~~
mjn
Possibly an obvious comment, but reading your post it struck me that any way
you slice it, you're more or less building a garbage-collection system, with
all the various tradeoffs those entail. Different performance issues since
it's S3 buckets rather than RAM, but still very dependent on who has
references to what, and when you check for it. Your two examples sound to me
like: You can have some set of known exhaustive roots and periodically delete
anything not still reachable from them. Or you could have a lifecycle option,
analogous to arenas (region-based memory management). Etc.

------
awinder

      For example, a financial application might 
      need to show users' account balances. When 
      users make a deposit, they want to see the result 
      of this deposit reflected immediately when 
      they view their balance (otherwise they 
      may fear their money has been lost!). There should 
      never appear to be more or less money in 
      aggregate in the bank than there really is. 
    

Why does this example continue to come up time and again, banking absolutely
doesn’t work this way. We feel like it should but deposits / transfers can
take days to post, bank balances can absolutely reflect “incorrect” values,
etc.

~~~
kuschku
Because banking in some couuntries actually does work like that.

In Europe, since November 2017 banks are moving towards the SEPA Instant
Credit Transfer system, which allows transfers in under 10 seconds, between
all banks.

So if you’re today using a European bank, your transfers might actually work
like this.

~~~
jnordwick
It could be under 100 ms, but that still doesn't tell you the consistency
model.

~~~
kuschku
Requiring by law transactions to complete (and show up!) in under 10 seconds,
EU-wide, provides some requirements for a consistency model, though, that
eventually consistent models can not provide.

------
mcguire
" _A strongly consistent solution for your mission-critical data_

" _For storing critical, transactional data in the cloud, Cloud Spanner offers
a unique combination of external, strong consistency, relational semantics,
high availability and horizontal scale. Stringent consistency guarantees are
critical to delivering trustworthy services. Cloud Spanner was built from the
ground up to provide those guarantees in a high-performance, intuitive way. We
invite you to try it out and learn more._ "

Is this a blog post or an advertising whitepaper?

------
vog
From the article:

 _> To quote the original Spanner paper, “we believe it is better to have
application programmers deal with performance problems due to overuse of
transactions as bottlenecks arise, rather than always coding around the lack
of transactions.”_

Isn't this a strong argument against the current Microservices trend?

~~~
imtringued
Microservices are about organisational structure, not performance. The primary
difference over modularising via libraries is independent release cycles.
Every time you update your libraries you also have to update all your
applications even if the change is backwards compatible. Microservices avoid
this problem.

~~~
fyi1183
That sounds wrong.

First of all, backwards compatible changes in libraries don't need application
updates, by definition.

Second, if your library changes are backwards compatible, you can maintain
separate release cycles - this is the normal way of doing things, actually.

Third, microservices simply don't allow backwards incompatible changes. This
is a big hint that perhaps your libraries shouldn't do backwards incompatible
changes either, as a matter of policy, even though doing them is theoretically
possible.

Perhaps you meant deployment instead of releases, and were assuming static
linking?

~~~
PeterisP
Probably what the parent post means by "backwards compatible changes" is that
the app doesn't need any changes to use the new behavior of the library, but
it needs to be rebuilt/repackaged/redeployed to get that new behavior.

An often seen scenario is that small changes (a one-line bug fix, a tweak in
behavior) in application functionality happen to be in the shared library
code, so to get that fix to production, you need to redeploy _all_ apps after
the library change. Microservices allow you to redeploy just that single
shared service.

~~~
vog
_> happen to be in the shared library code, so to get that fix to production,
you need to redeploy all apps after the library change_

Doesn't this simply mean that the deployment process is somewhat broken?

On a typical Linux distro, you run "apt-get" or "yum" to upgrade your
Heartbleed-fixed OpenSSL, the package manage automatically restarts all
affected daemons, and you're done with it.

Even if you deploy static images, a good build system should just re-run the
linker after the library change and be done with it.

~~~
PeterisP
We seem to be talking about quite different things - in the context of
discussion about monolith vs microservice architectures, the expected (at
least to me) context is deployment of in-house applications within an
enterprise.

This means that even on Linux distro's the packages generally are not managed
by apt-get or yum. This means that "shared libraries" don't necessarily mean
Linux/C shared libraries, but are likely to be packages of
Java/PHP/C#/whatever code that are re-used by multiple applications. This
means that, for starters, you need to choose when and how you'll push this
update to many different machines each running different applications. This
means that "automatically restarting all affected daemons" isn't trivial -
some of these services may have state (e.g. a deployment that also includes a
change to a database), for some of them the downtime matters, you likely don't
want to stop&restart all of the instances doing the same thing at once but
rather do a rolling deployment. This means that some of the instances that
need to be updated are out of your control/administration - you need to send
the package to some other organization so that _they_ can re-deploy it. Etc,
etc.

Sure, a good build system should just re-run the linker after the library
change and be done with it - that solves the packaging part. Deployment often
is much trickier; some organizations have a good, robust deployment process
but many (most?) don't.

------
ratsimihah
Am I the only one reading this on a mobile? What?

~~~
ryukafalz
Ironically for an official Google blog... it renders fine on Firefox for
Android, but not Chrome.

~~~
SerLava
Firefox, Chrome and Safari all broken on iOS.

Google could catch this automatically...

------
claar
What is the AWS equivalent to Cloud Spanner?

~~~
jamesblonde
There is none. DynamoDB would be the closest - it's a HA DB for OLTP workloads
(but without the transactions). AWS do not have the TrueTime API that Spanner
builds on to provide globally consistent transactions.

AWS's storage offering S3 is pretty limited in terms of its consistency model:
[https://medium.com/@jim_dowling/reflections-
on-s3s-architect...](https://medium.com/@jim_dowling/reflections-
on-s3s-architectural-flaws-71f14c05a5fa)

In contrast, Google's filesystem (yes, not an object store, but a filesystem)
is hardly known - Collosus. It provides a HA consistent metadata layer to a
distributed filesystem. The closest open-source architecture is HopsFS
(distributed metadata for HDFS) - disclosure I work on Hops.

~~~
elvinyung
I would add to this by saying that AWS Aurora "looks" the most similar to
Spanner by being a transparent scale-out relational database with strongly-
consistent transactionality, but it doesn't seem to be designed for geo-
replication.

~~~
nishantvyas
Agree. AWS aurora is closest to spanner and would work for 95% of use-cases...
master-master, cross-colo or geo-replication is great problem for less than 2%
of companies...

~~~
cosmie
Multi-master Aurora is actually in Preview right now[1]. Only for the MySQL
variant of Aurora though, not the Postgres variant.

[1] [https://aws.amazon.com/about-aws/whats-new/2017/11/sign-
up-f...](https://aws.amazon.com/about-aws/whats-new/2017/11/sign-up-for-the-
preview-of-amazon-aurora-multi-master/)

------
elvinyung
How does Spanner stay competitive with other NewSQL databases? Doesn't the
pricing and the API lock-in mean that people will just consider open-source
NewSQL databases first?

(Not a shill for any company, honestly just curious.)

~~~
donaldihunter
Spanner is Google's private in-house database so it's not competing with
NewSQL vendors.

Edit: Late to the news that Google is selling its spanner warez.

Well it's been in the game longer than anyone else and is battle tested by
Google.

~~~
elvinyung
> so it's not competing with NewSQL vendors.

But it... literally is?
[https://cloud.google.com/spanner/](https://cloud.google.com/spanner/)

------
polskibus
Great to see a come back to old values some of us been taught before the NoSQL
trends have gained strength.

One can only hope Google contributes some of their strong consistency
mechanisms to open source so that more developers can be swayed towards strong
consistency-based architectures. Spanner is great but creates huge vendor
lock-in and some businesses will not repeat old mistakes.

------
moxious
Well sure. Consistency is great and for years most DBs people really used gave
you all of the ACID guarantees. Then, huge scar came along. At extreme scale,
sometimes you have to trade things for speed. But that extreme scale is a
problem that frankly most people don't have.

So don't trade off a good attribute unless it's absolutely necessary. Seems
sensible

~~~
mjn
It's really hard to convince people they don't have (and aren't likely to
have) a Facebook-sized scale problem though, even if the vast majority of
people really don't. It's like trying to convince people they don't have "big
data". :-)

~~~
acdha
This bridges over to another thread
([https://news.ycombinator.com/item?id=16156972](https://news.ycombinator.com/item?id=16156972))
about the problem with many IT people thinking of their job as playing with
cool technology rather than business outcomes. I’ve run into even a fair
number of high-ranked people who habitually over-engineer things because
they’ve tied their ego to the wrong metric, and many places didn’t have
corrective pressure against that.

Hosted services have been good for counter-pressure: if your oracle cluster
can be replaced with a $50/mo RDS instance, it’s a lot harder to hand-wave
around the cost differential.

------
cube2222
Well, as a question fairly on topic:

Has anyone successfully used CockroachDB in production?

As it seems to be the only viable alternative to spanner at the moment.

~~~
humanfromearth
We use it as an insert only db for now for incoming webhooks which allows us
to de-duplicate events and significantly reduce writes to the main PG db.
We're running it on GKE which has been much easier to setup than PG.

There is also TiDB which is similar in some ways to CRDB:
[https://pingcap.com/en/](https://pingcap.com/en/)

~~~
cube2222
Have you had any major issues with it? It yes, how'd you dealt with them?

~~~
humanfromearth
No major issues. If you're moving from a single node PG to CRDB you should
expect CRDB to be slower because of all the consensus stuff. However almost
every update I see from them contains some improvements in that area so it
will get better with time probably. You just need to figure our if you're
willing to accept a 2ms - 1m (PG) query execution vs 20ms flat execution
(CRDB) - even after all the optimizations.

------
planetjones
I tried loading this on mobile safari and it looks ridiculous. The last cell
of the table is very narrow and the text piles up vertically. For a company
like Google the mess they make of how their pages display on mobiles is
astonishing.

------
ece
Global ordering is key, as briefly mentioned. To me, the proven distributed
hash table systems like Bigtable, Spanner, HBase, RocksDB have it, and those
that don't have it, aren't as strongly consistent (Riak, Cassandra, Mongo,
etc..)

Having global ordering lets you reason about where your data needs to go,
where it needs to be, and overlay different access patterns to see what will
work. Heavy write load? Have more shards. Heavy read load? replicate each
shard more. Random read/write? Nothing will get you as close to uniform load
distribution as global ordering.

------
kev009
I am always a bit surprised that HBase/Phoenix aren't more widely celebrated
in the scale out database scene vs Cassandra, MongoDB, etc but a lot of people
designing new systems as users of these DB systems have NFC what they are
doing and I guess are swayed by marketing stickers and tshirts? G's
BigTable/Spanner if you can afford to outsource..

~~~
dboreham
To be fair: these are three very different kinds of store, designed to meet
the needs of very different workloads, and trying to solve very different
problem sets.

~~~
kev009
Not the way I see them used. HBase is quite versatile, Facebook's HydraBase is
a good example of doing things people think they need Cassandra for
[https://code.facebook.com/posts/321111638043166/hydrabase-
th...](https://code.facebook.com/posts/321111638043166/hydrabase-the-
evolution-of-hbase-facebook/). It's also telling they as the creator abandoned
Cassandra, but C* has some very niche conceivable uses (eager replication
cache for metadata?). Mongo is just stupid.

------
jedberg
I used to agree until I was forced to use an eventually consistent system.

Both have their place, but I think you get better architected, more reliable
software with an eventually consistent system.

In and eventually consistent system, the programmer must consider what to do
in the case of data failure. I think this is a good thing. When building a
user authentication system, who better to decide what to do when you can't
access the user database than the person writing the user auth system? Should
it fail open or closed? Well if you're a bank, you should probably fail
closed. If you're Netflix? Fail open. The programmer should know the business
case better than the database, which will alway fail closed.

Sure, in a consistent system the programmer _could_ think about that, but they
aren't forced to, which I think is good.

Also, it means that joins must be done in your application. Sometimes its good
to have the programmer deal with that too, because it means they will have to
really consider what data they need and be much more familiar with how much
data is needed to create the join. If you're just relying on the database to
do the joins, it might be hiding a lot of load from you, especially if you
inadvertently create multiple table scans. In an eventually consistent system,
you _know_ if you created a table scan, because you have to pull back a full
table's worth of data (or not, because you saw that issue and fixed it).

I just feels like the programmer gets better insight into what is happening to
the data and can make better decisions on what to do when they are closer to
the business cases the software is being built for.

~~~
cookiecaper
I agree with the principle that good development practices will try to get the
coder to think through the potential failure modes of the code they've
written.

Unfortunately, I think you've come to the opposite conclusion in this case.
When jedberg writes the code for a system backed by an eventually-consistent
data store, he asks "What happens if this value is stale and how can I
structure the code to cope with that in a sane and reasonable way?"

But when $Random_Dev writes the code for such a system, he asks some
combination of "What does eventually consistent mean?", "What do you mean I
won't get back the right data?", and "Isn't that the DBA's problem?"

The core issue being that the average dev will punctuate this line of
questioning with things that shift the blame elsewhere; if not the DBA, then
"Well, it's good enough for [INSERT_TRENDY_COMPANY_HERE] [ reddit? ;) ], so I
think we'll be OK!" or "I don't think that will ever happen to us, our reads
get consistent quickly enough". Just whatever will punt the issue down the
road.

To be honest, I believe that this impulse is the impetus behind the rise in
NoSQL in the first place. Devs resent DBAs and SQL. They, very simply, want a
place to shove their data, a place to read it out, and to not know anything
else about the in between. I have seen many pitiful architectures that clearly
turned on this consideration (and we're seeing it all over again with cloud
orchestration).

Good developers will dig in and search out the meaning of the data whether the
failure case is handled at the DB level or not.

Highly-structured systems like a traditional RDBMS are very good for
resiliency because they require _someone_ to put some definition and formality
around the dataset to get basic functionality, and they are generally
extremely careful about ensuring the things that come in and out of them are
safe and accurate. That's the kind of backing that we really need, especially
when there are _many_ developers who can't be trusted to even try to
understand more than the bare minimum necessary to get a positive quarterly
review.

~~~
jedberg
Your point is valid, but it makes me sad.

We basically come from opposite viewpoints -- I think most devs will do the
right thing, you think most will do the wrong thing.

Honestly, you're probably right and I'm probably wrong, but I like to hope
that I will always work in a place where I'm right. :)

(It was Netflix BTW, not reddit. For the most part reddit used strongly
consistent databases (Postgres), but we did use some eventual consistency in
the caching layer).

~~~
lulmerchant
I have a slightly different perspective. I think defaulting to NoSQL is simply
removing a mature toolset from the developers kit, that they would have a lot
of great use cases for.

If you use it in the wrong places, then that almost certainly means increases
complexity where you don't necessarily have to. Whether or not that results
other positive behaviours seems a bit beside the point to me.

If you managed a team of rock climbers, you could say "I prefer it when my
rock climbers don't use safety ropes. It improves their climbing, because if
they make a mistake, they'll fall to their death". It could be 100% true, but
it doesn't mean you shouldn't simply be picking the right tools for the job.

~~~
AlexCoventry
Yeah, the dichotomy in this conversation is strange.

------
arnon
> "Cloud Spanner, in particular, provides external consistency, which provides
> all the benefits of strong consistency plus serializability."

What Google call 'external consistency' is what's known by relational theory
people as 'isolated serializability'.

------
lubesGordi
So the gist is that they can get better transaction throughput because they
serialize transactions with a distributed clock rather than by locking? Isn't
there also usually a query latency trade-off with higher consistency?

------
djur
This post would have benefitted from either explaining what Cloud Spanner is
or linking to such an explanation in the first paragraph.

~~~
aiiane
[https://cloud.google.com/spanner/](https://cloud.google.com/spanner/) fwiw.

------
marknadal
This article is very resourceful (I'll be returning to it again and again) but
no no no, its opinion is fundamentally flawed.

Ignoring the fact they obviously have a strongly-consistent product to sell,
which therefore informs their view, the flaw is that the article is the cliche
"I have a hammer therefore everything is a nail".

Honestly, if you go back to the beginning of the article, their first quote
sums it up "better to have programmers deal with performance problems ... as
bottlenecks arise" admits to the fact that this is the inevitable outcome of
strongly consistent systems!

If physics is against it, you're asking for trouble. I've given tech talks
around the world on this, just don't pick a fight against physics or else YOU
WILL LOSE FAITH in any/all database systems you use:
[http://gun.js.org/distributed/matters.html](http://gun.js.org/distributed/matters.html)
.

That is what these strongly consistent database systems run off of, faith
(religion) that the database is their god that will solve all their problems.
You only need to take a look at [http://jepsen.io/](http://jepsen.io/) to be
proven that basically every single system out there (except for like RethinkDB
and one or two others) fail miserably at being strongly consistent even when
they are suppose to be.

But hey, they need an excuse to get people to buy into Cloud Spanner - and
nothing sounds better than "Hey we're Google, let us manage your data for you,
because you aren't smart enough to deal with consistency."

~~~
dgacmu
I don't believe it's religion - it's experience. Google, like Amazon,
initially started building scale-out systems by sacrificing strong consistency
and transactions. Amazon did it with Dynamo, and Google did it with BigTable.
Over time - and with very substantial engineering investment - Google has
started walking that back, first with Megastore, and now with Spanner.

What we're seeing happen is a reinvention of a lot of classical transactional
systems from a "massive scale-first" perspective instead of a "local
transactions on spinning disk fast" perspective. The eventually and causally-
consistent systems have something to add, but I don't think it's wise to
discount Google's years of engineering experience in this as religion. Rather,
it's reflective of starting to operate at a number of engineers scale at which
it has become worthwhile investing very dedicated engineering and research
effort into supporting a transactional model, at scale, so that thousands of
other engineers can be more productive.

There's a chance this experience doesn't generalize -- recall that Google has
massive amounts of dedicated inter-datacenter bandwidth with good management
and redundancy, the ability to hire dedicated teams of database & distributed
systems experts, etc. But it probably generalizes to the other giant companies
in the field -- Microsoft, Amazon, Facebook, and Baidu have similarly huge
numbers of "non-distributed-systems-expert developers" who need to get stuff
done, correctly, and the ability to invest heavily in the infrastructure to
make them more productive.

Another way to summarize it is: At some point, you're going to have to fix
problems at the application and algorithm level, and not just hope that the
underlying storage system makes everything magic. It's easier when those
problems are _performance_ problems than when they're _correctness_
/consistency problems.

(Disclaimer: This is my view as a distributed systems professor, not a one-
day-per-week Google employee, but I do have that second hat around.)

