
Introducing Cloud Spanner, a Global Database Service - wwilson
https://cloudplatform.googleblog.com/2017/02/introducing-Cloud-Spanner-a-global-database-service-for-mission-critical-applications.html
======
ChuckMcM
Congratulations to the Spanner team for becoming part of the Google public
cloud!

And for those wondering, this is why Oracle wants billions of dollars from
Google for "Java Copyright Infringement" because the only growth market for
Oracle right now is their hosted database service, and whoops Google has a
better one now.

It will be interesting if Amazon and Microsoft choose to compete with Google
on this service. If we get to the point where you have databases, compute,
storage, and connectivity services from those three at equal scale, well that
would be a lot of choice for the developers!

~~~
snikch
Amazon's Aurora databases seem to be solving the same problem, and are MySQL
or Postgres compatible to boot.

~~~
johnsmith21006
It is not close to equivalent. But I do want to get a better feel for if
Google really has figured how to do basically the impossible. I want to see if
this truly scales horizontally but of it does then competitors better hope for
a much more detailed paper :)

~~~
jack9
> It is not close to equivalent.

It's equivalent, with different (unknown) constraints. Aurora is specifically
for scaling workloads in the same way. You can say it's horizontal (machine)
over vertical (resource) but it's all a matter of accounting.

The big nono is the Spanner pricepoint. I will stick with Aurora for scaling
based on traffic I use, over pricey timeslices.

You would have to have quite a load to justify the switch from cheaper de jour
solutions right now (AWS). Relying on the few that do, is a risk.

------
suprgeek
Really a CP system but with the Availability being five 9s or better (less
than one failure in 10^6)

How: 1)Hardware - Gobs and Gobs of Hardware and SRE experience

"Spanner is not running over the public Internet — in fact, every Spanner
packet flows only over Google-controlled routers and links (excluding any edge
links to remote clients). Furthermore, each data center typically has at least
three independent fibers connecting it to the private global network, thus
ensuring path diversity for every pair of data centers. Similarly, there is
redundancy of equipment and paths within a datacenter. Thus normally
catastrophic events, such as cut fiber lines, do not lead to partitions or to
outages."

2) Ninja 2PC

"Spanner uses two-phase commit (2PC) and strict two-phase locking to ensure
isolation and strong consistency. 2PC has been called the “anti-availability”
protocol [Hel16] because all members must be up for it to work. Spanner
mitigates this by having each member be a Paxos group, thus ensuring each 2PC
“member” is highly available even if some of its Paxos participants are down."

~~~
jpmattia
> with the Availability being five 9s or better (less than one failure in
> 10^6)

Anyone know how exactly this is defined for them? (Time? Queries? Results?)

~~~
the_cap_theorem
Well, this is what the spanner website says if you read the fine print:

>> This feature is not covered by any SLA

So I would guess that you don't get _any_ guarantess. Not five nines and not
even one nine.

~~~
ecnahc515
Currently yes, but that's because it's in Beta.

------
tedd4u
The team here at Quizlet did a lot of performance testing on Spanner with one
of our MySQL workloads to see if it's an option for us. Here are the test
results: [https://quizlet.com/blog/quizlet-cloud-
spanner](https://quizlet.com/blog/quizlet-cloud-spanner)

~~~
koolba
What's the SQL and wire compatibility level? MySQL?

EDIT: Found quite a bit of my answers in your linked article:

> Cloud Spanner uses a SQL dialect which matches the ANSI SQL:2011 standard
> with some extensions for Spanner-specific features. This is a SQL standard
> simpler than that used in non-distributed databases such as vanilla MySQL,
> but still supports the relational model (e.g. JOINs). It includes data-
> definition language statements like CREATE TABLE. Spanner supports 7 data
> types: bool, int64, float64, string, bytes, date, timestamp[20].

> Cloud Spanner doesn't, however, support data manipulation language (DML)
> statements. DML includes SQL queries like INSERT and UPDATE. Instead,
> Spanner's interface definition includes RPCs for mutating rows given their
> primary key[21]. This is a bit annoying. You would expect a fully-featured
> SQL database to include DML statements. Even if you don't use DML in your
> application you'll almost certainly want them for one-off queries you run in
> a query console.

> Though Cloud Spanner supports a smaller set of SQL than many other
> relational databases, its dialect is well-documented and fits our use case
> well. Our requirements for a MySQL replacement are that it supports
> secondary indices and common SQL aggregations, such as the GROUP BY clause.
> We've eliminated most of the joins we do, so we haven't tested Cloud
> Spanner's join performance.

This seems like it'd prevent any kind of easy switch over to Spanner.

~~~
deesix
Just to be clear, the JOINS were removed for the vertical sharding prior to
looking at Cloud Spanner. Cloud Spanner fully supports complex JOINS of many
times (e.g. INNER, OUTER)

Details - [https://cloud.google.com/spanner/docs/query-syntax#join-
type...](https://cloud.google.com/spanner/docs/query-syntax#join-types)

Disclaimer: I work on Cloud Spanner

~~~
koolba
Sorry for the confusion but I meant the DML portion.

~~~
MichaelGG
It sounded like you can only modify by primary key? Can you make a transaction
that contains a query and a bunch of updates by PK ?

And yeah it makes it sound like writing an OEM adapter will be much more
difficult.

~~~
obstinate
From reading the docs, the answer seems to clearly be yes, but I'm open to
being corrected.

------
jedberg
This release shows the different philosophies of Google vs Amazon in an
interesting way.

Google prefers building advanced systems that let you do things "the old way"
but making them horizontally scalable.

Amazon prefers to acknowledge that network partitions exist and try to get you
to do things "the new way" that deals with that failure case in the software
instead of trying to hide it.

I'm not saying either system is better than the other, but doing it Google's
way is certainly easier for Enterprises that want to make the move, and why
Amazon is starting to break with tradition and release products that let you
do things "the old way" while hiding the details in an abstraction.

I've always said that Google is technically better than AWS, but no one will
ever know because they don't have a strong sales team to go and show people.

This release only solidifies that point.

~~~
user5994461
Amazon: Create usual services and sell them.

Google: Make unique products that push the boundaries of what was previously
thought possible.

Amazon: Don't care about inefficiencies and usage. Inefficiencies can be
handled by charging more to the clients, usage doesn't matter because the
users are mostly the clients and they don't feel their pain.

Google: Had to make all their core technologies efficient, performant,
scalable and maintainable or they couldn't sustain their business.

~~~
edblarney
Not fair.

Amazon: IaaS

Google: PaaS

Amazon is philosophy is being 'close to the metal' to allow Enterprise
customer to migrate 'regular apps' into a 'regular environment' in the cloud.

Most of Google's offerings are (at least were) novel, but proprietary ways of
doing specific things.

Amazon is not a laggard: they have provided a number of interesting and useful
'helper' things to facilitate IaaS - as well as a number of 'pure cloud' type
things.

Amazon is very, very customer focused. Their products come from customer
demands.

Google often 'cool things they've done internally' and exposes them, hoping
that they might have some use-case in the rest of the world.

Google and Amazon are equally interested in profit.

~~~
computerex
Google is IaaS, their PaaS offering (App Engine) never gained much traction
AFAIK. I also find the comparison fair, Google is a software engineering
company, Amazon is a sales/marketing company.

~~~
Xorlev
At what point do infrastructure services become the platform? IMO, between
GKE, Spanner, BigQuery, etc., it's basically a PaaS for non-trivial
applications.

~~~
computerex
[https://cloud.google.com/docs/compare/aws/compute#iaas_compa...](https://cloud.google.com/docs/compare/aws/compute#iaas_comparison)

------
richdougherty
Some interesting stuff in
[https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...](https://cloud.google.com/spanner/docs/whitepapers/SpannerAndCap.pdf)
about the social aspects of high availability.

1\. Defining high availability in terms of how a system is used: "In turn, the
real litmus test is whether or not users (that want their own service to be
highly available) write the code to handle outage exceptions: if they haven’t
written that code, then they are assuming high availability. Based on a large
number of internal users of Spanner, we know that they assume Spanner is
highly available."

2\. Ensuring that people don't become too dependent on high availability:
"Starting in 2009, due to “excess” availability, Chubby’s Site Reliability
Engineers (SREs) started forcing periodic outages to ensure we continue to
understand dependencies and the impact of Chubby failures."

I think 2 is really interesting. Netflix has Chaos Monkey to help address this
([https://github.com/Netflix/SimianArmy/wiki/Chaos-
Monkey](https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey)). There's
also a book called Foolproof
([https://www.theguardian.com/books/2015/oct/12/foolproof-
greg...](https://www.theguardian.com/books/2015/oct/12/foolproof-greg-ip-
review-biggest-risk-is-safety)) which talks about how perceived safety can
lead to bigger disasters in lots of different areas: finance, driving, natural
disasters, etc.

~~~
fritzo
> perceived safety ... driving

I became a way better winter driver when I started intentionally fishtailing
in snow and ice (in low risk situations).

~~~
richdougherty
There's also research showing that removing safety features, e.g. white lines
between opposing lanes, increases safety.

[http://www.bbc.com/news/uk-35480736](http://www.bbc.com/news/uk-35480736)

------
karlding
I wonder how this will affect adoption of CockroachDB [1], which was inspired
by Spanner and supposedly an open source equivalent. I'd imagine that Spanner
is a rather compelling choice, since they don't have to host it themselves. As
far as I know, CockroachDB currently does not support providing CockroachDB as
a service (but it is on their roadmap) [2].

[1] [https://www.cockroachlabs.com/docs/frequently-asked-
question...](https://www.cockroachlabs.com/docs/frequently-asked-
questions.html)

[2] [https://www.cockroachlabs.com/docs/frequently-asked-
question...](https://www.cockroachlabs.com/docs/frequently-asked-
questions.html#does-cockroach-labs-offer-a-cloud-database-as-a-service)

~~~
bdarnell
(Cockroach Labs CTO here)

Google launching Spanner is generally a positive thing for our industry and
our product. It's more proof that what we're aiming for is possible and that
there's demand for it. We expect that in five years, all tech companies will
be deploying technology like ours.

One of the big differences is that Spanner only uses SQL for read-only
operations, with a custom API for writes. We use standard SQL for both reads
and writes, which means we also work with major ORMs like GORM, SQLAlchemy,
and Hibernate (docs should be live today or tomorrow). Spanner's custom write
API will make it difficult to work with existing frameworks, or to convert an
existing application to Spanner.

Cloud Spanner only works on Google Cloud and is a black-box managed service.
CockroachDB is open source and can be run on-prem or in any cloud on commodity
hardware. (We don't offer CockroachDB as a service yet, but may in the future)

At this point, both products are still in beta and are still missing features
like back-up and restore (according to the Quizlet blog post). We plan to
launch CockroachDB 1.0 with back-up / restore enabled.

* For anyone wanting to know more about how we make CockroachDB work without TrueTime, see our blog post: [https://www.cockroachlabs.com/blog/living-without-atomic-clo...](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/)

~~~
behrangsa
Would Cockroach 1.0 comply with SQL:2011?

~~~
bdarnell
(CockroachDB CTO here) We haven't implemented everything in the standard yet
(Nor will we by 1.0 - there's a lot of stuff there!), but we are aiming to
ultimately be compliant with the SQL standard. For example, when we introduced
"time travel queries" ([https://www.cockroachlabs.com/blog/time-travel-
queries-selec...](https://www.cockroachlabs.com/blog/time-travel-queries-
select-witty_subtitle-the_future/)) we adopted the SQL-standard syntax "AS OF
SYSTEM TIME" (as opposed to the non-standard out-of-band parameter used in
Cloud Spanner)

------
sudhirj
For those trying to compare this with AWS Aurora, Aurora is more a regular
database (MySQL / Postgres) engine with a custom data storage plugin that's
AWS/ELB/SSD/EFS-aware. Because of this the database engine can make AWS
specific decisions and optimizations that greatly boost performance. It
supports master-master replication in the same region, master-slave across
regions.

Global Spanner looks like a different beast, though. It looks like Google has
configured a database for master-master(-master?) replication, across regions
and even continents. They seem to be pulling it off by running only their own
fiber, each master being a paxos cluster itself, GPS, atomic clocks and lot of
other whiz-bangery.

~~~
sudhirj
From the technical blog post

> Does this mean that Spanner is a CA system as defined by CAP? The short
> answer is “no” technically, but “yes” in effect and its users can and do
> assume CA. The purist answer is “no” because partitions can happen and in
> fact have happened at Google, and during some partitions, Spanner chooses C
> and forfeits A. It is technically a CP system. However, no system provides
> 100% availability, so the pragmatic question is whether or not Spanner
> delivers availability that is so high that most users don't worry about its
> outages. For example, given there are many sources of outages for an
> application, if Spanner is an insignificant contributor to its downtime,
> then users are correct to not worry about it.

Basically, the underlying system is CP, but A is so high (because of the
custom fiber, paxos etc) that they're rounding it off to 100% and calling it
CAP.

~~~
zzzcpan
Except that A in CAP has nothing to do with overall system's availability over
time and using it as such is just confusing.

~~~
sudhirj
But it makes sense in this case. The system guarantees CP. But as customer it
looks like you're getting CA as well, because A is so high. If you drink the
kool-aid, you get C & A & P.

The kool-aid isn't too bad, though if they can measurably guarantee A >
99.999999%, I'm happy to round off to 100% and call it CAP.

~~~
thesandlord
The availability is "only" 99.999%, which IMO is still really high!

(I work for Google Cloud)

~~~
the_cap_theorem
Where does it say that? Does google cloud spanner guarantee three nines
uptime?

On the spanner page it says:

>> This feature is not covered by any SLA

;) ;)

~~~
thesandlord
Once Cloud Spanner leaves beta, it will be covered by an SLA. GCP Alpha/Beta
products don't have SLAs for the most part.

------
cipherzero
The white paper is available here:
[http://static.googleusercontent.com/media/research.google.co...](http://static.googleusercontent.com/media/research.google.com/en/us/archive/spanner-
osdi2012.pdf)

for anyone interested

~~~
cipherzero
Oh and the newer white paper from today:
[https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...](https://cloud.google.com/spanner/docs/whitepapers/SpannerAndCap.pdf)

~~~
amaks
I bet there will be a lot about CAP theorem in the comments:-)

------
runeks
I wonder why they charge a minimum of $0.90 per node-hour when they offer VMs
for as little as $0.008/hr. This is hugely useful even for single-person
startups, so why charge a minimum of ~$8,000 per year?

~~~
pramsey
Concur. The pricing makes sense for their "target market" of folks who
currently have a "bursting at the seams" MySQL or PgSQL instance, but it locks
out folks just getting started with a tiny database and low load. This seems
like bad positioning: the "bursting" folks will have to decide between the
cost of re-hosting their whole system on the Cloud Spanner and trying to
incrementally keep their current platform running; the small folks who would
like to organically grow on a platform without scaling limits are locked out
of the low end, so by the the time they are big enough to need Cloud
Spanner... they too with be forced into the "re-host or muddle on" decision.

~~~
stickfigure
This seems like a real problem. The "bursting at the seams" people will need a
pretty significant rewrite to run on Spanner, which seriously limits the
appeal.

The ideal market for Spanner seems to be new projects developed by big
companies that know that big traffic will arrive on day 1. In other words,
Google. Which I guess shouldn't be too surprising...

------
xapata
Amazon likes to respond to Google with it's own price drops and product
launches. It's telling that their announcements are orthogonal instead of
direct competition with Spanner.

When Google announced Spanner back in 2012, I'm sure Amazon and Microsoft
started teams to reproduce their own versions.

Spanner is not just software. The private network reduces partitions. GPS and
atomic clocks for every machine help synchronize time globally. There won't be
a Hadoop equivalent for Spanner, unless it includes the hardware spec.

~~~
elvinyung
Amazon already has Aurora:
[https://aws.amazon.com/rds/aurora/details/](https://aws.amazon.com/rds/aurora/details/)

You're right that there's literally nothing else out there that has tight
synchronization using atomic clocks, though.

~~~
user5994461
Aurora is a toy compared to Spanner.

Single region, limited backups and replication topologies, limited
performances.

There is yet to see if Spanner can achieve the expectations, if it does, it's
a game changer.

~~~
xapata
> achieve the expectations

It has for Google internally. No reason why they can't share the service
externally.

~~~
discodave
Actually, there are lots of reasons why it's hard to share a service
externally. Making a service public means you have to deal with lots of new
problems like billing, abuse, dealing with lots of small users instead of a
few large ones, stronger backwards compatibility requirements and so-forth.

~~~
xapata
Yes, but those are orthogonal to the scaling issues that I presume user5994461
was thinking of.

~~~
discodave
That's kind of the point, when you make a service like Cloud Spanner public
you create _new_ scaling dimensions. New scaling dimensions means new problems
to solve that are not trivial.

------
emersonrsantos
Thomas Watson in 1943 amd his famous quote: “I think there is a world market
for about five computers".

If he was alive, he could say these computers are Google, Apple, Microsoft,
Amazon and Facebook.

~~~
jonstokes
I beat you by 7 years :)

[https://arstechnica.com/civis/viewtopic.php?f=21&t=1109206](https://arstechnica.com/civis/viewtopic.php?f=21&t=1109206)

~~~
emersonrsantos
You win :D

------
abalone
How does this compare to AWS Aurora in terms of pricing and performance?

With Aurora the basic instance is $48/month and they recommend at least two in
separate zones for availability, so it's about $96/month minimum. Storage is
$.10/GB and IO is $.20 per million requests. Data transfer starts at $.09/GB
and the first GB is free.[1]

Spanner is a minimum of $650/mo (6X the Aurora minimum), storage is $.30/GB
(3X), and data transfer starts at $.12/GB (1.3X).

Of course with Aurora you have to pick your instance size and bigger faster
instances will cost more. Also there's the matter of multi-region replication,
although it appears that aspect of Spanner is not priced out yet. So maybe as
you scale the gap narrows, but it's interesting to price out the entry point
for startups.

[1] [https://aws.amazon.com/rds/aurora/](https://aws.amazon.com/rds/aurora/)

~~~
xapata
Aurora replicas appear to be read-only, according to that link.

~~~
abalone
Sure, but to compare accurately we should look at where that impacts
performance in practice. How many writes can Aurora's largest instance handle?
What's the write latency from other parts of the globe?

------
tabeth
Forgive my ignorance, but could someone explain in layman's terms in which
situation this would be helpful? E.g. if I have 1TB of data would I use this?
If I have 1GB with a growth rate of 25GB/daily would I use this?

~~~
mdolan
The rule of thumb I have always been told is that you can push MySql to ~200
GB total and between 99.9% and 99.99% availability (between 1 and 9 hours of
downtime per year). Your milage might vary, but these are probably the right
orders of magnitude. There is also an iops limit but that's harder to put a
clear limit on because it's workload dependent.

If you need more storage, availability, or iops, the current recommendations
are shared relational stored or NoSql.

Sharding relational databases means instead of turning on 1 MySql machine, you
turn on 16 MySql machines and store 1/16th of the data on each machine. It
only allow queries patterns that can be responded to by a single shard (so if
you are only doing queries on a per user basis, your golden, otherwise, not so
much) and require that your hottest shards (e.g. most active users) should
only be a few orders of magnitude larger than your coldest (Justin Bieber
probably needs his own machines at Twitter because he has so many users).
There are ways to get around this, but they are tricky and will require a lot
of developer time.

With NoSql (riak, cassandra, dynamo, etc.), you have to give up many features
that make developers lives easier in order to scale beyond a single machine.
Each flavor has it's own set of tradeoffs, but the biggest problem is that you
loose (1) committing multiple rows at the same time (atomic commit) and (2)
ensuring queries don't overlap (isolation).

Spanner offers an alternative where the developer (for the most part) doesn't
have to know that her data lives on multiple different machines. It gives you
99.999% availability and theoretically unlimited scaling to any size, with an
API that is still relatively developer friendly (compared with NoSql or Shared
Sql). One cost here is latency / performance. You need to write to multiple
computers across the world so write speed will be bounded at a minimum by the
speed of light between those datacenters.

Short answer, there are a lot of companies that could benefit from a system
like this. It could give higher availability to a small Sql dataset or
stronger guarantees to a current large NoSql database. There is even an
argument to be made that the majority of problems companies face today are
best solved with this type of system and relational or NoSql optimized
workloads are the exception, not the rule.

~~~
ec109685
1 to 9 hours of downtime a year for MySQL seems on the high side. With Aura,
the failover times are much less and outside of the cloud, if there's a "noc",
they can failover quicker than that.

------
Ajedi32
> Today, we’re excited to announce the public beta for Cloud Spanner, a
> globally distributed relational database service that lets customers have
> their cake and eat it too: ACID transactions and SQL semantics, without
> giving up horizontal scaling and high availability.

This sounds too good to be true. But it's Google, so maybe not. Time to start
reading whitepapers...

------
jdcarr
Link to the actual OSDI paper (not the simpler whitepaper)
[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-
osdi2012.pdf)

------
kennethmac2000
Looks cool, but the pricing seems a bit non-cloud-native (or at least non-GCP-
native).

"You are charged each hour for the maximum number of nodes that exist during
that hour."

We've been educated by Google to consider per-minute, per-instance/node
billing normal - and presumably all the arguments about why this is the right,
pro-customer way to price GCE apply equally to Cloud Spanner.

~~~
advisedwang
The per-minute billing is an advantage when you are scaling up and down
rapidly. If you use VMs just for 5 minutes, per minute pricing is 20x cheaper
than hourly billing.

However with a database it is rare to scale up and down rapidly. Rather you
expect change over the order of days. Imagine you go from 10 instance to 15
instances over a week. per minute billing only saves a possible 5 instance-
hours over the week compared to hourly billing, which is less than 1% saving.

------
bowmessage
Interesting reading: Spanner Whitepaper

[https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...](https://cloud.google.com/spanner/docs/whitepapers/SpannerAndCap.pdf)

------
elvinyung
While everyone is puzzling over how Spanner seems to be claiming to be CA, I
would like to take this opportunity to bring up PACELC[1].

The idea is that the A-or-C choice in CAP only applies during network
partitions, so it's not sufficient to describe a distributed system as either
CP or AP. When the network is fine, the choice is between low latency and
consistency.

In the case of Spanner, it chooses consistency over availability during
network partitions, and consistency over low latency in the absence of
partitions.

1: [http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf](http://cs-
www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf)

~~~
meddlepal
It's not claiming CA. It's claiming CP with A strongly implied thanks to a
massive amount of Google engineering and private networking.

[https://cloudplatform.googleblog.com/2017/02/inside-Cloud-
Sp...](https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and-
the-CAP-Theorem.html)

~~~
zzzcpan
I think A is wrongly implied, tried to explain it here:
[https://news.ycombinator.com/item?id=13645925](https://news.ycombinator.com/item?id=13645925)

------
andy_ppp
> clients can do globally consistent reads across the entire database without
> locking

How is this possible across data centres? Does it send data everywhere at
once?

Seems too good to be true of course but if it works and scales it might be
worthwhile just not having to worry about your database scaling? Still I don't
believe it ;-)

EDIT: further info...

> Spanner mitigates this by having each member be a Paxos group, thus ensuring
> each 2PC “member” is highly available even if some of its Paxos participants
> are down. Data is divided into groups that form the basic unit of placement
> and replication.

So it's SQL with Paxos that presumably never get's confused but during a
partition will presumably not be consistent.

~~~
jdcarr
> In terms of CAP, Spanner claims to be both consistent and highly available
> despite operating over a wide area, which many find surprising or even
> unlikely. The claim thus merits some discussion. Does this mean that Spanner
> is a CA system as defined by CAP? The short answer is “no” technically, but
> “yes” in effect and its users can and do assume CA. The purist answer is
> “no” because partitions can happen and in fact have happened at Google, and
> during some partitions, Spanner chooses C and forfeits A. It is technically
> a CP system.

~~~
jhugg
I would expect more from Brewer.

"CA except when there are partitions" is CP. It's not "effectively CA".

~~~
xapata
No, he's saying it's effectively CAP because the A downtime is so small.

It's one thing to do that for a key-value store. Entirely another to support
joins on a _globally_ distributed database. This ain't just one availability
zone. Spanner is amazing.

It took them a few years to make it a service, but when they announced its use
internally a few years ago, it seemed like the nail in the coffin for in-house
database hosting.

~~~
jhugg
I understand what he's saying. It's marketing.

There's nothing wrong with saying it's CP, but since we control everything
there's extremely rare P. Then he can show availability numbers (which he
kinda does).

Saying it's "effectively CA" defeats the point of the CAP theorem, which says
you have to make tradeoffs. See: [https://codahale.com/you-cant-sacrifice-
partition-tolerance/](https://codahale.com/you-cant-sacrifice-partition-
tolerance/)

~~~
xapata
> It's marketing.

No, it's engineering. It's the recognition that if periods of unavailability
are too small and too rare to be noticed, then the system behavior is
indistinguishable from an "available" system in the sense of the CAP theorem.

It's like the "Retina" display you're probably reading from. There are pixels,
you just can't see them.

------
sebringj
I was like YES!!! Then I read for a single node it is 90 cents per hours then
I was like NO!!! so absolute minimal cost for me is $648/month? I was hoping
there was like a dev version. Maybe I didn't read the fine print?

~~~
therealmarv
I was thinking the same thing! Maybe it's better to look into some smaller
service which offer e.g. PostgreSQL as a Service like
[https://www.elephantsql.com/plans.html](https://www.elephantsql.com/plans.html)
or
[https://www.databaselabs.io/pricing/](https://www.databaselabs.io/pricing/)
(I'm sure they are even more, just done a quick googling). Does somebody has
experience with such services?

~~~
sebringj
PostGres at least offers full text indexing. I couldn't find that on Cloud
Spanner.

------
tamalsaha001
One thing to note is Spanner's transactions are different compared to what you
get with a traditional RDBMS. See
[https://cloud.google.com/spanner/docs/transactions#ro_transa...](https://cloud.google.com/spanner/docs/transactions#ro_transaction_example)

An example is the rows you get back from a query like "select * from T where
x=a" can't be part of a RW transaction. I believe because they don't have the
time-stamp associated with them. So, you have to re-read those rows via
primary key inside a RW transaction to update them. This can be a surprise if
you are coming from a traditional RDBMS background. If you are think about
porting your app from MySQL/PostgreSQL to Spanner, it will be more than just
updating query syntax.

Disclaimer: I used F1 (built on top of Spanner,
[https://research.google.com/pubs/pub41344.html](https://research.google.com/pubs/pub41344.html))
few years ago.

------
the_cap_theorem
>> Remarkably, Cloud Spanner achieves this combination of features without
violating the CAP Theorem.

This is the best weasel PR language I have seen in a long time.

Note that the sentence does not actually proclaim that they solved (the
previously "unsolvable") problem of achieving distributed consensus with
unreliable communication while maintaining partition tolerance and
availability.

The blog only says they don't "violate" the CAP theorem -- whatever that
means. So the statement is technically correct. Still the intention is
obviously to mislead the casual reader (why else would you start the sentence
with "Remarkably"?).

A litmus test: The same statement is true for MySQL - or _any other_ database
in fact:

    
    
      >> "Remarkably, MySQL achieves this combination of features without violating the CAP theorem"
    

It's a bit like saying

    
    
      >> "Remarkably, MySQL is not a perpetuum mobile"

------
dhd415
Given that CockrochDB is based on Spanner and F1, this DBaaS sounds like it
will compete directly with them.

~~~
assface
> Given that CockrochDB is based on Spanner and F1, this DBaaS sounds like it
> will compete directly with them.

Except that it's not. CockroachDB uses MVCC. Spanner uses 2PL and F1 uses OCC.
Not the same at all.

------
wcdolphin
Is JSON data type support in the works? Seems to be a very commonly requested
feature these days.

~~~
deesix
Yes, it's something we have on the roadmap. We will adjust priority based on
the level of demand so thanks for your vote ;-)

(disclaimer: I work on Cloud Spanner)

~~~
brandmeyer
Dumb question: Since strings are supported, couldn't you just store some JSON
as a string? Or are you talking about supporting queries that involve parsing
the JSON on the "server"?

~~~
wcdolphin
Not dumb, I was being inexact. I was talking about Spanner understanding the
semantics of JSON, similar to Postgre's JSONB or MYSQL's JSON.

~~~
seangrogg
I think the OP was saying "dumb question" as in "I have a dumb question"
rather than "this is a dumb question". At least that's the context I inferred.

------
gonyea
I can vouch for Spanner: it's a badass piece of Google's infrastructure.

------
nodesocket
Related, I wrote a blog post on the network latency between Google Compute
Engine zones and regions. I'm assuming Cloud Spanner will still have these
latencies once multi-region is deployed. Cross-zone latency on GCE is very
good though.

[https://blog.elasticbyte.net/comparing-bandwidth-prices-
and-...](https://blog.elasticbyte.net/comparing-bandwidth-prices-and-network-
latency-between-google-compute-zones-and-regions/)

------
BinaryIdiot
Oh this looks really compelling! Though I'm guessing this is targeted to
companies? I'd love to use this for some personal projects but the pricing
seems really high. Am I reading it right that a single node being used at
least a tiny bit every hour is about $670 a month?

Maybe I'm misunderstanding how the pricing works here. Any clarification would
be highly welcomed :)

~~~
advisedwang
It's targeted to large datasets more than companies. There isn't really any
advantage of a single node Cloud Spanner instance over Cloud SQL. Cloud
Spanner becomes worthwhile when you have more data/throughput than a single
node system can support, at which point the pricing is competitive with other
options.

~~~
BinaryIdiot
Fair enough. I just kinda wanted to play with it on my next project :)

------
pagade
What is TrueTime really? Are their Distributed Systems 'sharing a global
clock'?

~~~
philip1209
From the Spanner paper:

> The underlying time references used by TrueTime are GPS and atomic clocks.
> TrueTime uses two forms of time reference because they have different
> failure modes... TrueTime is implemented by a set of time master machines
> per datacenter and a timeslave daemon per machine. The majority of masters
> have GPS receivers with dedicated antennas; these masters are separated
> physically to reduce the effects of [GPS] antenna failures, radio
> interference, and spoofing. The remaining masters (which we refer to as
> Armageddon masters) are equipped with atomic clocks. An atomic clock is not
> that expensive: the cost of an Armageddon master is of the same order as
> that of a GPS master.

Source:
[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/39966.pdf)

------
executive
Does it support spatial objects / can it replace PostGIS?

~~~
brightsize
No. [https://cloud.google.com/spanner/docs/data-
types](https://cloud.google.com/spanner/docs/data-types)

------
rdtsc
> This leads to three kinds of systems: CA, CP and AP,

What is a distributed system that is CA? Can you build a distributed system
which will never have a partition.

~~~
halflings
They answer your question just two lines under the one you are quoting:

> For distributed systems over a “wide area,” it's generally viewed that
> partitions are inevitable, although not necessarily common. If you believe
> that partitions are inevitable, any distributed system must be prepared to
> forfeit either consistency (AP) or availability (CP), which is not a choice
> anyone wants to make. In fact, the original point of the CAP theorem was to
> get designers to take this tradeoff seriously. But there are two important
> caveats: First, you only need to forfeit consistency or availability during
> an actual partition, and even then there are many mitigations. Second, the
> actual theorem is about 100% availability; a more interesting discussion is
> about the tradeoffs involved to achieve realistic high availability.

~~~
rdtsc
> If you believe that partitions are inevitable, any distributed system

How does that answer it? Are they implying that partitions will not happen if
you don't believe in them?

~~~
robryk
A CA system is not a system that doesn't have partitions, it's a system that
works under the condition that there are no partitions (ie. it is a non-
partition-resistant system).

~~~
rdtsc
> , it's a system that works under the condition that there are no partitions

But that is not a choice with distributed systems, unless as soon as a
partition happens the system shuts down immediately. That is it effectively
disappears and never re-appears again. But that's not a CA system then?

Or saying that it is not partition resistant is also difficult because the
system in case of a partition will do _something_. The typical choices is that
it either responds to clients (trying to be AP) or it doesn't (trying to be
CP).

That is why I understand CA systems as equivalent to the belief that
"partitions can't happen". Which I think is unrealistic.

~~~
robryk
This is a system which is guaranteed to be consistent and available as long as
there are no partitions. If there are partitions, the system isn't guaranteed
to be either.

Yes, it's a pretty much useless system.

------
mallipeddi
Few questions from reading the docs:

1) How big can all the colocated data for a single primary key get before they
don't fit within a split? Can I implement a GMail-like product where all the
data for a single user resides within one split?

2) Is there a way to turn off external consistency and fall back to
serializability? In return you get better write latencies. This is similar to
what CockroachDB provides?

------
Beldur
Here is a very interesting video from 2013 of Martin Schoenert explaining the
Google Spanner White Paper (In german though):
[https://www.youtube.com/watch?v=2QKewyoOSL0](https://www.youtube.com/watch?v=2QKewyoOSL0)

Now he works for Google as an Engineering Manager.

------
tdrd
Doesn't seem possible to use this yet. No client libraries and no samples:
[https://cloud.google.com/spanner/docs/tutorials](https://cloud.google.com/spanner/docs/tutorials)

Have they documented the wire protocol? I couldn't find it.

~~~
mairbek
I work on Cloud Spanner and client libraries are rolling out right now, but
API definitions are available.

RPC:
[https://cloud.google.com/spanner/docs/reference/rpc/](https://cloud.google.com/spanner/docs/reference/rpc/)
Rest:
[https://cloud.google.com/spanner/docs/reference/rest/](https://cloud.google.com/spanner/docs/reference/rest/)

~~~
michaelsbradley
Anyone working on a Rust lang client-library?

------
dorianm
Spanner's paper:
[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-
osdi2012.pdf)

------
sandGorgon
> _If you have a MySQL or PostgreSQL system that 's bursting at the seams_

Postgresql ? How does this work for people _migrating_ from traditional SQL
databases - typically people use ORM. How would this fit in with, say , Rails
or SqlAlchemy ?

~~~
deesix
There is real work to migrate you application to use Cloud Spanner. Your
schemas and queries will work with some tweaking but we don't have ORM support
at beta.

We are going to produce some additional collateral on migrating from popular
RDBMS, but the Quizlet post is the best reference right now for migrating from
MySQL.

[https://quizlet.com/blog/quizlet-cloud-
spanner](https://quizlet.com/blog/quizlet-cloud-spanner)

We have released client libraries for Java, Go, Node and Python on Github, but
we haven't used those clients to implement support for popular ORMs.

Basically, the open-source ecosystem will need to add support for these ORMs
but we will contribute wherever we can to push these initiatives. If we get
lots of demand for a specific ORM, we will look into doing something special
for that.

Hope that helps.

(disclaimer: I work on Cloud Spanner)

------
rattray
Very interesting. How does this pricing compare to AWS Aurora?
[https://aws.amazon.com/rds/aurora/pricing/](https://aws.amazon.com/rds/aurora/pricing/)

~~~
tedd4u
Not sure. If you need to scale beyond a single master, Aurora won't help in
the same way Spanner does though. You can dial up the number of nodes in
Spanner dynamically under load with good results.

------
nodesocket
So does Cloud Spanner replace the existing Google Cloud SQL offering [1]? What
are the pros/cons of each?

[1] [https://cloud.google.com/sql/](https://cloud.google.com/sql/)

~~~
floatboth
That offering is just normal managed MySQL, the new thing is a custom built
database for huge scalability, much more expensive of course.

~~~
nodesocket
Right that's what I was getting at, a typical web app does not and probably
should not start with Cloud Spanner (ignoring the fact it costs $0.90 per hour
per node). Cloud Spanner seems like it is attacking the big data market
correct?

~~~
floatboth
Something like that. But not necessarily the Big Data that's being analyzed,
that's usually done with Hadoop/Spark/whatever is the big thing now. It seems
to be aimed at, like, huge apps that actually need horizontal scalability.

------
jallriddle
Is this similar to AWS Aurora or is this something else completely different?

~~~
agwa
Aurora is not globally distributed. Spanner is, and is based on Google
research which takes advantage of atomic clocks installed in each server:
[https://research.google.com/archive/spanner-
osdi2012.pdf](https://research.google.com/archive/spanner-osdi2012.pdf)

Edit: not every server has an atomic clock; see replies by Google employees

~~~
Scaevolus
There are only atomic clocks in _some_ master servers: "TrueTime is
implemented by a set of time master machines per datacenter and a timeslave
daemon per machine. The majority of masters have GPS receivers with dedicated
antennas; these masters are separated physically to reduce the effects of
antenna failures, radio interference, and spoofing. The remaining masters
(which we refer to as Armageddon masters) are equipped with atomic clocks. An
atomic clock is not that expensive: the cost of an Armageddon master is of the
same order as that of a GPS master."

The timeslave daemons running on each machine keep them synchronized with the
master time servers, and maintain tight bounds on their inaccuracy.

(Disclaimer: I work at Google)

------
danols
Interesting but without INSERT and UPDATE it just isn't worth it for me. When
can we expect it to handle data manipulation language (DML) statements?

------
DivineTraube
"What if you could have a fully managed database service that's consistent,
scales horizontally across data centers and speaks SQL?"

Looks like Google forgot to mention one central requirement: latency.

This is a hosted version of Spanner and F1. Since both systems are published,
we know a lot about their trade-offs:

Spanner (see OSDI'12 and TODS'13 papers) evolved from the observation that
Megastore guarantees - though useful - come at performance penalty that is
prohibitive for some applications. Spanner is a multi-version database system
that unlike Megastore (the system behind the Google Cloud Datastore) provides
general-purpose transactions. The authors argue: We believe it is better to
have application programmers deal with performance problems due to overuse of
transactions as bottlenecks arise, rather than always coding around the lack
of transactions. Spanner automatically groups data into partitions (tablets)
that are synchronously replicated across sites via Paxos and stored in
Colossus, the successor of the Google File System (GFS). Transactions in
Spanner are based on two-phase locking (2PL) and two-phase commits (2PC)
executed over the leaders for each partition involved in the transaction. In
order for transactions to be serialized according to their global commit
times, Spanner introduces TrueTime, an API for high precision timestamps with
uncertainty bounds based on atomic clocks and GPS. Each transaction is
assigned a commit timestamp from TrueTime and using the uncertainty bounds,
the leader can wait until the transaction is guaranteed to be visible at all
sites before releasing locks. This also enables efficient read-only
transactions that can read a consistent snapshot for a certain timestamp
across all data centers without any locking.

F1 (see VLDB'13 paper) builds on Spanner to support SQL-based access for
Google's advertising business. To this end, F1 introduces a hierarchical
schema based on Protobuf, a rich data encoding format similar to Avro and
Thrift. To support both OLTP and OLAP queries, it uses Spanner's abstractions
to provide consistent indexing. A lazy protocol for schema changes allows non-
blocking schema evolution. Besides pessimistic Spanner transactions, F1
supports optimistic transactions. Each row bears a version timestamp that used
at commit time to perform a short-lived pessimistic transaction to validate a
transaction's read set. Optimistic transactions in F1 suffer from the abort
rate problem of optimistic concurrency control, as the read phase is latency-
bound and the commit requires slow, distributed Spanner transactions,
increasing the vulnerability window for potential conflicts.

While Spanner and F1 are highly influential system designs, they do come at a
cost Google does not tell in its marketing: high latency. Consistent geo-
replication is expensive even for single operations. Both optimistic and
pessimistic transactions even increase these latencies.

It will be very interesting to see first benchmarks. My guess is that
operation latencies will be in the order of 80-120ms and therefore much slower
than what can be achieved on database clusters distributed only over local
replicas.

~~~
wsh91
Spanner's p90s are, for at least one user, consistently lower than 50 ms.
[https://quizlet.com/blog/quizlet-cloud-
spanner](https://quizlet.com/blog/quizlet-cloud-spanner)

(Disclaimer: I work on Google's cloud.)

------
etherealG
I'd love to see a jepsen test. Maybe the spanner team would be able to sponsor
an independent test?

------
aladine
Great product from Google. I wonder what is the difference between Cloud
Spanner and Google CloudSQL

~~~
kyrra
Cloudsql is managed MySQL. Spanner is a custom in-house distributed database.

------
darkerside
> Unlike most wide-area networks, and especially the public internet, Google
> controls the entire network and thus can ensure redundancy of hardware and
> paths, and can also control upgrades and operations in general

I know this is a single system, but I'll still say it. This seems like another
step in a scary trend for our internet.

~~~
runeks
Why is it a scary trend that Google has made their part of the internet more
resilient?

~~~
aanm1988
because we keep giving up more and more control to a single Ad company.

------
gigatexal
The sql syntax reference looks similar to that of Postgres' syntax reference.

~~~
advisedwang
Actually it is most closely related to the new ["Standard SQL" in
BigQuery]([https://cloud.google.com/bigquery/docs/reference/standard-
sq...](https://cloud.google.com/bigquery/docs/reference/standard-sql/)).

------
esseti
does anyone know if it works with django or a way to make it working? it
should be a matter of a connector, no?

------
kozikow
SqlAlchemy engine please :) ?

------
crooked-v
Neat.

How long until it gets shut down with a month's notice?

------
koolba
> Today, we’re excited to announce the public beta for Cloud Spanner, a
> globally distributed relational database service that lets customers have
> their cake and eat it too: ACID transactions and SQL semantics, without
> giving up horizontal scaling and high availability.

This is a bold claim. What do they know about the CAP theorem that I don't?

Separately, (emphasis mine):

> If you have a MySQL _or PostgreSQL_ system that's bursting at the seams, or
> are struggling with hand-rolled transactions on top of an eventually-
> consistent database, Cloud Spanner could be the solution you're looking for.
> Visit the Cloud Spanner page to learn more and get started building
> applications on our next-generation database service.

From the rest of the article it seems like the wire protocol for accessing it
is MySQL. I wonder if they mean to add a PostgreSQL compatibility layer at
some point.

~~~
marknadal
Theoretically it means they are giving up on being Partition Tolerant. There
was a popular post a while ago about how the P can't be sacrificed. Because if
it is... everything else will fail.

Being Google they are probably prideful enough to think their servers could
never have an outage. Which yes, I agree with you, that is a very scary claim.

~~~
CobrastanJorji
This is thoroughly wrong. Cloud Spanner sacrifices the "A", not the "P." The
cool thing being accomplished here is that the sacrifice to the A is greatly
reduced (five or more 9s). There are several documents on the subject linked
right off that page and elsewhere in these same comments, like this one:
[https://cloud.google.com/spanner/docs/whitepapers/SpannerAnd...](https://cloud.google.com/spanner/docs/whitepapers/SpannerAndCap.pdf)

------
mankash666
While the product is compelling (acid compliant, horizontally scanning DB), it
does seem expensive.

If you use 2 nodes/hour, Cost = (2 _0.9)_ 24 * 31 = $1400/month not anointing
for storage and network chargers.

------
tajen
> $.90 per node per hour

That makes $700 per month. Is this the minumum? or can we have 0 node when the
lambda is idle ?

------
williamle8300
I see there's "data layer encryption" but the data is still readable by
Google. Why would anyone want to keep feeding the Google beast with more data?

Software is about separating concerns, and decentralizing authority.
Responsible engineers shouldn't be using this service.

------
Avloss
Amazing! But why does this feel like such a de ja vue all over again.. (surely
I'm missing something).. They've spent 5 years telling us that we just CAN'T
scale SQL.. Now they'll tell us that actually.. they've figured it out! :)

------
api
Given the CAP theorem I wonder what trade-offs they make and how much
visibility they give you into these trade-offs.

In any case this is much better than Amazon's offerings... when they actually
ship it. :)

~~~
jrowley
Check out this post mentioned in the original post:

[https://cloudplatform.googleblog.com/2017/02/inside-Cloud-
Sp...](https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and-
the-CAP-Theorem.html)

Of note:

They say Spanner is "both consistent and highly available despite operating
over a wide area". So not 100% availability but they've got it to "more than
five 9s of availability (less than one failure in 1066)."

~~~
mikeyouse
I didn't pick up on it until just a moment ago, but when you say "They say",
it's actually Eric Brewer saying that -- who's most known for coming up with
the CAP theorem. I think they've got a pretty good understanding of it!

~~~
jrowley
ha whoops good catch! I knew the post was written by Brewer, but slipped up
with the they... yeah I think we have it on pretty good authority that they
system doesn't so how violate CAP yet addresses it's challenges.

------
kozak
I wonder how many people will get a seizure from that red-blue blinking
rectangle in the video :(

Upd: Downvoting this warning will only increase that number.

------
theptip
> Does this mean that Spanner is a CA system as defined by CAP? The short
> answer is “no” technically, but “yes” in effect and its users can and do
> assume CA.

It's somewhat ironic that Brewer, the original author of the CAP theorem, is
making this sort of marketing-led bending of the CAP theorem terminology. I
think what he really should be saying is something in more nuanced language
like this: [https://martin.kleppmann.com/2015/05/11/please-stop-
calling-...](https://martin.kleppmann.com/2015/05/11/please-stop-calling-
databases-cp-or-ap.html)

But perhaps Google's marketing department needed something in the more popular
"CP or AP?" terminology. I don't see what would be wrong with "CP with
extremely high availability" though.

It's certainly wacky to be claiming that a system is "CA", since as the post
admits it's technically false; to me this makes it clear that CP vs. AP (vs.
CA now?) does not convey enough information. I'd prefer "a linearizably-
consistent data store, with ACID semantics, with a 99.999% uptime SLA". Not as
snappy as "CA" (I will never have a career in marketing I suppose), but it
makes the technical claims more clear.

~~~
wsh91
You're omitting what immediately follows:

> The purist answer is “no” because partitions can happen and in fact have
> happened at Google, and during some partitions, Spanner chooses C and
> forfeits A. It is technically a CP system.

> However, no system provides 100% availability, so the pragmatic question is
> whether or not Spanner delivers availability that is so high that most users
> don't worry about its outages. For example, given there are many sources of
> outages for an application, if Spanner is an insignificant contributor to
> its downtime, then users are correct to not worry about it.

~~~
theptip
My point is, why call this "CA", if it's not CA? Especially when you
immediately go on to explain why it's not really CA?

