
How Cloud Storage uses Spanner for consistent bucket listing - deesix
http://cloudplatform.googleblog.com/2018/02/how-Google-Cloud-Storage-offers-strongly-consistent-object-listing-thanks-to-Spanner.html
======
wavesplash
This is Google's dig at AWS S3 which is eventual consistency for object
creation/listings.

If you want strong consistency for S3 consider something like JuiceFS:
[https://JuiceFS.io](https://JuiceFS.io)

It wraps your S3 in a strongly consistent and fast POSIX API (aka regular
mountable filesystem) and its way less expensive than EFS.

~~~
mrep
> This is Google's dig at AWS S3 which is eventual consistency for object
> creation/listings.

Correction: S3 is strongly consistent after new object puts, but not lists,
modifications, and deletes.

Definitely not as convenient but you can get strong consistency if you only
rely on new objects.

That being said, I am curious as to how available/consistent S3 is versus
Google cloud storage.

~~~
moduspwnens14
Additional correction: S3 is consistent after new object puts only if you
haven't already asked for that object. Otherwise you may get a stale "no
object" response.

------
trevyn
Spanner is great, but I really wish there was a way to use it for less than
$600/month. :(

~~~
clhodapp
Yep. I also wish that you didn't have to use a special proprietary RPC API for
writes...

These are connected because it means that it's tough to prototype on something
cheaper and then switch over.

~~~
jopsen
Yeah, it makes Amazon aurora far more attractive

~~~
anderspetersson
GCP also offers Cloud SQL with similar features to AWS Aurora.

~~~
clhodapp
Cloud SQL is more like plain RDS, no?

------
jacksmith21006
The Spanner paper is one of my favorites. I love how it took several different
things to come together to make possible.

A great example is Google needed a determinate network so standard TCP/IP
would not cut it. But then other things in the infrastructure was needed.

This is the ultimately full stack solution.

[https://research.google.com/archive/spanner.html](https://research.google.com/archive/spanner.html)

------
jacques_chester
But I still can't set a bucket-wide policy for caching to be applied
automatically to new files. So in other ways, consistency is inconveniently
eventual.

~~~
justinph
This is pretty easy to do with a cloud function. I was a little apprehensive
at first, but have been using it across many buckets and haven't seen any
issues. Here's a version of the script I'm using:
[https://gist.github.com/justinph/acb6301d63b407e1fb205cbb42e...](https://gist.github.com/justinph/acb6301d63b407e1fb205cbb42e08cba)

Google recently announced some changes with cloud storage and pubsub triggers
that potentially means that you could only have this trigger when a file is
newly created, not every change. I haven't looked into that yet.

~~~
jacques_chester
The code I've picked up from a predecessor uses a Concourse task to achieve a
similar end. But it's just annoying that I can't set a policy on the bucket.

As an aside, it looks as though Google's golang API for GCS has a way to set
the policy when you provide the file, so maybe at some point I can close this
issue: [https://github.com/concourse/semver-
resource/issues/61](https://github.com/concourse/semver-resource/issues/61)

