
How to escape lock-in with a multi-cloud stack - mikecb
https://cloudplatform.googleblog.com/2016/07/how-to-escape-lock-in-with-a-multi-cloud-stack.html
======
jakozaur
It takes a lot of engineering time and effort to be able to switch between
mentioned alternatives. E.g. Druid and Drill vs. BigQuery.

This article is more probably to convince ppl to use Google Cloud rather than
how to run multicloud strategy.

~~~
stephenr
ding ding ding, we have a winner.

If their "cloud" offerings were all just hosted instances of open source,
there would be no lock-in, but they would have no 'competitive edge'.

I think this is more about letting Google fanboy dev's sell higher-ups on GCP
because "hey look they're not trying to lock us in like AWS is!"

~~~
milesward
OP/fanboy here; i'm interested in your thoughts on our products like Cloud
BigTable: OSS (HBase) API, but unique goog performance bits under the hood.
Valid? How can we best move the state of the art forward?

~~~
jakozaur
Though I agree some Google products are great, but cloud industry lacks
standardized APIs allowing to switch between products.

E.g. in the past we had OpenGL, PC which spur a lot of innovation and created
whole industries. Common subset that allows seamless migrations.

E.g. I wish we have standard for big table like databases:

1\. Google Cloud BigTable.

2\. DynamoDB.

3\. HBase/Cassandra.

~~~
milesward
I'm with you! We didn't want to ship yet another one so Cloud BigTable uses
the HBase API: [https://cloud.google.com/bigtable/docs/bigtable-and-
hbase](https://cloud.google.com/bigtable/docs/bigtable-and-hbase)

~~~
jakozaur
Kudos for doing that. I didn't realized about it.

Common API even in non-perfect way is so much better than everybody
reinventing the wheel from scratch, because some tiny use case doesn't fit in
the previous API.

------
harlowja
Well that's odd, they sort of forget to mention the whole premise of (all the
projects composed in) openstack (which is to provide the same APIs across
providers, vendors... by its very nature escaping lock-in by doing this).

Also sadly (and maybe I'm taking this wrong) but calling the holy land 'Google
Infrastructure For Everyone Else' seems condescending ...

~~~
milesward
Author here: yeahhhh we don't know what to call it. CoreOS folks said this and
it sorta made sense, we call out the name ookyness right in the post.
AsKSHBDDVBTFRcsM seemed, well, dumb. Ideas?

~~~
harlowja
Call it 'googles view of a stack', at least that doesn't make it sound like an
elitist view on (one of many) versions of a stack (that yes is just one
version of many stacks that have and do work for the <wider world>).

------
piotrkaminski
After all the noise about Firebase at the last Google I/O, it's odd that the
article doesn't mention it at all. I guess its proprietary API and
undocumented wire protocol with no open source alternatives would be off
message...

~~~
stemuk
I know everybody would love to see this, but for me it's a bit much to ask
from Google to open-source Firebase since they invested quite a lot of time
and money into it to get to the current version. Apart from that, I can
definetly recommend [http://deepstream.io](http://deepstream.io) as a Firebase
alternive, although it currently lacks an Android/iOS client (Java client is
in the works).

~~~
piotrkaminski
Yeah, I agree that open sourcing Firebase is too much to ask. But there's
other things they could be doing: a) document the wire protocol and license
the clients for free use with any server, b) make the API definitions public
domain and provide a compatibility test suite for alternative implementations,
c) lead a standardization effort on real-time JSON databases, etc...

~~~
stemuk
True, maybe [https://github.com/firebase](https://github.com/firebase) could
help with some reverse engineering. However, I discovered the Firebase
document and the deepstream.io record structure to be fairly similar, so with
the help of [https://github.com/firebase/firebase-
queue](https://github.com/firebase/firebase-queue) the migration from Firebase
to deepstream.io + RethinkDB might be possible. As a more general note, I
would expect some of the stuff you mentioned to happen in the upcoming
months/years, since Firebase is right now set to become the universal
plattform for app devs (analytics, dynamic links etc.) rather than a
standalone database offering.

------
mattbillenstein
I think the key thing is data interchange so you can get off of product X when
you have business reasons to do so without running into a huge data migration
project.

Standard SQL databases you can use pretty much the same tooling as if you were
running it yourself. Cloud storage, you can download/upload to a new thing,
but it may take some time and cost some bandwidth. BigQuery/Redshift you
should be storing your data as .json.gz on S3/GCS as a backup anyway, so an
export shouldn't be needed -- you'd just import the cloud storage data into
another system.

I did a smallish BigTable dump a couple months ago -- kinda a PITA to do -- I
don't recommend it; maybe I was doing it wrong, I'm not that familiar with
hbase and it was a one-off thing.

I also did a rather small scale data migration out of app engine (~100GB)
awhile back -- total PITA -- datastore dumps are protobuf, so you need to
build special tooling to import the data into another database.

And generally, I don't use the other services to avoid lock-in. DNS round-
robin+nginx for load balancing and ssl termination, Saltstack/Ansible
deploying to Ubuntu VMs for config management and deployment. I run the
databases myself so I can run it on any cloud I want -- or localhost; having a
full dev environment you can create and destroy on your own machine using the
same tools that you deploy to is really nice. This also enables doing things
like running dev workloads on cheaper clouds (Linode) while running production
in a VPC on AWS.

------
itaysk
As someone who worked with Google from a customer perspective, they take pride
in the fact that their big data (BigTable, BigQuery, DataFlow) is proprietary
and superior the their OSS alternatives (I believe they actually are). I am
not saying that they should OSS it, if they build something so awesome that
differentiate them, they should know how to monetize it. But at least don't
come off with a post like this.

~~~
jpatokal
Dataflow is the hosted version of Apache Beam, which is OSS.
[http://beam.incubator.apache.org/](http://beam.incubator.apache.org/)

BigTable is compatible with Apache HBase API, so it's trivial to swap out.
[https://cloud.google.com/bigtable/docs/bigtable-and-
hbase](https://cloud.google.com/bigtable/docs/bigtable-and-hbase)

Even BigQuery now has a standard SQL interface, although it's still Beta:
[https://cloud.google.com/bigquery/sql-
reference/](https://cloud.google.com/bigquery/sql-reference/)

Working at Google Cloud myself, I find the company philosophy to be to provide
standard interfaces to superior implementations. This keeps us honest, because
if the door to leave is open, then you have to actually _want_ to stay.
(Standard disclaimer: my opinion, not the company's.)

------
jacques_chester
I'm surprised they didn't mention Cloud Foundry, which runs on OpenStack,
vSphere, AWS, Azure and ... GCP. It's owned and managed by an independent
foundation and is the most mature of the open PaaSes, having a head start of
several years.

I guess they see it as a competitor to Kubernetes. Or, plausibly, Google is
very large and the engineers who are working on Cloud Foundry integration with
GCP are not known to Miles Ward, Head of Global Solutions.

I work for Pivotal, we donate the majority of engineering to Cloud Foundry. We
make our money selling a distribution, Pivotal CF.

Cloud Foundry's ability to switch cloud providers, or to take a cloud inhouse,
and in future to do so seamlessly, is _very_ attractive to our customers. IaaS
providers remind them the bad old days of RDBMS and middleware lockin and they
do _not_ want to go back.

As for the services, it's good that Google are pointing to open alternatives.

------
jssmith
Google's support for a multi-cloud vision would be a lot more meaningful if
they were offering the open source alternatives as part of their cloud
platform and contributing to their development. Google has done so in a few
key cases (Kubernetes and Tensorflow), but in other areas it's hard to get
comfortable with the belief that there will be smooth transition to other
infrastructure.

~~~
troymc
You've got a point. BigTable is an example; they've never open-sourced it and
instead other OSS projects (notably the Hadoop ones) were inspired by it.

Other examples are GFS, Colossus, Spanner and F1. The OSS analogue of GFS is
HDFS (AFAIK), but I know of nothing equivalent to the others. (CockroachDB is
a bit like Spanner, but is much more pragmatic about its clocks.)

------
xorgar831
Many companies would be better off however using cloud services since the
"lockin" still costs them less than the overhead of running xyz service them
self. Lock-in is only important if you have a cheaper way to run something,
not if you're hiding costs in IT salaries for operations. :)

~~~
stephenr
> Lock-in is only important if you have a cheaper way to run something

False. Lock-in is a business risk, and anyone who thinks it isn't is kidding
themselves.

If you are dependent on a single vendor for a non-standardised service, you
are also at risk if the vendor goes out of business, if the vendor decides to
raise prices significantly, if the vendor decides to discontinue that service,
or even if the vendor decides to change the mode of operation for that
service.

You are also beholden to that vendor's internal procedures, and tech
decisions, which aren't always in your best interests. You may remember last
September when there was a fucking massive AWS outage, where customers
couldn't access the console, EC2 instances wouldn't spin up, etc. The fault
for _all_ of that, was a network disruption which caused DynamoDB to flip it's
shit and make "the worlds biggest cloud provider" into the worlds biggest
brick shitting machine.

They've probably made changes so that type of error is less likely in future,
but that doesn't matter because you don't get to make that decision if you're
a single vendor customer on AWS. Whatever choices they make, they make for
you.

> hiding costs in IT salaries for operations

Here's a pro-tip for you: use of AWS/Azure/Google Cloud/etc doesn't absolve
you of the need for operations staff. If you think your single nodejs
developer is equally skilled and has enough time to setup and maintain your
infrastructure just because it's dynamic and has a browser interface, I have a
great investment offer for you, in magic beans futures.

~~~
FooHentai
All of this is accurate. Also on the infrastructure deployment and maintenance
side, virtualization already vacated the need for any ordinary organization to
spend time worrying about infrastructure. You just rent colo and slap
hypervisors in.

There are still organizations messing around with overly complex internal
server systems because they have bad admins or are heavy in technical debt,
but these are no longer the rule.

~~~
stephenr
> You just rent colo and slap hypervisors in.

For most clients I work with (who often have this "oh but shouldn't we use
AWS, they're the best right?" attitude) even a few rented VPS (i.e. on shared
hardware) is sufficient.

I'm a big fan of the way some providers like Rimu allow more control, which
helps with scaling up - by default most customers just use one or more Xen
VM's on shared physical hosts, but you have the option to rent dedicated
hardware (either existing stock, or custom orders), and then you can run one
or more VMs - without noisy neighbour concerns.

The "single vm on a host" seems weird to most people at first, until you
realise that it allows you to migrate the VM between physical hosts.

Sure, the new hotness is servers-as-cattle and we should just provision a new
instance, but that doesn't always work for smaller outfits.

------
ipsin
The blog entry covers a lot of options, but as a cheapskate who wants to play
with cloud infrastructure, I'm wondering if there are any good resources for
pricing comparison?

I could use AWS or Google, but if I'm just keeping a toy cloud service going.
I'm not particularly worried about reliability, and I don't want to spend
$30+/month.

~~~
theDoug
I also recommend Digital Ocean for quick/cheap personal toy services, but for
projects of enduring duration and reliability pricing calculators are totally
public for AWS[1] and GCP[2]. I haven't seen any direct comparison calculators
as everyone's particular mileage/needs may vary.

The good news is that at scale, the days of compute, storage, and data
management becoming easy to use utilities to plug into are here, or near. The
bad news is it still feels a bit too 'big' to plug small lamps into the main
AWS or Cloud Platform grids, but I suspect those days are numbered as well.

[1]:
[https://calculator.s3.amazonaws.com/index.html](https://calculator.s3.amazonaws.com/index.html)
[2]:
[https://cloud.google.com/products/calculator/](https://cloud.google.com/products/calculator/)

(Disclaimer: I’m a Google employee, dealing with Cloud, loving all options,
but especially being able to have options.)

------
mikecb
I don't really understand why the title was changed to this from my original
or the title of the post, since it is less informative and more
inflammatory...

~~~
brazzledazzle
It looks like the current title is in the URL but not on the page itself.
Maybe someone at Google changed it but retained the URL for obvious reasons.

------
niemyjski
This is one of the main reasons we built
[https://github.com/exceptionless/Foundatio](https://github.com/exceptionless/Foundatio)
as we didn't want to be locked into azure or aws and have a great dev
experience.

------
quickben
Gentlemen, I would like to interrupt this "google promotes open-source anti
lock-in" PR party, and ask for a minute of silence for the recently departed
GoogleCode.

Their actions speak louder than their PR blogs.

Edit: spelling

~~~
strcat
There's already an open standard for the core functionality of code hosting
sites: Git. Projects migrate between GitHub, GitLab, Bitbucket and other
alternatives all the time by simply changing the Git remote. There's some
lock-in via extras like issue trackers and wikis but that's not difficult to
migrate, especially with Markdown as a common standard so markup can be
preserved.

I'm not really sure why you think that's relevant to this. Google Code simply
failed to compete with the alternatives and was obsoleted. It has nothing to
do with lock-in.

