

An update on container support on Google Cloud Platform - proppy
http://googlecloudplatform.blogspot.com/2014/06/an-update-on-container-support-on-google-cloud-platform.html

======
jbeda
We are particularly excited about Kubernetes. We are taking the ideas for how
we do cluster management at Google and creating an open source project to
manage Docker containers.

[https://github.com/GoogleCloudPlatform/kubernetes](https://github.com/GoogleCloudPlatform/kubernetes)

~~~
boundlessdreamz
Slightly offtopic - I haven't used docker yet but can docker be used to run
Ubuntu on GCE? That seems to be possible by using a ubuntu base image?

~~~
wmf
Ubuntu can run directly on GCE. [http://doit-intl.com/blog/2014/5/31/how-to-
install-ubuntu-se...](http://doit-intl.com/blog/2014/5/31/how-to-install-
ubuntu-server-on-gce)

~~~
nivertech
Note that some Google's Debian Wheezy images are specifically optimized for
their Andromeda SDN [1], especially when they running on us-central1-b or
europe-west1-a zones, where Andromeda is enabled [2].

Also Ubuntu 14.04 LTS has much worse performance than Ubuntu 12.04 LTS, so
it's better to stick to 12.04 LTS for now.

In my opinion it's better to port to Debian or to run Ubuntu image inside
container hosted on Debian, until they will have native Ubuntu support.

[1] [http://gigaom.com/2014/04/02/google-launches-andromeda-a-
sof...](http://gigaom.com/2014/04/02/google-launches-andromeda-a-software-
defined-network-underlying-its-cloud/)

[2] [http://googlecloudplatform.blogspot.co.il/2014/04/enter-
andr...](http://googlecloudplatform.blogspot.co.il/2014/04/enter-andromeda-
zone-google-cloud-platforms-latest-networking-stack.html)

~~~
sciurus
"Also Ubuntu 14.04 LTS has much worse performance than Ubuntu 12.04 LTS"

Citation needed.

~~~
nivertech
[http://www.phoronix.com/scan.php?page=article&item=ubuntu_14...](http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_ocean)

------
michaelmior
"Everything at Google, from Search to Gmail, is packaged and run in a Linux
container." Was this something which Google had previously disclosed? Seems a
bit surprising to me.

~~~
jbeda
Yeah -- I talked about it a couple of weeks ago at GlueCon. Also shared that
we launch over 2 billion containers every week.

My slides from the talk: [https://speakerdeck.com/jbeda/containers-at-
scale](https://speakerdeck.com/jbeda/containers-at-scale) PDF:
[http://slides.eightypercent.net/GlueCon%202014%20-%20Contain...](http://slides.eightypercent.net/GlueCon%202014%20-%20Containers%20At%20Scale.pdf)

~~~
zmanian
I went to a talk on Omega on Box.com about a year ago. At that point Omega was
still about managing Google's giant statically linked binaries.

Did Google switch to containers in a year? Maybe that answer is in your
slides? If so crazy...

~~~
zeroxfe
The statically linked binaries run inside the containers. Static linking gives
you a certain kind of portability (no need for library dependencies on the
machine.) The containers give you isolation, resource management, etc.

~~~
zmanian
The way it has been explained to me is that Google's fat binaries have no
dependencies beyond libc. The task scheduler would just have to deploy that
one file to one or more machines to get it to run.

Containers are much more flexible that statically linked binaries. You could
have multiple binaries in a container sharing a common set of dynamically
linked files.

Fat binaries inside containers sounds a bit like the worst of both worlds...

~~~
menage
If you want unrelated jobs to share library binaries, you need to stick to
explicit release schedules for those libraries, which means more coordination
between the various app teams and the teams that own the libraries.

When the size of the libraries (megabytes) is compared with the typical heap
size of running jobs (gigabytes - a small number of large instances per job is
typically a lot more efficient than a large number of small instances) the
space savings of shared libraries become pretty negligible.

Back then, one of the main bottlenecks in the system was the central
scheduler, in particular the amount of work it had to do tracking what binary
packages were installed on each machine and what were needed for the candidate
jobs it might plan to run on those machines. Having many packages per job just
makes the scheduler bottleneck worse.

There were two places where it did actually make sense to share packages
between jobs:

\- Java Virtual Machine and supporting libraries \- libc

These were external (so changed much less frequently), quite large, and were
needed by very large numbers of jobs, so the space savings of having one copy
of each needed version per machine outweighed the extra scheduling load
required for them.

------
planckscnst
This post was written by Eric Brewer, the author of CAP theorem.

~~~
mp99e99
Is that the same Eric Brewer from Inktomi?

~~~
michaelmior
Yes, that's him :)

------
PanMan
I know Google runs at a huge scale, but isn't even for them 2 billion
containers / week a LOT? I assume a lot of these only run for a really short
time? Are containers the new scripts?

~~~
derefr
If docker images are fancy static binaries, then docker containers are fancy
OS processes. Going through two billion OS PIDs in a week doesn't seem that
hard.

~~~
martinp
Still seems like a lot, 2 billion a week is over 3300 every second. Says a lot
about the scale at which Google operates.

~~~
planckscnst
_The Datacenter as a Computer_ [1] is a free book written by Googlers that
helps get your head around that scale.

[1]
[http://www.morganclaypool.com/doi/abs/10.2200/S00516ED2V01Y2...](http://www.morganclaypool.com/doi/abs/10.2200/S00516ED2V01Y201306CAC024)

