
Interview with Eric Brewer - agonzalezro
https://medium.com/s-c-a-l-e/google-systems-guru-explains-why-containers-are-the-future-of-computing-87922af2cf95
======
jedberg
A funny and somewhat off topic story -- back in 2003, before the Google IPO,
Google was doing a recruiting event at Berkeley. They brought a few of their
folks with them: their founder Larry, one of their female engineers, Marissa,
and some others. They did a little talk, and during the Q&A, professor Brewer
told Larry that there was an opening in the PhD program and he was welcome to
it. Larry politely declined.

Afterwards I asked Larry, "so, do you think you'll ever finish your PhD,
either here or at Stanford?". He said, "If this Google thing doesn't work out
I might, but I have a feeling it will work out ok."

It amuses me that Professor Brewer is now working for Larry. :)

~~~
bostonpete
In 2003, Google had over a billion dollars in revenue. I suspect that answer
was tongue in cheek.

~~~
voronoff
I suspect the question was as well.

------
mark_l_watson
Great Interview!

I worked as a contractor at Google in 2013 and loved their infrastructure. It
was amazing to fire off a Borg job that used hundreds to thousands of servers,
and the web based tools for tracking the job, fantastic logging to drill into
problems, etc.

And, Borg was two generations ago!

Even though I am very happy doing what I am now, sometimes I literally wake up
in the morning thinking about Google's infrastructure. I now use lesser but
public services like AppEngine, Heroku, nitrous.io (like Google's web based
IDE Cider, a bit) but it is not the same.

BTW, not to be negative, but while Google is a great home for someone like
Eric Brewer, it is a shame that many hundreds of future students at UC
Berkeley will not have him as a professeur.

~~~
philip1209
How hard was it to learn these tools as a contractor? I hear that Google is
more aggressively making their tools open-source to help the portability of
their engineers' skills. Specifically, friends have told me that onboarding at
Google is difficult because every tool is a proprietary one built on top of
more proprietary systems. In addition, people who leave Google have trouble
interviewing because they rely heavily on tools that are unavailable outside
of Google. By beginning to contribute more to open-source, I think that it has
the potential to make interviewing at and joining Google a smoother
experience.

~~~
mark_l_watson
Good question. I found onboarding at Google really fun and interesting. I took
several great classes, the best being the end-to-end class that in 8 hours let
you write code that used most of Google's infrastructure - that class was so
awesome that I would have payed Google for that day at work :-)

Also, they had code labs that are self paced modules for learning specific
tech. I didn't spend much time at work doing code labs, but I could access
them at home with my corporate laptop and I went through about a dozen of them
at home. One other nice thing is even mentioning that you couldn't figure out
something from the documentation or code labs would cause someone to jump in
to help you.

Also, I don't think that people leaving Google have problems getting other
jobs :-) The retention rate at Google is surprising low, given the pleasant
atmosphere there. People leave to go elsewhere, start their own companies,
etc. I was 63 when I worked there, and although it was probably not the most
awesome place I worked, it was really great.

Apply for a job if you are interested, or go the easier route and get a
contractor position.

------
nickpsecurity
One thing that bothers me about the article is that it shows a recurring
problem: IT not knowing what it knows. The NoSQL movement didn't notice that
NonStop Architecture scaled linearly to thousands of cores with strong-
consistency, five 9's, and SQL support. In the mid-80's. Instead of making a
low-cost knockoff, like cluster movement did for NUMA's, they ditched
consistency altogether and launched NoSQL movement. Now, I see man who
invented CAP theorem discuss it while referencing all kinds of NoSQL options
to show us the tradeoffs. Yet, there's Google services in production and tech
such as FoundationDB doing strong consistency with distributed, high
throughput and availability.

[http://www.theregister.co.uk/2012/11/22/foundationdb_fear_of...](http://www.theregister.co.uk/2012/11/22/foundationdb_fear_of_cap_theorem/)

So, why aren't such techs mentioned in these discussions? I liked his
explanation of the partitioning problem. Yet, he and NoSQL advocates seem
unaware that numerous companies surmounted much of the problem with good
design. We might turn CAP theorem into barely an issue if we can get the
industry to put the amount of innovation into non-traditional, strong-
consistency architectures as they did into weak-consistency architectures.
There is hope: Google went from a famous, NoSQL player to inventing an
amazing, strong-consistency RDBMS (F1). Let's hope more follow.

[https://static.googleusercontent.com/media/research.google.c...](https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41344.pdf)

~~~
jeswin
> The NoSQL movement didn't notice that NonStop Architecture scaled linearly
> to thousands of cores with strong-consistency, five 9's, and SQL support.

Also incredibly expensive. Take this case, Google took off because they were
able to scale-out with off-the-shelf hardware, compared to the millions banks
were pouring in for scale-up configurations which handled much less load.
Scale-up can quickly hit hard limits, before it becomes exponentially
expensive to continue on the path. This is true even today.

NoSQL movement, if you wanna call it that, took off because most apps
(including Google, including Financial Services, even Health Care) don't need
some types of consistency offered by relational databases. Many large apps are
significantly de-normalized and have many foreign-key less tables, often
filled up by scheduled jobs. That's fine for most apps; NoSQL architectures
recognize that and users consider that in design.

For the majority of use-cases out there, NoSQL databases offer enough
consistency. For the remaining use-cases, there are tools available in NoSQL
databases to make them work, though it requires a bit of work.

~~~
AlisdairO
> For the majority of use-cases out there, NoSQL databases offer enough
> consistency.

For applications with a nontrivial data model, ensuring that each logical
operation only does a single document update (or that multiple
nontransactional updates cause no conflict in maintaining consistency in the
presence of other logical operations) is actually really challenging - and it
adds a substantial design overhead to every new feature added. I think you're
being extremely optimistic in your assertion that NoSQL systems are that
widely safely applicable. My experience has been that NoSQL-based systems stay
mostly-consistent because they happen to experience very little concurrent
activity, not through understanding/design on the users' part.

This is not to make light of the situations where NoSQL systems shine, but the
idea that higher levels of consistency are rarely useful does not match my
experience at all.

> NoSQL movement, if you wanna call it that, took off because most apps
> (including Google, including Financial Services, even Health Care) don't
> need some types of consistency offered by relational databases.

I'd say that Mongo (for example) took off because:

\- They really nailed the setup experience for new users (which RDBMSs
historically sucked at).

\- The data model is much easier for simple apps.

\- They had some fairly creative techniques for making their system look good
- unacknowledged writes, and claims of support for replication which didn't
really fully work.

\- Most programmers don't really understand the ins-and-outs of maintaining
consistency.

------
MCRed
I'd like a real explanation for why containers are better than unikernels.
Yes, unikernals are still early, and containers are convenient, because you
have all of linux there... but it seems that running several linuxes on a
linux machine is a bit much. One operating system plus XEN plus several
applications in unikernels seems more efficient, and more exciting.

But it's the less common choice.

I am guessing convenience is more important than the better solution that
would ultimately be more just as convenient and more efficient if it gets
enough eyeballs?

~~~
jacquesm
I'd like a real explanation for why containers _and_ unikernels are better
than regular run-of-the-mill applications running on dedicated servers. It's
almost as if the wild west of the web isn't quite enough and we now need to
add another explosion of layers-of-abstraction but this time on the server in
order to pretend we have infinite hardware which then becomes it's own reason
for existence rather than to simply run efficient software configured properly
on properly utilized hardware. As if regular virtualization alone doesn't give
enough headaches in trying to figure out why some subsystem does not perform.

At this rate we'll end up shipping containers as 'apps' to the clients
machines with a suitable emulator at some point.

All this luxury comes at (considerable) cost and not everybody seems to be
doing the math before deployment which more often than not leads to terrible
efficiency.

But we're technology fans, so 'oohh! shiny!'.

~~~
mbesto
Do you have experience using containers?

Here's my answer for "why": DRY. Once you've deployed hundreds of servers
using the same exact Ubuntu 12.04 LTS kernel base, why not just completely
abstract the OS away and focus the attention on scaling the OS services that
matter? Why is that when I decide that I need to scale out, I need to copy
every library of the OS and every line of code for the kernel and redeploy it
every time I add a node?

> _At this rate we 'll end up shipping containers as 'apps' to the clients
> machines with a suitable emulator at some point._

That's exactly the point. Care to elaborate on the downside of such a promise?

~~~
jacquesm
> why not just completely abstract the OS away and focus the attention on
> scaling the OS services that matter?

Because it adds a layer that makes no sense unless you have very specific use
cases. Though I see the point regarding people efficiency, that one makes good
sense (see other comment in this sub-thread)

> Why is that when I decide that I need to scale out, I need to copy every
> library of the OS and every line of code for the kernel and redeploy it
> every time I add a node?

If you're doing it that way then you are simply doing it wrong. See: chef,
configuration management and various deployment services (of which you could
argue containers are one off-shoot, but they focus (imo) on the wrong level
for all but the largest companies). Containers are like sandboxes with
significant overhead for applications that focus on ease of deployment (but
that's strange to me because I see that as a one-time cost for most of my own
use cases, though I can see how that equation would change if you deploy lots
of things configured by lots of different people to a single set of servers,
especially if there are conflicting requirements between those deployments).

> Care to elaborate on the downside of such a promise?

That's my personal view of hell, if you don't see any downside there please
ignore my vision and continue as if nothing was said.

From open, text based standards to shipping arbitrary binaries in a couple of
decades. And I thought GKS was about as bad as it got ;)

~~~
mbesto
> _See: chef, configuration management and various deployment services_

A chef script is basically the automation of _" I need to copy every library
of the OS and every line of code for the kernel and redeploy it every time I
add a node?"_ I'm sorry if you didn't pick up on my implied remark. Two
problems are then introduced when automating those actions: (1) it doesn't
negate the fact that I need to store and deploy a 700M sized OS layer every
time I want to add a node (which takes minutes, not seconds with non-
containerized configs) and (2) maintaining config scripts can (not always) be
painful (version control, rollbacks, etc)

> _unless you have very specific use cases._

> _Containers are like sandboxes with significant overhead for applications_

Again, do you have experience using containers? You seem awfully dismissive
("you're doing it wrong!") in a way that suggests that you might not entirely
understand how they actually work...

~~~
15thandwhatever
What I do on our app servers is create a new user for each app (via
configuration management).

That gives me both environment separation (A needs Ruby 1.9, B needs Ruby
2.0), resource accounting on a per-app basis, and a repeatable foundation in
case I need to re-deploy the server or spin up new instances.

~~~
vidarh
That is only true if you use no parts (e.g. shared libraries etc.) of the host
environment. That guarantee that you're not inadvertently depending on
something on the host system that might change is what containers give you.

------
zallarak
"Heroku took off for doing things with Salesforce" \- This is not my
impression at all.

------
jesstaa
Containers, like virtual machines before them, aren't the future of computing.
They're how we manage legacy apps.

The future of computing is not this horrible kludge.

~~~
TheIronYuppie
Disclaimer: I work at Google on Kubernetes

The problem containers are trying to solve is not isolation of environments
(though, that's a tremendously good outcome).

The problem is how do you develop microservices that are self-contained,
easily composable and inherently portable. It doesn't matter if you're running
on Java 1.1 on AIX 3.0 or Node on Ubuntu 26, if I have a ball of computing
providing a service in Dubai, and want to move it to Ireland to take advantage
of computing space that just opened up, containers make that trivial (and
about a million other scenarios).

------
digitalzombie
I saw a kubernete talk at a local meetup.

Google have 40 programmers dedicated to that project. It's still very beta btw
all programmed in Go.

There's also mesos and I think you can use both in tandem since they're
targeting a different thing.

Anyway if anybody is doing or thinking about containers check Kubernete and
Mesos out. Also of course docker and rocket. Kubernete officially support
docker and will be supporting rocket.

There are also article about how rump kernel are better than containers. Just
fyi.

------
voidr
I have been using Amazon Web Services and other cloud platforms for over a
year now, and I never really felt that VMs were the bottleneck in any way. Can
someone explain to me the advantage of containers here?

I know that containers are faster because they don't virtualize the hardware,
however it comes at the cost of security.

~~~
TheIronYuppie
Disclaimer: I work at Google on Kubernetes & containers.

I tried to address it here:
[https://news.ycombinator.com/item?id=9570639](https://news.ycombinator.com/item?id=9570639)

But in summary, the basic thing that containerized software offers is inherent
portability and simple composition. VMs aren't going anywhere; running a
container on top of them just makes everything more powerful.

------
dunkelheit
Can someone explain or provide an educated guess about what is the google's
strategy with kubernetes here? Surely containers are hot now and it is nice to
have a stake in the game but borg has been one of their key competitive
advantages. What is the profit in making an open-source alternative?

~~~
inquisitiveio
I have been wondering the same thing. I suppose it's partly because it was
only going to be a competitive advantage as long as as containers where not
commoditized which seems to be the direction its now heading.

------
kayman
I love the idea of using containers. Due to linux popularity and google's
backing, containers will be next.

But FREEBSD had jails since back in the day. What's the benefit of containers
over bsd jails?

~~~
otterley
Linux has copy-on-write block devices that make it possible to efficiently
layer container filesystems. FreeBSD has no such thing as far as I know; the
best you can do involves hard links (correct me if I'm wrong).

~~~
bch
[https://en.wikipedia.org/wiki/UnionFS](https://en.wikipedia.org/wiki/UnionFS)

Edit: addendum

Additionally, a hardlink solution (as you point out) really doesn't sound too
outrageous to me. A trivial tool to write to manage immutable things like
binaries. Or is there some trouble I'm failing to see (possible)?

~~~
otterley
According to the mount_unionfs(8) man page:

> THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK) AND
> USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM.

~~~
bch
You're right. My personal reference for unionfs is NetBSD, which doesn't
include such a warning [0]; apparently freebsd's implementation needs some
love.

[0] [http://netbsd.gw.com/cgi-bin/man-cgi?mount_union++NetBSD-
cur...](http://netbsd.gw.com/cgi-bin/man-cgi?mount_union++NetBSD-current)

------
andyidsinga
imo, if containers are the future ( very plausible ), then, things like Aws
lambda are just as plausible if only just a bit further out.

I think this is the case due to granularity of workloads and what apears to be
a continuum in the workload container from metal > vm > containers > lambdas
(as first class workloads).

fun stuff

~~~
TheIronYuppie
Disclaimer: I work at Google on Kubernetes

True, lambda is DEFINITELY an advance, but it's more like the salmon at a
buffet that includes steak and chicken. Some applications will need just the
ability to run code (lambda), some will need defined environments (containers)
and some will need total isolation (VMs/bare metal). You'll see a mix of all
of these in every mature environment - some things do not fit. For example,
it's super unlikely that you'd be able to run a trading app on Lamdba (needs
10GB/sec of direct network access & memory); similarly it'd be totally
unnecessary to run a thumbnail processor on bare metal (though you could, of
course).

~~~
andyidsinga
good points but I'm not sure anything about the fundamental lambda
architecture prevents it from being applied to high bandwidth or low latency
uses. especially if lambda functions are orchest rated to run in the metal
...think: kernel module implements lambda, then add orchestration glue up in
user land

------
la6470
With all these container talks people forget Solaris zones which were pretty
advanced sort of containers in Solaris. However Sun was honest about not
mentioning zones as the solution for everything. The most important problem
with containerization is that due to dependence of multiple containers on the
same kernel of the host or VM OS , any kind of upgrade specially the security
patching is virtually impossible to do without taking full downtime.

