
Immutable Infrastructure is the future - geerlingguy
http://michaeldehaan.net/post/118717252307/immutable-infrastructure-is-the-future
======
falcolas
> Ultimately in the end it means more time for everyone to focus on business
> logic versus thinking about deployments, so I’m a fan of that.

I find this amusing: our company is migrating to Docker right now, and over in
operations I'm spending more and more time thinking about deployments than
I've had to since I started doing operations work.

I have to think about "what major functionality has changed in Docker in the
last three months, and how does that impact any/all of our images?"

I have to think about creating frequently updated base images, and how to
deploy these updated base images across our regions.

I have to think about making the image layers as small as possible, to limit
the bandwidth impact of updating images.

I have to think about making as few image layers as possible to avoid
DevMapper IO overhead.

I have to think about externally mapped volumes to keep necessary IO as fast
as possible.

I have to think about asymmetric TCP, docker vs host network interfaces,
natting, and service discovery.

I have to think about private registries, token based authentication systems,
and redundancy.

My developers have been able to focus on business logic since they hired an
ops team. Will the amount of thinking go down over time? Some of it: the bits
which can be automated fade to the back (at least until the core Docker
functionality changes yet again), but there are bits and pieces will
necessarily change with every deploy.

~~~
dasil003
The difference is the pain you are experiencing with Docker is potentially
transient, whereas the problems with configuration management on the wild
internet are intractable, and even unquantifiable. With Docker the scope of
changes you have to deal with is at least knowable.

I say "potentially" because I think the jury is still out whether Docker can
live up to its promise. Clearly it has a real purpose and value add above
traditional configuration management, but the churn is an indication of how
hard a problem it is. Plus, even in an ideal case, you still have to deal with
security updates and other inevitable version updates and things that
invalidate your current images.

~~~
KaiserPro
The big problem is that docker/container image creation is now buried deep
inside an app's build system. This means that your build system is now a
critical part of your infrastructure.

~~~
parasubvert
Isn't that the case with any continuous deployment pipeline?

Best to use the CD system to do the deploy, and avoid "hold my beer, I've got
to emergency deploy this via SSH" which is high risk for minimal gain.

~~~
KaiserPro
No I don't mean pushing patches to your App, I mean critical system libraries.

Because they are baked into the container, to do critical patches you need to
rebake the image.

The Docker bit will be mostly painless, unless you've don't something stupid,
however the bits around the app can and will changes.

The amount of times I've tried to use a build job from 3-6months ago and it
fails horrifically is far too often.

------
geerlingguy
A couple relevant snippets:

    
    
      Caveat - I think images must come from a build system.
      Using anybody’s canned images as a baseline is reckless.
    
      Config tools still have a place - in building the automated
      systems that allow folks to focus on business logic, but I
      think there are going to move down-stack to doing things
      like automating OpenStack.  The longer term future is about
      cloud/virt systems like this provisioning themselves, but
      they are still yet a bit too complicated in the way they are
      going about it, but I feel it’s coming.
    

(The post's author, Michael DeHaan, wrote Ansible in 2012, but left the
project earlier this year to pursue other interests.)

My thoughts on the subject are much the same—but I think it'll take a few
years for the tooling to reach the level of polish we currently have with
best-of-breed CM and general provisioning tools. It's not as easy to go from
0-to-infrastructure with a Docker-based workflow (especially if you have your
own registry, etc.) as opposed to a VM-plus-CM-based workflow.

~~~
mpdehaan2
author here.

"but left the project earlier this year to pursue other interests" \- Hi Jeff!
I haven't really shared reasons for leaving the company to date and can't go
into that.

I also don't think it will take years and think immutable is there today.

I think we're there today with a bit of skill and minor bit of work on top,
but a little bit of work from cloud/virt solutions will push us over the hump
- to be able to codify the upgrade-flip idiomatically within cloud/virt
systems, so you don't need something else to do it for you. I do look forward
to that being surfaced.

Things like Amazon's ECS in particular strike me as particularly interesting.

~~~
geerlingguy
> I think we're there today with a bit of skill and minor bit of work on top,
> but a little bit of work from cloud/virt solutions will push us over the
> hump.

I think that's true, but my timeline is based on how long I see it taking
'mainstream' enterprise companies to adopt immutable infra—for many of the
orgs I've worked with, they've only recently completed a sea change from
dedicated servers to virtualized 'cloud' deployments.

So it really depends on who you're working with; many businesses are making
the move today, but cautiously. Once the early adopters complete rolling
things out to production and working out the kinks, we'll finally get to the
late majority/stability, and I won't have to commiserate with the thousands of
sysadmins stuck with dozens, hundreds, or thousands of VMs :)

------
KaiserPro
I quite like immutable infrastructure. It means you have a rollout/roll back
with a flip of a dns/connection.

However I see a lot of things about docker and think two things:

1) most people really wany a mainframe 2) shared filesystems really are
brlilliant.

Let me unpack that:

a mainframe provides a massive amount of hardware, a hypervisor and scheduler.
This means that you can control a bunch of isolated processes through one one
system easily. Something that you can't really do in docker yet
(fleet/kubernetes arn't there yet. in a mainframe you can use a single script
to control an entire system)

Quite a lot of web config management is about shipping packages to different
machines. Almost all of this config, and therefore complication can be removed
by using a shared filesystem.

For example we have many version of java. For some reason they are in seperate
repos. This means that if you want to change version (openJDK to oracle etc)
you have to change config. With a shared file system, in the $PATH, you'd just
say _oracalejava-1.8 $programme_ or _java-1.7 $programme_

If you make your deploys to a similar shared filesystem, then to change config
is basically a case of ssh machine, kill old process, launche new process. Job
done.

This makes each machine a dumb execution system, minimal state is contained in
each node.

With NFS coming to AWS, that'll be the model for my new infrastructure. the
reason why? because it works. In HPC thats how we've been managing high scale
workloads for years.

------
runlevel1
I'd like to see a polished common configuration API.

Currently, we have Chef, Puppet, Ansible, and Salt (amongst others) all
working to create a sane common abstraction on top of a rainbow of
configuration files and command-line arguments.

Each of them have their strengths, and frankly they're all impressive in their
own right. But there's only so much you can do to reign in the chaos of the
bazaar.

The Docker workflow certainly has its merits. It's great for narrowing dev-
prod parity and repeatability. I've embraced it, but I'm a bit anxious that
we, as a community, might end up creating bigger black boxes -- Dockerfiles
upon Dockerfiles.

~~~
mpdehaan2
Hmm, I think your boxes can be still pretty transparent in that case - if
anything more so, as you've got single-bash files versus perhaps directories
and directories of content?

In most organizations, all those Dockerfiles should be yours, and in source
control.

Relying on the blobs from others shouldn't be neccessary for most
applications. Yes - in large organizations, a management problem probably does
arise, but I think if you find copy/paste rising up you have to push that down
into the base image, basically?

I don't think a configuration API helps to be honest - what you would do is
turn them all into a "greatest common divisor" type scenario, where as they
feel different because they were trying to be different.

I think you're best solved by organizationally mandating one of them within
your company, as painful as that may be.

------
nickbauman
> While PaaS has not become a reality for everyone...

A telling aside. Much of what he wants already exists in Google App Engine
(and probably in Elastic Beanstalk, too, but I haven't tried it) — sans the
need to manage the underlying OS.

Interesting Google is going the other way with Managed VMs running on top of
Kubernetes. Perhaps they will meet in the middle somewhere?

~~~
w4tson
I thought Google was running containers in kubernetes? AFAIK you can install
Kubs on VMs

~~~
kevinschumacher
"Managed VMs" is a service offering within Google App Engine that lets you use
any runtime (e.g., Python, Node.js, Go, C) but get the benefits of the GAE
PaaS auto-scaling, versioning, etc.

It actually works by shipping a docker container to the Google Container
Registry (privately scoped to your project), spinning up a "VM" that shows up
in your Compute Engine instances list that you can then SSH into if you want
(this is why you're both right; they call it Managed VMs, it shows up in your
GCE instance list, but they literally are, I think, running your docker
container, probably with Kubernetes under the hood).

~~~
nickbauman
This is exactly what's happening as I've seen it.

------
ABS
For those looking into this topic only now we interviewed 6 people who have
been doing this for some time, collected their answers and published them +
did a hangout with them here: [https://highops.com/insights/immutable-
infrastructure-6-ques...](https://highops.com/insights/immutable-
infrastructure-6-questions-6-experts/) (pure content, no sales pitch anywhere)

------
akurilin
Very interested in this subject. Is there a comprehensive guide on how I can
pull this off TODAY with e.g. AWS?

~~~
eropple
It's only one piece of the puzzle, but at my last company I wrote
deploy_thing[1] for exactly this. An immutable config bundle and an immutable
artifact tag combine to create a deploy that can be launched and put into
production with an AWS auto-scaling group. Need to change a configuration, you
burn down the auto-scaling group. This was a transitional step for us before
moving to Mesos, but it's a good example, I think, of how to approach the
problem from a minimum-viable level.

(I'm currently playing with moving it to OpenStack.)

[1]
[https://github.com/eropple/deploy_thing](https://github.com/eropple/deploy_thing)

~~~
bender80
The URL is throwing 404, probably due to it being private.

Would love to see it. I am working on something similar.

~~~
zerocrates
It looks like it was supposed to a hyphen, not an underscore:

[https://github.com/eropple/deploy-thing](https://github.com/eropple/deploy-
thing)

~~~
eropple
Whoops, yeah. My bad, I switched from an underscore to a hyphen halfway
through the project. It's still inconsistent, because we had a case of
company-going-unders and I haven't needed it since.

------
andrewchambers
Stateless servers with kubernetes seem like a great thing to do this sort of
thing with.

~~~
nickbauman
Why go that low, though? Why not go with Managed VMs first?

~~~
andrewchambers
Kubernetes gives you freedom about where to run your stuff since it works on
multiple providers and your own hardware. Once you learn how to set it up, it
isn't too bad.

Google managed kubernetes cluster gives you the best of both worlds anyway.

~~~
KaiserPro
A decent VM system will give you HA and load balancing (ie migrating VMs from
overloaded hosts) for free.

Something docker/kubernetes still can't do.

Its also worth bearing in mind that the "vm penalty" is almost trivially small
compared to the cost of dev time.

~~~
deathanatos
> […] (ie migrating VMs from overloaded hosts) for free. > Something
> docker/kubernetes still can't do.

This doesn't solve issues arising from the VM outright crashing though, does
it? (I don't see how it could.) At the end of the day, I still need to solve
that myself, and if my application can gracefully withstand being crashed,
then docker/kubernetes _can_ migrate from overloaded hosts: just crash
container.

I could probably also throw in something to gracefully die on a SIGTERM, but I
feel like once you can withstand a crash, throwing in a SIGTERM is fairly
straightforward. The only difference is that one trains traffic, the other
just outright 500s it. (Though, perhaps there's more work involved there than
I realize.)

Ultimately, I want crash resistance. I want to run Chaos Monkey. I can't,
right now, because I know all too well what would happen.

~~~
KaiserPro
In HA mode, you specify that this VM must be running. If its not do a list of
certain actions.

But the thing is, making your app crash gracefully can be incredibly hard, and
can almost always take a lot of dev time.

its cheaper in terms of planning and execution to have two have an HA pair
defined in the cluster, and another HA pair defined in software (i.e. behind a
varnish node or the like)

You can have virtual IPs so that if a node is not responding the HA partner
picks up the traffic. If thats properly locked up, the hypervisor can kill the
machine and restart it.

as for: >migrate from overloaded hosts: just crash container.

try doing that with a clustered something, the host becomes overloaded, kills
the app. not only does the traffic shift off to other nodes(which might
already be started to overload), it takes resources to sync back into the
cluster.

You don't want this behaviour as it causes failure hysteresis. (that is, as
soon as you start to reach breaking point, the whole thing collapses and
refuses to come back up, without stopping incoming traffic)

------
dorfsmay
Yes, I agree, but the issue is still how do you build that infrastructure (be
it an AMI, a docker image etc..)?

Docker file sare simplistic, so you end up using the same tools that were
available before (ansible, salt, bash etc...).

What we're doing right now is move what we used to do at "configuration time"
to "build time".

~~~
mpdehaan2
My point was you don't need to use the tools you use today, _except_ to do the
version flip (and probably to deploy the subtrate - unless you are doing a
cloud-image solution and not trying to do something on-premise). And this will
get better when cloud providers embrace higher level cluster operations more
natively.

And build time is the way to go for everything you can push down into build
time. There still may be some need for service-discovery type applications.

~~~
dorfsmay
Agreed, moving more and more stuff to the build, lighting up an image should
be nearly instant, so we can do auto-scala etc... so no more "pip install -r"
when bringing up an image, that's now part of the build.

But the landscape for tools to build that image, be it a docker image or an
AMI, right now, is the same. I find myself "going back" to ansible but now
using it to build images rather than change online VMs (could be any tool, I
know ansible and it suits my way of thinking).

------
tomc1985
So by immutable he means... systems you don't patch? That's it?

------
kolev
Wrong! It's the present!

