
Docker 1.12: Now with Built-In Orchestration - sashk
https://blog.docker.com/2016/06/docker-1-12-built-in-orchestration/
======
d33
I chose Docker for my project with hope that it would help me create an easily
reproducible development environment... And I'm not really sure if it was a
good idea. There's so many moving parts and I can't name one that actually
works well. Configuring storage is a hell (running out of disk space can lead
to errors that are really difficult to debug), the dance with bridges makes
configuring firewall a terrible experience. I definitely wouldn't recommend it
as a stable container solution.

~~~
redrummr
Why is storage an issue? With databases, for example, you store data upstream
(on the host) and keep the dockerised app/thing mostly stateless, or at least
capped with regards to storage. More details about your environment would be
appreciated.

~~~
jdoliner
Keeping data on the host is exactly the reason that it's an issue. Swarm
offers not guarantees that containers will be rescheduled on the same host
that it was initially scheduled on and assumes that it can reschedule
containers as failures arise... this doesn't work when your database gets
rescheduled and emptied along the way.

~~~
kerr23
In docker 1.12 you can 'tag' a node and tell a 'service' to only run on the
tagged nodes.

Not saying that's a good idea, but it is getting closer.

You could, for example, have a node that's only for DBs that has volumes on
it. You could then use DRBD on the host to clone that data to a secondary
node. then in the event that node 1 dies swarm would bring the DB up on node2.

With the mesh network stuff they've added the endpoint would remain the same,
so all your apps would need to do would be re-connect.

~~~
jdoliner
> Not saying that's a good idea, but it is getting closer.

No, that isn't getting closer it's getting farther away. The whole point of
containers is that they make the host machine completely fungible. If I can
only schedule my DB containers on a specific machine then I might as well just
run my DB on that machine and be done with it.

~~~
drdaeman
My personal opinion is, Docker's good when you need to be sure you deploy
exactly the same stuff on multiple hosts. This includes ensuring that the code
you try on local machine would be the same on the production hosts. Docker is
completely worthless for deploying singleton services that you already have
packaging and deployment recipes for - it just doesn't add any value for such
use cases.

That is, Docker is just a packaging system + a tool to spawn the deployed
stuff in an isolated environment. Now, 3 years after, they've improved
dependencies (previously done by docker-compose + hacks for using it in
production).

That is, I use Docker only because its images are somewhat easier to build and
deploy, compared to .deb packages or plain tarballs (and it's more OS-
agnostic, since Docker Machine's available on Windows and OS X, so I can just
build an image with dev. environment and don't care what OS is used). Doubt
it's something more than this.

~~~
Annatar
> My personal opinion is, Docker's good when you need to be sure you deploy
> exactly the same stuff on multiple hosts.

There is a technology for that already: it's called OS packaging. Docker does
not maintain enough metadata to allow for in-place upgrades of individual
components in an image (software and configuration lifecycle management). The
best you can do with Docker is install lumps, you can forget upgrades. Docker
is not a replacement for configuration management, and specifically, Docker is
not a replacement for the operating system's software management subsystem.

~~~
orf
> Docker does not maintain enough metadata to allow for in-place upgrades of
> individual components in an image

Your missing the point of a container, you don't upgrade them in place

~~~
Annatar
Containers are actually resource constraints, and were first introduced in
Solaris 10, along with zones, which is what you appear to understand under
containers. Even on GNU/Linux, containers are implemented via cgroups, which
are resource controls and not a virtualization solution.

On SmartOS, when you provision a zone, you get a fully virtualized UNIX
server, and you can apply a container to it by defining resource constraints,
but that is both pointless and unnecessary there. Once you have a fully
virtualized server provisioned from an image with the imgadm(1M) and
vmadm(1M), it is only logical that you will want to service individual
components via pkg_rm and pkg_add, rather than baking an entire image all over
again, and redeploying it, all over again. It's the rule of modularity: "write
simple parts connected by clean interfaces" [Kernighan-Plaguer], and it
applies particularly well to lightweight virtualization.

------
csears
Just based on the details mentioned in the keynote, the services API and swarm
features look very similar to what's offered by Kubernetes... service
discovery, rolling updates, load balancing, health checks, auto-healing,
advanced scheduling.

I guess this was to be expected, but it's also kind of sad. I think it would
have been a more strategic move to embrace Kubernetes instead of trying to
compete with it.

~~~
cerisier
Have you ever tried running Kubernetes yourself ? Kubernetes is cool yes, but
only on GCE, and managed by Google people. Apart from that, its nowhere near
usable.

Docker 1.12 does just what Docker does best: Making complex stuff simple to
use and give control back to people.

~~~
advisedwang
Kubernetes can totally run outside of GCE. For example here is the official
k8s-on-EC2 doc: [http://kubernetes.io/docs/getting-started-
guides/aws/](http://kubernetes.io/docs/getting-started-guides/aws/). You can
run it on your own VMs or physical machines if you want.

The only GCE specific thing is Google's _managed_ kubernetes offering, Google
Container Engine (confusingly called GKE).

------
namidark
I wish they would address stability issues first, like this bug[1] that has
been around 2 years and is still happening on the latest versions.

[https://github.com/docker/docker/issues/5618](https://github.com/docker/docker/issues/5618)

~~~
SEJeff
Or this nasty one[1] that has been around just over a year now. It didn't take
me more work than using docker as a build server to run into this. There are
some udev hacks that make it better, but the underlying issue is ultimately
docker here.

Doesn't seem that docker inc upstream takes stability seriously from my
standpoint.

[1]
[https://github.com/docker/docker/issues/13885](https://github.com/docker/docker/issues/13885)

~~~
cpuguy83
Hard problems are hard. We are looking at all kinds of issues related to this,
and in reality reports from that issue are likely caused by different things
(some things are fixed, others obviously not).

There are several people looking how we can resolve issues like this.

------
ShakataGaNai
This seems like a really exciting, albeit natural, evolution of the Docker
"platform". It's really true that no one cares about containers, everyone just
wants their apps to work.

That being said the one giant omission from Docker is still seems to be
management of volumes/data. Great, we can run 100 nodes on AWS in a matter of
minutes, but if your system has data storage requirements (ex: Almost every
database ever) ... You're kinda left up to your own still. How does Docker
Orchestration migrate the data volumes?

They really tried to sell this as "You don't need to do anything but the AWS
beta, the DAB and a few commands" which would be wonderful. However with the
need for reliable data storage... you're still stuck doing everything "the old
fashion way".

(Edit: No, I don't mean store data IN the container, I presume no one is that
silly. I meant the attached volumes. No volume mgmt = less greatly less
helpful).

~~~
gongfupotass
It seems to me they are more focused on NOsql databases with multiple nodes
and shared data like cassandra. So that you don't have to worry too much about
volume/data mgmt because you will have multiple copies of the data on other
containers when you lose a docker host or a container.

You also should conside that docker might not be the best option as opposed to
just some vms on openstack for your much needed databases.

I'm stuck trying convince my co-workers that docker is not the answer to
everything.

~~~
ShakataGaNai
In no way did I mean to convey "store data IN the container" as I had simply
assumed no one was that silly.

What I actually meant was as "there is no way to deal with volumes". So
spinning up 100 instances and hot failover doesn't help if it doesn't deal
with the attached volumes. Without that functionality, the orchestration is
less magic and more "eh".

------
hosh
I'm not surprised the Kubernetes primitives got copied into this. I had stated
before that Kubernetes was what Docker wants to be when it grows up -- and
maybe that time is here. Having used Kubernetes in production, I don't know
robust the Docker orchestration primitives are in comparison. But I'll
probably find out soon.

The big advantage having a built-in Docker orchestration is that Kubernetes is
painful to install from scratch. (Yes, there are scripts to help mitigate
this; yes, GKE is effectively hosted Kubernetes). I'm involved in another
Dockerization projection, but we don't know if we want to invest the time into
setting up Kubernetes (though GKE is an option). This would be a good time to
check things out.

Just to show how quickly things change. In 2015, I tried the docker compose,
AWS ECS, and Kubernetes for production deployment, and of all of them,
Kubernetes addressed the pain points in a practical way. The Kubernetes
project ended in late 2015, and now six months later, things changed again ...

~~~
lobster_johnson
Having set up Kubernetes to test, I'd say it's a bit less complicated than
many people claim, though still obviously complex. I had a two-node cluster up
and running in an hour using Flannel and Kismatic's prepackaged binaries for
Debian/Ubuntu [1].

The biggest hurdle, ironically, is the official documentation, which seems to
have been written by someone with no understanding of explaining things from
first principles, or any talent for organizing ifnroamtion. You'd think the
"setup from scratch" guide [2] would be the most detailed and useful for
someone who wants to see how each piece fits into the next one, but it's
actually terrible. It skips a lot of implicit information and makes some odd
assumptions/suggestions. Almost every part of Kubernetes is multiple choice,
which affects later steps. For example, if you use Flannel and you're not on
GCloud, much of what Google's docs say about networking is moot.

Everything is also made harder by the fact that Kubernetes is changing awfully
fast in non-backwards-compatible ways, and chances are that if you google some
config problem, the answer is long out of date — the necessary kubectl option
no longer exists or whatever. A lot of config recipes are useless now.

[1]
[https://github.com/kismatic/repo_server/find/master](https://github.com/kismatic/repo_server/find/master)

[2] [http://kubernetes.io/docs/getting-started-
guides/scratch/](http://kubernetes.io/docs/getting-started-guides/scratch/)

~~~
hosh
I set up Kubernetes on AWS, a staging and a production cluster. Staging had 1
master, 3 worker nodes, Production has 3 masters, 5 worker nodes, with two
compute classes (fixed and burst, so pods could be scheduled to what was
needed). All of this was on CoreOS, and included independent etcd clusters. So
all total, we're talking 18 AWS EC2 instances. I put live users on it, with
real traffic and tinkered with it as I learned things from a production
workflow.

This was back in mid-2015, which meant figuring out things like how to get
overlay networking with AWS's routing, fiddling with AWS's IAM settings, VPC
settings, figuring out how to split the nodes across availability zones. This
included integrating with AWS AutoScalingGroups to bring up new CoreOS/K8S
nodes whenever something goes down. I never got EBS working as a Kubernetes
volume type (at the time, you have to add hackish tools into CoreOS to detect
unformatted EBS blocks and format them when they get mounted; forget it), but
fortunately, used AWS RDS for persistent stores. Persistent storage is
something I look forward to trying on GKE.

I _could_ have tried creating CloudFormation scripts but I was making my way
through documentation, blog posts, and Github issues to pull it all together.
It was worth it when I was done.

I didn't say it was complicated. I said it was a pain in the ass, and it's
something that Kubernetes developers have acknowledged and are working on it.

------
waitwhatwhoa
At first I was worried that this is yet another Raft implementation, but it
seems they're building on etcd's version [1] which should be a win for
everyone trying to prove correctness for the various implementations.

1\.
[https://github.com/docker/swarmkit/blob/a78a157ef66adf0978f0...](https://github.com/docker/swarmkit/blob/a78a157ef66adf0978f031e7bbc9bff2c16df9a0/manager/state/raft/raft.go#L19)

~~~
gtirloni
I like it that etcd is self-contained and easy to troubleshoot, as opposed to
a blackbox with tons of magic.

The demo for `docker swarm` was great but as an infrastructure engineer, it
worries me. First because of the blackbox effect I mentioned above and
secondly because I'm not sure how much flexibility is compromised to achiveve
that.

Sure, it's awesome to quickly have a cluster up for your app and the user
experience is phenomenal... but what about storage, network, firewalls,
integrating with my linux distro, etc. Things are never this simple when they
reach a certain scale and/or number of requirements deviate from those of a
simple app.

From a quick look at the docs and watching the demo, Kubernetes still feels
like a safer option, today. That being said, it's nice to see competition in
this space. I guess this will put pressure on Kubernetes and others to improve
the user experience further.

------
geodel
I also noticed that now 'Docker for mac' is available without beta invite. I
just downloaded and installed though its version is 1.12.0-rc2.

Maybe they will make formal announcement when 1.12 for mac is released.

~~~
ingve
They have announced a Public Beta:

[https://blog.docker.com/2016/06/docker-mac-windows-public-
be...](https://blog.docker.com/2016/06/docker-mac-windows-public-beta/)

~~~
geodel
Ohk, great. I did not notice that blog post.

------
eddd
I wish docker released lite version of docker. Only containerisation, no
networking configurations, weird cache mechanisms - just something small and
easy to use locally.

No one sane will run docker on prod anyway.

~~~
ams6110
Just use libvirt (virsh utility command) if that's all you want.

~~~
sciurus
Oh gosh, no. Whatever faults Docker may have, they're outweighed by its very
approachable UI compared to the discombobulation that is libvirt's interface.

[https://libvirt.org/drvlxc.html](https://libvirt.org/drvlxc.html)

------
lvh
The new orchestration looks based on Swarm and possibly Compose; have they
resolved the issue where the lack of lifecycle semantics means that your
services don't work if they have any sort of cluster semantics, like Riak,
Cassandra or Zookeeper? These previously always required outside hacks to
work, and work flawlessly with Kubernetes. It looks like there's some service
discovery support; which may or may not fit the bill.

~~~
shykes
Yes, all of that works seamlessly now. Each container gets an IP and is
discoverable via DNS.

~~~
falcolas
Due to caching implementations which change between OSes, programming
languages, and routers, DNS is actually a really terrible discovery mechanism
at scale.

For example, I'd never want to implement service discovery via DNS in Java,
which ignores expiration times and caches queries according to its own
settings. I've also seen some server configurations which result in every
gethostbyname call to a fully recursive query against DNS.

~~~
shykes
> _Due to caching implementations which change between OSes, programming
> languages, and routers, DNS is actually a really terrible discovery
> mechanism at scale._

Only if you rely on the client's resolver caching implementation, which we
don't. Docker networking guarantees that a given hostname lookup by a given
container in a given service discovery scope will always return the same IP.
In that configuration, using DNS is a robust fit for production.

Believe it or not, Docker is built by people who have actual operational
experience :)

~~~
bboreham
> a given hostname lookup by a given container in a given service discovery
> scope will always return the same IP

Did you mean this to cover the situation where N containers have registered
the same name? I thought Docker DNS returned all such results in random order.

Given that, if one of those containers goes offline and Java has cached its
address, you have the problem the parent was alluding to.

~~~
shykes
That's incorrect. You are describing DNS-based load-balancing, which would
indeed rely on the container's resolver implementation. But Docker doesn't do
that. Instead it always resolves the service name to the same IP, which is
load-balanced by IPVS. That way even the world's crappiest dns caching
implementation will still be handled properly.

So when I said that Docker doesn't rely on the container's DNS resolver, I
really meant it. We have seen in past lives the consequences of "DNS abuse"
and have been careful to avoid it.

~~~
bboreham
I was going by [https://blog.docker.com/2016/04/docker-
engine-1-11-runc/](https://blog.docker.com/2016/04/docker-engine-1-11-runc/)
which advertised "DNS round robin load balancing".

EDIT: calmer tone

~~~
theod
Docker 1.12 built-in load-balancing supports both VIP based LB using IPVS and
also DNS-RR and it is configurable per-service. VIP based LB is the default
though. All of these will be fully documented shortly.

------
boulos
I find it super amusing that GCE isn't mentioned, that ostensibly Kubernetes
is the source of lock-in (despite being open-source and likely having more
cycles on EC2 than on GCE/GKE), and that this proudly uses gRPC for http/2 and
protobuf goodness. So which is it, is Google an evil vendor trying to lock you
in, or are we actually doing work in the open and just hoping you'll choose us
when you want infrastructure?

Disclaimer: I work on Compute Engine, and think of the Kubernetes folks as
friends.

~~~
shykes
Disclaimer: I work at Docker.

You are reading way too much into this. We're introducing orchestration in
Docker because it solves a problem for our users. We haven't called your
employer or anyone else "evil", and frankly this announcement is not about
Google or any other company. It's about improving Docker for its users.

Speaking personally, I think GCE is a great product, Docker wouldn't exist
without Go, and although grpc is not my personal favorite, it gets the job
done and it's quite popular with Docker engineers. The libcontainer project
(now runc/containerd) started with a very fruitful collaboration between
Docker and Google engineers. The current Docker networking model is also
heavily based on early feedback from the Kubernetes team (one IP per
container, remove nat between containers, etc.)

Yes, some features of Docker overlap with other products. One of those
products is Kubernetes but there are dozens of others. That kind of healthy
competition is normal and good for users!

------
foxylion
If someone else also want to see the keynote, they have a recording:
[http://www.ustream.tv/recorded/88610090](http://www.ustream.tv/recorded/88610090)

~~~
jjn2009
Thanks, I was looking all over for this! Much appreciated.

------
cdnsteve
I have two code bases that are microservices. I thought using Docker would be
easy to setup and share a redis instance between these services. After
spending a day trying various thing I just gave up and did 'brew install
redis' and got back to real app dev. Docker isn't streamlined enough, does
have great examples and is still changing lots. Sometimes doing it the old
fashioned way is what lets you ship features faster.

------
kstenerud
Omg their navigation bar covers half my screen! Seriously, why do companies
inside upon reducing my screen real estate for stupid shit?

~~~
rcarmo
I actually think this is a valid point, even if somewhat colloquially
expressed. I had a lot of trouble getting past that menu bar on Opera Mini and
even on an iPad, it's entirely too much.

~~~
wallace_f
Agreed. To be honest, I think the colloquial nature of the post is fine in
this case.

------
heurist
Great, now make it a first class citizen in compose and mix in docker-
machine's provisioning capabilities!

------
Annatar
The article is lots of blah blah plus "lock-in" for good measure, with exactly
zero code or configuration examples, and it reads like a snake oil salesman
brochure. I'm still skeptical about the purported advantage of Docker / Linux
over zones / SmartOS.

------
ssijak
I was using osx beta but uninstalled it for now because docker was eating all
my space without recovering it after deleting containers. Anybody using rkt
for dev wants to share their experiences, I am thinking about trying it but
would not invest time if it is not ready yet.

~~~
foxylion
There may be images or volumes you should remove when no longer required.

~~~
ssijak
Nope, it was known bug and still not fixed by the time I removed it.

~~~
avsm
Sorry about that -- this was a bug in one of the betas where log rotation was
broken. It's been fixed about 3 or 4 betas ago via a mid-week hotfix, so the
latest open beta should be just fine. Hope you get a chance to try the latest
series again soon!

------
estefan
Have they sorted out storing secrets yet? That's one thing I noticed that
Kubernetes manages that docker is still scratching it's head over.

But this looks like it could be great progress. Orchestration was always the
main pain that stopped me from using docker.

------
rcarmo
This is great. I can't wait for an ARM build, though.

~~~
PanosJee
Try resin.io!

~~~
rcarmo
I did. I want the mainstream stuff.

------
tapirl
Docker is great (wonderful in fact) for building development/CI environment. I
never use docker for production.

------
curiousDog
Maybe i'm not uptodate or misunderstanding, but doesn't Mesos/Mesosphere do
something similar?

~~~
wmf
There are a dozen tools that do something similar.

------
webaholic
is docker as good as rkt nowadays?

