
Docker in Production: An Update - smackay
https://thehftguy.com/2017/02/23/docker-in-production-an-update/
======
crb
It's great that hftguy thinks Google Container Engine is stable (I work on it)
but I'm sorry to say it's very easy to prove that is, in fact, running Docker
on the nodes.

You can just SSH into one and see for yourself.

Kubernetes was built from the ground up to orchestrate Docker. CoreOS did a
lot of work to make it possible to trade rkt in for Docker's engine, and the
cri (Container Runtime Interface) is now generalising that so that there is a
clear abstraction between the kubelet and the engines it orchestrates. Read
about it here: [http://blog.kubernetes.io/2016/12/container-runtime-
interfac...](http://blog.kubernetes.io/2016/12/container-runtime-interface-
cri-in-kubernetes.html)

If you want to do things that are different to what we provide support for on
GKE as a Managed Service (tm), you're able to run your own Kubernetes clusters
on GCE. (We do let you run a Kubernetes alpha version, but only on non-
supported clusters that self-destruct after 30 days.

~~~
SEJeff
Is it eventually going to run rkt/cri-o/lmctfy?

~~~
crb
If you're a customer and you're interested in either of the first two, please
ping me an e-mail.

GitHub's lmctfy README states "We are not actively developing lmctfy further
and have moved our efforts to libcontainer."

------
mwpmaybe
I am neither a Docker expert nor evangelist, and I have my own gripes and
frustrations with it, but this article is full of misinformation and FUD. To
wit:

 _> CoreOS is an operating [system] that can only run Docker and is
exclusively intended to run Docker._

No, it isn't. (Maybe if you substitute "Docker" with "containers.")

 _> First, the main benefit of Docker is to unify dev and production. Having a
separate OS in production only for containers totally ruins this point._

No, it doesn't. Completely the opposite, in fact: Docker makes it possible to
not care about OS differences between dev, test, and prod.

 _> Docker on Debian is major no-go

> Docker is 100% guaranteed suicide on Debian 8 and it’s been since the
> inception of Docker a few years ago

> Debian froze the kernel to a version that doesn’t support anything Docker
> needs and the few components that are present are rigged with bugs._

 _rubs temples_ It's Linux. It's Debian. You can run any kernel you want.
There are plenty of repositories out there with binary kernels, including
Debian's own back-ported 4.9 kernel, or you can build one from source.
Because, you know, open source. I've run many instances of Jessie on Linode
with their 4.8 and 4.9 kernels and it's no problem. Hundreds of companies are
running Docker in production on Debian and Debian-derived systems.

 _> I am not aware of any serious companies than run on Ubuntu._

Just because you are not aware does not mean they do not exist. How about
Netflix, Snapchat, Dropbox, Uber, and Tesla for starters?

 _> I cannot comment on the LTS 16 as I do not use it._

It's been out since last April. If you're serious about this survey, this is a
no-brainer.

 _> I received quite a few comments and unfriendly emails of people saying to
“just” use the latest Ubuntu beta_

What? No. Just use the latest LTS.

 _> I am moderately confident that there is no-one on the planet using Docker
seriously AND successfully AND without major hassle._

LOL

~~~
cjhanks
I have tried to use Docker on very busy processing systems with short to
medium running tasks. My efforts were on Ubuntu 16.04... because any non-LTS
is almost a non-starter.

There were two major obstacles; the file system driver and the Docker daemon.
I ended up settling on 1.11.{i forget}, because virtually every other version
was unusable. As sad as it may sounds, I was tracking two metrics; probability
the container will start and time until the docker daemon reaches a state of
irrecoverable dead-lock.

Personally I found containers (on a highly contentious system) launched at
~98% success rate with a time-to-dead-lock somewhere around 6500 container
start/stop (it was a pretty fat tailed distribution). For a busy system...
those numbers equate to an administrative headache.

And on Ubuntu/Debian the defaults (the thing the majority of non-specialists
are probably using) were far worse. The only filesystem driver which seemed to
work at all was devicemapper direct-lvm.

-

If you have developers on your team who can spend their time figuring out the
one magic incantation that makes Docker work most of the time, you're fine.
Everybody else should follow his advice for using service providers.

~~~
mwpmaybe
Were you using Device Mapper purposefully, or just because you couldn't get a
more typical storage driver to work? The general setup instructions recommend
either 1. installing linux-image-extra and using AUFS (for kernel versions <
4) or 2. modifying the dockerd invocation to specify the OverlayFS storage
driver with `-s overlay` (for kernel versions >= 4).

Admittedly, the latter strategy is a bit frustrating and counterintuitive on
16.04. When you install Docker it will start automatically and hang because it
can't find the AUFS driver and Device Mapper isn't set up. The solution is to
either 1. modify policy-rc.d to prevent services from automatically starting
or 2. set up Device Mapper (install dmsetup and run `dmsetup mknodes`) before
installing Docker and changing the storage driver. Unfortunately, these
workarounds are not particularly well-documented.

I would be interested to see how Docker 1.12 or 1.13 with a modern kernel and
OverlayFS would be able to handle your described workload.

~~~
user5994461
So you are saying that all default settings are unusable in production, to the
point docker might not start at all, and the only cure is series of 5 obscure
advanced system setup/configuration that are almost impossible to figure out
by oneself and not documented, yet they should be totally obvious to anyone
using Docker, right?

FYI: It's because of this sort of bullshit that there are articles called
"Docker in Production: An history of failure".

~~~
mwpmaybe
No, that's not what I'm saying. I'm saying that Ubuntu Server 16.04 LTS comes
out of the box without AUFS and Device Mapper, and that you either need to
enable one or the other before installing Docker or prevent Docker from
starting automatically at install time. It sucks, yes, but your description of
the problem and its solution is beyond hyperbolic. Googling "docker ubuntu
install hang" takes you right to the GitHub issue with the solution at the
bottom. And I think it's fixed in Docker 1.13 (it will prefer OverlayFS if
AUFS and Device Mapper are not available), although I have not tested it.

~~~
user5994461
The GitHub issue:
[https://github.com/docker/docker/issues/23347](https://github.com/docker/docker/issues/23347)

And there goes my hope of ever running Docker on Ubuntu with any success.

------
gtaylor
I get the sense that this person wrote a sarcastic, vaguely entertaining piece
that drew lots of views. Hoping for lightning to strike twice, they did this
again because who doesn't love traffic? That seems to be the entire motivation
behind this post, from what I can tell.

There's just not a ton of substantiated content here. It's mostly really lousy
anecdata like:

> Sadly, I am not aware of any serious companies than run on Ubuntu.

I am troubled by the direction of Docker and there are serious issues, some of
which were raised in this post. However, whatever signal there is in this post
is lost in a sea of noisy ranting.

My advice to anyone who doesn't already have strong opinions:

* Complement any reading that you do with your own research.

* Don't try to invent your own container orchestration system.

* If you aren't sure how to best do something, ask someone!

And for the love of everything holy:

* Don't run stateful systems on Docker if you can't handle failure or data loss!

Docker and orchestration systems like Kubernetes can be an excellent pairing.
It's going to require research, a change in how you develop, build, test, and
deploy systems, and a gradual building of operational experience. It will not
be a quick process, and it's not for every org. But for some orgs and usage
cases, it's an excellent way to go!

~~~
awinder
I'm not sure I've personally been able to figure out if it was sarcasm or
willful ignorance, but yeah, this article annoyed the pants off me just as
much as the first in the series. The fact that it's now worked twice to hit
the front page is also a little unsettling, because apparently one of the best
ways to reach people on HN now is to just post a bunch of misinformed FUD
(with !!Attitude!!) and jump on board the rocket ship!

------
kordless
> CoreOS is an operating that can only run Docker and is exclusively intended
> to run Docker.

This is a patently false statement and needs to be revised, especially given:

> I will not comment on it.

P.S. I'm getting downvoted for speaking the truth, no matter how pedantic it
may seem. CoreOS can run rkt containers and is not exclusively intended to run
Docker. One possible reason is because Docker is the 800 lb. gorilla in the
room and having options is always good.

P.P.S. Gotta love the Streisand Effect. :)

~~~
wrs
It is an especially silly statement considering that CoreOS had a much-
publicized falling-out with Docker and created rkt to replace it.

------
bitwalker
> CoreOS is an operating that can only run Docker and is exclusively intended
> to run Docker.

This shows an astonishing level of ignorance for someone who claims to have
done their research.

> First, the main benefit of Docker is to unify dev and production. Having a
> separate OS in production only for containers totally ruins this point.

What? This makes no sense. Your images will be the same between dev and prod,
even if the host running the containers is different - which is really the
whole point - if you build and run an image in dev, it should run identically
in prod.

> If you like playing with fire, it looks like that’s the OS of choice.

We spent the last year+ running containers on Centos7 with no problems from
the OS. Whatever issues we did encounter were either transient bugs with
Docker or our own configuration. Perhaps we got super lucky, but we were
running 120+ containers on 12 hosts, so I would've expected at least some
evidence of significant problems within that timeframe if it were really such
a risky setup.

> It’s not possible to build a stable product on a broken core, yet both
> Pivotal and RedHat are trying.

We've been running OpenShift Origin since March of last year, it's been very
stable during that time - the few issues we did encounter were due to our own
mistakes, and were usually fixed just by changing some configuration and
restarting the host.

While there are undoubtedly problems with Docker, and likely many of the
issues you brought up are very real, there are many teams like mine that use
it successfully, and painlessly. Docker isn't the tire fire you want to make
it out to be.

~~~
StreamBright
>> 120+ containers on 12 hosts

it is virtually irrelevant what you use to run and operate this sort of
cluster. it gets tricky when you pass 100 nodes and even trickier when you get
to 1000+ nodes in a non-linear fashion. Docker certainly has issues that are
pretty severe when you are in the financial space and not a big deal if you
are running a popular blog like medium.com for example.

~~~
user5994461
12 nodes running 12 containers (1 per node) each bringing in 12$ a second and
you're a multi billion dollars company.

Let's multiply that a few times for dev, test and support systems. Still, no
need to have hundreds of nodes.

------
apeace
I have to agree when it comes to the business perspective. I can't imagine how
Docker is going to survive financially.

I use Docker at work, and I 1) don't have any loyalty to their brand and 2)
try as hard as I can to abstract away their specific APIs.

For example, we use Convox [0] to deploy containers to AWS. I could care less
what Convox and/or AWS are doing under the hood. They could switch out Docker
for rkt under my feet, and I probably wouldn't even notice.

It is kind of like POSIX to me. My apps are designed to run in a POSIX
environment, not specifically CentOS or Debian. And just like it's easy for a
new Linux distro to come along, give me POSIX, and give me some other shiny
features I like, it will be easy for any competitor to come along and replace
the Docker interfaces I use.

[0] [https://convox.com/](https://convox.com/)

------
gregmac
> I am moderately confident that there is no-one on the planet using Docker
> seriously AND successfully AND without major hassle.

Though the tone of this article is very negative, the conclusion is
interesting to me as someone that's considered Docker but hasn't done much
with it beyond experimenting locally.

I'm assuming if there's any forum that has users that can speak to their use
of Docker in production, this would be the place.

~~~
fnbr
We're technically using it in production for my site, www.bugdedupe.com, but
we're still in beta, so we haven't experienced heavy usage.

We've been using Kubernetes to deploy Docker containers on Google Container
Engine, and while there have been a few issues due to Docker/Kubernetes
(namely, getting the containers to expose localhost to each other, and to
expose themselves to the world), the issues that we've had so far have been
issues that we would have been bit by eventually. Namely, if we hadn't been
forced to deal with the issues now, we would have been screwed later. There
have been some weird bugs due to the internal environment that containers use
(we had extremely slow DNS lookups that caused our request times to shoot up
to 9s each). These issues have been transient though, so it's not clear that
it's Dockers fault, or if we're making mistakes in our code.

Docker has made our deployment much easier. You just build & push your
container, and you instantly have a versioned deployable instance of your
code. Kubernetes makes it extremely easy to rollout or rollback containers, so
I have nothing but good things to say about containers.

~~~
gregmac
An interesting comment also from the article:

> Google offers containers as a service, but more importantly, as confirmed by
> internal sources, their offering is 100% NOT Dockerized.

> Google merely exposes a Docker interface, all the containers are run on
> internal google containerization technologies, that cannot possibly suffer
> from all the Docker implementation flaws.

~~~
daenney
As pointed out by
[https://news.ycombinator.com/item?id=13715358](https://news.ycombinator.com/item?id=13715358),
that's not true.

------
random3
I think Docker was and still is a great way to popularize the concept of
containers along with the smarter tooling around it, like resource schedulers.
By the time we adopted Docker (because it was popular and realized it's a good
vehicle to push some concepts across organizations) we knew we're going to use
it as a package system in a distributed scheduling environment (based on
Mesos, Zookeeper, HDFS) and that it may get replaced later on. However, it
took longer for Docker Inc. and the community to depart from the original
ideas (initially very developer task centric, including trying to figure out
how to store data inside containers) and figure how to enable scheduling,
service discovery, health checking etc. I think Kubernetes did a good job
popularizing better practices.

IMO two of the few systems that were sound were Mesos and Kubernetes (and
their roots are somehow interleaved) and neither puts the Docker as the
central piece, nor the container, which is just a building block to achieve
the actual goals of running distributed, highly available jobs and services
efficiently in a shared environment.

------
merb
the problem is, that the docker (community) or rather the company behind it,
focuses on too much, instead of just doing one thing and doing it great. no
they need to reimplement the world and create their own kubernetes (or some
sort of it). their new website is less accessible as it was in the old
version, etc... and all the new projects and people. and the various ways to
configure docker. why does a simple container engine needs to have so many
parameters to configure it? the network experience is still not as good as it
could be and docker is still way more complicated to setup than a simple aws
and some AWS AMIs.

~~~
coding123
I'd like to see Rkt/Coreos folks start showing how their stuff IS accessible
than keep spewing this flamewar crap on HN. I've had nothing but excellent
comments from the Kube developers, but every time a Rkt dev or fan comes on
here, this is the crap we get.

~~~
jzelinskie
I think you might be conflating pot stirrers with actual rkt devs. CoreOS
(along with many other companies) are the Kubernetes developers.

FWIW, there was drama with the initial rkt announcement, but we've actually
seen docker adopt most of the original criticisms and that should be applauded
for both CoreOS and Docker.

------
Artemis2
> Google merely exposes a Docker interface, all the containers are run on
> internal google containerization technologies, that cannot possibly suffer
> from all the Docker implementation flaws.

Google running their own containerization tech with a Docker interface seems a
bit far-fetched given the level of integration of Kubernetes with Docker.
That's totally possible though, I'd like to read more about it.

~~~
raesene6
It is far-fetched 'cause it's not AFAIK true. My experience for this is having
set up a GKE Kubernetes cluster last night, it was definitely running Docker.

------
sandGorgon
TLDR of the first 60% of the blog post - " _I cannot comment on the LTS 16 as
I do not use it. It’s the only distribution to have Overlay2 and ZFS
available, that gives some more options to be tried and maybe find something
working?_ "

Second part TLDR - " _If you are locked-in on Docker and running on AWS, your
only salvation might be to let AWS handles it for you._ ". Use Docker-For-AWS
that sets up Cloudformation + Swarm mode.

 _Google offers containers as a service, but more importantly, as confirmed by
internal sources, their offering is 100% NOT Dockerized. That is a huge label
of quality: Containers without docker._ Actually, this is deliberate.
Containerd. For example, rkt support on GKE is "coming soon" .
[https://www.mail-archive.com/google-
containers@googlegroups....](https://www.mail-archive.com/google-
containers@googlegroups.com/msg00417.html)

------
callumjones
> As confirmed by internal sources, they experienced massive troubles to get
> Docker working in any decent condition

I call B.S. on this. Amazon wouldn't have spent so much effort on ECS if this
was true.

~~~
yebyen
Has Amazon spent a lot of effort on ECS? I am totally ignorant here, but the
people who I consider more knowledgeable about AWS things have said in a
nutshell that "in 2015 the things I heard about ECS were not good things, and
I have basically not heard any new things since then."

Basically stating that ECS was Amazon's attempt to plant a flag in the
container-space and that it was half-hearted, not done with the rigor of
solutions like k8s that had additional advantages of also being cloud-
agnostic, and finally that ECS was not a solution that anyone they knew was
recommending or would recommend.

~~~
abbyaws
hey yebyen- i was one of the very first beta users of ECS, and I now work for
AWS. first off, sounds like maybe we're not doing the best job we can be with
educating users on ECS- that's on us, and we'll focus on improving. to dig
into your actual question: when ECS launched, the goal was to do a smaller
number of things really well, and then listen to the community on what _they_
wanted to see from a container management platform, and grow accordingly. I've
seen a number of significant improvements over the last year or so. Off the
top of my head: ECR (thanks @coding123), task placement policies and
strategies to give developers more control over how they place tasks and use
resources, IAM roles for tasks, event stream for cloudwatch events, service
level autoscaling, ALB support, and a number of smaller configuration changes,
like multiple network modes, and out-of-the-box support for and awslogs
driver. also discovered while writing a workshop the other day that there is a
pretty sweet first-run wizard for users just trying out ECS for the first
time. in any case, a couple of main takeaways here: i'm seeing a focus on
adding features and services that reduce the operational work for developers-
let AWS worry about scaling and managing your cluster infrastructure, and you
can focus on building cool stuff. beyond that, though, developers have asked
for more control, flexibility, and extensibility, and i think ECS is working
on delivering that: the cloudwatch event stream can be consumed by other
services, the blox open source project (and the already opensourced ecs-agent)
let you build custom schedulers/functionality on top of what AWS offers,
placement policies let you customize how ECS consumes your cluster resources.
would love to know where you get your news and why you haven't heard much
about ECS, so we can make sure we fix it. if you want to talk more, you can
also DM me

~~~
yebyen
From an engineering POV, this line makes the most sense out of all:

> when ECS launched, the goal was to do a smaller number of things really
> well, and then listen to the community on what _they_ wanted to see from a
> container management platform, and grow accordingly

I'm coming from a small CoreOS cluster on bare metal, onto Kubernetes and Helm
on EC2 nodes. I was a Fleet user before and I loved it! But always with the
understanding that when things got better in the cluster space, I'd move from
Fleet toward some resource aware scheduler.

I love to read how you're all going down a similar road, however you get
there! Thanks for the blox links and I think I will be able to make immediate
use of ECR with the rest of my AWS stack.

~~~
abbyaws
awesome! keep me posted on how ECR works out for you. If you build something
sweet, write about it and let me know! we love blog posts/write ups/all that
jazz.

------
mdekkers
Every three months or so I tell myself "right,get your finger out, and do
something production quality with containers" and come to the same conclusion
this guy did.

~~~
cdelsolar
Have you tried? Make some Dockerfiles, upload to ECR, deploy to an ECS
cluster. Hooray, no more dealing with dependency hell and config management.

~~~
StreamBright
Actually a single JAR and Ansible deployments solve the same problem as well.
The number of times I was running into issues with Docker >>> the number of
times I was running into issues with single jars and Ansible. Until this stays
like that it is not justified to move to Docker. The business use case is to
have a reproducible & reliable infrastructure and not to use containers at all
cost.

------
erikb
Lot of memes, but a little more facts, statistics, links to bug reports etc
would have been nice. It may all be true, but how can I decide if I believe it
or not? It's not that I know you, dear author, and therefore could trust your
personal judgement. Make your point a little more clearly. Please.

------
stevecalifornia
This 'Hitler uses Docker' video seems to be a pretty similar viewpoint on
Docker and the world.

[https://www.youtube.com/watch?v=PivpCKEiQOQ](https://www.youtube.com/watch?v=PivpCKEiQOQ)

~~~
kordless
Don't cry, you can run bash on Windows 10 now!

~~~
jhasse
You could always run bash on Windows using Cygwin or MSYS before.

~~~
nunez
LXSS (Bash on Ubuntu on Windows) is an honest-to-goodness installation of
Ubuntu running on top of Windows using the Linux subsystem. Cygwin/MSYS are
more limited in comparison (or at least I have found them to be so).

~~~
jhasse
Have you tried MSYS2?

------
dekhn
The claim that GKE doesn't use Docker internally is false. You can ssh to the
master and see all the docker containers.

~~~
crb
You can't ssh to the master with GKE, but you can check the nodes (as
mentioned at
[https://news.ycombinator.com/item?id=13715358.](https://news.ycombinator.com/item?id=13715358.))

~~~
dekhn
Sorry, yes. You used to be able to ssh to the master.

------
parzivalm
The author says that the AWS ECS AMI they provide is based on ubuntu but it is
an amazon linux AMI which is based on centos/rhel.

------
_Codemonkeyism
While other people just use Docker to serve hundreds of thousands of users
(mid size startup) and make money.

~~~
StreamBright
Yeah like running a CI job for customers. Absolutely great with Docker, nobody
cares about long term reliability. Container crashes? Now worries we spin up
10 more and re-start the CI job. Perfect match for Docker.

Running financial infra is quite the opposite. Long term reliable processes,
not much space left for failures and mishaps, especially not in the lowest
level of your infrastructure. What is the benefit for using Docker for this
sort of services? Almost zero.

------
dman
Can anyone comment on whether its safe to run ceph inside docker?

------
jhasse
What about Fedora?

------
holydude
I still think that Solaris zones are / were a superior technology. Sadly it
died with the demise of Solaris as an operating system. Still in use in some
"must have enterprise" types of companies. I would not run anything serious on
docker (definitely not yet) if I would not be heavily invested in
understanding and contributing to the technology (docker codebase + related).
Docker is a nice to have but absolutely abstracts too much and takes control
from you. I am wondering what people like Brendand Gregg think about docker.

~~~
marios
Solaris within Oracle is as good as dead from the latest reports. However,
Illumos is a Solaris descendant that is very much alive. It has all the good
systems stuff that came out of Sun (zones, ZFS, DTrace) and is backed by
Joyent (now part of Samsung).

There's also FreeBSD jails that can be used to contain applications. I'm not
sure about the timeline, but I think jails came first. Sun engineers wanted to
achieve the same so they ported the same technology over to Solaris.

~~~
StreamBright
Oracle is the company where great projects go and die.

------
necklace
Yikes, I could never work in aerospace.

