
Docker in Production: A History of Failure - user5994461
https://thehftguy.wordpress.com/2016/11/01/docker-in-production-an-history-of-failure/
======
user5994461
Author here.

A bit of context missing from the article. We are a small shop with a few
hundreds servers. At core, we're running a financial system moving around
multi-million dollars per day (or billions per year).

It's fair to say that we have higher expectations than average and we take
production issues rather (too?) seriously.

I updated and fixed a few points mentioned in the comments. (Notably: CoreOS).

Overall, it's "normal" that you didn't experience all of these issues if
you're not using docker at scale in production and/or if you didn't use it for
long.

I'd like to point out that these are issues and workarounds happening over a
period of [more than] a year, summarized all together in a 10 minutes read. It
does amplify the dramatic and painful aspect.

Anyway, the issues from the past are already in the past. The most important
section is the Roadmap. That's what you need to know to run Docker (or use
auto scaling groups instead).

~~~
boulos
Can I ask (given the summary / roadmap at the end and your blog post about
GCE's pricing): Are you moving to Google Cloud? If not, why not?

For what it's worth, I agree with you that a lot of the value of Container
Engine comes from having a team ensure the bits all work together. That should
be (one of) the positives of any managed service, but I think the Container
Engine team has done a particularly good job in a fast moving and often bumpy
space.

Disclosure: I work on Google Cloud ;).

~~~
mayank10j
Based on past discussions on HN and I feel google has huge terrible support
for it's offerings. May be it will improve with time. But currently I am
really concerned with running GCP in production.

~~~
andybak
I wonder if the problem is more that Google has got terrible support for it's
free offerings - maybe for justifiable business reasons. However the damage to
their brand when they want to sell business critical services is an unintended
consequence. (Similarly I think killing Reader was justifiable from their
perspective but the damage to their reputation amongst a vociferous and
possibly influential demographic has probably outweighed the cost of keeping
it going... They can't launch a service without someone popping up to remind
us about Reader)

~~~
boulos
I miss Reader (though I'm amused that it let Feedly take off, and they're a
Cloud customer!).

One thing often not considered ("Why not just have someone keep it running?")
is that you really have to keep a team of people on it (in case there's a CVE
or something) or at least familiar enough with the code to fix it. There's
also the double standard of "you haven't added new features in forever!"
(Maybe this wouldn't have applied to Reader though).

But, I agree if we could have kept it on life support somehow, we wouldn't
have (as many) people asking "What if they shut down Cloud?!?". Conveniently,
as people get a sense of how serious Google Cloud is about this business, even
on HN I'm seeing this less.

------
amouat
Well, this person seems to have had a hard few months. I have a lot of
problems with the post though:

\- Docker do not own CoreOS and did not create the distro

\- Docker did not write OverlayFS

\- There are issues with storage drivers. Don't use AUFS for the reasons
outlined. I've had no issues with Overlay2, but note there is also
devicemapper (which Red Hat have poured a lot of time into) and BTRFS. And ZFS
now IIRC.

\- It's not correct to say Docker was designed "not" to do persistent data.
You do need to be aware of what you're doing, and I suggest not running DBs in
containers until you have the solved the FS issues and have experience with
orchestration.

\- Kubernetes has been around for a while now and is fairly stable in my view.
Swarm isn't rubbish, but the new version is very young and you may run into
issues in the short term.

\- With microservices, when you are running lots of small containers, you have
to expect them to crash now and again. It's just the law of averages. You need
to design your application to accept this and deal with it when it happens.

If you hit this many problems with any given tech, I would suggest you should
be looking for outside help from someone that has experience in the area.

~~~
X86BSD
"Issues with storage" is an understatement. Linux has major issues with
storage. I see no way forward for Linux in that area unless they adopt ZFS.

How many filesystems has the Linux world created in the last 20 years? Of
those how many are rotting piles of tire fire?

Filesystems are hard to get right.

ZFS is the way forward. For cross compatability, for reliability, for
stability, for lots of use cases.

Btrfs is another tire to throw on the burning pile of other filesystems that
won't work out in Linux.

Let the down votes begin.

But Linux has some major architectural issues that eventually the Linux
faithful will have to admit to.

Containers.

Filesystems.

Observability. Though bless him Brendan Gregg is trying his hardest to help
here.

Portability.

And this is what is so painful for the rest of us to watch or deal with. This
inability for the Linux folks to admit these glaring architectural problems.
Problems they refuse to look outside their bubble to see how others solved
these problems and adopt solutions that have already solved these problems.
They just want to continue to bury their heads in NIH soil. And double down on
trying to solve these issues, poorly, on their own without any awareness of
how others have solved these issues. People outside Linux land just might have
the right ideas on how to solve these problems!

But I expect no one in that camp to acknowledge this and down vote away.

Sigh.

~~~
mancerayder
_And this is what is so painful for the rest of us to watch or deal with. This
inability for the Linux folks to admit these glaring architectural problems._

Alright, the suspense is killing me: what OS is the one that gets it right? I
ask this to "the rest of us" as you put it.

~~~
X86BSD
SmartOS, FreeBSD, Tredly, etc

~~~
SSLy
Tredly is "just" a FreeBSD distribution. And Linux can't have ZFS in-tree
because CDDL.

~~~
X86BSD
It is not just a version of FreeBSD. The base is FreeBSD but the tooling for
containers is not stock. It's value added bits that are shaping up to be very
nice for containers on a more stable, secure, and storage friendly stack.

~~~
SSLy
Yes, but the storage parts which the discussions are about are from FreeBSD

------
jconley
This is why DevOps became a thing. It's complicated managing systems at scale.
If you were using these tools and aren't at scale, well, that sucks.

Unplanned downtime is the main drawback to both hosting your own OS's and
using leading edge tooling for your critical systems. It doesn't matter what
the underlying system, stuff like this happens. This stuff is complicated. You
will find major warts in every new system. Everything will break, often,
leaving your users stranded. It takes a very long time for software to mature.
[0]

That's why you see engineers with 15+ years of experience patiently waiting
for new technologies to mature before putting them in place in our critical
infrastructure. Sure, we'll play with them, but putting new tech in production
that could easily make your entire system unavailable is too risky. Yes, three
year old tech is new.

[0]
[http://www.joelonsoftware.com/articles/fog0000000017.html](http://www.joelonsoftware.com/articles/fog0000000017.html)

~~~
inopinatus
It takes a fair bit of experience before you can recognise the warning flags,
though.

Most folks don't realise how simple their requirements are. Most folks need to
put files on a server and start a process. What they don't need is a whole
mass of rickety scaffolding and control plane infrastructure that gets them
halfway (and only halfway) to running an in-house PaaS.

So having kicked the tyres on Docker and been appalled by the messy design and
dreadful tool quality, I went back to using OS-native packages and isolating
our microservices via (gasp) user IDs.

Judging by their product direction I suspect Docker's board want them to
challenge VMware, which (rubbing my crystal ball) suggests to me that their
future is a bloated and overcomplicated piece of Enterprise, targeted at
companies who think you can buy devops.

~~~
digi_owl
Well container came out of slimming down VMs.

Meaning that rather than moving a whole OS from hardware to hardware, or spin
them up or down as load required, you would spin up or down processes and
their runtime requirements from a known functioning image.

Thus removing the overhead of the VMs hardware emulation and the need to run
so many kernels.

The response from the VM world to containerization seems to be to push
unikernels. Effectively going back to the DOS days where you have just the
minimal kernel and userland, all living "happily" in the same (virtual) memory
space.

~~~
KaiserPro
Well, thats what cgroups and chroot is for, but then docker is just a fancy
wrapper.....

As for performance, if its critical then get a real server, if you've got
decent change management (ie ansible, puppet et al with no manual
intervention) Then its no effort.

(seriously network boot, and vlans are your friend here.)

------
davexunit
Docker's reliance on overlay filesystems is one of the biggest problems I have
with Docker. Stacking opaque disk images on top of each other just isn't a
great design, and it makes for a cache strategy that is all-too-often
invalidated (because a Dockerfile is _linear_ , there is no dependency graph).
The file system is just the wrong layer of abstraction for solving such an
issue.

If you change paradigms from the imperative sequence of mutations in a
Dockerfile to a declarative specification that produces immutable results and
a package dependency graph structure, you get a much better cache, no need for
disk image layering, a real GC, etc. For example, the GNU Guix project (purely
functional package manager and GNU/Linux distro) has a container
implementation in the works named call-with-container. Rather than using
overlay file systems, it can just bind mount packages, files, etc. inside the
container file system as read-only from the host. Not only is this a much
simpler design, but it allows for trivial deduplication of dependencies.
Multiple containers running software with overlapping dependency graphs will
find that the software is on disk exactly once. Since the results of builds
are "pure" and immutable, the software running inside the container cannot
damage those shared components. It's nice how some problems can simply
disappear when an alternative programming paradigm is used.

[https://www.gnu.org/software/guix/news/container-
provisionin...](https://www.gnu.org/software/guix/news/container-provisioning-
with-guix.html)

~~~
fpoling
One can get very similar experience with Docker. The idea is to make single
Docker image with all the software one needs on the server and start all
containers using it with read-only flag. Then one use volumes for persistent
data.

~~~
davexunit
That's not really the same thing. What about different applications that share
dependencies? With Docker, they would have to share an exact subset of a
linear Dockerfile to take advantage of any deduplication, so you're back to
image layers. It's telling to me that when proposed solutions involve
completely avoiding one of Docker's primary features that maybe Docker isn't
very well designed to begin with.

With Guix, any sub-graph that is shared is naturally deduplicated, because we
have a complete and precise dependency graph of the software, all the way down
to libc. I find myself playing lots of games with Docker to take the most
advantage of its brittle cache in order to reduce build times and share as
much as possible. Furthermore, Docker's cache has no knowledge of temporal
changes and therefore the cache becomes stale. Guix builds aren't subject to
change with time because builds are isolated from the network. Docker needs
the network, otherwise nothing would work because it's just a layer on top of
an imperative distro's package manager. Docker will happily cache the image
resulting from 'RUN apt-get upgrade' forever, but what happens when a security
update to a package is released? You won't know about it unless you purge the
cache and rebuild. Docker is completely disconnected from the real
dependencies of an application, and is therefore fundamentally broken.

~~~
fpoling
Docker needs network only when one uses Dockerfile for deployment, a rather
bad idea. Instead Docker images should be used. It allows to verify them on
development/testing machine before deployment. And with this setup all "bad
pieces" of Docker are located at developer's notebook. In production
everything is read-only and shared among all containers.

------
tzaman
That's odd, we've been using Docker for about a year in development and half a
year in production (on Google Container engine / Kubernetes) and haven't
experienced any of the panics, crashes yet (at least not any we could not
attribute as a failure on our end).

Clearly, Docker itself has a a few annoyances, like no garbage collection and
poor tagging process, but we're using it on a daily basis, and nobody
complains about it (nothing a few bash scripts can't fix).

Finally, the biggest benefits of not just docker but containers in general are
consistency and immutability which brings me to the final point I agree with
the author: Your database shouldn't be containerized. It's not your pet (from
pets vs cattle), it's your BFF.

~~~
fpoling
What is wrong with DB in container? As long as one uses host volumes for
persistent data, docker allows to isolate DB setup rather nicely.

~~~
adenot
Reverse question that no-one seems to ask: What's the advantage of running a
DB on a container?

~~~
etherealG
Spinning a new db in continuous integration for integration tests.

------
zwischenzug
I work for a large enterprise 100K+ employees using Docker in production [1]

It's certainly had its challenges (different versions of Docker clients
causing havoc with different manifest versions, race conditions in the engine
etc etc), but this article takes things a little far.

If you take pretty much any new technology like this make it a base part of
your platform, you'll want support from a big vendor. To think otherwise would
be naive.

There is definitely a debate to be had about Docker Inc's approach to
production/enterprise rollouts though, but as a technology I'd say it's
developing pretty much as I'd expect.

I've also seen Docker succeed in production despite the technical challenges.
If you don't like the heat...

(1) See my blog:
[https://medium.com/@zwischenzugs](https://medium.com/@zwischenzugs),
especially: [https://medium.com/zwischenzugs/a-checklist-for-docker-in-
th...](https://medium.com/zwischenzugs/a-checklist-for-docker-in-the-
enterprise-d69f2ae3091b) and [https://medium.com/@zwischenzugs/docker-in-the-
enterprise-ec...](https://medium.com/@zwischenzugs/docker-in-the-enterprise-
ec662d296695#.sik7etvcp)

~~~
knite
From the article: "The impossible challenge with Docker is to come with a
working combination of kernel + distribution + docker version + filesystem.
Right now. We don’t know of ANY combination that is stable (Maybe there isn’t
any?)."

What combination of Docker version + distro + kernel + filesystem are you
running, and how stable is it?

~~~
zwischenzug
RHEL7.x - docker versions 1.8.x-1.10.x, mostly ext4. Stability is OK, aside
from the things I mentioned.

~~~
dserodio
One thing that bit me is that on RH 7.1 you _should_ use ext4[1], but on RH
7.2 you _need_ to use xfs[2]

[1] [https://access.redhat.com/documentation/en-
US/Red_Hat_Enterp...](https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/7/html/7.1_Release_Notes/chap-
Red_Hat_Enterprise_Linux-7.1_Release_Notes-File_Systems.html#TP-OverlayFS) [2]
[https://access.redhat.com/documentation/en-
US/Red_Hat_Enterp...](https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-
file_systems.html)

------
awinder
"The only way to clean space is to run this hack, preferably in cron every
day"

[https://github.com/spotify/docker-gc](https://github.com/spotify/docker-gc)

"Many attempts can be found on the internet, none of which works well. There
is no API to list images with dates"

[http://portainer.io/](http://portainer.io/) seems to be able to do it, docker
images lists it... I mean, I don't want to call bullshit without knowing the
full story, but...

"So, the docker guys wrote a new filesystem, called overlay"

[https://github.com/torvalds/linux/commit/e9be9d5e76e34872f0c...](https://github.com/torvalds/linux/commit/e9be9d5e76e34872f0c37d72e25bc27fe9e2c54c)
written in 2014 by the linux kernal team / Linus, but OK cool story

"It affects ALL systems on the planet configured with the docker repository"

OK, this is why people use AWS' container repo, or they use the open source
software and maintain their own repo... this happens with any of the public
repo services, and it was 7 hours.

"The registry just grows forever"

S3 is a supported backend, highly would advise anyone running their own repo
to use it (or have a similar expanding-space story). There's also a config
flag for the repo to allow removing images. Obviously wouldn't want to use
this if you're hosting a public repo but internally, go for it, it's off by
default, seems sane enough.

"Erlang applications and containers don’t go along"

I'm certain that people are running erlang containers successfully.

"Docker is a dangerous liability that could put millions at risk. It is banned
from all core systems."

Daily FUD allowance exhausted.

I guess the tone of this article really bugged me because there's obviously a
point to be made from this experience that running Docker is more difficult &
error prone than it should be. And maybe it wasn't a great fit for this
company culture. But this article crossed the "we didn't have a good
experience, therefore, it's BS software" line, and frankly, that attitude may
very well have been to blame for the lack of success just as much as docker's
shortcomings were...

~~~
salex89
Take a look at Rancher also. It is a bit more mature project, and it did save
me some headache. To be honest I didn't use Docker everywhere and everywhen
but where I did, do real issues.

Also there is DCGC from yelp!

~~~
awinder
[https://github.com/Yelp/docker-custodian](https://github.com/Yelp/docker-
custodian) looks awesome, thanks for the tip! Rancher looks great too -- would
love to know if you looked at other options (kubernetes etc.) & how Rancher
won out

~~~
thallian
FYI, Rancher can use Kubernetes for orchestration (Mesos or Swarm too)[0]. No
idea how well that works though, I am using it with their own Cattle and that
is more than enough for my lowly needs.

I looked at Kubernetes before Rancher but it was hard for me to really
understand where to start and how it all fit together. Now that I understand
much better what I am doing I should probably take another look at it.

[0]: [http://rancher.com/rancher-now-supports-
kubernetes/](http://rancher.com/rancher-now-supports-kubernetes/)

~~~
nikolay
I love RancherOS - why do you think it's so undervalued compared to CoreOS?

------
jsz0
> Docker Issue: Can’t clean old images

The worst part of this problem is when you do run out of disk space docker's
image/container database often gets corrupted resulting in the great cryptic
'could not find container entity id' error. The only fix I'm aware of is
rebuilding all your images or mucking around with the docker sql database. The
lack of basic features to manage the problem creates a worse and harder to
troubleshoot problem.

It's an easy enough problem to avoid in the first place so it's really not big
of a deal. It hurts docker as a company more than the users in the long run
because it makes them look bad. There are a lot of other small things like
this with the docker cli tools especially. They aren't deal breakers but they
are definitely eyebrow raisers. For example if you mess up your docker run
command it usually just fails in strange ways instead of telling you what's
wrong with your command.

I'm actually pretty happy with docker overall but they really need to fix some
of these basic usability problems.

------
atemerev
I also work in HFT, and I also attempted to use Docker in production and
encountered all sorts of problems (different from what are described in this
article, though).

However, as it turned out, these problems had nothing to do with Docker.
Containerization is a great idea, and Docker's approach is sane (one container
per service — fine).

The offender was docker-swarm. Like many others, I chose it as the default
container management approach (it is written by the same team, so it should
work the best, right? Wrong.) Docker-swarm is indeed buggy and not ready for
production, or at least it wasn't 10 months ago. And if you use "one container
per service" approach, orchestrating large groups of containers is a
necessity.

Then I discovered Kubernetes, and became significantly happier. It's not
without its own set of quirks, but it orders of magnitude smoother experience
compared to docker-swarm. And it works with AWS just fine. (I didn't get to
use DCOS, but I heard some nice things about it, too).

Tl;dr the source of all "Docker is not ready for production" rants seems to be
docker-swarm, at least from my experience. Use Kubernetes or DCOS and
everything will be so much better.

------
apeace
Here is the smoking gun, showing that this organization shot itself in the
foot:

> Docker first came in through a web application. At the time, it was an easy
> way for the developers to package and deploy it. They tried it and adopted
> it quickly.

Unless you've got a lot of spare time on your hands, it's never a good idea to
adopt unfamiliar tools where they're not needed. Stick to your traditional
deployment techniques unless you're equipped, and have the time, to take a
roller coaster ride.

Docker is young software.

That said, it seems the author did experience many legitimate breaking changes
and abandoned projects. It would be great to hear a perspective from someone
on the Docker team about how breaking changes are managed, how regressions are
tracked, how OS packaging is handled (e.g. the dependencies the author
mentioned wrt filesystems, as well as the bad GPG signature), etc.

~~~
matt4077
> Unless you've got a lot of spare time on your hands, it's never a good idea
> to adopt unfamiliar tools where they're not needed.

That's so broad, it's meaningless. Nothing is ever "needed". Given enough
time, you could run everything on a stack limited to technology released
before 1995. And without "adopting unfamiliar tools", they'll remain
unfamiliar forever.

~~~
dasil003
True, however if you mix in maturity I think you can come up with a reasonable
formula for new tech adoption. I only just started using Docker this year
because I was giving it time to mature. When the Docker for Mac Beta came out
I started playing with it because it promises to solve real problems with
using Vagrant. If it was more stable I might jump in and start replacing
Vagrant boxes faster, but because of the memory leaks / freezes / CPU
overhead, I'm not pushing forward aggressively. Meanwhile, let the cool kids
bang their heads against the wall until it stabilizes a bit more and then I
can reap the rewards with a lot less pain.

------
markbnj
Like the author I've also been running Docker in production since early 2015.
Given the 'HFTGuy' part of his email address I'll guess I don't do anything
nearly as edgy as he has to do on a daily basis, so it would be tiresome to
just say that I haven't had the problems he's experienced. A lot of his issues
probably do stem from his organization pushing the limit and wanting to run
very specific configurations for performance reasons.

The one thing I do want to respond to is the notion that Docker cannot run
stateful apps. Among the stateful apps I have run on Docker in production:
elasticsearch, redis, postgresql, mysql, and mongodb. Containers and
application state are orthogonal topics. The contained process is reading and
writing a file system on persistent media just like all the other processes on
the host and that file system doesn't mysteriously disappear when the
contained process exits. Naturally stateless apps are simpler. They're simpler
to run on VMs or bare metal too. When you need data in a container you do the
same thing you do when you need it on a VM or on bare metal: you mount
appropriate storage for it.

The main issue with most stateful applications isn't the existence of the
state. The main issue is that if you have state then its probably important,
you probably want to make it durable and highly available, you probably want
clustering and replication, and now you're into discovery and peering, and
that can be an actual issue with Docker, especially for finicky beasts like
redis.

------
audleman
I haven't seen anybody speak to one of his points and I'm curious about what
HN thinks: if you're already using a cloud provider that provision's instances
of the exact size you need, what benefit do containers bring?

I'm excited by containers and all, but this point has always stopped me from
going forward. If I were a huge company running bare metal, then yes I'd want
to squeeze every last ounce out of my hardware. But if I'm running inside a
hypervisor and somebody else is paying the cost, what are the benefits?

As for the article: it comes across as a blend of frustration and real
technical challenges. Some good points, though the hostile tone weakens the
argument.

I don't think Docker should worry too much about backwards compatibility right
now, just power forward until it's got a solid formula. In the meantime,
caveat emptor!

~~~
buzer
If you are running 1-2 of copies or applications that very large amount of
memory while being pretty small themselves, not much (and that's admittedly
pretty common).

However, if you, for example, need to run 20x copies of 50MB application that
uses 100MB memory at runtime in somewhat isolated environment, you would only
provision machine with 20x100MB+50MB=2050MB + whatever OS needs (~100MB?) to
keep everything in memory. If you made VMs for each of them, you would need
20x100MB+20x50MB+20xOS overhead=5000MB'ish, 150% increase in this case.

Also, starting up a container is a lot faster than starting a VM. If image is
in the local cache, your application likely starts loading within 500ms. VM
startup times are usually measured in minutes.

------
pbecotte
The problem with this article is it is the kind of thing I could see myself
writing after about three weeks of playing with the tool, when I did not yet
understand the best ways of using it... but it is written from the POV of
someone who does not know what they are talking about.

Half of the article is various complaints about AUFS, and then Overlay. There
are at least 6 storage drivers off the top of my head, and I feel like there
are one or two more I am forgetting about. What made you choose AUFS? What
were your thoughts on Overlay? Instead of going with the default and
complaining, maybe research your options? Even worse though, the filesystem
driver is completely beside the point. If you want to store things on the
filesystem from a Docker container, you don't use the copy-on-write storage
system! You create a volume, which completely bypasses the storage driver.
There are even a bunch of different volume drivers for different use cases-
say mounting EBS volumes automatically for your container to persist data to?
His whole rant about data magically disappearing has nothing to do with the
tool, but that he didn't take the time to learn how to use it correctly.

Further, while I am sure he is telling the truth and has machines randomly
suffering kernel panics and crashing multiple times per day, it seems hard to
believe that it is docker causing this. He is running 31 containers spread
across 31 aws hosts... I am running thousands, across hundreds of aws hosts,
and have been for a long time, and cannot recall a single kernel panic caused
by docker.

------
pjmlp
> Google invented containers long before it was a thing

I had to laugh at this one, then again I am an old dog.

~~~
Thaxll
Solaris zones ~~

~~~
justinsaccount
Does it count if you implement something but the UX is so terrible that no one
ever uses it?

~~~
devonkim
Nobody really remembers the Diamond Rio Karma but everyone knows about the
iPod. History is only acknowledged by historians, perception is reality
otherwise.

~~~
justinsaccount
I think the fact that all portable mp3 players had terrible UX and were not
successful until the iPod kind of proves my point here.

------
api
When I first saw Docker I saw something with all the characteristics of a
developer fad, namely that it was a magic new thing that in reality was
nothing but a barely-novel rearrangement of old things but promised to make a
lot of very old very hard problems go away.

Even worse, it was a fad that _adds a whole lot of complexity_ to the system.
Anything that adds complexity always gets a jaded glance from me. Unless
there's a very clear and good reason for it, increasing complexity is
typically a sign that something is wrong. (I use this rule in my own code and
designs too... if I'm building a tower of Babel then I'm not seeing something
or making a mistake.)

"Let's take preconfigured Linux system images and treat them like giant
statically linked binaries."

Don't get me wrong-- it _is_ a somewhat interesting hack and does have some
utility. (and it IS a hack) But the problems it solves are in fact a lot
harder than they seem and stem from fundamental 30+ year old issues in OS
design and the Unix way of doing things. Our OSes date from the days when a
"box" was a very long lived stationary thing that was provisioned and
maintained manually, and they're full of things that _today_ are bad design
decisions because of this... like sprawling filesystems full of mutable data
with no clear mutable/immutable division.

But the hype on this thing was just insane. I've never quite seen anything
like it except maybe Java in the 1990s and that had more substance (a popular
Smalltalk-ish language). For a long time I've joked that devops conference
conversation was sounding like:

"Docker docker Docker docker docker docker Docker docker docker?"

"Docker docker docker."

"Docker, docker?"

"Docker docker docker Docker docker."

"Ahh, Docker."

Luckily it seems to be dying down and a lot of people are realizing it's not a
silver bullet. This will probably inoculate a generation of devs and admins
against over-hyped fads, but never fear... a new generation of devs and admins
are coming in now and are eager to rewrite everything again since this time
they'll be able to make a few trivial rearrangements of things that are
already there and this time they'll get it right.

~~~
joeevans1000
Java is the most 'popular' language in the world in 2016 (TIOBE), so maybe the
hype was well placed.

------
marcosdumay
Well, I lost nearly half the article asking myself why would somebody ever run
a single container per VM... They got a bug that made container-running VMs
unreliable, thus did a non-test using what they got available.

I'd be more interested on articles that actually tested it running how it was
meant to. Or, if you can't, just stop there and say "there's a showstopper
bug, it wouldn't work for us" \- those are useful articles too, and require
much less effort.

About databases, mostly anything¹ you put between them and the hardware will
hurt you. VMs are bad too. What I can't imagine is how you would create and
destroy database instances at will - I might live in a different reality, but
around here you'll always want as few database nodes as possible, and as a
consequence, as powerful and reliable nodes as possible, what means as little
moving parts as possible.

1 - What won't hurt is a disk abstraction layer designed for performance and
as expressive as the original physical stuff. This is widely used with the
"SAN" name, and I'm willing to bet you can mount one inside a container. But
well, on AWS you really won't.

~~~
user5994461
The critical bugs left are actually race conditions. They happen rarely, more
often under high load.

Everything was fine in testing, everything was fine in production... at first.

Fast forward a bit. More application put in docker, a lot more load in both
testing and production. Things go wonky.

If we ran our current systems in Docker/Debian from January, a host would
crash every hour and we'd have figured out the unstability. Instead the docker
adoption happened slowly, while a few patches (and temporary regressions) were
released along the year.

------
edpichler
It seems I have seen a lot of posts on HN relating that Docker is not ready
for production. I am using Docker in production on my small company for one
year and it's working pretty well (maybe because we are still small).

------
FuNe
Being caught in the containerization craze myself I'd love to hear whether the
story is exaggerating or is painfully accurate.

So far I've been bitten by the inability to clean up images of certain age.

UPDATE: Another really annoying thing is the inability to tag an image
directly into a registry (AFAIK). You need to pull, tag and push back again.
Given that images can be GBs long you end up with really heavy net traffic for
a simple task.

~~~
mgliwka
The team at spotify has built this (see above):
[https://github.com/spotify/docker-gc](https://github.com/spotify/docker-gc)

~~~
bostonvaulter2
Why isn't that just built-in?

------
rzimmerman
A lot of the issues in this article seem to stem from running Docker on old
kernel versions.

> We are using Debian stable with backports, in production. We started running
> on Debian Jessie 3.16.7-ckt20-1 (released November 2015). This one suffers
> from a major critical bug that crashes hosts erratically (every few hours in
> average).

If you're stuck on an older system that's poorly supported by Docker then it
may be a bad choice for you.

~~~
mmagin
Which actually stable distribution includes a sufficiently new kernel? For
important stuff, a lot of us stick to Debian stable, Ubuntu LTS releases,
CentOS/RHEL, etc.

~~~
user5994461
CoreOS only. They're on 4.8 right now and they update the kernel whenever is
necessary to keep up with Docker needs. That's the only safe bet on the long
term. (as stated in the Roadmap in the article).

Read: Even if Ubuntu 16-LTS and Centos/RHEL 7 may be up to date enough now
(not sure), they won't keep up with the latest minor kernel updates and that
will become a problem in 6 months when docker 1.1X will require THE newest
kernel.

~~~
Yeroc
It'll be interesting to see how this plays out. RedHat is pretty invested in
Docker right now via OpenShift so they have a lot incentive to backport
required kernel fixes to Centos/RHEL 7 but at some point they'll likely have
to drop support for the latest and greatest Docker release.

~~~
StillBored
What makes you think they aren't just stabilizing/standardizing a single
version of docker and backporting fixes?

After all, about 1/2 the gripes stemmed from the fact they docker itself is
basically the engineering branch minus any serious regression/etc testing. Of
course things are going to break from release to release, that is why test
departments exist.

------
scresswell
What a very strange article. TES were using docker in production in October
2014. It was certainly possible to run docker containers (docker run -d) in
the background and to view them (docker ps) as early as October 2013 when we
first evaluated it at the Daily Mail.

------
andmarios
Author's company should hire a proper devops person and then docker, magically
—from the devs pov— will start working.

AUFS on production? Seriously? Create a BTRFS partition. Docker isn't a
shortcut to learning a linux based OS, it is a useful tool in the hands of
people who know what they are doing.

~~~
andmarios
To the people downvoting:

If your company lost thousands of man-hours on a tool and has nothing more to
show than a blog article with half the facts coming out of the author's head
and the other half boasting about bad production practices, would you prefer
me to sugar coat it?

A hire that knew linux and docker, would solve the problem.

I work with docker since 2013. I came to a team with devs that already used
docker. I didn't know anything about it at the time. They gave me a docker
image with a JVM based app, that had a two years old JDK, ran as root and the
container was ran with privileges enabled. Did I have to know docker to tell
them that this was nuts? Nope, I had to know ops. For the same reason I knew
AUFS before I learned about docker and I knew very well that I would trust
AUFS on my NAS for media storage but not as storage for a production database.

Did I ever tell the devs how to write java? No. It isn't on my job description
for a reason.

~~~
user5994461
There is a dissonance here.

A linux guru will know that AUFS is unstable... but the problem is mentioned
nowhere and Docker still uses it as the default filesystem for [most of] all
cases.

The linux guru will know to avoid AUFS... but the replacement [overlay2] was
rolled out very recently and it's only available in the latest OS and systems.

~~~
andmarios
AUFS isn't included in the default setup of many Linux distributions. CoreOS,
CentOS, RHEL do not have it. If I recall correctly, CentOS and RHEL will use
the device-mapper driver by default and CoreOS the btrfs. I would speculate
that the docker team gave so much attention to AUFS to lower the barrier to
entry but it isn't the only option.

My choice is btrfs and it has been rock solid for me since the start. Of
course one has to know a bit about btrfs and always leave a healthy percentage
(I'd say 10% or more) of the filesystem free.

I think AUFS and overlay2 do have an advantage of their own though (apart the
works-out-of-the-box). Due to the way they work, you can have shared memory
between containers. Thus if I had to run many containers with a JVM app, I
would give them a try to lower the tax on RAM.

~~~
ysh7
> If I recall correctly, CentOS and RHEL will use the device-mapper driver by
> default and CoreOS the btrfs

CentOS & RHEL uses device-mapper with loopback device by default (and logs
"Usage of loopback devices is strongly discouraged for production use. Either
use --storage-opt dm.thinpooldev or use --storage-opt
dm.no_warn_on_loop_devices=true to suppress this warning". Also related
article [1]

CoreOS uses overlayfs by default now from a while [2]

[1] [http://www.projectatomic.io/blog/2015/06/notes-on-fedora-
cen...](http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-
docker-storage-drivers/)

[2] [https://lwn.net/Articles/627232/](https://lwn.net/Articles/627232/)

~~~
andmarios
Thanks for the correction. It has been a long time since I had to setup docker
manually.

------
finid
The author made some really good points, but _CoreOS is made by Docker for
Docker_ is not one of them.

CoreOS is the company behind a competing container tech called rkt (rocket)
and CoreOS Linux.

------
ubercore
I have to disagree on the pain of working with Kubernetes. There are
definitely gaps in documentation, no doubt about that.

But it took me under a day to containerize an application I'm working on, and
get it up and running flawlessly on Google's container engine. And this is
coming from having zero Docker experience.

~~~
eggie5
Not as fast as you, but I requested myself for about 2-3 weeks and went from 0
Docker/k8s experience to having a modern SOA rails app up and running on a GKE
cluster. Are you using some type of CI/CD tool to deploy? How did you setup
DNS?

~~~
ubercore
I didn't get that in-depth, which is why I was able to do it so quickly :) I
just deployed to Google Container Engine using some docker images I built to
replicate a rethinkdb/node app we've been working on. I used DNS for service
discovery in the app, and didn't hook into a CD tool. Just used the google
command line tools.

------
dominotw
>Making a new filesystem in 1 year is still an impossible mission. Docker just
tried and failed. Yet they’re trying again! We’ll see how it turns out in a
few years.

Is this just fear mongering or are there any actual issues with overlay2. we
have been trying out overlay2 on our less critical systems and haven't
experienced any issues so far but we have stressed it enough to know for sure.

------
0xmohit

      That was a 7 hours interplanetary outage because of Docker. All
      that’s left from the outage is a few messages on a GitHub
      issue. There was no postmortem. It had little (none?) tech news
      or press coverage, in spite of the catastrophic failure.
    

This is incorrect. There were github issues, HN posts, and so on. Solomon
Hykes posted on Github [0] and HN [1]:

    
    
      Hi everyone. I work at Docker.
    
      First, my apologies for the outage. I consider our package
      infrastructure as critical infrastructure, both for the free
      and commercial versions of Docker. It's true that we offer
      better support for the commercial version (it's one if its
      features), but that should not apply to fundamental things like
      being able to download your packages.
    
      The team is working on the issue and will continue to give
      updates here. We are taking this seriously.
    
      Some of you pointed out that the response time and use of
      communication channels seem inadequate, for example the
      @dockerststus bot has not mentioned the issue when it was
      detected. I share the opinion but I don't know the full story
      yet; the post-mortem will tell us for sure what went wrong. At
      the moment the team is focusing on fixing the issue and I don't
      want to distract them from that.
    
      Once the post-mortem identifies what went wrong, we will take
      appropriate corrective action. I suspect part of it will be
      better coordination between core engineers and infrastructure
      engineers (2 distinct groups within Docker).
    
      Thanks and sorry again for the inconvenience.
    

[0]
[https://github.com/docker/docker/issues/23203#issuecomment-2...](https://github.com/docker/docker/issues/23203#issuecomment-223326996)

[1]
[https://news.ycombinator.com/item?id=11823231](https://news.ycombinator.com/item?id=11823231)

~~~
paulddraper
So what did go wrong?

~~~
user5994461
As said, they pushed packages with invalid signatures.

Why did this happen? There wasn't any explanation given. I'd love to see a
postmortem from Docker.

~~~
paulddraper
Exactly.

------
vogonogov
We've been using docker in production for two years now. Started with docker
1.4 and Rancher v0.15. Docker has had it problems with backwards
compatibility, but we haven't had the mentioned filesystem problems.

Our containers are completely stateless. We use AWS's RDS and S3 to store
state.

------
thaeli
Since OP is running on AWS anyway, why not just use Amazon ECS? The
orchestration (including Elastic Beanstalk) is pretty good, and we haven't had
any stability issues in production with it. Same thing for registries; just
use AWS ECR instead.

~~~
nzoschke
Here here. I operate hundreds of ECS clusters and it's great.

> The impossible challenge with Docker is to come with a working combination
> of kernel + distribution + docker version + filesystem.

Amazon has met this challenge with the ECS Optimized AMI. It works, gets
frequent upstream updates, and you can open tickets against it and get great
support.

[http://docs.aws.amazon.com/AmazonECS/latest/developerguide/e...](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-
optimized_AMI.html)

* Amazon Linux

* Linux Kernel 4.4.23-31.54.amzn1.x86_64

* Docker version 1.11.2, build b9f10c9/1.11.2

* Device mapper

------
julienchastang
I also have been steeped in Dockerland for the last couple of years. Here are
my own Docker pros/cons. Caveat: I am not nearly the power user that thehftguy
is or understand Linux kernel issues in any real depth. I’m a lowly high-level
language programmer (elisp/Python/Clojure/Java/shell).

Pros:

When Docker works, it’s great. The concept of being able to install software
on the users behalf in the container and have the user install the container
really does make life easier. The unwieldy command lines are nicely abstracted
with docker-compose(.yml). Software upgrades via containers becomes a nearly
trivial process.

I work on scientific software that can be a huge pain to install and
configuring the packages correctly often times requires the help of the
software creator. The fact that you can codify this “dark knowledge” in a
Dockerfile is of tremendous benefit. And generally speaking, I like the
concept of the Dockerfile. Whatever container technology ends up winning, I
hope there will always be something like the Dockerfile.

Cons:

\- Docker version upgrades never go smoothly. Firing up docker after apt-get
update && apt-get upgrade never works with respect to Docker. You get some
obscure error and after spending 30 minutes in Google you end up having to rm
-rf /var/lib/docker/aufs. My solution is sometimes to throwaway the VM and
start over, but this is unacceptable.

\- File ownership issues inside versus outside the container are (or at least
used to be) a huge pain in the ass! I am referring to files mounted from the
Docker host file system into the container. (And I am also referring to
running docker as unprivileged user that is part of the docker group.) The
solution I settled on is to enter the container as root, run chmod/chown on
the appropriate directories and drop down to a “regular” user with gosu. At
that point, you proceed on your merry way by starting your service, process,
etc. This work flow solves permission problems inside the container, but
outside the container is a different story. At least on earlier versions of
Docker, you could lose permissions of files created via the container because
at that point they are owned by root as seen from outside the container! More
recent versions of Docker seem to solve this problem as I have gleaned
empirically and from experimentation but I have yet to see adequate
documentation about any of this and I have spent a long time searching! With
more recent versions of docker I have _observed_ that the files in question
are no longer owned by root (again as viewed outside the container), but by
the user that started the container, the behavior I would expect. I’m also not
crazy about the gosu solution which seems like a bolted on rather than baked
in solution.

------
kordless
> My first encounter with docker goes back to early 2015. Docker was
> experimented with to find out whether it could benefit us. At the time it
> wasn’t possible to run a container [in the background] and there wasn’t any
> command to see what was running, debug or ssh into the container.

Having worked all of 2015 in the container space, I can state this comment is
a false statement. While it may not have been possible for this person, the
technology has existed for quite some time to see what process are running in
a container, debugging became possible with sysdig and ssh'ing into a
container has been a thing for YEARS.

------
edsouza
Excuse my ignorance on docker, but why did the OP not use a proxy cache for
the registry? [0]

If they were running this on a single machine with backups they could restore
to a time before the incident.

Also querying and downloading from the public registry everytime time seems
like a waste of bandwidth.

[0] [https://blog.docker.com/2015/10/registry-proxy-cache-
docker-...](https://blog.docker.com/2015/10/registry-proxy-cache-docker-open-
source/)

------
carrja99
> My first encounter with docker goes back to early 2015. Docker was
> experimented with to find out whether it could benefit us. At the time it
> wasn’t possible to run a container [in the background] and there wasn’t any
> command to see what was running, debug or ssh into the container. The
> experiment was quick, Docker was useless and closer to an alpha prototype
> than a release.

Anyone who has used docker at all knows the above statement is 200% rubbish.
FFS

~~~
zwischenzug
Yeah, it is! I was using it pre-2014 and docker top was available then and the
standard way to debug was to set up an ssh server in the container. You only
had to google it.

~~~
vitalysh
Or just use lxc-attach

------
pjwal
We're successfully running Dokku instances for various dev/qa environments,
but I have not attempted containers in production as of yet.

While Kubernetes seems to be the orchestration choice du jour, I had been
eyeing [http://deis.io/](http://deis.io/) (which I believe to be at least as
mature). Can anyone comment on their experience or thoughts on Deis vs
Kubernetes? TIA.

~~~
sagichmal
Deis is built on top of Kubernetes.

~~~
pjwal
Ok, I see that now, it was not always the case...it has been awhile since I
looked through their documentation. TY!

~~~
bacongobbler
Yep, we made the shift over from Fleet (a CoreOS project) to Kubernetes in v2.
We were just evaluating Kubernetes at that time.

------
ungzd
> Making a new filesystem in 1 year is an impossible mission

If this is "real" filesystem for disks, not proxy for other filesystems,
mostly read-only.

------
iamleppert
Over the course of the years I've tried to like Docker. I've found the
benefits to be not worth the stress and aggravation for the associated "magic"
that it takes to provide the kind of user (developer?) experience they're
after but have thus far failed to achieve. It's like the uncanny valley of
sysadmin tools.

The concept of getting away from snowflake machines and making your
infrastructure more code-like with deployment scripts is the way to go, but
you certainly don't have to use docker for that. At the end of the day, it's
more about forcing your team to be regimented in their processes and having a
working CI system for your systems related stuff. You should be able to start
from nothing and with a single command have a developer's environment up, all
the way to a fully scaled prod system.

Docker is just a tool to try and get to that point of evolution (albeit a poor
one).

------
Ideabile
After reading your post I think that you miss understood the purpose of
Docker.

Even if people try to sell it has flexible silver bullet, made up for
everything; like any other tools around, they need to apply some sort of CAP
theorem. Because underwoods we're just talking in how moving bits.

For instance I use containers like AWS lambda's functions. And I use tar
streams to pass the contents inside them (So no virtualised mounted filesystem
needed).

With that I mean you can find different way to use them, in short: Law of the
instrument.

If you're wise (and according with you article you got ton of experience), you
can find good purpose for it.

Docker is just 'meant' to make your life 'easier', like any other tool, not
always they satisfy the expectations.

In other hands you gain experience with it, I'm sure you would use them back
for a better purpose, where they can perform better for you, there is nothing
to worry about this is just experience.

Thanks for sharing.

------
bergwolf
Interesting! A relevant topic ("Hyper is Docker Done the Right Way" by
TheNewStack) came up in HN first page as well.

Is it how Docker should have been designed to embrace production in the first
place?

[https://news.ycombinator.com/item?id=12873089](https://news.ycombinator.com/item?id=12873089)

------
crad
I've been using Docker in production for over a year at this point. I can't
say that it's been problem free, but nothing so dramatic as this post
outlines. For example, I run Erlang apps in it, with heavy production use.

Heck, it is possible even remove images from the docker repository without
corrupting it, just not from a single API endpoint. FWIW I think that it is a
big failing of the v2 repository project to not have that functionality from
the get go.

It's also worth noting that I've only run Docker in production using CoreOS
and Amazon's ECS AMI. Both have their drawbacks, but nothing so dramatic as to
keep me from recommending Docker in production for "cattle" style
applications.

------
minitech
The section on not running databases with Docker isn’t really fair. Of course
you wouldn’t use something that crashed all the time to run your database
server, but Docker’s not _designed to crash_. It has volumes and you can do
regular crash recovery with it. Crashing hard and taking the host’s data with
it shouldn’t be a likely scenario and should be prepared for about the same as
disk failure – have redundancy and make backups.

~~~
SadWebDeveloper
From the article itself:

> Containers are not meant to store data. Actually, they are meant by design
> to NOT store data. Any attempt to go against this philosophy is bound to
> disaster.

~~~
minitech
Yes, I am replying to that point. It’s incorrect, or at least misleading.
Containers are meant to store any data that they have to externally; volumes
are a facility to do this. Storing your data in an ephemeral container would
absolutely be stupid, and the risk of unrecoverable corruption after a crash
should be low if you’re not experiencing frequent kernel panics.

------
Qerub
> The impossible challenge with Docker is to come with a working combination
> of kernel + distribution + docker version + filesystem. Right now. We don’t
> know of ANY combination that is stable (Maybe there isn’t any?).

You can do worse than use Red Hat Enterprise Linux 7 (RHEL7) that has a tested
and supported blend of those components. Or CentOS if you just want to try it
out and don't care about support.

------
ererekererkerk
Docker is unstable trash and does not belong anywhere near a production
environment.

Unsurprisingly the linked article is fraught with glaring errors and
misinformation. This has become typical of most things I read that is docker-
related; it's a circus.

It saddens me to see the Linux community get dragged into and overwhelmed by
this mess.

------
barnacs
It's funny how the article mentions AWS lock-in but fails to recognize the
infrastructure lock-in aspect of relying on a whole
container/orchestration/virtualization stack that is so fragile that
(according to the article) the only way to reasonably use it in production is
to outsource its maintanenace.

Once you go down this path, it seems there is a very a limited number of
providers capable of providing the infrastructure you require (at least
compared to vps/dedicated/datacenter providers) and they can just keep pushing
you into using more and more fragile infrastructure components that you would
never want to maintain yourself.

Eventually, you may very well find yourself paying so much for all the
hardware/virtualization/os/container/orchestration maintenance and resource
overhead, with a number of backup instances of everything due to the
unreliability of all these components, that you wish you could just go back to
placing a few pieces of metal in a few datacenters and call it a day.

------
nhumrich
I feel like almost all issues in this article stem from not following ops best
practices. And similar issues would happen no matter what technology was being
used, they just happen to blame their issues on docker because that's the tool
they were using at the time.

------
okket
Clean docker images:

    
    
      docker rmi $(docker images -f dangling=true -q --no-trunc)

~~~
justincormack
Docker 1.13 is shipping with cleanup commands built in.

~~~
anshargal
Just saw you comment and googled for details:
[https://github.com/docker/docker/pull/26108](https://github.com/docker/docker/pull/26108)

------
corv
Meanwhile BSD jails & Illumos zones are humming along nicely on ZFS.

------
ex3ndr
Does anyone know is rkt much better in terms backward compatibility?

------
z3t4
I think Docker is meant for apps in a shared environment (with other apps). If
you run dedicated app servers Docker is an unnecessary abstraction.

------
k2xl
I've been running docker in production for about 3 years.

Almost every time I've hit a problem, it has had nothing to do with docker
(though everyone likes to blame it). 99% of the time the issues I've
encountered are due to docker configuration (and yes the learning curve is
steep) or the app itself crashing.

The one time I've encountered an issue with docker with databases (used
elasticsearch) was a kernal issue (where the host kernal was out of date).
That was the only time I had to care about the host OS in debugging a docker
issue.

One thing that I recommend for anyone getting into docker - use docker-compose
for everything. _Don 't_ use the docker CLI.

One of the major problems I have with docker is that (especially for
beginniners) the docker command line tool can cause unintended consequences.

Every time you run _docker run <image>_ you are technically creating a new
container from an image. Once that finishes, the container still exists on
disk. Using the _\--rm_ flag will cause the container to actually be removed
after running it. I sort of wish docker would emphasize that flag, because
often when developers are debugging or trying things, they will continue to
run _docker run <image>_ multiple times - not realizing that they are creating
a ton of containers.

Yes, the containers are usually very light, but it caused a lot of confusion
for me when I started out. And part of my superstitious self believes that the
less dead containers you have the less likely you'll have filesystem issues.

A second source of confusion for new devs I've noticed is that the concept of
volumes are somewhat confusing. "Why would I ever do COPY in my docker file if
I can just volume it?"

Once you've worked with Docker, like any tool, you become more adept at
avoiding a lot of the mistakes mentioned in this article. Which leads me to a
point I can't disagree with the OP about - upgrading docker is annoying as
hell.

Almost everytime I upgrade docker I end up having some obscure error. When I
look on docker github I notice I'm not the only one, and often times the
issues are just a few hours old.

But, what the OP didn't seem to realize, is that to avoid these issues you
need to _lock_ your docker version in production to one that you know works.
Additionally, you need to _not build your dockerfile in production_. _Build
your docker containers with a build system and upload them as artifacts_. You
can either use an internal docker hub or do _docker save_ to save them as a
.tar.

Building while in production is a real no-no, even though it may seem more
attractive than moving around 300-500MB images. You never know if that one
dependency you _apt-get_ in your dockerfile is offline for some reason. And
you can't always depend on devs of dockerfiles properly versioning them
anyway.

~~~
user5994461
the --rm flag only removes the container, not the image. That doesn't help.

------
siegecraft
Ha, docker wishes that they had the staff to create all the things this
article claims they create.

------
leandot
Is it just me or this article mostly shows serious incompetence from the
author/company?

------
mamcx
If not Docker, then what?

~~~
jacques_chester
Well there's LXC, LXD and rkt to begin. Docker have themselves helpfully begun
extracting libraries.

The Garden container API which underpins Concourse CI and Cloud Foundry has
switched to runC as the container runtime. So far works fine, with no Docker
daemon required.

Disclosure: I work for Pivotal, we sponsor Concourse and donate the majority
of engineering to Cloud Foundry.

------
SadWebDeveloper
Docker has been the go to solution all my python-for-web-development friends
have been using as flagship to "kill php et al" and sharing this with every
single one of them is making smile again. Thank you OP.

------
discordianfish
That such a bad article ends up here shows the biggest problem with Docker:
It's irrational hype. Whether it leads you to hate it or want to dockerize all
the things, it's unhealthy.

~~~
Yeroc
Sensationalism and hyperbole sells. We all know that. We also know that the
truth almost always lies somewhere in the middle. The writer of the post
clearly bears a lot of scars. Self-inflicted? Maybe. Avoidable? Possible. Is
Docker at fault in some ways? Definitely. In any case, such articles still
serve to balance all the other hype and remind us where we're at on the Hype
Cycle curve with respect to Docker containerization. ;)

------
Unbeliever69
Let's not forget to LOL about those 2 animated GIFs. Just about ejected my
morning coffee through my nostril.

------
m3kw9
Docker SHALL NOT run databases in production, by design."

You need to use external volumes

~~~
amouat
I have some sympathy for the OP here, as I have heard of a lot of people
running into pain with DBs. But yes, they definitely should not be writing to
the UFS, as the OP seems to be doing. However, even if you're using volumes,
you need to be careful about what happens when images change etc. It's not
trivial.

------
X86BSD
I recommend he try a better solution such as tredly.com, or perhaps SmartOS.
Some things actually work OP, don't give up hope because of docker.

------
frenck
I'm not getting in to this discussion... nevertheless, this guy just made my
day!

This was a fun read :)

------
Daviey
Automatically updating core infra from public repositories without CI'ing.....

No... losing a day of development is on their infra team.. not docker.

------
ben_jones
Can we please, please, please get a follow-up titled: "Bash in Production: A
History of Failure"? With a comedic yet tragic theme of course.

------
moondev
Docker in Production: How we did it wrong

This is a typical case of blaming the tool. There are so many bad practices
mentioned I don't know where to start. No orchestration... One container a
host... Not using volumes correctly... Apt-get installing docker every time ci
runs... Not using an optimized base os for containers... It's amazing what
people will blame their problems on when they don't do their due dilligence
with a new platform.

~~~
mschuster91
> One container a host...

As said in the post, this was a consequence of Docker failing them when
running multiple containers on a host.

> Not using an optimized base os for containers...

On a server that might be possible, but I thought Docker's advantage was
providing a reproducible system that can also be run on your dev machine?
Sorry, if Docker doesn't run stable on my dev machine, no way I'm trying it.

~~~
moondev
The topic is running containers in production. Why not use a base os designed
for that?

~~~
mschuster91
Because the base OS needs to be something an admin can work comfortably on,
esp. if the host is supposed to run other things besides docker (e.g. a
database, you do not want to run mysql over a network if possible).

And for this you need a "real" base OS, which will be either apt-flavor
(debian/ubuntu) or rpm-flavor (rhel, sles), depending on the organization.

~~~
moondev
Why not run your database on a different machine then. Separation of concerns.
Have a cluster a machines for containers so you can take advantage of it's
strengths instead of trying to duct tape everything together on machine

~~~
mschuster91
> Why not run your database on a different machine then. Separation of
> concerns.

While true in general, you always end up introducing latency (simply because
it's on a different host) and possible packet losses/retransmits (e.g. when
the switch buffer memory overruns). Given the "hft" in the blog name (high-
speed financial transactions), both can equal losses of money.

------
billyjobob
Is 'An History' in the title some sort of meme joke like 'An Hero' [1] or does
the author not know basic grammar?

[1] [http://knowyourmeme.com/memes/an-hero](http://knowyourmeme.com/memes/an-
hero)

~~~
n0us
No it's not a joke and yes it's valid grammar. If you're going to correct
someone's grammar, at least know what you're talking about.

[https://www.quora.com/Which-one-is-correct-a-history-or-
an-h...](https://www.quora.com/Which-one-is-correct-a-history-or-an-history)

~~~
Benjamin_Dobell
Hmm, if anything that link seems to imply it's invalid - plus it's Quora, not
exactly a high quality reference.

The rule is "an" is used in front of a word that starts with a vowel sound[1].
"An hour" is correct because the "h" is silent. All modern (native) English
dialects pronounce the "h" in history, ergo "an history" is incorrect grammar.

[1]
[https://en.wikipedia.org/wiki/English_articles#Distinction_b...](https://en.wikipedia.org/wiki/English_articles#Distinction_between_a_and_an)

~~~
n0us
"In British English, either is acceptable"

from the first sentence.

> Quora

> not a high quality reference

> proceeds to reference Wikipedia

~~~
Benjamin_Dobell
What's your problem with Wikipedia?

~~~
n0us
none but I wouldn't call it a more reliable source than quora for something
like this.

