
Running Docker in production for 6 months - swap32
http://racknole.com/blog/running-docker-in-production-for-6-months/
======
nzoschke
Thanks for sharing your experience as another cautionary tale.

At Convox we have been running Docker in prod for 18 months successfully.

The secrets?

1\. Don't DIY. Building a custom deployment system with any tech (Docker,
Kubernetes, Ansible, Packer, etc) is a challenge. All the small problems add
up to one big burden on you. 6 months later you look back at a lot of wasted
time...

2\. Don't use all of Docker. Images, containers and the logging drivers are
all simple great. Volumes, networks and orchestration are complex.

3\. Use services. Using VPC is far simpler than Docker networking. Using ECS
is much easier than maintaining your own etcd or Swarm cluster. Using
Cloudwatch Logs is cheaper and more reliable than deploying a logging
contraption into your cluster. Use a DB service like RDS is far far easier
than building your own reliable data layer.

Again thanks for sharing your experience as a cautionary tale.

If you are starting a new business you should not take on building a
deployment system as part of the challenge.

Use a well-built and peer reviewed platform like Heroku, Elastic Beanstalk or
Convox.

~~~
ekidd
At Faraday, we're also been running Docker in production for over a year, and
we're very happy with how well things have been working.

As nzoschke recommends, we rely heavily on ECS, RDS and other managed
services. We are very careful about using exotic new Docker features until
somebody else has successfully used them in production. We use DNS, load
balancers and regular networking for discovering containers and communicating
between them.

And it all basically just works. The worst problem we've encountered is that
twice a year, we deploy a new version of ecs-agent to our staging cluster and
need to revert it because of issues.

For us, the biggest challenge has been setting up a good local workflow for
developing Docker apps with multiple services and multiple underlying git
repositories. The docker-compose tool is great but it doesn't go far enough
and doesn't provide enough structure. We've open sourced some our internal
Docker dev tools here [http://blog.faraday.io/announcing-cage-develop-and-
deploy-co...](http://blog.faraday.io/announcing-cage-develop-and-deploy-
complex-docker-applications/) but we think there's a lot more which could
happen to make it easy to develop complicated apps.

~~~
dkastner
As another Faraday employee, I'll add that we didn't start right away with
deployment. Instead, we used docker to provide a consistent way to test our
apps. Once all of our apps were properly dockerized to run tests, we were able
to move to deploying the app with a simple docker-machine + docker-compose
setup. After that, it was a relatively easy move to ECS. docker-compose can
probably handle 80% of your development needs, but the tool we wrote allowed
us to quickly switch our apps from a canonical "image mode" to "source mode"
while developing. It also provided a way to run acceptance tests locally with
our entire app cluster running locally. A little more info:
[https://dkastner.github.io/2015/07/25/accidental-
continuous-...](https://dkastner.github.io/2015/07/25/accidental-continuous-
delivery-and-docker.html)

------
vuyani
"we didn’t find enough reason to figure out if kubernetes was a better
option."

Then proceeds to name every problem kubernetes fixes

~~~
swap32
Yes. My bad! We really should have invested some time evaluating Kubernetes.

~~~
gtaylor
If you don't want to stand up your own cluster for eval, fire up Google
Container Engine for a few days. It's cheap and very well done.

------
lizxrice
I'm a big Docker fan, but this is interesting to me mostly because it shows
how it's hard to get started, particularly the opening few sentences about the
paucity of documentation for helping you get going from scratch. There have
been good guides, but they go out of date quickly! There's so much change
(because it's still very much under development) that it's hard for someone
coming to it fresh to figure out what's up-to-date and what isn't. For those
of us working in container tech and tools this is a good lesson about making
sure the entry curve isn't too steep.

~~~
k__
On the other hand, which alternatives are easier?

I'm a NixOS fan, but have to use Docker at work occasionally and I have the
feeling they are both difficult, but in "their" ways.

~~~
Ixiaus
What's hilarious is that we use _both_ nixos and docker...

~~~
k__
Why not?

I the configuration.nix of NixOS is a nice, small way to get your dev machines
configured and installed with the stuff you like. Just copy it on your new
machine and off you go :)

I don't know if I would ever use Nix instead of the more language specific
packet managers.

~~~
Ixiaus
Nix derivation and nix store managed artifacts is a superior model to many
language specific package managers (though the learning curve is non-trivial),
we use it with Haskell extensively, for instance.

~~~
fizzbatter
I can't wait for someone to write a great book on Nix. I was so lost in it. I
had it running on my home server, and needing to compile my own things because
of what Nixpkgs lacked was a constant struggle.

It was a very frustrating experience. A frustration which was led by the fact
that i could tell how powerful Nixos was - if only i could grok it.

~~~
Ixiaus
Yeah a book on nixos would be valuable I think. Understanding nixos really
comes down to reading all the docs then trying to accomplish what you need to
do, learning by doing and reading source code.

I love nix, my biggest gripe would be that the Nix language is dynamically
typed...

------
coldcode
Reading these articles and comment is fun, we are a very large organization
and starting to use docker, and we love making things even more complex than
other folks, so I can tell this is going to a big mess. Fundamentally we (as
in this industry) seem to love simple ideas that become massively complex.

~~~
qaq
Well that's how we ensure availability of employment opportunities :)

------
qaq
Not being a fan of docker but "Since images sizes can be as high as few GB,
its easy to run out of disk space. This is another problem with docker which
you have to figure out yourself. Despite the fact that everyone who’s ever
used docker seriously has to come across this issue sooner or later; no one
tells you about this at the outset" \- monitoring free disk space is something
you definitely want to do regardless of using docker or not.

~~~
kpcyrd
It's not that hard to work around this:

I've deployed a global unit into our coreos cluster that regularly executes:

    
    
        #!/bin/sh
        while :; do
            # remove stopped containers w/ volumes
            docker ps -a -q -f status=exited | \
                xargs -r docker rm -v
            # remove dangling images
            docker images -f "dangling=true" -q | \
                xargs -r docker rmi
    
            sleep 6h
        done
    

You need to be sure that you don't lose important data when running something
like this in your setup, but it works nicely to remove old images. This script
is deployed to non-coreos servers as well.

~~~
skrebbel
I want to use Docker in production so I don't have to write hacks like this.

I want to put my application, any application, in a nice tidy box, ship it to
a server, any server, an be confident that it'll work. That's the promise of
Docker. To me, Docker doesn't deliver on that promise if I have to copy&paste
a 12 line bash script from HN to be able to do that or else face out-of-space
problems at surely the worst possible moment. If I have to jump through hoops
like that, then what's the gain?

Might as well just install an Ubuntu VM, apt-get everything I need by hand or
with ansible and add my app to the startup script. I'll have similar
complexity in a more mature and well-understood environment.

~~~
qaq
If you are running a simple homogeneous setup there is literally no gain.

------
zwischenzug
A lot of this was the motivation for writing the book we wrote on Docker in
Practice [1]. The reality of implementing such a technology requires a
combination of improvisation, technical nous, bluffing, and a willingness to
work through the inevitable problems [2].

I've talked about the relative immaturity of Docker as a used system (outside
of dev) [3] and am struck often by how rarely people understand that it's
still a work in progress, albeit one that can massively transform your
business. The hype works.

That said, Docker can work fantastically in production, but you need to
understand its limits and start small.

[1] [https://www.amazon.com/Docker-Practice-Ian-
Miell/dp/16172927...](https://www.amazon.com/Docker-Practice-Ian-
Miell/dp/1617292729) \- working on 2nd edition, if anyone has any suggestions
@ianmiell

[2] Blog: [https://medium.com/@zwischenzugs](https://medium.com/@zwischenzugs)

[3]
[https://www.youtube.com/watch?v=Yj5I-yOaDao](https://www.youtube.com/watch?v=Yj5I-yOaDao)

------
afandian
Linked at the end is this article:
[https://thehftguy.wordpress.com/2016/11/01/docker-in-
product...](https://thehftguy.wordpress.com/2016/11/01/docker-in-production-
an-history-of-failure/)

As someone starting considering Docker (and possibly Swarm), these seem to be
pretty serious criticisms. Any experiences to corroborate / counter these two
posts? Going by what's written here it would be suicide to use Docker, but
many people are...

~~~
atemerev
My advice — do not use Swarm. Start directly with Kubernetes or DCOS. You
might think that those are overkill for smaller projects — well, I also
thought this way, and got into similar issues as this article describes. When
I started again with Kubernetes, life became much simpler.

~~~
agmcleod
Sorry I'm really confused and rather new to the devops world, aside from
pushing an app to a single VPS. What problem does Kubernetes solve exactly? Is
it to tie a single application spread among multiple servers?

~~~
atemerev
Docker way is structuring your apps in "one service per container" — Docker is
designed around this. Even the simplest applications require multiple
containers (one for front-end, one for back-end, one for the database etc) —
even without clustering or high-availability, which complicate things even
more.

When you have multiple containers, it is imperative that they are managed as a
group, with clearly defined dependencies. This is why Docker requires an
orchestration tool. And when the number of your production nodes is greater
than one, Docker on its own becomes inadequate for that.

~~~
ohstopitu
Ignoring availability & mirroring (and all other cloud enabled features) for
conversation's sake:

Can't the above be orchestrated via docker-compose?

(I've been using docker with docker-compose for my side projects but not been
using it excessively in customer facing production)

~~~
atemerev
It can, if you have a single production host and only one development machine,
i.e. if you are a solo developer.

For anything more than that, native Docker tooling is inadequate.

~~~
ohstopitu
I can understand the use of Kubernetes to spread over multiple hosts (but
that's generally for scaling & availability).

If you have multiple devs...how does docker-compose fall short? (I'm asking
because I wanted to suggest using docker for my team and wanted to know it's
short comings before I suggested it).

From my understanding: you can use compose to orchestrate databases, cache,
app all on one machine quite easily (and pass environment variables, get them
linked etc.)

~~~
wstrange
In addition to multi-node distribution, there are a bunch of things Kubernetes
gives you. Things that you may think you don't need, but you probably will if
you ever intend to put something into production. To name just a few:

\- Secret distribution

\- Managing persistent volumes

\- Monitoring containers for failure, restarting according to policy

\- Service discovery and DNS integration

\- Integration with load balancers, setting up routes, etc.

\- Managing affinity / non-affinity for containers

\- Sharing resources on a cluster via namespaces

~~~
kossae
_\- Monitoring containers for failure, restarting according to policy_

This alone warrants an orchestration solution, even on a single machine.

------
crb
> _Orchestration across multiple physical server gets even more nasty where
> you’d have to use something like Swarm. We’ve since realised that swarm is
> one of the lesser preferred options for orchestrating clusters._

Could you elaborate on this? Did you settle on an orchestrator, and if so,
which one?

~~~
swap32
We didn't use orchestration across multiple physical machines so it was
possible for us to get away with just docker-compose. Otherwise we would have
had to go with swarm. Kubernetes is a way better choice than both compose and
swarm but it has a even more steep learning curve.

~~~
crb
I work on Kubernetes, and we're putting a lot of effort into making it easier
to get started.

Here are some things that might help:

* minikube: [http://kubernetes.io/docs/getting-started-guides/minikube/](http://kubernetes.io/docs/getting-started-guides/minikube/)

* the new tutorial: [http://kubernetes.io/docs/tutorials/kubernetes-basics/](http://kubernetes.io/docs/tutorials/kubernetes-basics/)

* deploying in 2 steps with kubeadm: [http://kubernetes.io/docs/getting-started-guides/kubeadm/](http://kubernetes.io/docs/getting-started-guides/kubeadm/)

I'd love any honest feedback on how we can either (a) fix the learning curve,
or (b) remove the perception that it's steep.

~~~
vmarsy
Just a quick heads up: On chrome v54.0.284.0.68 on my Android nexus 7 tablet
the new tutorial page is half cut. I only see "asics" instead of what I assume
is supposed to be Basics, all the text below is also cut:

    
    
           Overview 
    
        asics
    
        through of the basics of the Kubernetes cluster        
        module contains some background information 
    

...

There's no way to zoom out, and there's 2 hamburger menu : one white on black
on the left and one blue on white on the right, both are broken The 2 other
links work well

~~~
crb
Do you see the "Create Issue" button at the bottom of the page?

If so, please raise this here (with a screenshot perhaps)

------
baconizer
This article reflects our transition experience with Docker, it took us two
years to finally feel comfortable (old chaps from decades of corp dev), but
end result is very positive and rewarding. It is fair to say that this should
be expected for any infrastructure migration, there is nothing wrong with
being slow and careful as long as we are moving forward. Based on experience
from our team, the problem has never been finding help/answers, problem is we
were facing, one one hand, an encyclopedia of single page documentation like
Dockerfile/docker run command, while practical guidances and gotchas are
scattered around rest of teh Internet, blended with personal and business
specific opinions. It is hard but eventually we figured the best way to work
with containers from git to build server to deployment, that fits well with
our productivity workflow;, as well as where to not use docker.

~~~
swap32
It'd be great if you can share your learnings and increase in productivity
with docker AND where not to use docker.

------
20andup
I have been playing with docker for the past week, and its really hard to get
started. I am only going to use it for integration test for now.

------
brown9-2
_We spent hours figuring out a good way to use a databases in both dev and
production with docker. It was tricky since docker containers don’t support
persistence unless you use a mount-point. There were a few patterns documented
which didn’t work for us or we didn’t really like. We had to figure it out by
ourselves. This is another area where you’re expected to figure out by
yourself whether it is a good idea to use docker to run your production
database. Hint- Its not._

I think that a problem that leads to a lot of these articles being written is
that the motivation of the authors to use Docker is unclear.

Why bother to put your database inside a container if the way you ran your
databases before worked just fine?

You shouldn't just rush to put all your processes in containers because it is
cool and Containerization Is A Good Thing. You should use technology that
makes sense given the problems you are trying to solve at a give moment.

~~~
dglass
Completely agree. I don't know where or how people started thinking that
putting a stateful service inside of a stateless container was a best
practice, but no one should be doing that in production.

~~~
swap32
As I said, if it is really as obvious as you say it is, please point me to a
single mention in official documentation that databases should not be
containerized.

~~~
hiou
Why would documentation about docker include basic fundamental knowledge about
deploying services in production? This isn't docker knowledge, this is service
based architecture 101. Why is it docker's responsibility to teach you general
software engineering concepts?

------
kossae
If anyone is looking into orchestration systems, I can't speak highly enough
of Kontena
([https://github.com/kontena/kontena](https://github.com/kontena/kontena)).
While it is still in the early stages of development, it is a great "small-mid
level" platform with a ton of features. The config and concepts are also easy
to wrap your head around. We chose Kontena as Swarm is still unstable (imo not
"production-ready" but I've seen counter-anecdotes) and Kubernetes was lacking
some features we required. The devs have been super helpful with any problems
we've had in deployment as well.

------
gigatexal
There's a lot of "figured it out ourselves" without actually sharing what they
figured out. It's like this post is a whole lot of nothing: we know docker in
production is not easy but how about telling us more details

~~~
swap32
Isn't every "figured it out ourselves" followed with the sentence describing
what we figured? I see only one section where I didn't describe what we
figured, which is the workflow part and I mentioned there that I would write a
separate blog.

------
orf
I agree with most of the points here, but some stuff is misleading. For
example the "you have to rebuild after everything that code change" is
ridiculous. Just use "COPY SRC/ /app" In your dockerfile, and in dev mount
SRC/ as a volume over /app. There, hot reloading sorted for development.

Don't get me wrong, docker is one of the most frustrating technologies I've
used (partly because it shows such promise), but a lot of the problems he
describes can be sorted with the most cursory Google.

~~~
gtirloni
What do you use to restart the app inside the container when the code changes?

~~~
swap32
Because since the source code folder was not mounted on the container, we had
to rebuild the image with the updated code and then restart the container. On
dev machines, since the source folder is mounted, django runserver's
autoreload function works as usual.

------
romanovcode
> Since the client wasn’t keen on spending on getting a more private repos, we
> managed with the single repo to save our base image with the most of the
> dependencies bundled into it.

What's the point then?

~~~
swap32
Well, most of your client are bound to do this. The point is we managed to get
away with only one private repo. My guess is you'd need more than one private
repo very rarely.

------
pmontra
Why is running a db in a container with the data directory mounted from the
host is such a bad idea? Dropping the container on a new host with a copy of
the data directory looks easier than having to install the db from scratch,
even with automatic tools. Are there performance penalties at run time?

~~~
swap32
It is fine for dev, but not for production. To quote from the link at the end
of the article- "Docker is meant to be stateless. Containers have no permanent
disk storage, whatever happens is ephemeral and is gone when the container
stops. Containers are not meant to store data. Actually, they are meant by
design to NOT store data. Any attempt to go against this philosophy is bound
to disaster.

Moreover. Docker is locking away processes and files through its abstraction,
they are unreachable as if they didn’t exist. It prevents from doing any sort
of recovery if something goes wrong"

"A crash would destroy the database and affect all systems connecting to it.
It is an erratic bug, triggered more frequently under intensive usage. A
database is the ultimate IO intensive load, that’s a guaranteed kernel panic.
Plus, there is another bug that can corrupt the docker mount (destroying all
data) and possibly the system filesystem as well (if they’re on the same
disk)."

~~~
cpuguy83
All these statements are patently false.

1\. Containers are not ephemeral. They have a lifecycle. Data written in the
container is persisted to disk and available after the container is stopped
and then started again.

2\. Processes/files/etc are not locked away as if they don't exist. See `ps
aux` on the host. You will see all the processes running. You can inspect the
filesystems for each container, etc. There is no magic here.

3\. A database crash could cause data corruption inside a container or not.
This has nothing to do with the container, and chances of a database crash are
not made worse by being in a container.

That said, I would let a volume driver manage persistent storage rather than
manually managing this through the host fs... but that's my preference.

\--- EDIT --- Disclaimer: I work at Docker Inc, and am a maintainer on the
Docker project.

~~~
swap32
>> Data written in the container is persisted to disk

Even if you don't mount any folder from the disk onto the container? Are you
sure? Then everything I know about containers is just wrong.

~~~
vel0city
I think there's a bit of mix up between what an image is and what a container
is. You normally don't write data to an image, but you can write data to the
container. You can keep this data so long as you keep the container, and you
can commit that container's data to the image if you wish.

I've found a good place to visualize how the filesystems work was this blog:
[http://merrigrove.blogspot.com/2015/10/visualizing-docker-
co...](http://merrigrove.blogspot.com/2015/10/visualizing-docker-containers-
and-images.html)

------
pacuna
Kind of very basic issues IMO. The only one that's hard is logging. We are
using K8s on AWS and I haven't found a good solution for logging
centralization. Personally I don't like the Kibana interface for that kind of
stuff.

~~~
nepotism2016
I blame this on the way kibana is advertised. Some of web2.0 GUI with fancy
color schemes and graphs.

I use kibana to monitor logs over 60 containers. Sure, the initial dashboard
is a pain to setup, but once you get going going its very useful

~~~
pmontra
My criticism to Kibana is that it runs all the queries for a dashboard in
parallel. It overloads the Elasticsearch machine(s) even when there are only
few data. A small single ES server would be enough if the queries would be run
sequentially. I wonder if Elastic is in the business of selling multicore 32
GB servers that sit idle most of the time.

A max_parallel_queries setting would be great.

------
vkat
Logging we’ve frequently experienced issues with django autoreload and delay
in flushing logs

>> Try using "ENV PYTHONUNBUFFERED 1" in your dockerfile

------
yahyaheee
Sounds like you need k8s! I went though a similar progression: loving docker
-> hating docker -> finding k8s

------
yotamoron
Honestly, this whole post sounds like an amateur complaining about how hard it
is to become a proffesional. Grow up.

~~~
swap32
Please enlighten us with your professional wisdom and do point out which parts
of the article are amateur.

------
vivekjuneja2010
Good write-up considering that you had to manage the adventure of learning and
scaling Docker along with the pressure of commitments from clients.

I would like to share some pointers on how I and my team deals with the issues
that you mentioned in this post.

1\. Orchestration :-

For us, Swarm never became a choice, as we started our journey adopting Docker
in early 2015. That time there was no Swarm. We resolves to Mesos and Marathon
for our orchestration. Both worked out well for us in the long run. We have
production systems running this setup for last few months. Swarm is mature
now, and with Docker 1.12 its been made more easier to use. The good thing
with Swarm is that you could avoid adding another new system in your
infrastructure like Mesos, Kubernetes etc. if you have reasonably simple
requirements. We found that Orchestration also established service discovery
and routing capabilities for us. We use HAProxy and Mesos DNS for our routing
and discovery needs. Marathon-lb project is used to allow us to reconfigure
HAProxy everytime a new Docker Container is deployed by the CD Pipeline.
Marathon manages our service ports across the cluster, and every new Docker
container gets its own unique service port. This service port is then informed
to HAProxy, and reload happens. This setup worked good for us, although we had
some initial trouble. We also practice Zero downtime deployment with our
stateless services using the ZDD script inside the Marathon-lb project.

2\. Running out of disk space :-

This is a common problem especially with the idea of rebuilding and deploying
disposable containers with the CD pipeline. In our case, we use Monit to
gather system wide metrics at all times. We use a Garbage collection script
that we developed in house to remove the old Docker images and Containers
periodically whenever Monit detects file system usage beyond the set
thresholds. We do continuous production deployments as often as we need, so
this allows for our Docker image diff to be minimal. We avoid big bang
releases so that the latency for docker push on the Build server and docker
pull on the cluster is minimal. The Spotify Docker-GC project is a good choice
according to me.

3\. Docker registry :-

We use Docker Registry container that runs on the Mesos cluster via Marathon.
The Docker Registry is backed by a shared volume on the Docker hosts. We share
the same volume on all Docker hosts in our cluster. So, if the registry
crashes on one host, Marathon is able to redeploy the Registry Container on
another host which has the access to the shared registry volume. We tried
moving our Registry backed to S3, but never in production. For the systems we
manage, we need the Docker images in house due to compliance requirements.
Therefore, we could not use Gitlab Registry or Docker hub for our production
deployments. But I heard good things about Gitlab registry.

4\. Logging :- We use Logspout on each Docker hosts. It forwards the logs to
our managed Logstash and further to Elasticsearch service. We use Kibana for
log dashboard. Logs are rolled over on each Docker container, so that we avoid
storing the logs on the host for long time. However, any distributed logging
introduces log ordering and latency issues. So, we are tackling them as of now
through some optimisations.

5\. Dependency and Base Images :-

We use hierarchical model of managing Base Images : One top-level Registry
(Global), and isolated docker registry for each project.

Every Base image gets into our Top-level Docker registry which is curated, and
the associated Dockerfile for that Base image is checked into our Git
Repository. We insist using these Base Images from our registry for all
projects.

Each project can then inherit the base image, and customize to the local needs
of the project. We follow CI and CD for our Base Images as well. Each project
gets a notification when the Base image changes. They are free to opt in or
opt out. This model works for us, but many not be that interesting for smaller
setups. I had written about it here:- [http://thenewstack.io/bakery-
foundation-container-images-mic...](http://thenewstack.io/bakery-foundation-
container-images-microservices/)

6\. DB and Persistence :-

We avoid running stateful services in production on Docker Container, as we
have qualms about the persistence support in Docker. However, we do use Docker
volumes for all purposes in non-prod environments including CI, Elasticsearch
and other services. We are very interested to pursue this further with
ClusterHQ and Flocker based offerings in the Docker ecosystem. I may blog
about it in the coming days on this. But so far, I don't have any production
experience with the Database in Docker. But I am optimistic that this will
happen soon.

7\. Longer build times :-

This is correct as per your assessment, but widely varies across deployments.
As said earlier, we want the team to have faster build times, so we build and
release as often as possible.This allows to not have to deal with build
latency. We use lightweight Base images like Alpine, and prevent the use of
Configuration and Package mangers like Puppet inside the container. In our
base images, we prevent bloating by avoid installing irrelevant packages. This
has backfired some times in production, but we have found ways to go around it
most times.

Overall, I am aware of the challenges that you had, and can connect with all
of them. Docker is not the panacea for all the infrastructure woes, but its
certainly gives the taste to me and our team on how software development and
delivery will change for good in the coming days.

~~~
swap32
There are some excellent suggestions and workarounds here and looks like
you've been using them in production for some time. Thanks for sharing.

------
eggie5
Orchestration:

A lot of the issue I see him describe in production are fixed by Kubernetes.
Compose works fine for local dev orchestration but pattern doesn’t work for
deployment. The ideal world would be that I can run my compose file on my
cloud provider, but Swarm isn’t there yet. I have to rewrite my compose file
using kubernetes configs — it’s not a 1:1 mapping but the high level
connection are there if you think of Cabernets Pods as Docker Containers. He
mentions orchestration across a cluster with Swarm is nasty, but it’s elegant
w/ Kubernetes.

Docker Registry:

Obviously, there is no constraint permitting him to use a 3rd party service.
Why to let Google Container Engine (GKE) or AWS ECR handle it for you?

Longer build times:

I think this is really where he is missing the mark. It sounds like he has a
fundamental misunderstand that if you mount the source code in dev you have to
do it in prod too and that you have to have 1 container. Not true: You can
mount the source code in dev using compose, so you don’t have to rebuild every
time you change a line. Also, I think it’s a pattern in docker to try to keep
your containers as atomic units of your app architecture. It sounds like they
are trying to bake all competent of their app into 1 container (app + db +
service, etc). Just break them up into containers, link them up w/ compose.
This architecture them translates cleanly to one of the cloud providers for
production: GKE or AWS.

DB and Persistence:

Yes, I think it is very clear that containers are stateless. So, yes if you
want to run a DB in a container you’d have to mount an external drive
somewhere. There merits and risks of that are another discussion, but as he
states it’s generally frowned upon to containerize a DB. (not completely sure
why, some argument about stability of container and corruption of data…) I
talk more about this here:
[https://news.ycombinator.com/item?id=12913198](https://news.ycombinator.com/item?id=12913198)

Logging:

I think 12-factor style app containerized fit more smoothly into the docker
compose style architecture. Accordingly if all you containers are logging to
std out it, it’s all conveniently merged and printed out to the terminal if
you run compose. Then on production, GKE handles it nicely too w/ the Logging
system.

In conclusion, I think most of his problems would been avoided if he didn’t
skip researching Kubernetes and if he didn’t make the mounting oversight. The
other big oversight I think he did, at least he didn't mention, is that he has
no deployment tool. I wouldn't be able to effectively deploy w/o a build tool
like Jenkins. I talk about a lot of these issues and how to fix them here:
[https://news.ycombinator.com/item?id=12860519](https://news.ycombinator.com/item?id=12860519)

------
Walkman
Yet another guy who doesn't know how to use Docker properly and just brags
about it. There are a lot of misinformation in this blog post.

The conclusion from these is not that Docker sucks, but YOU HAVE TO LEARN it.
I agree that it's a very steep learning curve, but after the pieces come
together, Docker solves quite a lot of problem and actually very useful.

~~~
atemerev
You have to learn it — where? Documentation is sketchy and mostly obsolete,
docker-swarm is buggy, genuine production issues are routinely closed with
WONTFIX resolution.

Could you point me to good piece of documentation or success story of running
Docker in production?

(I do use Docker, but it requires to learn Kubernetes or DCOS to do anything
production-related. And those are separate projects).

~~~
cpuguy83
Maintainer here, can you please point to a production issue that is WONTFIX?

~~~
atemerev
(Transferring ~/.docker wasn't an option as I already had a set of my own
hosts on my computer, and manually editing undocumented configuration files is
harder than it sounds — for some reason I forgot, I couldn't partially
transfer some hosts but not the other. This is after I have spent 2 days in
even deciphering what has happened, as error messages are uninformative and
Docker CLI just hangs in an endless loop in such situation).

Nothing of it is documented, of course.

~~~
cpuguy83
Indeed, would definitely be cool to be able to export a specific machine
config to a tarball that can be imported on another machine. I don't work on
machine, but anyone made a proposal for this (probably with warnings about
copying ssh keys and such)?

I know this doesn't help with your current issue on docker-machine, but... I
think the issue here is docker-machine's primary intent is as a developer tool
to spin up dev environments quickly and easily (zero-to-docker as we say), as
such the datastore and security model is tailored to this. We are working on
production-level infrastructure management, the base-layer of which you can
find here:
[https://github.com/docker/infrakit](https://github.com/docker/infrakit)

~~~
atemerev
Docker-swarm is positioned as a production-grade orchestration tool, yet it
depends on docker-machine. :(

~~~
cpuguy83
Why do you say it depends on docker-machine?

