
Things to avoid in Docker containers - 0xmohit
http://developerblog.redhat.com/2016/02/24/10-things-to-avoid-in-docker-containers/
======
fatherlinnux
The large yum update is a symptom, not the problem:

1\. The upstream base image maintainers aren’t doing their job of keeping the
image up to date, so it is huge. If the upstream maintainer had this base
image updated, there would be no updates. Not the developer’s fault. Also, if
the developer puts it into production, it’s their responsibility to make sure
there are no CVE’s in their code, and hence their responsibility to do a yum
update. You touched it, you own it.

2\. The Docker tools don’t have a way of flattening the image easily (without
doing a save/load). This can be fixed down the road.

Telling developers not to do yum updates terrifies me as a security person….

~~~
gtrevorjay
In no way should the advice be taken to be _never_ `yum update`.

Perhaps the article could have been clearer, but the advice is more subtle
than it first appears. It ties into point 4 ("Don't use a single layer
image"). You need to `yum update` at the right time to make good use of
layering and to be able to control updates separately from builds.

Imagine you build both Nginx and Apache containers off the same distro image.
Rather than having `RUN yum -y update && yum -y clean all` as the first step
(which is easy to mess up or forget), you can instead build an "update" layer
as a named image with it's own Dockerfile consisting solely of the `yum
update`. Now your two application Dockerfiles can simply `from base-update-
date`. This means you control when you get updates explicitly orthogonal to
the application build and lets you run the two apps off slightly different
versions, if there is indeed an issue you need time to resolve regarding
backwards compatibility. This also means you get the most out of the caching
mechanisms in Docker.

Full disclosure - I am both a hatter and a sec guy. [edit, spelling, grammar]

~~~
fatherlinnux
Again, this is recursion, if the upstream maintainer for the Apache and Nginx
images is doing their job, the "yum update" should do nothing.

Though, I agree, it's very important to highlight that too much branching can
cause massive environment bloat and way too many permutations.

If we just tell everyone to do yum updates at their layer this problem goes
away. Ahh, recursion...

~~~
garethadams
Well, apart from "6) Don’t use only the “latest” tag".

If I'm not using latest, and I don't know that the Apache and Nginx images
have been updated, then it's very likely that "yum update" might do something.

------
api
Don't run more than one process needs nuance. Apache and Postgres work fine in
containers and are fork based, and helper apps or threads that are purely
supportive don't matter either. No more than one job or "thing" makes sense.

I hate ham fisted advice like this that will always end up being dogmatized
and multiplying complexity needlessly.

~~~
conradev
The author does make a valid point in that logs are harder to collect the more
processes you add.

For example, I run gunicorn inside of a container with worker processes, and I
had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the
worker processes to the stdout of the master process so that they would show
up in `docker logs`.

~~~
stephenr
It's _almost_ like the idea of relying on stdout from processes in a container
to collect logs is fucking stupid.

With all the hoops people jump through so they can say "we use Docker, we're
cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now
on.

~~~
XorNot
If you drill through the docker source code, it becomes apparent that their
are some rather serious issues to doing something other then this.

I really really want a LOG directive for docker where I can just setup some
pipes which the daemon will collect and tag into it's log files (would make
dealing with old Java processes a lot easier).

But actually finding somewhere to hook this logic in - huge task. A lot harder
then grabbing stdout/stderr of the init process in the container.

~~~
Gigablah
You can use different logging drivers:
[https://docs.docker.com/engine/admin/logging/overview/](https://docs.docker.com/engine/admin/logging/overview/)

~~~
ibotty
That does not answer the question on how to capture _different_ application
logs.

~~~
elbear
Why not have the application log to a file on a data container?

~~~
lisivka
Who will manage these volumes, directories and log files? Why not just use
existing infrastructure?

I use containers for 10 years (they are called VServer), and I newer had a
problem with standard setup: some less important logs are stored directly in
container and rotated with logrotate, some more important logs (and error
messages from less important logs) are forwarded to central server for
analyzing, some are just disabled or discarded.

------
Perceptes
Any security folks have some specific practical examples of how running
processes as root inside the container is dangerous? I understand how root in
the container mapping to root on the host is risky (this is what user
namespaces are for) but I don't have enough experience to know why it matters
inside the container.

~~~
gtrevorjay
Given the possibility of a container escape, it's far better for the escapee
to find himself on the outside as a normal user. Just like with kernel
exploits, most escapes rely on access to esoteric, complicated, or poorly
understood devices or functions. Obviously, there are fewer of those in
userspace. Imagine, for example, an escape that required arbitrary network
control. That would be much harder to pull off (if not impossible) from
userspace.

There have been and will continue to be parts of the kernel hard-coded to
respect UID 0. Until these are all found and fixed (likely many years from
now) using usernamespaces to remap root will not provide all the safety one
might assume. It is super handy for other users though.

Full disclosure - I am both a hatter and a sec guy.

~~~
yAnonymous
The attacker can install a keylogger as normal user and then wait for the sudo
password. How much better is that, really?

~~~
tokenizerrr
How would a normal user go about installing this system-wde keylogger?

~~~
NoGravitas
It wouldn't be system-wide. You'd need to compromise an interactive user
account, install and run the keylogger as them, and wait for them to su or
sudo. So it's a means of escalating privileges from a regular user account.

~~~
tokenizerrr
Right, and your daemons use sub-1000 UIDs and user UIDs start at 1000. In
Debian, at least. Other distros have other starting points, but the concept
remains the same.

------
derFunk
All is quite valid, yet I disagree with "7) Don’t run more than one process in
a single container". This is perfectly doable with supervisors like ...
supervisord, which ensure that the multiple processes you want to run stay
alive. In some environments it's just virtually impossible to only let one
process run (eg webserver/proxy, log forwarder, fastcgi etc).

~~~
derFunk
Oh, I want to add why supervisord makes so much sense: If your "main process"
(PID 1) dies, your container would stop. Even if you only have a single
process to run, using supervisord makes sense to monitor that single process,
because restarting a stopped container can be a pain because it mostly
involves some manual work. Supervisord itself is written in python and is very
stable (because simple) itself.

~~~
azylman
Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle
restarting failed containers - that's generally something you want to be
outside of your container, not inside.

If you don't use one of those, you can still have supervisor run outside of
your containers, and manage a set of "docker run" commands.

~~~
derFunk
> Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle
> restarting failed containers - that's generally something you want to be
> outside of your container, not inside.

Not sure if I understand correctly. I'm using ECS. How would you provision the
Host with supervisord (automatedly)? ECS and the other systems are taking care
of restarting the containers anyway, but they don't necessarily have any logic
attached (like restart it 5 times, then stop and notifify if the container
doesn't come up again).

From my POV it's definitely better to directly have it deployed within the
container. Maybe it's depending on the use case.

The problem is that it often happens that the container cannot restart because
the main process (invoked by ENTRYPOINT or CMD) cannot start up anymore, e.g.
because of corrupted (config) data which has to be loaded on startup or other
suddenly unsatisfied dependencies.

~~~
fatherlinnux
I think they mean this: [https://blog.phusion.nl/2015/01/20/docker-and-the-
pid-1-zomb...](https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-
reaping-problem/)

------
anentropic
Does anyone have any advice on using "init" scripts as CMD for the container,
vs something like Ansible for post-provisioning?

Example: a Django web app... after deploying the new version container you
need to run migrations etc

Similarly for say RabbitMQ you may need to ensure your vhosts are defined
etc... commands that need to be executed against a running server, can't be
done in the Dockerfile itself.

So far I've ended up with a bash script in each container that does the
necessary init tasks before exec-ing the main process. Or in some cases
(Rabbit, Mongo) they first run the server in a limited state, do the init
tasks against it, then stop it and exec again as the container process.

Have started to feel it might be cleaner to instead just start the main
process and then provision it with Ansible afterwards. But then I lose the
ability to have a system up and running just via docker-compose...

~~~
goldbrick
You should think of all these operations in terms of the state they persist.
For example, a database migration doesn't modify the application at all, but
only the data directory wherever your database is running (whether mounted in
another container or across a network).

Whatever process is orchestrating your containers should take responsibility
for running setup chores in a separate, ephemeral container from the eventual
application container.

docker-compose is often not enough, unfortunately.

------
dcw303
I'm a docker newbie and I had a hare-brained idea to install postfix and
dovecot in a container. I thought it would be neat to version/deploy the
various config necessary on top of a clean OS install. As I'll probably mess
it up the first time, a quick way to blast the system clean is appealing.

If I can keep credentials and data outside of the container, is this still a
bad idea? Would I run into trouble with other containers needing to use SMTP?

~~~
contingencies
The biggest problem you will have is that a lot of critical SMTP functionality
(SPF, DKIM, DMARC, antispam blackhole lists, SSL certificates, email addresses
themselves) rely on DNS. So instead of testing with just a container, to get
any meaningfully useful notion of reliability and/or applicability to the real
world for your container you will have to test with entire fake DNS
infrastructure and multiple hosts. While you could technically do it, the
reality is you're entering a world of pain.

Currently setting up a mail-server myself, the majority of clients are in
China but I want reliable sending to foreign servers: it's quite difficult.
Like finding a host, when so many Chinese IP ranges are blanket blacklisted by
western mailservers, and the China/foreign link is often
slow/overloaded/lossy. Taking the time to update Gentoo postfix documentation
en-route, for the other DIY'ers with an NCH (not compiled here) problem. Not
using docker.

------
fpoling
I do no think "Don’t create large images" is a good one. I prefer to have a
single image that I use to run multiple containers with different services
that forms a pod. This greatly simplifies distribution as I need just single
image file. It also greatly simplifies updates.

But with such image it is useful to add few tools so docker exec -ti container
/bin/bash creates a shell that one can use for debugging or verification.

------
gizi
> Don’t store data in containers

You could manage your own data volumes or else let docker do it for you. I
don't see the difference. In the end, you will still need a backup policy. I
snapshot all data every 30 minutes for storage on another device. Nothing
would change to that policy when using external data containers.

> Don’t ship your application in two pieces

Internally -and externally provided software have different dynamics. Issues
are different. I update my own software with bug fixes and new features quite
frequently. I don't need to do that for externally provided software. If that
happens, I will indeed rebuild the container. In all practical terms, my own
software sits in a host folder where I can update it, without rebuilding the
container.

> Don’t use only the “latest” tag

I use debian:latest. I don't see a problem with that. It may lead to trouble
some day, but then I just change the tag to the latest but one version before
rebuilding. The problem has not occurred up till now.

> Don’t run more than one process in a single container

I have containers that internally queue their tasks. The queue processor is
then a second process, besides the main network listener process. There are
many reasons why it could be meaningful for a program to use more than one
process. A categorical imperative in this respect is misguided. Other
engineering concerns will take precedence.

You see, if I can reasonably split a container into two, I will. It's just
like with a function. If it is possible to split it in two, I will most likely
do so. But then again, such design recommendation should never be phrased as a
categorical imperative.

> Don’t store credentials in the image. Use environment variables

Both are pretty much the same problem. If the attacker can read files
containing credentials, he will also be able to read environment variables
with them.

> Containers are ephemeral

A network-based service uses a long-running listener to process requests. Why
shut it down? That would just disrupt the service. Containers may very well be
long-lived. They are not necessarily ephemeral.

This subject is not part of a domain such as morality where categorical
imperatives are the norm. There are pretty much no categorical imperatives in
software engineering. Software is mostly subject to just an Aristotelian non-
contradiction policy. If your choices are non-contradictory, feel free to go
with them. Furthermore, unmotivated, categorical imperatives simply have no
place in this field.

~~~
jarfil
> Containers are ephemeral

It doesn't matter how long they are up, it means you should expect them to go
down at any moment and lose all data stored inside. Plan for it by storing
your data and config somewhere else.

* Custom volumes and system mounts are persistent.

* Images are persistent-ish, they get replaced by updates.

* Containers are ephemeral, intended to scale up, down, sideways or whatever.

> Don't store data in containers

As stated above, you should expect your containers to puff out and into thin
air at any moment.

Also remember that docker auto-created volumes get auto-deleted when no more
containers are using them. Use custom created volumes or system mounts if you
don't want to wake up to a nasty surprise.

> Don’t use only the “latest” tag

It will bite you in the ass if you do. It may not today, nor tomorrow, but it
will some day, the exact day that a bigger version upgrade happens. It's not a
"hasn't happened to me yet, so it may not happen" thing, it _will_ happen to
you if you keep using the "latest" tag, no doubts about that.

So just plan accordingly and don't use the "latest" tag (for deployment).

> Don’t run more than one process in a single container

This could be more of a "really try not to" than a "don't", but the truth is
the more stuff you put in a single container, the harder it becomes to manage
and scale. It is much better to have self-contained containers (see what I did
there?) that you can match and mix with stuff like docker-compose.

> Don’t store credentials in the image. Use environment variables

No, they are not the same problem, and it has nothing to do with security.
When you want to scale and spin up a new container... now you can't because
your credentials are baked into the image.

Don't store credentials in the image, period. If you don't like environment
variables, you could mount a per-container config dir, but it's usually much
easier to manage config passed as environment variables to the container.

~~~
mateuszf
> No, they are not the same problem, and it has nothing to do with security.
> When you want to scale and spin up a new container... now you can't because
> your credentials are baked into the image.

Another reason - once image is built - why rebuilding it? There is always a
chance that you will end up with different versions of os or your packages.
Immutable image which doesn't store credentials can be used on multiple
environments with different credentials / configurations without changing /
rebuilding the image.

~~~
lisivka
You can always add layer with new credentials. E.g.

    
    
        FROM image:staging
        COPY ./production-credentials/ /
    

When implemented in this way, staging image won't be able to run in production
code until it will be tested and accepted (sealed) using formal process.

If you will use environment variables, you (or your deployment team) will be
able to run staging images in production at any moment.

~~~
mateuszf
Haven't thought about it - that's a nice solution, thanks.

------
grantlmiller
we open-sourced www.fromlatest.io as a way to share docker best practices in a
really easy to use way

------
cardoni
Great article. Very well written! Nice.

------
XxFurCornxX
The article is very well written, and I mostly agree with it. Upvoted. Some
other people will disagree with me.

~~~
philjackson
Apart from this one, your comment was the most pointless in this thread. Just
use the up arrow!

