The large yum update is a symptom, not the problem:
1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is huge. If the upstream maintainer had this base image updated, there would be no updates. Not the developer’s fault. Also, if the developer puts it into production, it’s their responsibility to make sure there are no CVE’s in their code, and hence their responsibility to do a yum update. You touched it, you own it.
2. The Docker tools don’t have a way of flattening the image easily (without doing a save/load). This can be fixed down the road.
Telling developers not to do yum updates terrifies me as a security person….
In no way should the advice be taken to be never `yum update`.
Perhaps the article could have been clearer, but the advice is more subtle than it first appears. It ties into point 4 ("Don't use a single layer image"). You need to `yum update` at the right time to make good use of layering and to be able to control updates separately from builds.
Imagine you build both Nginx and Apache containers off the same distro image. Rather than having `RUN yum -y update && yum -y clean all` as the first step (which is easy to mess up or forget), you can instead build an "update" layer as a named image with it's own Dockerfile consisting solely of the `yum update`. Now your two application Dockerfiles can simply `from base-update-date`. This means you control when you get updates explicitly orthogonal to the application build and lets you run the two apps off slightly different versions, if there is indeed an issue you need time to resolve regarding backwards compatibility. This also means you get the most out of the caching mechanisms in Docker.
Full disclosure - I am both a hatter and a sec guy.
[edit, spelling, grammar]
Well, apart from "6) Don’t use only the “latest” tag".
If I'm not using latest, and I don't know that the Apache and Nginx images have been updated, then it's very likely that "yum update" might do something.
The issue is one of process/collaboration/composition. Even at the largest of distros, package maintainers usually act with pseudo independence. They can change, QA, and push their packages basically at will. Images are collections of those pushes. Even with automation the aggregate nature of images makes them qualitatively different and---especially organizationally---provides a process challenge.
That solves part of the problem. The problem with base images, are they are the root - I call them Adam and Eve images. They do not solve all of the supply chain problems.
Layered images such as Apache and Nginx still may be maintained by operations or even Middlware teams.
Each team is responsible for making sure they update their piece of the supply chain. If the operations team (or image supplier) doesn't, the "yum upgrade" (or apt-get) downstream (by the developer or middlware maintainer) will bloat the environment.
The problem is, it's a symptom, not the problem. Telling people to NOT do yum updates, just promotes security problems because people upstream aren't keeping up with updates.
All of that said, the ONLY sane way to do all of this is with Automation. Hence, the reason for ImageStreams [1], BuildConfigs, DeployConfigs, and Triggers inside of OpenShift.
When any piece of the spider web detects a change, all of the dependent layers kick off jobs to rebuild themselves. Also, having good tests is critical. Yeah, most people on the planet haven't thought through this. Yes, Docker images make your life more convenient, at the deploy stage, with the downside that you need to do a LOT more thinking (automation and testing) at build phase.
Like shipping containers, we pack them at the factory, but can move them around at will after that. Instead of having dock hands load the ship at the port (which took 45 days in 1906). The supply chain problems are the same either way, but at least Docker gives you a good format to solve them.
This is exactly the conclusion we arrived at. We re-build our base images nightly (including all available updates) and then base the application images on those.
Yes to both points. Also, a lot of base images are pretty huge. I am looking forward to the move to alpine-based images (as well as assisting with that process)
Also, though I love Alpine, it uses Musl instead of Glibc. Some giant amount of the supply chain uses Glibc. You could convert everything to Musl, but then you would probably have performance hits on all kinds of things, so you would be trading CPU for Disk. Also, this could take TONS of time and money to figure out what is missing in Musl (including syscall mapping (not even sure it exists in other c libraries), etc)
Nothing is free, nothing. Building a cool new C library (musl) is fun. Maintaining one for 10, 20, or 30 years (glibc) is work (and fun)...
Careful, it's not the base image size. It's the environment size. If you don't put enough in your base image, you will have bloat everywhere. If you put too much, it will cause slow start (caching takes a while).
It's a balancing act. It's not easy. But, if you use a package exactly twice in your environment, then putting it in the base image actually makes sense.
Don't run more than one process needs nuance. Apache and Postgres work fine in containers and are fork based, and helper apps or threads that are purely supportive don't matter either. No more than one job or "thing" makes sense.
I hate ham fisted advice like this that will always end up being dogmatized and multiplying complexity needlessly.
The author does make a valid point in that logs are harder to collect the more processes you add.
For example, I run gunicorn inside of a container with worker processes, and I had to do some trickery (writing to /proc/1/fd/1) to redirect the logs of the worker processes to the stdout of the master process so that they would show up in `docker logs`.
It's almost like the idea of relying on stdout from processes in a container to collect logs is fucking stupid.
With all the hoops people jump through so they can say "we use Docker, we're cool" I think I shall refer to Docker fan-boys as Olympic Gymnasts from now on.
Why is it stupid ? Is the same approach that heroku advocates.
I have loads of containers logging to stdout and another container which collects all stdouts and send to a logging-as-a-service. Why is it a bad idea ?
Let's compare logging via stdout, to say syslog, with a simple test.
How do you identify the difference between a warning and an error or an emergency, in a string sent to stdout? What happens if the message contains a newline. How do you know if what follows is a continuation of the log entry, or is a new log entry?
If your application dares to have two processes in the same container (i know, crazy concept) how do you separate them.
The "literally one process per container" idea is what both causes the issue, and causes the obvious solution to the issue to be even harder to solve for the cases where software doesn't natively support the syslog protocol.
Were you the one who downvotted me because you disagree? o.O
Anyways, this is dealt with in the logging-as-a-service (LAAS) that receives the logs... say you are using the python logger, you have different types of logging (DEBUG, INFO, WARN, ERR). If you just start the line with one of those statuses, your LAAS will know it should email you or something like that.
The print, or file, or whatever, it's just how your data is getting somewhere for the proper analysis.
You know if it's a new line or not because every log message starts with the same format, something like {APP, TIME, SEVERITY} and then the message...
So instead of having actually structured information in defined fields, you're relying on a specific service's format. So presumably you then have to try to get all the services you use (e.g. Apache / NGinx / HAproxy / Varnish / (my|postgre)sql / redis / memcache etc) to format logs in the same way, and do it to stdout...
> Were you the one who downvotted me because you disagree? o.O
Down voting on HN is broken[1], so I don't use it, ever[2].
1. In the sense of, it does the wrong thing, not in the sense of it doesn't do anything.
2. Except when i hit the wrong thing because of the complete lack of usability on a mobile device.
yup, I will tell you tho, it isn't really hard once you have a format... it's pretty easy to share the format between different services as it's just string formatting.
I currently have logs from the django app and nginx and was pretty easy to set up.
It might not be the best solution but I don't think it's stupid
If you drill through the docker source code, it becomes apparent that their are some rather serious issues to doing something other then this.
I really really want a LOG directive for docker where I can just setup some pipes which the daemon will collect and tag into it's log files (would make dealing with old Java processes a lot easier).
But actually finding somewhere to hook this logic in - huge task. A lot harder then grabbing stdout/stderr of the init process in the container.
Who will manage these volumes, directories and log files?
Why not just use existing infrastructure?
I use containers for 10 years (they are called VServer), and I newer had a problem with standard setup: some less important logs are stored directly in container and rotated with logrotate, some more important logs (and error messages from less important logs) are forwarded to central server for analyzing, some are just disabled or discarded.
I'm a heavy docker (ab)user, but I haven't gone through the source code - i.e. perhaps there is some complexity I'm overlooking.
But couldn't Docker provide additional file handles for extra logging streams?
Programs which understand they live in a Docker world could hook into an special system call directly. Non-Docker-Logging-Enabled apps could have a helper program (conceptually similar to 'ip netns exec') remap STDOUT & STDERR.
Normally, if you run Docker you won't have a single container anyway, normally you want to scale out. Hence you'd be better off using a centralized logging infrastructure like ELK or one of the many commercial solutions like Splunk etc.
True, but it's not uncommon to only route the stdout/stderr of each container to fluentd, logstash and the like such that the containers do not have to care about where the logs go (other than just writing to stdout/stderr). And now that we have logging driver I definitely would want to avoid custom in-container log routing.
Yes, it's good practice to have a separated logging container (eg logstash forwarder, filebeat etc), which accesses the shared log folder of the other container you want to actually forward the logs from. I'm recommending using file forwarder because this way you can simply categorize your logs by filename. Pushing all to stdout is a pain.
Yeah, the problem is getting them from each process in the container, then get them to the centralized logging. It can be done either way though. Especially with the syslog driver in Docker...
If you put Apache and Postgres into the same cgroup/pid namespace/mount namespace/network namespace, an attack on Apache could result in access to the mount namespace of your Postgres. (Presumably that is something you are trying to prevent).
I think what the comment you're replying to was saying is that even putting just one of them in a container would violate the rules here, because Apache and Postgres are multi-process applications (at least in many common configurations), and the rules say flat-out no more than one process.
Referring to these as "rules" when they aren't is exactly what great-grandparent was warning against:
> […] will always end up being dogmatized and multiplying complexity needlessly.
If more than one process helps you get your job done, great. If it's all one self-contained application, go for it. You start feeling the pain when you have unrelated or loosely-coupled applications in the same container.
Any security folks have some specific practical examples of how running processes as root inside the container is dangerous? I understand how root in the container mapping to root on the host is risky (this is what user namespaces are for) but I don't have enough experience to know why it matters inside the container.
Given the possibility of a container escape, it's far better for the escapee to find himself on the outside as a normal user. Just like with kernel exploits, most escapes rely on access to esoteric, complicated, or poorly understood devices or functions. Obviously, there are fewer of those in userspace. Imagine, for example, an escape that required arbitrary network control. That would be much harder to pull off (if not impossible) from userspace.
There have been and will continue to be parts of the kernel hard-coded to respect UID 0. Until these are all found and fixed (likely many years from now) using usernamespaces to remap root will not provide all the safety one might assume. It is super handy for other users though.
Full disclosure - I am both a hatter and a sec guy.
Can you speak to the security implications of building the container as a remapped root and running as a user?
My understanding is that the most important reason to have root in the container is to install software through the standard measures, but obviously, we don't want to have to run our build process for containers as root on the real host.
Given the comparatively restricted behavior, is this a good practice or are there implications of using a remapped root during container build time that would linger on to run time?
That should be fine. The only "lingering" effects would be possibly suid or SELinux labeling oddities, but those are easy to watch out for.
And I should have been clear that a remapped root that drops drops privileges and/or transitions into another user is still better than an unremapped root doing the same. It's just that the remapping is not a panacea.
Value for each delta will vary, but every extra step (the need to install a keylogger) or dependency requirement (that this be an interactive node and someone eventually logs in) will decrease your overall chance of penetration.
Almost no one has the resources to stand up to a targeted attack from even a mildly capable hacker. Therefore, the risk profile most administrators are working against are attacks of opportunity. In that race, as in the old joke about outrunning your friend instead of a lion, you need only be be some small amount more difficult to crack than the next potential victim.
It wouldn't be system-wide. You'd need to compromise an interactive user account, install and run the keylogger as them, and wait for them to su or sudo. So it's a means of escalating privileges from a regular user account.
Right, and your daemons use sub-1000 UIDs and user UIDs start at 1000. In Debian, at least. Other distros have other starting points, but the concept remains the same.
Though things have improved considerably a while back with docker dropping most capabilities before starting a container, but without user namespacing on a shared kernel, ultimately root in the container == root in the host machine.
More specifically, uid 0 on the container == uid 0 on the host, so if container breakout occurs then you have root on the host.
Apart from that Docker doesn't really utilize user namespaces until recently, user namespaces are tricky. There have historically been buts and ifs involved. Don't rely on it unless you really understand it!
But why would you want to run things as root anyway? You probably didn't before Docker, so why do it now? With Docker, there's even less reason to muck about with capabilities for binding low ports etc. It's absolutely an unnecessary risk to take.
If one use host volumes, Docker has no way to setup them in a way that are usable for non-root container. There are workarounds, but running as root is a simplest thing to do that I think many follow.
You can obviate almost all the issues by running docker with --drop-cap all to drop all capabilities. You still share the uid, but with the default seccomp policy blocking ptrace as well it is pretty secure. Obviously running as a user as well is better.
All is quite valid, yet I disagree with "7) Don’t run more than one process in a single container". This is perfectly doable with supervisors like ... supervisord, which ensure that the multiple processes you want to run stay alive. In some environments it's just virtually impossible to only let one process run (eg webserver/proxy, log forwarder, fastcgi etc).
I think they picked the wrong term and meant something closer to service than process. If your service is say postfix, it consists of a half dozen or so processes that make up one service. I don't think 'they' would say that you should run each of the processes in a separate docker. So sometimes a single service is more than one process, and sometimes they are not as tightly coupled as postfix...
Agreed, I was trying to help a friend implement the Actor model inside a container, and if you want to put every process in a different container, you basically have to talk tot he Kubernetes Daemon to do it (not fun).
I am fine with services. I am also fine with larger applications under duress. I think the format still makes operations lives easier. Containerize everything, is better than some things...
Oh, I want to add why supervisord makes so much sense: If your "main process" (PID 1) dies, your container would stop. Even if you only have a single process to run, using supervisord makes sense to monitor that single process, because restarting a stopped container can be a pain because it mostly involves some manual work. Supervisord itself is written in python and is very stable (because simple) itself.
Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside.
If you don't use one of those, you can still have supervisor run outside of your containers, and manage a set of "docker run" commands.
> Most container orchestration systems (Kubernetes, Mesos, ECS, etc.) handle restarting failed containers - that's generally something you want to be outside of your container, not inside.
Not sure if I understand correctly. I'm using ECS. How would you provision the Host with supervisord (automatedly)?
ECS and the other systems are taking care of restarting the containers anyway, but they don't necessarily have any logic attached (like restart it 5 times, then stop and notifify if the container doesn't come up again).
From my POV it's definitely better to directly have it deployed within the container. Maybe it's depending on the use case.
The problem is that it often happens that the container cannot restart because the main process (invoked by ENTRYPOINT or CMD) cannot start up anymore, e.g. because of corrupted (config) data which has to be loaded on startup or other suddenly unsatisfied dependencies.
Integration with host systemd/machinectl is nice, but it requires modification of host setup, while my solution can be run on unmodified docker, i.e. it is universal. I will manage my containers using Kubernetes, so integration with host systemd is useless for me.
Does anyone have any advice on using "init" scripts as CMD for the container, vs something like Ansible for post-provisioning?
Example: a Django web app... after deploying the new version container you need to run migrations etc
Similarly for say RabbitMQ you may need to ensure your vhosts are defined etc... commands that need to be executed against a running server, can't be done in the Dockerfile itself.
So far I've ended up with a bash script in each container that does the necessary init tasks before exec-ing the main process. Or in some cases (Rabbit, Mongo) they first run the server in a limited state, do the init tasks against it, then stop it and exec again as the container process.
Have started to feel it might be cleaner to instead just start the main process and then provision it with Ansible afterwards. But then I lose the ability to have a system up and running just via docker-compose...
You should think of all these operations in terms of the state they persist. For example, a database migration doesn't modify the application at all, but only the data directory wherever your database is running (whether mounted in another container or across a network).
Whatever process is orchestrating your containers should take responsibility for running setup chores in a separate, ephemeral container from the eventual application container.
docker-compose is often not enough, unfortunately.
I'm a docker newbie and I had a hare-brained idea to install postfix and dovecot in a container. I thought it would be neat to version/deploy the various config necessary on top of a clean OS install. As I'll probably mess it up the first time, a quick way to blast the system clean is appealing.
If I can keep credentials and data outside of the container, is this still a bad idea? Would I run into trouble with other containers needing to use SMTP?
The biggest problem you will have is that a lot of critical SMTP functionality (SPF, DKIM, DMARC, antispam blackhole lists, SSL certificates, email addresses themselves) rely on DNS. So instead of testing with just a container, to get any meaningfully useful notion of reliability and/or applicability to the real world for your container you will have to test with entire fake DNS infrastructure and multiple hosts. While you could technically do it, the reality is you're entering a world of pain.
Currently setting up a mail-server myself, the majority of clients are in China but I want reliable sending to foreign servers: it's quite difficult. Like finding a host, when so many Chinese IP ranges are blanket blacklisted by western mailservers, and the China/foreign link is often slow/overloaded/lossy. Taking the time to update Gentoo postfix documentation en-route, for the other DIY'ers with an NCH (not compiled here) problem. Not using docker.
I've run simple SMTP from a container and not had any issues. I've not tried a more complex stack (but I have thought about it... ) and I was looking at this project
I'm afraid I can't account for it - having never tried it, but it looks like the sort of thing you might be interested in!
PS If you are a newbie, do try something simple first - it took a while to get my head around the differences in the docker tooling (for example this project uses docker-compose which is another tool on top of the docker cl tool).
I do no think "Don’t create large images" is a good one. I prefer to have a single image that I use to run multiple containers with different services that forms a pod. This greatly simplifies distribution as I need just single image file. It also greatly simplifies updates.
But with such image it is useful to add few tools so docker exec -ti container /bin/bash creates a shell that one can use for debugging or verification.
You could manage your own data volumes or else let docker do it for you. I don't see the difference. In the end, you will still need a backup policy. I snapshot all data every 30 minutes for storage on another device. Nothing would change to that policy when using external data containers.
> Don’t ship your application in two pieces
Internally -and externally provided software have different dynamics. Issues are different. I update my own software with bug fixes and new features quite frequently. I don't need to do that for externally provided software. If that happens, I will indeed rebuild the container. In all practical terms, my own software sits in a host folder where I can update it, without rebuilding the container.
> Don’t use only the “latest” tag
I use debian:latest. I don't see a problem with that. It may lead to trouble some day, but then I just change the tag to the latest but one version before rebuilding. The problem has not occurred up till now.
> Don’t run more than one process in a single container
I have containers that internally queue their tasks. The queue processor is then a second process, besides the main network listener process. There are many reasons why it could be meaningful for a program to use more than one process. A categorical imperative in this respect is misguided. Other engineering concerns will take precedence.
You see, if I can reasonably split a container into two, I will. It's just like with a function. If it is possible to split it in two, I will most likely do so. But then again, such design recommendation should never be phrased as a categorical imperative.
> Don’t store credentials in the image. Use environment variables
Both are pretty much the same problem. If the attacker can read files containing credentials, he will also be able to read environment variables with them.
> Containers are ephemeral
A network-based service uses a long-running listener to process requests. Why shut it down? That would just disrupt the service. Containers may very well be long-lived. They are not necessarily ephemeral.
This subject is not part of a domain such as morality where categorical imperatives are the norm. There are pretty much no categorical imperatives in software engineering. Software is mostly subject to just an Aristotelian non-contradiction policy. If your choices are non-contradictory, feel free to go with them. Furthermore, unmotivated, categorical imperatives simply have no place in this field.
It doesn't matter how long they are up, it means you should expect them to go down at any moment and lose all data stored inside. Plan for it by storing your data and config somewhere else.
* Custom volumes and system mounts are persistent.
* Images are persistent-ish, they get replaced by updates.
* Containers are ephemeral, intended to scale up, down, sideways or whatever.
> Don't store data in containers
As stated above, you should expect your containers to puff out and into thin air at any moment.
Also remember that docker auto-created volumes get auto-deleted when no more containers are using them. Use custom created volumes or system mounts if you don't want to wake up to a nasty surprise.
> Don’t use only the “latest” tag
It will bite you in the ass if you do. It may not today, nor tomorrow, but it will some day, the exact day that a bigger version upgrade happens. It's not a "hasn't happened to me yet, so it may not happen" thing, it will happen to you if you keep using the "latest" tag, no doubts about that.
So just plan accordingly and don't use the "latest" tag (for deployment).
> Don’t run more than one process in a single container
This could be more of a "really try not to" than a "don't", but the truth is the more stuff you put in a single container, the harder it becomes to manage and scale. It is much better to have self-contained containers (see what I did there?) that you can match and mix with stuff like docker-compose.
> Don’t store credentials in the image. Use environment variables
No, they are not the same problem, and it has nothing to do with security. When you want to scale and spin up a new container... now you can't because your credentials are baked into the image.
Don't store credentials in the image, period. If you don't like environment variables, you could mount a per-container config dir, but it's usually much easier to manage config passed as environment variables to the container.
> No, they are not the same problem, and it has nothing to do with security. When you want to scale and spin up a new container... now you can't because your credentials are baked into the image.
Another reason - once image is built - why rebuilding it? There is always a chance that you will end up with different versions of os or your packages. Immutable image which doesn't store credentials can be used on multiple environments with different credentials / configurations without changing / rebuilding the image.
You can always add layer with new credentials. E.g.
FROM image:staging
COPY ./production-credentials/ /
When implemented in this way, staging image won't be able to run in production code until it will be tested and accepted (sealed) using formal process.
If you will use environment variables, you (or your deployment team) will be able to run staging images in production at any moment.
If you store data in containers, update your application inside the container, and treat them as long-running persistent entities, then what benefit are you getting out of a container?
You're essentially treating a container like a vps, so why not just run your app directly on a vps?
> I use debian:latest. I don't see a problem with that.
That's not the advice he was giving. He was talking about tagging the images you create. If you always use latest, you're not keeping a history of images in your registry that allows for easy rollbacks. Latest is an entirely appropriate tag to use so long as it's always a synonym for some other longer-lived tag.
> There are many reasons why it could be meaningful for a program to use more than one process.
There are, but there are people (like Phusion) that want to run lots of processes inside a container. That is, IMHO and that of the story author, a bad idea. I may be more lenient than the author, but I believe a single container should represent a single logical process, whether that makes to a single OS process or not. But when you have many distinct logical tasks, they should be separated into their own containers.
> Both are pretty much the same problem. If the attacker can read files containing credentials, he will also be able to read environment variables with them.
This is very, very wrong. Credentials stored inside the image are less flexible and stored in more places than credentials that are supplied as environment variables. Images are designed to be copied around easily and trying to add authorization to the copying process is never going to be particularly successful. In contrast, credentials supplied as environment variables only live on the machine that's running the container and are only visible to those with access to the machine.
> A network-based service uses a long-running listener to process requests. Why shut it down?
Because software is constantly changing. At my last job, we released many times per day. Trying to patch a running container is just a recipe for pain. It's much simpler to start new containers and kill the old ones. You can even do it without downtime if you've got a system to route to the correct container (something like a long-lived HAProxy process that watches etcd/consul for the container it should route to. More and more, the notion of a non-distributed system is becoming obsolete. And the one lesson I've learned about distributed systems is that state is what makes them hard and you have to be incredibly intentional about how you manage your state. This also means that any time you can avoid adding state, be it in code or infrastructure, you should avoid doing so. Ephemeral, immutable containers follow that philosophy.
Yet beside Docker's :latest tags also using "FROM xy:latest" is a very bad practice and should be avoided. Always stick to one certain version of the dependencies you're having and you should be safe that you're getting the reproducible behaviour you're expecting to get when creating your containers. (of course if the base image is yours then you might be knowing that using :latest is well tested and safe)
A Timestamp or git hash would be good options. And it's not that you shouldn't tag as latest, it's that you shouldn't only tag as latest. You can tag as latest in addition to other tags.
1. The upstream base image maintainers aren’t doing their job of keeping the image up to date, so it is huge. If the upstream maintainer had this base image updated, there would be no updates. Not the developer’s fault. Also, if the developer puts it into production, it’s their responsibility to make sure there are no CVE’s in their code, and hence their responsibility to do a yum update. You touched it, you own it.
2. The Docker tools don’t have a way of flattening the image easily (without doing a save/load). This can be fixed down the road.
Telling developers not to do yum updates terrifies me as a security person….