I understand the problem with docker image sizes. I worked at a company where we had a ~1GB image and our CI tool didn’t support caching of docker images so it would take a good 15 minutes to do a build every time. But when we were faced with the option of using another smaller OS, like alpine, we decided not do it because we would give up a lot of flexibility that the OS was providing us.
If you’re running a statically linked binary produced by go and that’s all you want on your pretty much empty image, why not just scp the file and run it manually under a cgroup? Or good ol choot/jails/zones?
In a way you are answering your own question. Sure you can give up Docker and use something else, but you are giving up benefits of using the Docker infrastructure and ecosystem.
If you are just using Docker for one app, then yes I agree, but if you have other apps running through Docker then it’s certainly beneficial to do so even for statically linked executables, to keep everything consistent.
A new sysadmin doesn't need to learn that custom way your hand-rolled deployment system handles dependencies, how to see what's supposed to be running on a box, etc. A security person who's wondering whether something is supposed to be listening on a port can at least see that it's something someone went to the trouble of exposing. That QA team or the person who inherited your maintenance coding can avoid learning a new way to ensure they have a clean CI build.
(That doesn't mean that Docker's always the right answer — maybe you've identified an area like networking where there's a reason not to use it — but in most cases the differences are below the threshold where it's worth caring)
There are plenty of ways to create reproducible minimal images (such as using the nix package manager to create the rootfs), but the official docker images don't use those techniques and the docker tooling and ecosystem actively fight against it.
Docker is clearly focused on usability / first-user-experience at the expense of reproducibility and immutability. They encourage the use of the 'latest' tag, they encourage the use of procedural network-heavy build steps, and they have made not attempt to promote or create alternative tooling which tackles reproducibility correctly.
You can call `docker build --network=none` to disable network access during the build step.
Why would you make a lock file from a Dockerfile? You can specify everything inside it already; from the version you pull with yum to explicitly COPY'ing RPMs/debs you keep locally.
Being able to use consistent tooling with the rest of the containers is another plus.
> But when we were faced with the option of using another smaller OS, like alpine, we decided not do it because we would give up a lot of flexibility that the OS was providing us.
Alpine is getting easier to work with. It's been a while since I haven't been able to find a pre-built package for something I needed for instance.
That said, I definitely agree that you don't want to do 'FROM scratch' unless you're definitely not re-using the various upper layers. Having a fat base image is a one-time cost that potentially pays itself back many times over.
- There will only ever be one running kernel with docker.
- The base filesystem layer, if identically hashed, will be shared as an overlay filesystem.
- The memory footprint of whatever each container runs (which will generally not be a full from runlevel0 system) will not be shared, except in the sense that binaries loaded into ram from the same overlay filesystem will have some of their disk pages shared.
If I need more facilities from an OS, then I try to use a micro-distribution like Alpine. This could be because I have a more complex go binary, or if I have a python script that I want to execute.
If Alpine isn't cutting it, then I go for something like Ubuntu. This is typically because Alpine doesn't have some library that I need, or because musl libc isn't behaving properly.
This uses the awesome clux/muslrust Docker image as a build environment, then copies the result into a new ‘from scratch’ layer.
Works great with Haskell statically compiled binaries. Running the binary through UPX i've managed to get small HTTP microservices down to a 2MB docker image with just Scratch.
Edit: I really should have read the article first. It uses Go binaries as the example. Good to know Haskell folks are also using it.
You can use `setcap` to grant capabilities to the binary or the `pam_cap` module if you need to do capabilities per user.
I haven't run across the need to run most containers as root for a while now.
init comes from: http://appfs.rkeene.org/web/artifact/ecb8eda1cfb32ecc
And just sets up some symlinks and starts appfsd, followed by running bash (which is cached and run transparently).
For example I created a cntlm base image (linked in another comment)
From there I can do
and then add layers of services
first one is proxy
second one could be queue service (for example http://nsq.io)
then a message server, that just sends notifications
etc. etc. etc.
The same could be achieved downloading and configuring the static binaries, but Docker packaging, security and network separation makes evrything a little bit easier