A 3 MB nginx docker container is pretty impressive. I'm going to test this approach vs. the phusion baseimage-docker (http://phusion.github.io/baseimage-docker/) images this week, I think, just to see what difference there is in performance. (Beside the size of the image and the amount of time I'll save pulling images down the first time, I can't think of any other differences I'd see - nginx in a container is nginx in a container, or so I'd assume...)
OP: What are the technical limitations that stop builds from working inside a single Dockerfile? Just curious.
I strongly recommend against using the Phusion base image. That image is designed to treat the container as a virtual machine, complete with its own init system, cron, and SSH daemon. The best practice with Docker, as I understand it, is to treat a container as an isolated process and little more. The latter approach is much lighter, but requires some changes from the way things have typically been done on Unix systems; for example, processes shouldn't daemonize, and logs should go to stdout/stderr rather than syslog or a file.
So comparing the performance of my nginx or PostgreSQL images with images built on the Phusion base wouldn't be fair, since the Phusion base is heavy by Docker standards. A better point of comparison would be the Orchard images (https://index.docker.io/u/orchardup/).
My nginx build uses about 600 KB less RAM per worker than the Ubuntu package (which is what Orchard uses), because mine is statically linked, and musl generally has less overhead per process than glibc. So nothing spectacular.
As for why my images can't be trusted builds, the problem is that a single Dockerfile can't produce multiple images, or specify that only a portion of the final filesystem should be kept. So I have to do the build in two steps: First I produce, say, an nginx-build image, which has all the build-time layers and dependencies, and installs the final run-time artifact in a subdirectory (/runtime). Then I take that subdirectory and add it to the muslbase-runtime or muslbase-static-runtime image (depending on whether the particular build needs shared libraries) to produce the final runtime image. I can't do all of that in one Dockerfile, so it can't be a trusted build.
I think one strong feature is that it gives less for crackers to play with if they exploit the application. The less environment there is that is usable or recognisable, the better. So I am also in favour of stripping things down to the minimum.
Intuitively this seems plausible. In particular, I'm guessing that return-oriented programming is more difficult in a build with fewer libraries. And just having a custom build based on musl libc, as opposed to the omnipresent Debian and Ubuntu images, might throw off some attackers. I'd appreciate input from a real security expert though.
OP: What are the technical limitations that stop builds from working inside a single Dockerfile? Just curious.