Do you know if Quay (or anyone else) solves the compilation issue?
The issue is that if you compile anything in your Dockerfile, you end up installing the compiler as well as producing unnecessary build artifacts, which will still remains as a layer that must be downloaded even if you uninstall the compiler and clean up after yourself. In other words, a bunch of unnecessary cruft. This applies not just to compiled languages, but to any language (Node.js, Ruby) that relies on a build phase as part of getting dependencies.
The proper fix is to perform the compilation outside of the main container (for example, by starting a throwaway build container that you only use for compilation) and then copy the final artifacts into the final container. But I don't know of any hosted solutions that support that workflow.
The trick around this, without requiring volumes and staging binaries in a host directory between containers is to...
Have a builder container that contains your build-deps (compiler, etc). Have it build your code, but have the CMD be a `tar` command that just tars your "output" to standard out, along with your "final container"s Dockerfile. Then, you run the builder container and pipe it's output as a new context to a docker build command.
This gives me minimal "Prod" containers that only include the built binaries. It doesn't include any of the SDK tooling, and it doesn't rely on staging the binary in some location on the host in between running the builder and building the final container.
For example, my frontend "builder" container winds up being hundreds of megabytes (thanks npm/nodejs) but the final product after doing the Angular2 build is a handful of kilobytes. Makes a big difference.
(It was also the only way to achieve this in certain setups before docker-machine and the new Docker for {Mac,Win} became available. I still use it because it feels "cleaner".)
I still think Dockerfile syntax should be extended to support this, but there have been proposals for well over two years now and nothing has happened.
It's especially great for compiled stuff, but we use it heavily with Python as well. In the case of the former, you do all of your compilation in a build container, then use a Docker build container to copy the compiled bin to your image. So you end up with an image that is just the base + your binary.
With Python, I've had great results with wheel dirs.
I believe App Engine Flexible (neé Managed VMs) does this (source: I looked at the code a while back). Basically your package.json (or requirements.txt for Python, etc) and your code get sent up as part of gcloud app deploy, and then a builder VM does the npm install bits for you, before producing your output "code plus packages" container that it stores in GCR (Google container registry).
This isn't just for fun though, if you're writing code on a Mac and deploying to App Engine, it's not very enjoyable to produce the x86-64 ELF libs you may need for your dependencies (I do admit this is why Vagrant, Virtual Box, etc. became so popular).
All that said, I'm not sure if the result of that is as minimal as we'd like ("final artifacts"). There's probably some amount of extra cruft, but certainly no more than roll your own.
Disclaimer: I'm one of the engineers on the Quay team
Quay doesn't (yet) support the idea of multi-step builds, but it is definitely on our roadmap.
One recommended approach would be to make use of our squashed image support [1]. You can build the artifacts using the in-image compiler toolchain and then delete it (in the same step or a followup step). If you then download the image as a squashed image, all deleted and overwritten files are automatically removed from the layer when it is created. Not a perfect solution, but definitely better than downloading all the unneeded binaries.
Cool, thanks! But if I understand this correctly, you squash everything into a single layer? If so, that's unfortunately not a workable solution for production environments, because it would cause every deploy to create a completely unique image, and so there'd be no caching of layers and containers wouldn't share any data, thus preventing effective use of file caching and shared lib reuse.
I'm also concerned that your cache behaviour wouldn't play well with an automatic workflow; whatever glue sits between Quay and Kubernetes would have to do a dummy pull just to "prime" your cache.
I haven't checked it out yet personally. I'm not sure what else I'd want to put on top of Kubernetes, though. The only thing I'm really left wanting right now is distributed cron (which is supposed to be landing in 1.4).
