The main reason why containers are (a lot) better than the former status quo is shockingly simple: Dockerfiles. They list the steps that you need to build a working image. And yep, there are some caveats, but most of the time they do not cause problems (example: I have containers where :latest has been fine for over 5 years.)
I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways.
But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.
I really don't understand what Dockerfiles offer in terms of reproducible builds that a minimal install of a distro + a config manager didn't.
I feel like we took Ansible roles (for example, it could be Puppet, CFEngine, whatever) and spread them in uncountable and not reusable Dockefiles and bash scripts (oh the irony). But we still have the config managers too, because who could have imagined, you still have to have bare metal underneath.
Docker (like every other tool before it) started nice, clean and simple, because it didn't cover all the real needs. As those where added on we ended up with tens of tools on top of k8s and now here we are in yaml hell with version compatibility still not solved. And a new generation will come along and repeat it with another set of tools becase "this time it will be different". You can not get rid of complexity, you can only move it from here to there. And if there == YAML then may God have mercy on you.
Basically, configuration management is thinking about the problem backwards. Taking a mutable filesystem and trying to flip things until its in the desired state is complicated and prone to a lot of problems, I used SaltStack for 4 years in production, it's hard to enumerate the number of ways that didn't work well.
Building an image from the ground up is the other direction. The difference is subtle, but if you have an exact known starting point, then declarative configuration management is not only unnecessary, it's downright worse than just running a few commands to get to the desired state.
The proof for me is in the pudding. Container building is a superset of configuration management. You can definitely use Ansible or what have you to build a container root image, but it would be long, overcomplicated, and more fragile than just starting from a known state and mutating from there.
The problem with config management is that it turns into a sprawling unmaintainable mess when you include everything in it. Back in 2014 I was working for a growing startup that was heavy into puppet. Everything was done through puppet. The puppet codebase was a gigantic mess. It has to care about a hundred different types of node. We containerized all of the internal apps and many of the other services (except for things like DNS, database and a few other bits and pieces). So yes, of course we "still had config management", but it shrunk from being a huge mess that we struggled to release weekly (and it always broke shit) to a simple codebase that was only concerned configuring the OS. Every developer was responsible for understanding their own container, and how it ran in production. And the infra team was responsible for providing a fleet of machines for that shit to run on. And a devops team took charge of building a system that scheduled containers on the fleet (pre-kubernetes-and-nomad). Simple, daily releases, and a nice separation of concerns.
Just for the sake of discussion: couldn't the same thing be achieved with sepparate code bases of puppet for each component and the infra? I'm no big fan of monorepos either.
If it worked, it worked, though. I'm not arguing with results.
It could, ish. The problem with puppet is that the setup is basically a single agent on the host that talks to a single puppet service. Puppet agent grabs "the one true manifest" and applies it. You could possibly break that up, but it would be a pain, and I wouldn't like to be troubleshooting two possible clashing puppet codebases on a production server.
What we actually did, back in the day, was host our own yum repo and build RPM packages. For the most part, the only differences between the nodes were the installed packages. But having to define 100s of node types in puppet and force every developer to know at least some puppet, and troubleshoot failed runs, and understand RPM packaging foot-guns. Like, if you screw up your package uninstall scripts or file lists you can make it impossible to remove a package. Or you can leave unwanted files behind. Or two different developers can try to "own" the same file, so one of those RPM installs is gonna fail.
Fundamentally, this also leans towards long-lived servers that you tend to and care for. We had teams with long-lived servers that deployed incremental app updates every few days. You know how many times we tried to build a node from scratch, only to find out that it failed to build? Or worse, it completed the build but the app is broken in some way? Eg: some historical package gets removed but leaves files behind, and this files are actually useful, so everything still works. But on a clean install it doesn't, so you end up on a sev1 incident using strace, ldd, hunting in logs etc etc.
You also wouldn't believe the number of incredibly stupid screwups that broke dozens of servers at a time when developers were building their own RPM packages and using Puppet to deploy them. This was the cause of most of my work as an SRE.
What made docker so good was that developers weren't given the ability to package up something dangerous into an RPM that gets installed by root, potentially fucking up the host.
Also, building a container either works or doesn't. This forced our developers, for the first time ever, to know 100% of the steps required to get the software to work in isolation.
And my Alpine container can peacefully coexist with your Ubuntu container on a CentOS system without filesystem clashes.
And instant rollback meant running the previous container tag.
Honestly, the puppet > docker change for applications was the biggest leap in reliability and simplification that I ever experienced.
They both achieve the same aims - reproducible setup - but (on Linux) a container is uses less resources than a VM, so it's lighter to run them both side by side. This matters for developer machine setup.
When it comes to production deployments, I haven't even started looking at k8s because I don't need that level of complexity.
Every new generation of devs is determined to learn the hard way that "complexity kills".
No, you cannot remove it, but you can definitely add more. Every time I have to work with the modern Javascript tooling upon tooling upon tooling, I want to punch a baby rabbit.
Even if I wasn't using containers for production, the ability to make a repeatable build allows you to make software with complex dependencies extremely easy to develop in. Making it possible for a new developer in a new environment to get a build working on their own laptop, in the first day on the job, didn't use to be simple.
And being able to make hermetic environments for integration tests used to be almost impossible, and today, depending on your stack, it is trivial with libraries like testcontainers.
As a long time user of VMs for this purpose, this is definitely missing the point. Containers are sort of a technology, or perhaps some combination of technologies, but the actual value of containers for development has nothing to do with cgroups, seccomp, etc. In fact, using firecracker/ignite/whatever would probably be just as effective as using Docker or Podman.
It really does come down to the concepts that underlie OCI images and Dockerfiles. Why?
- Immutable root design promotes hygienic containers that keep state only in mounts. This makes upgrading, backup, etc. of containers very easy. Containerized software can be written around the assumption that this is how things work, by having a way to e.g. run database migrations at startup.
- Composable images: Rather than having every environment ever made try to perfect making a reproducible Go development setup, you can just use FROM statements to use other images. With multi-stage builds you can even compose multiple images together to an extent.
- Single-process design: Prior to containers, most VMs operated like full machines that could standalone, with just emulated or paravirtual hardware. Containers are designed around running a single process. In practice, this dramatically lowers the complexity of containers and their lifecycle. Of course, you Can and sometimes even will need to have multiple processes in containers sometimes, but the general mantra holds well.
- Composable containers: Because of the previous point, it's natural to compose containers too, using features like Podman or Kubernetes pods, or Docker Compose. Containers in the same pod can share resources that would otherwise be isolated across typical containers, such as a network namespace. This makes for concepts like sidecars that allow composable functionality, and the ability to e.g. re-use PostgreSQL and Redis containers.
- As a bonus, these approaches avoid plenty of typical pitfalls of configuration management systems often used with e.g. Vagrant such as Ansible. Because the environment is built up from scratch to your specifications, rather than working backwards from a mutable Linux system and trying to bring it to match a certain specification. Some of the problems that can lead to configuration drift still exist, but they are somewhat contained, especially if you use tags and/or image digests to pin your images.
Virtual machines are technically fine, but containers elegantly offer solutions for both development and production deployment issues that VM setups never really did prior.
Indeed, you could make something like testcontainers but for VMs... but then what do you use in place of OCI images and Dockerfiles for the base images? And you need to handle networking between VMs and the host, probably across multiple different VM runtimes. Eventually you realize you're just making worse containers!
I won't go point-by-point, just point out that they are so similar, that there are products enterprises are buying that allow them to run their VM workloads in k8s like a container. It's also worth pointing out that containers sold themselves as VMs without the overhead (that was Google's goal with cgroups and namespaces that enabled things like Docker to come after)
Much of what you talk about is tooling and patterns around containers, not containers themselves, such as networking and volume mounts. I would also point out that there are better options to Dockerfiles for containers these days as well, that unlock the real potential of buildkit.
But that is the whole point! It's not about the technology or the isolation (while running). It's about the ability to simplify the generation of a new set of containers. Yes that's mostly tooling. But tooling is the single most important difference between something that's cool and something that's useful!
We moved from "here is an (incomplete) description of what I did", which rounds to "please guess all the previous stuff that I did to my environment, before even starting to work on getting my instructions to work" to "download this binary and execute these 5 lines". The incompleteness problem of the description of _the actual thing you want to do_ hasn't gone away, but the largely not even mentioned description of where you have to do the thing... that has been replaced by Dockerfiles
You seem not to have worked with the VM tools that do the same thing as a Dockerfile. Had you, you would realize that your points are invalid and I can have the same guarantees with VM as containers
If you think Dockerfiles are great, then you haven't hit the day 2 problems yet or explored the better alternatives for defining containers.
Also, the whole point of the original post is to point out that what you are saying is not magically true for containers
That's exactly what I have been saying, though. Prior to Docker, there was nothing like Dockerfiles or OCI images. The closest thing I can think of was Vagrant, but it's missing a lot of what makes Docker good.
The reason why containers are good is entirely about the concepts, NOT the actual engine or technologies. Again, you can literally run OCI images in a virtual machine; Ignite will run them in a KVM. That's as literal as you can get.
Docker existed and did a lot of the same things it does today in 2013.
The ONLY thing docker did was make a small amount of questionable UX around the low-level container primitives. They did not invent anything, definitely not containers. I could share VM images before Docker, nothing new there... the concepts all existed before containers.
You can literally run... anything... on a VM, why make a point about being able to run OCI images? This is like a no-op.
On one hand, layered docker builds mean that with some care you can only care about the top layer and treat base layers as immutable. As long as they are not repulled.
On the other hand, to have actual reproducibility you need to self build and self host everything down from base userland. However, once you achieve that, reproducible machines are one `apt install devenv` away.
What docker/containers do and excel at, compared to traditional workstation provisioning, is reduction of dependency trees via isolation. With one single user process running dependency trees are shaken and chances of accidental dependency collision drop. Does this count as solving dependency problem? Personally, I say no, but understand the other side of the argument.
Yeah, I understand what you mean; bit-for-bit reproducibility is a great idea, and to be clear, I was not talking about that sort of reproducibility, though we should certainly strive for it.
Yes I remember the problems with Vagrant. I'm unsure about what's making Docker more predictable across machines than Vagrant. Possible reasons
- it's usually headless
- it comes with a layer that mounts the host file system, instead of installing extensions
- better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac)
- it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine.
Not sure I get the last point. Spawning a shell in a container, persisting the changes made and distributing such edited container is trivial. Of course once you want to rebuild it reproducibly from scratch you need to edit the Dockerfile, but then it's exactly the same in Vagrant, isn't it?
Perhaps the difference is that storing your changes is an explicit operation that needs to happen consciously, and that data is usually persisted outside of the container.
The scenario is: write a Vagrantfile and create the VM on Linux, then manually fix it or evolve it and never destroy and rebuild the VM. Then give that Vagrantfile to somebody on Windows a couple of years later. Chanches are that it won't work as intended.
That's more difficult to happen with a container and its image.
What you suggested about the listed steps is a bad suggestion. Docker should crash if you don't use a sha256 in a Dockerfile, or at least make some sort of lock file. But it instead allows you to make mistakes.
I recently contributed to the Linux kernel and they often get irate over the contributor not following manual steps. They could automate a lot of it with basic features like CI, and your suggestion that it is easy to make things reproducible if you just follow a list of instructions is part of this problem. "Just follow a list of instructions" will never be a solution to a bad workflow, and it is no replacement for a good methodology.
If you do not force best practices in the tool, it permits mistakes. Something you probably don't want to allow in a tool that builds software and manages the software supply chain. Docker doesn't provide a language for describing things correctly, you can make all the mistakes you want to make. Nix, for example, is a domain specific language which won't permit you to commit crimes like not pinning your dependencies, at least in 2023 with pure evaluation mode (on by default in flakes).
> They list the steps that you need to build a working image.
No they don't. They typically list the steps that you need to build a working image yesterday, not today, or in 5 years, which is a very important thing to be aware of, otherwise you might assume the instructions in the Dockerfile were crafted with any intention of working tomorrow. There's no reason to believe this is true, unless you know the author really did follow your list of suggestions.
Nix crashes when you make mistakes in your .nix expression. `docker build` won't crash when you make mistakes in your build, it is unaware and doesn't enforce a reproducibility methodology like Nix outlines in the thesis, an obvious example being the unconditional internet access given by the Docker "sandbox".
Docker does not make distinctions between fetching source code and operating/building that source code. In Nix these happen in two separate steps, and you can't accidentally implement your build instructions in the same step as fetching your program, which would otherwise lead you to execute the build on untrusted/unexpected input. This is just one part of the methodology outlined in the Nix thesis.
TL;DR Nix doesn't suggest you follow a list of instructions to make things reproducible, it just doesn't permit a lot of common mistakes that lead to unreproducibility such as not pinning or hashing inputs.
Dev would get the thing working on their machine configured for a customer. We'd take their machine and put it in the server room, and use it as the server for that customer. Dev would get a new machine.
Yes, I know it's stupid. But if it's stupid and it works, it isn't stupid.
DLL Hell was real. Spending days trying to get the exact combination of runtimes and DLL's that made the thing spring into life wasn't fun, especially with a customer waiting and management breathing down our necks. This became the easiest option. We started speccing dev machines with half an eye on "this might end up in the server room".
nm -D is not hard. Debugging missing symbols isn't even that hard.
Biggest damn challenge I see is we've completely dropped the ball at educating people about how linkers, loaders, and dynamic library symbols work.
...and so people understand something here... I've transplanted/frankensteined userspaces that were completely hosed/disjoint into working before. In fact, every now and again I do it just to remain in practice. It's actually gotten to the point I don't even worry about dependency hell anymore. I just find the right version of the library/source and expand my archives just in case I need it down the road.
> "I don't know whats wrong, it works on my machine"
I had one of these years ago where QA had an issue that I couldn't reproduce.
I walked over to his desk, watched the repro and realized that he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select.
Yeah, it's been way too long to remember (maybe a decade ago). Best guess is the click event was propagating up the dom and triggering something. These days, it looks like chrome generates a click even if you don't let up on the mouse.
I have never been able to realize the alleged ergonomic gains of containers. Ever. It always adds more friction to actually getting something initially stood up, prototyped, and deployed.
I'm guessing it may be one of these things where it only starts to make sense after something has matured enough to warrant being replicated en-masse in a data-center environment.
Then again, I tend to live in a world where I'm testing the ever-loving crap out of everything; and all that instrumentation has to go somewhere!
The power of docker compose for the development process is unparalleled. Being able to declare my entire local environment in a single file in a consistent way, create and destroy it with a simple command, and share it with my co-workers eliminates so many issues.
Then being able to package my application in way that is repeatable and predictable is great. Then that same artifact is run across multiple environments spanning data centers around the world and on other developers’ machines via their compose files if needed.
We test the crap out of all of this as well. Docker helps there to generate ephemeral environments to run the bulk of the integration tests against every time we push to git.
Its just so damn powerful compared to what we had before…
Are you using bazel as your build system? If not, for almost all languages, your build system itself is probably not reproducible. Gcc uses random numbers. Lots of compilers add timestamps to builds (javac does).
For almost all non-Google use cases, repeatable is generally good enough.
It's reproducible enough for most purposes most of the time. When is the last time your Python web app serving cat pictures went down because of a subtle change in the Debian Bookworm container?
I was cobbling together build scripts for a mail system for Raspi/Armbian the last couple weeks. Very similar packaging stuff, but the number of little subtle differences in install/postinstall/prerm/postrm scripts took a generous level of spackling over to get just right.
Hell, once I get everything nailed down, I'm writing test frameworks for my bloody build scripts if you can believe it.
Same with static types. Sometimes it really helps to limit yourself, other times it's setting up unnecessary obstacles with no benefit. Sometimes you want Idris 2 or Rust, and sometimes you want Ruby or Clojure.
Those sound the same to me. Are you drawing a distinction between a repeatable/reproducible process vs result? Like if you run the same command to fetch a dependency, you wind up with different results if the dependency maintainer releases a new update and you aren’t pinning the dependency?
To use the apt analogy further up in the thread, `sudo apt install git` is repeatable in your dockerfile, but often not reproducible. Later on you will get a different build. Across say 500 packages and 1,000,000 containers (or say 1000 container images if you are deploying images) over even a week this becomes extremely... varied...
On smaller scales, this is perfectly fine. How often does Git actually release breaking changes of features that you actually need to use inside your Dockerfile? How often does Debian pull in such a version into their stable OS? And why didn't you just version-pin Git like Hadolint told you to do?
Exact reproducibility is nice for two scenarios: 1) academic research, and 2) very large-scale applications and deployments. For regular people writing boring small web apps, choosing a stable base image and pinning dependencies is good enough.
Consider also that your preferred programming language will also very likely not provide particularly reproducible package builds.
I've never looked seriously into it, but my feeling is that distros will delete old versions as newer ones are uploaded: When I run "apt-cache policy git" in my Ubuntu, I only see a couple versions available to install, often other packages show only a single one (so, the latest).
I know that Debian has Snapshot for older packages but you are still at the will of other people and people are fickle, and Nix should allow you to use specific versions to build your base images from to pin to.
However, much in the same way that if you actually take your build system seriously you'll store your application dependencies in a local proxy, you can run a mirror or proxy to hold these historical packages too.
Take a look at something like apt cacher, however it is a proxy cache so you can reproduce builds using the exact same package versions but if upstream delete old packages, and you want to roll back to one you haven't previously downloaded, then you are out of luck.
Yes. This can be a problem in scientific research involving numerical code or random number generation, where results can vary even due to small inconsequential-seeming changes, leading to your results not being reproducible scientifically because they're not reproducible computationally.
Agreed, and the fact that I can launch and develop my web app from seven years ago, no matter how much my environment has changed, is a lifesaver.
There's a lot of code I've written that's not deployable any more because the versions of the dependencies it was using don't exist on my OS any more, and are too old to install.
Snapshotting a VM is a pain in the ass. It takes forever and the tooling sucks. The concept of snapshotting is also completely wrong. Snapshots are tied to a VM with the specific VM being front and center, you know, the thing that is disposable with containers.
Also, from my misadventures with Ubuntu VMs I always ran into the problem of them breaking over time.
For me, it solved A LOT of cases where users said "your software install fails", "the software doesn't run", or similar claims. They tend to be complicated to troubleshoot: people try setting up new software in an already customized development environments, and all sorts of mistakes are done.
It was refreshing being able to say: this works perfectly from a base Ubuntu 20.04 container, launching this and that commands to install and run the software. Anything that diverges from there, you're on uncharted territory and/or is a problem in your system, not a defect in the software.
Containers aren't about you. They're about the person you're trying to hand things to.
I got stuck creating VMs for testers for a distributed workflow (several services, several tools), and keeping those VMs up to date and working was a PITA, but far less painful than dealing with them filing bug reports based on thinking they were doing X when they were doing Y.
I ended up creating a workflow that felt like layers, so when Docker came along I didn't need a salespitch. I could just do what I'd done with a script instead of a runbook. Where do I sign up?
Guess what? That's what I do. Still seems more like an anti-pattern, because the "easier and more thoughtless" I make the entire process, the more my testers don't pay attention to those details, or figure out how to make new tooling, making me the central point of load bearing failure.
Docker doesn't fix that. In fact, sometimes it makes it even harder for my testers to keep track of what is happening where. A shockingly high number aren't even able to visualize the logical boundaries between physically cohabiting, but logically partitioned systems.
It's too many damned layers.
And developers do awful things with docker containers to the point I end up having to rip apart multiple projects Dockerfiles to rebuild those, but with different values to work around the fact that oh
.. look at that, this one spins up a postgres database that wants to bind to ports we're already using. Looks like I get to go dig into the base image to change that port and rebuild...
Again, not saying it doesn't work. As someone who put in the work to become a generalist though, I put in an asinine amount of time working around other peoples things that quote "make things easier".
Maybe for the person writing it. Certainly not for someone learning from it or having to analyze it.
They did, many times. they're called flatpaks/snaps/self executing zip files.
Even windows reinvented the exe with msi's, and python has dists and wheels, and *nix/bsd has jails and vms have existed forever, and java has the jvm with jars and previously, applets.
Of course, none of those are actually as user friendly nor cross platform compatible as docker. Heck even docker isn't perfect and will leak abstractions once you start hitting syscalls (network, filesystem, etc), or if you're like 99.9...% of developers who don't host their own artifactory and cryptographically tag every step in their dockerfile
Show me an .exe that can self deploy an elk stack across windows, Darwin, and Linux with minimal system environment issues
> They did, many times. they're called flatpaks/snaps/self executing zip files.
Honestly, getting some desktop software in the form of one of those is wonderfully refreshing. Just take an AppImage file or whatever, make it executable and run it.
Sometimes you don't care about the benefits of shared libraries and want something to just run regardless of what's going on with your distro, for which these technologies are great. Ideally, in addition to the proper package manager approach for those that value different things at different times.
But for shipping things like WebApps, OCI containers and Docker/Podman is amazing.
It's more than just creating a distributable, though. It's also a dev fixed environment and a cross-compiler when paired with qemu, and a test suite all rolled into one. It's hugely powerful for how simple it is.
Being able to run self contained postgres with a single command is an easy ergonomic win. You don't necessarily need to be an engineer or write code for containers to be useful.
So my issue is that now I have PostgreSQL running... inside a container? So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up.
And like, that's just minimum: I generally care deeply about the filesystem that PostgreSQL is running on and I will want to ensure the transaction log is on a different disk than the data files... and I now have to configure all of this stuff multiple times.
At some point I am going to have to edit the configuration file for PostgreSQL... is it inside of the container? Did I have to manually map it from the outside of the container?
The way you access PostgreSQL locally--maybe if you find yourself ready to add a copy of pgbouncer, but also just to run the admin tools--is via unix domain sockets. Are those correctly mapped outside of the container, and where did I put them?
I honestly don't get it for something like PostgreSQL. I even use containers, but I can only see downsides for this particular use case. You know how easy it is to run PostgreSQL in some reasonable default confirmation? It is effectively 0 commands as you just install it and your distribution generally already has it running.
I literally don't have to do any of that if I just install PostgreSQL the normal way as the confirmation file is in the same place the documentation for PostgreSQL says it is in, the daemon is probably already running, and all of the tools know how to find the local domain socket. Why am I having to configure all of this stuff manually in some new "compose file"? Oh, right: because I am now using a container.
It's really hard for me to not make some incredibly dismissive "okay, boomer" comment here, but you are not giving me much to work with and this really reads as obstinate resistance to learning new things. Are containers different? Without a doubt. The benefits are, however, impossible to ignore.
The amount of work you are complaining about is, objectively, trivial. It's no harder than learning how to deploy on another distro or operating system, except in this case, your new knowledge is OS-agnostic.
I agree. All he is doing is making people uncomfortable for using containers. The only real major downside with containers is that the building process is quite expensive so you should get familiar with docker run -it --rm <id> bash or the same thing with docker exec. Oh and also that your container might not come with diagnostic tools.
> So my issue is that now I have PostgreSQL running... inside a container? So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up.
And how is that different from running directly on the OS?
the OS will have an understood abstraction behind the filesystem structure, whereas container-systems often create an entirely new abstraction that they view as the best way.
this makes sense when you're trying to be deployable universally, but it increases the amount of onboarding that someone needs to receive before they're proficient with the container system; onboarding they may have not actually needed to get the software working and well understood, simply 'docker overhead'.
from personal experience : i'm a long time old linux person, the insistence on going 'all in' on Docker (or whatever) just to run a python script that has two or three common shared dependencies gains me nothing but the hassle of now having to maintain and understand a container system.
if you're shipping truly fragile software that is dependent on version 1.29382828 rather than version 1.29382827 then I understand the benefits gained, but just to containerize something very simple in order to follow industry trends is obnoxious, increasingly common, and seemingly has soured a lot of people on a good idea.
p.s. : I can also understand the idea of containerizing very simple things as parts of a larger mechanism; I just don't get it with the promise that it'll reduce end-user complexity, it isn't that simple.
I like your point of view. I started working with software more than 20 years ago and did my fair share of VMs, bare metal, ESXi 3.5, and so on.
However, we live in the world where the choice we have for new hires is: a) teach them all of those OS fundamentals, b) give them Docker.
This doesn’t mean we shouldn’t strive to teach said new developers all those underlying concepts, but when we talk about training juniors and have them contribute relatively quickly, it’s a much smaller surface area to bite through.
Docker doesn't solve having to know OS fundamentals as both the inside and the outside of the container have an OS, and now you additionally have to know how to communicate between the former and the latter.
> This doesn’t mean we shouldn’t strive to teach said new developers all those underlying concepts
That's what I said.
Furthermore, let's be pragmatic. The operations team needs to know, yes. We want new hires to contribute and feel productive. They'll naturally learn while working on software. A junior person can contribute almost immediately with a limited surface knowledge.
Less friction: no need to understand systemd/upstart/rc what have you, /etc, /opt, /usr, mount, umount, differences between various distros, build-essentials, Development Tools, apt, dpkg, rpm, dnf, ssh keys, ...
The right time will come but give them an easy way in. Containers provide exactly that.
All they need to know: it's somewhat isolated so under normal circumstances whatever you do in the container doesn't affect the host, how to expose ports, basics of getting your dependencies in, volumes, basic understanding of container networks - for things to talk to each other they need to be in the same network. Enough to start.
Database in container solves a few problems for me:
1. I can start it and stop it automatically with Compose. That's a big increase in ergonomics.
2. I don't have to write my own scripts to set up and tear down the database in the dev environment. The Postgres image + Compose does all that for me.
3. Contributors who don't know much about Unix or database admin stuff (data analysts learning Python & data scientists contributing to the code) don't have to install or mess with anything. It works on everyone's machine the same way.
Volume and port mapping are basically trivial concepts anyway. There's been zero downside for me in using it, even though I personally have the skills and knowledge to not "need" it. Why would I go without it? It saves me time and effort that could be significantly better used elsewhere.
> So my issue is that now I have PostgreSQL running... inside a container? So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up.
You bind a mount to where the data is onto the external system. That way only the important data is exposed. It's very clean, although it requires you understand docker a bit to know to do this. But for things like postgres and similar sprawling software that basically assume they're a cornerstone of your entire application and spread out as though this was the case, it's actually a very neat way of using them a bit without having them take over your machine.
This is something that can be useful for a developer too. Like my search engine software assumes it owns the hardware it runs on. It assumes you've set up specific hard drives for the index and the crawl data. But sometimes you just wanna fiddle with it in development, and then it can live in a pretend world inside docker where it owns the entire "machine", and in reality it's actually just bound to some dir ~/junk/search-data.
Although I guess an important point is that using docker requires you to understand the system better than not using docker, since you both need to understand the software you run inside the container, as well as docker. It's sometimes used as though you don't need to understand what the container does. This is a footgun that would make C++ blush.
All my automated integration tests use a real Postgres database in a known state. I don’t mock or stub or use an in-memory database pretending to be Postgres. It’s nice.
Yup. And you can make sure to run your tests in the exact same Postgres version you are running in production. No need to go help your teammate whose machine is for some reason behaving different, due to some weird configuration file somewhere that they forgot they changed it 6 months ago.
> So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up.
As a user, this is why I love Docker. The configuration is explicit and contained, and it's well documented which directories and ports are in play.
I don't need to remember to tweak that one config file in /etc/ which I can never remember where is. Either it's a flag or it's a file in a directory I map explicitly. And where does _this_ program store its data? Don't need to remember, data dir is mapped explicitly.
That said I haven't tried to use PostgreSQL myself directly, just tools that uses it like Gitea.
Yes this is the hidden beauty of docker... "I don't need to remember". Someone can reverse engineer exactly how I have a goofy custom postgres setup instantly, just by looking at my original Dockerfile I committed a year ago. No hunting around on the OS!
As someone else said, docker isn't about you, it's about everyone else. The extra complexity up front is so worth it for the rest of the team.
How you inform your backup system where to get backups ? How you set pg_hba and other configs? Few other "how?" and you're doing what you'd be doing on VM anyway
> How you inform your backup system where to get backups ? How you set pg_hba and other configs?
Simple answer: with configuration. In Docker Compose or Kubernetes. Less often in Mesos. Maybe I want to run it on a fleet of VMs, maybe on bare metal.
> and you're doing what you'd be doing on VM anyway
Right. But with containers I can have different apps using different dependency versions. Some things use nginx, some use some other web server. Some things run with node 12, some with node 16, some things use MySQL, some other things need Postgres. This is so easy with containers.
> Being able to run self contained postgres with a single command is an easy ergonomic win. You don't necessarily need to be an engineer or write code for containers to be useful.
Postgres is a poor example; it's far far easier to do `apt-get install postgresql` than to run it from a container.
The latter needs a container set up with correct ports, plus it needs to be pointed at the correct data directory before it is as usable as the former.
Is this some kind of joke? Those things are absolutely trivial and once you wrote them in your bash script or compose file you can forget about these things.
It solves the "we can't possibly package the software for all versions of libc out there" and the "there are all those programs that must run at the same time, it's hard to correctly install them all" problems. People push it to solve the "this program needs executable A version 1, that program needs executable A version 2, both programs must run at the same time" one, that should never be a big issue anyway.
But then, we have none of those problems at work and people still believe it's a panacea. I don't know what people believe they gain by it.
Dependency management is one thing Docker handles, and I don't really think it does a great job of that. To me containers solve the challenge of getting a development environment that closely matches production. It standardizes how deployments are performed, how testing environments are set up. Before this we had complex, immutable ansible scripts, READMEs, install scripts that had to be babysat, snowflake servers that no-one knew how they were configured. It's also an extremely widely used technology, so if you use it in your stack you can expect many people you hire to know how to use it from day one versus everyone learning a bespoke set of custom scripts.
It also solves the "we have limited developer-hours available and just want to deploy some stuff" problem. Docker is amazing for teams with limited resources. If you can deploy one image, you can deploy any image.
I don't know what you are talking about. My production setup is dumber than you would think. A bunch of bash scripts in a git repository. You just do git pull and then run a single bash script. ./app build builds the entire thing. If you want to run it then do ./app start.
Yes you do need to configure some environment variables like in almost any other form of deployment.
Messing around with tar files or flat binaries doesn't work for me. I tried that. I build a python app into an executable with Nuitka but it generates a folder anyway. Ok throw it in a tar file. Nope, doesn't run because the glibc version is too new on the developer machine. Nice try. I have to build it on the target machine. Amazing.
I feel the same about your post. I have no idea how deploying a flat binary or Python tarball is better than a container image. The entire system is effectively a dependency. Docker (or equivalent) lets me ship the entire system. Moreover I can take quite literally an image of that one system and deploy it to any number of machines identically, without any rebuilding. And I can host it interchangeably on any number of cloud providers with only small changes in configuration. The benefits go on and on. I would only ever bother with a "plain" deployment in the future if I had complete control over my deployment environment (+ the in-house expertise and time budget to manage it) and the performance overhead of the container was unacceptable.
This was already trivially solvable by establishing a new sysroot, which in all fairness is 90% of the core benefit of containers in my view. The rest of the container features are bloat for most of my use cases.
You answered your own question. When you join a large project and there's a perfect dev environment available in 3 minutes with `make init`, it makes sense. If you're the solo dev, it doesn't make sense to spend 5 hours figuring out the docker build system.
Every time I see this, I peek at the init target and it's always doing something weird like writing to my SSH config or something. I have learned to never trust installers written by colleagues.
I have yet to be handed a containerized dev env that "just works".
One company came close, they had someone dedicated to maintaining the dev tooling including the containers. It still didn't "just work" but they handled the troubleshooting, and the fix made it into the repo for the container so it wasn't just an unwritten adhoc fix I needed to remember. Close enough.
I totally get the concept, I wish it worked so seamlessly that the promise was realized. But since I usually work with tooling I am familiar with, there is no point for me personally. I can stand up a local env faster than I can troubleshoot a broken container. Since I am not interested in the infrastructure and just want to get to work, that is what I do.
Well there is one unarguable one, although with consequences.
Instead of keeping some old machine with old dependencies because project is on life support and doing update to latest version of language/tools is unviable, you can just keep a docker container for it while machine it is running on is up to date.
It doesn't solve the problem of application, but it is no longer ops problem that app is old.
You can also do that partially, like keep old PHP version running with just fcgi socket exposd in a container, while rest of the app lives on "normal" VM (or other container, if that's what you want).
That's what got me on the Docker kick. Old unsupported programs that are tightly coupled to a kernel version, a Windows thing, some combination of Java / JDK shenanigans, an old SoundBlaster DLL, "The Python Carnival", or, really, what-have-you. Honestly, every big ERP demo ever should be done via container, because - and I swear I am not making this up - I have sat in months-long meetings with dozens and dozens of staff, both mine and theirs, trying to get Big ERP X up and running on-prem under whatever weird-ass policies the customer has on all their environments.
Granted, a lot of THAT benefits from the fact that the container actually has to be a documented flavor of good solid *NIX, unlike the "Windows Image X" which is usually just a big "whatever" of old iron, management fad spoor, and seven different kinds of antivirus software. Hell, last time I did this, IT couldn't even find me any kind of description of what the "official" windows image actually was. So no fault of Windows there, it's just that, as the big bus everyone rides in, it gets all the goop from everyone wrenching on it.
I guess we live in entirely different worlds, because, for me, containers have been a godsend. Standing up the average application is a matter of reading a compose file and modifying it according to instructions, or using it to construct a proper manifest for something like k8s for full production.
For toy/evaluation use, it's hard to beat "tweak compose file, docker-compose up". You now have a golden source of truth for how the entire application and all of its moving parts were set up.
Going back to deploying applications on bare metal feels positively medieval by comparison. From the admin side, configuration drift is effectively not a thing anymore. That's huge.
It feels like admins just offloaded the job of deploying apps back to developers. And as a developer I woudn’t even mind, if not for the f-ing slowdown of, well, the pace of development. All external teams that I worked with and who were using containers for development were like “oh you need that trivial endpoint or a bugfix? No problem, we’ll do it in few hours and it will autodeploy tomorrow”. And I didn’t ask why, because I know. As if we weren’t snail-paced money burners already.
Provided all that, I don’t get why “devops” is even a high-paying job at most places except few really web scale ones. They are basically former sysadmins who reject anything except plugging colored squares into square sockets.
I'm on the other end of the spectrum. I don't want to deal with written instructions. A shell program will do but a container is the best. Give me an environment with all dependencies, software, and some decent configuration method. A config file over a volume will do, env vars are better. Dockerfile tells me everything I need to know about the environment the app requires.
I can run that container in CI, in tests (with stuff like testcontainers), on my laptop, in production, on Kubernetes, in Mesos, Lambda, whatever.
Instrumentation usually ends up in Prometheus and Jaeger.
To me, containers are like OOP. Nice ideas on paper but a pain the ass in practice. but again, like OOP, a lot of people do great things with them but I don't think they can't do the same things with VMs.
My theory is eventually things will come full circle and there will entire app ecosystems running in a unikernel executable which is running on physical hardware and the hardware itself is segmented. I can imagine companies like netflix and cloudflare having racks of servers with 1TB ram with no disk, booting a unikernel from the network for maximum throughput and latency and then everyone in tech follows along.
No, i haven't. (not OP but) The last decade or more 90% of people i collaborated with were on the same operating system. Which moves the docker development setup solidly into "nice to have" category for me, especially when working on small teams <10.
It is the best solution. In my particular situation, at last.
Another solution would find a senior developer who can manage his dependencies properly, but I can't afford him. However, I can pay a junior ML developer wage and just throw what he develops in a docker image instead of wasting time and money on doing things the hard way for a prototype.
As engineers, we're supposed to be rational people, but unfortunately, we tend to forget that time and money are often happen to be the most important constraints we have to work around.
I have an app that needs a media server, a webrtc STUN/TURN server, a PostgreSQL database, a JVM based web server, a worker that may or may not run on the same host and a way to run liquibase for migrations and certbot for letsencrypt certificates.
The whole app can be started with a single command and it works on most Linux distributions. I can't imagine wasting the time of users or newbie developers demanding that they install all these things separately and with no easy way to clean it up if they want to undo everything.
Instead of composing libraries into an executable you can use containers to compose services into an application [0]. As noted in the sibling comment, this still can have a high utility for just a single workstation/service. Also as you noted, scaling to a datacenter is already partly automated. The entire environment is captured in code and reproducible/portable. But you need to be familiar with the tools and ecosystem to make use of them.
[0] We don’t have concise terms for the types of distributed, complex systems which have become standard.
> Instead of composing libraries into an executable you can use containers to compose services into an application
This is the most concise and exact definition I've heard lately.
My only problem with the consequences of this approach is that the amount of overhead for performing the same operations is astonishing.
Messages pass on a network instead of stsying in RAM, CPU context switches every other ms to handle IO for what could have been a simple function call, etc.
It's amazing when it's needed, but seeing this approach becoming the standard for a big part of the industry almost turns it into an environmental problem. How much energy is wasted juggling bits around?
It doesn't have to be a library replacement. It could be that your app uses a DB server and a message queue for async tasks. Containers allow you to easily run those servers locally for dev if you don't care about performance and maybe in prod you'll have a dedicated server for them, or use some DB as a service offering.
Yeah, agreed: in that case spinning a container is evidently an operational advantage.
However, I was specifically answering to the different case highlighted by the parent comment: how a potentially cohesive application is cut along some of its internal APIs, and some functionality is allocated to different processes living in different containers.
It is an extreme point in the continuum "single thread" -> "multi thread" -> "multiple processes" -> "fully distributed". In that continuum scalability increases, while efficiency progressively decreases.
Cornering oneself to a specific point in the design space is problematic, and for some cases has direct implications on how many resources are wasted.
A very didactic experience is, for example, running a simple local application under a microarchitecture profiler, such as Intel Vtune. It is not uncommon seeing that even straightforward C/C++ programs use a core resources less than 10%.
What I am reflecting about is that the choice of fragmenting that program among tens of systems (maybe in a scripting langiage) should be conscious, and done after encountering performance or scalability bottlenecks.
How much of the resulting total system workload would be useful work?
The quantity of potentially wasted resources is astonishing if you think about it.
Not at all. They make cross compiling fairly trivial, and give teams a way of managing cross compiled dependencies. This was vital a few years ago for things like torch and even opencv, where deploying on anything other than x86 meant you were probably on your own, and you almost certainly wanted control over the compiler flags.
That's just one example, though. It also lets you standardize the dev environment (work directly out of the build container directly, or use your host OS and run unit tests in a local build), and it allows for easy, standardized testing within the same environment.
That's basically what Smalltalk was back in the 20th Century.
The OS, the development environment and the application (both code and live objects) where one and the same thing. To ship an "app" you would export the image and the user would load it into their Smalltalk VM.
Owner hired an extremely “senior” developer. Was told to let him do his thing.
After he spent 3 months building a web app, I asked him how he wanted to deploy it.
Perfectly straight face he said we would take his developer machine to a local data center and plug it in. We could then buy him a new developer machine.
It went downhill from there.
I ended up writing the application from scratch and deploying it that same evening.
And dependencies are about the way that we went from a blank slate to a working system.
If you can't retrace that path, you can't debug. If you don't have tools to track the path, you will make mistakes. At best, you can exactly replicate the system that you need to fix -- and fixing is changing.
Not in general, no. Docker image layers are just snapshots of the filesystem changes that result from build commands. But there is nothing that guarantees that the effects of those commands are reproducible.
For example, it's incredibly common for Dockerfiles to contain commands like:
RUN apt-get update && apt-get install -y foobar
which means when you build the container, you get whatever version of foobar happens to currently be in the repository that your base image points to. So you can easily end up in a situation where rebuilding an image gives you different behavior, even though nothing in your Dockerfile or build context changed.
Even the Docker support team is confused by this. Most of why I stopped engaging is that they would reject a feature because it lead to dockerfiles they said were 'not reproducible', but were just as reproducible as many of the core features.
Which is to say, not in the slightest.
Every time you do something in a Dockerfile that involves pulling something from the network, you don't have reproducibility. Unless you pull something using a cryptographic hash (which I've never actually seen anyone do), the network can give you a different answer every time. The answer might not even be idempotent (calling it once changes the result of calling it again).
My argument was the same as yours. Virtually every dockerfile has an 'update' call as the second or third layer, which means none of the repeatable layers afterward (like useradd or mkdir) are actually repeatable. You're building a castle on the sand.
Docker images are reproducible. Docker builds are not. And that's okay, as long as everyone accepts that to be true. And if Docker contributors can't accept that then we are all truly fucked, until something else comes along that retreads the ideas.
Is there really a significant number of folks who are confused by the idea that networks can serve different files under the same name at different times?
> which means when you build the container, you get whatever version of foobar happens to currently be in the repository that your base image points to.
Well no, the layer won't be rebuilt if you already have it from yesterday or last week and that + previous lines in the Dockerfile didn't change. So yours would still be using the old version, but someone new to the project would get the most up-to-date version. Have to remember to skip the cache to get the most recent version.
2) even if you don't apt-get a specific version, if two developers build and then run the container (within some time frame), they will get the same version. It would be rare to get a different one on the same day.
And 3) You can still apt-get a specific version
And 4) That's still far and a way greater chance two developers will have the same version when using docker in this way than if they're both using two crazy different versions of Ubuntu locally on their desktop.
Also most developers that are working on the same codebase will push to their branch and circleCI will make it so there's only one docker image, that's tested and moves along to prod eventually.
You can apt-get a specific version, but that is not a 100% guarantee that the contents of the package will be identical to what was there yesterday. It is absolutely possible to delete a package from repo and replace it with another one with same version.
It is nice when your package manager reliably alerts you to changes like that, though! Binary package managers can't, really— although this feature could be added— but sourced-based ones often do. Nix and Guix enforce this categorically, including for upstream source tarballs for source builds. Even Homebrew includes hashes for its binaries, and provides options to enforce them (refusing to install packages whose recipes don't include hashes).
If I tell you that there's a remote code execution in libfoobar-1.03 through 1.15, how long does it take you to verify where libfoobar is installed, and what versions are in use? Remember, nobody ships an image layer of libfoobar, it's a common component, though not as common as openssl.
Is there one command, or one script, which can do that? You need that, basically daily.
Is there one command to rebuild with the new libfoobar-1.17, and at most one more to test for basic functionality? You need that, too.
I mean you're not gonna like the answer but in real life when herding cats the answer is setting up an image scanner and renovate, and calling it a day.
It's not like OS images are any better in this respect. I have bit my teeth long enough on software that depends on the base OS and not being able to infra upgrades. Bifurcating responsibility into app/platform is a breath of fresh air by comparison.
....I can tell you which ones are used by the current linker on the system in what order.
ldconfig -p | grep libfoobar
If you go and spread the linkers all over hither and yon and make it nigh impossible to get results out of them, or don't bother to keep track of where you put things... Welp. Can't help ya there. Shoulda been tracking that all along.
Oh, excuse me... That'll only work for dynamically built things in the abscence of statically linked stuff or clever LD_PRELOAD shenanigans.
Hope ya don't do that.
Fact is. You're really never going to get away from keeping track of all the moving pieces.
You're just shuffling complexity and layers of abstraction around.
Especially for linux hosts this misses that containers will still run on the same kernel (version) as your host OS, inheriting a lot of settings and limitations from there as well (for example net.core.somaxconn). The most obvious ones are sysctl settings, many of which must be set system-wide. Common ones which can drastically change characteristics of database software are vm.nr_hugepages (postgres - why are we OOM or latency is spotty?) and vm.overcommit_memory (redis - why does background saving not work?).
I also looked at this topic, see [1]. Some points are similar to the article posted by OP. My findings were:
- Docker Desktop and Docker engine (CE) behave differently, e.g. bind mounts, or file system ownerships.
- CPU/Platform differences (ARM vs. AMD64): many devs don't realize they use ARM on their mac, thus ARM images are used by default, and tools you run in it (or want to install) may be have differently, or may be missing entirely
- Incompatible Linux kernel APIs (when containerized binaries make syscalls not supported by the the host's kernel, for whatever reason)
- Using the same version tags, expecting the same result (--> insanity, as you know it :D)
- Different engines (e.g. Docker Desktop vs. colima) change the execution behavior (RUNNING containers)
- Different build engines (e.g. kaniko vs. BuildKit vs. buildah) change the BUILD behavior
For anyone who is interested: more details in [1].
I think a lot of this comes down to a broader difference between Mac/Windows Docker Desktop and "plain" Docker on Linux. The former is actually backed by a VM, so a lot of the painless simplicity comes from having a true virtual machine involved, rather than just a layer of namespacing.
A lot of people are in here complaining about how Docker is not reproducible enough. But reproducibility of image builds is a matter of diminishing returns, and there are other problems to worry about, like the ones you are pointing out.
Speaking of which, it's probably good to get in the habit of installing some Linux OS in a VM and trying to run your container images inside that (with "plain" Docker, no inner VM), before pushing it to your cloud host and waiting for it to fail there.
I feel like most of the problems raised in this blog post can be solved with a proper reproducible build system - for example NixOs (or guix if you will) derivations.
It's true that Dockerfiles are not reproducible, but at least they're human friendly and easy to deploy. If you need something more deterministic, I really encourage you to try NixOs. It's (almost) 100% reproducible and works for any real-world use-case that I've ever had. Dockerfiles have a different use case - they are a formal version of a installation instuction that you would give to a new hire in the older times.
Reproducibility and build determinism are slightly different. Nix is not really about bit-for-bit reproducibility, that is a side-goal, and one that is being worked towards. It is far more pragmatic than that. If you use `--rebuild` like `nix build nixpkgs#hello --rebuild` it *will* warn you if the output is not deterministic FWIW.
Instead, Nix ensures the inputs are fetched deterministically via Fixed Output Derivations (FODs) that guarantee the output of building/compiling against that input can only vary so much, and in ways often insignificant to code flow in the output program, (timestamps/byte layout, etc). In the Nix thesis this is referred to as "the extensional model" in Section 5 (Page 87), otherwise known as input-addressing rather than the intensional model which is in reference to content-addressing.
Of course we'd all like build outputs to be deterministic, but that's not very pragmatic, or achievable. Many compilers and toolchains don't produce the same results twice, but this often does not effect the behavior of the program. There's some tracking of that goal on r13y.com, and it's probably a goal that will always be limited to a small subset of deterministic outputs, as it's unlikely that all toolchains used in the dependency graph will exhibit deterministic behavior.
Sometimes things that do effect code flow leak in, but are squashed by the Nix stdenv where possible, and also by the culture of fixing/patching those problems in Nixpkgs. The result is a pragmatic set of 100,000+ packages that work properly and can be reproduced anywhere.
There are a number of incorrect statements in this post.
1) One should neither be using the "latest" nor just the "version" tag as the version can still vary depending on when it is pulled.
Instead, one should use a combination of version + hash, say alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 for reproducibility reasons. This provides for human readable versions as well as the specific hash.
2) Next, afaik, Compose has removed the need for version tags. All of the compose.yml files that I now use do not specify versions.
"version + hash" is ugly though. I trust the publisher of my base image to keep compatibility even if they update their image and trust my test suites to detect any issues, so I just use version without the hash nowadays.
Using hash doesn't protect you from supply chain attack either. If the publisher is compromise, any updates could potentially be malicious. The alternative is to never update at all, which can be even worse.
The article seems to miss the point that we are able talk about these differences in terms of containers as they are abstracted into a purely software system that are reproducible. Versus a customized hardware+software system with no means to reproduce it. Containers were a huge step forward because they raised so much more of the software stack into a simple, repeatable, defined systems than were previously much harder to obtain.
Controversial opinion but I still deeply believe that any building tools offering a “latest” option in its build configuration file and not as an option to update said file is doing something deeply wrong and just poorly designed. Everything pooled from outside should be using a checksum.
People want and need the ability to pin their environment. If you want to avoid silly surprises when going to production or through your CI system, you need everything to be identical: same versions, same configuration. Tooling should help you do that not introduce new way to shoot you in the foot.
I got -3 points on a comment on a different post where I stated what I didn't like about docker. I was bullied by docker fan boys. Glad to see many here agree with me. It sucks to be surrounded by a mob.
took a peek at your comment, and i do agree with you. docker can add unnecessary bloat to projects.
docker shouldn't be used for everything. if you provide a docker version of something, it's a smart idea to also publish (for example) an appimage or deb file for people who can't or don't want to use docker.
like for example, at my work we don't want to use docker because we will have to get approval from corporate for every little script we want to run in a container because corporate identifies a container as a separate application so it must go through the approval process (which takes 4-8 weeks).
In short, make sure to pin the exact version of dependencies, set the user and keep in mind OS-specific restrictions.
Containers are still a great improvement over running on dev machines directly: besides cleanly separating environment per project, an unsung benefit is being able to get rid of the environment when you're done working on a project instead of accumulating bloat over time.
While the comments here talk lot about pinning and locking everything down, I'll offer alternative viewpoint: test your application against wider range of environments and versions. Docker is great for that too, you can easily spin up your application in Debian oldstable or latest Fedora and see how it behaves. Your software will become less fragile as a result and better in the long term.
Somehow the old adage of portable software being good software seems to have been lost to the ages now that we as developers have attained such precise control over the environment our software runs in.
That's the beauty! The build process is exactly the same for in and out of container stuff. In both cases result is either binary or binary + static files (nowadays, as it is mostly ops stuff, binary gets embedded static files).
Now you are back to the “it works on my machine” issues. Seen a couple of times where a precompiled binary works okay on one OS but the behavior changes when ran on another OS.
In my case, the bottom bid contracting firm that delivered the code had special logic for windows but otherwise wouldn’t happen on unix based machines.
> Now you are back to the “it works on my machine” issues. Seen a couple of times where a precompiled binary works okay on one OS but the behavior changes when ran on another OS.
Sure, but containers don't protect you from that - you're still exposed to differences in the host kernel. For that kind of issue you'd want a full VM rather than a container.
By that logic, we'll need to ship bare metal, to avoid differences in processors.
You've gone from being exposed to OS and kernel differences to being exposed only to kernel differences. For most apps, that's acceptable. Others will need to ship VMs. Some will even need to ship bare metal.
> What "OS differences" are you exposed to if you're shipping a static binary?
I dunno, probably stuff like ulimits and selinux rules that you don't think about until it burns you. Not to mention whatever idiosyncratic configuration your customer might perform.
> [It's] absolutely dogma, enforced by kubernetes and friends.
The dogmatic people are wrong to be dogmatic, but in the context of this particular conversation, I would like to point out - meaning no disrespect - that it is you who is advocating the more maximalist and less pragmatic view.
> The dogmatic people are wrong to be dogmatic, but in the context of this particular conversation, I would like to point out - meaning no disrespect - that it is you who is advocating the more maximalist and less pragmatic view.
How so? The thread started with talking about having a statically linked binary (not a specific one with special requirements, but a generic one), and described putting it in a container with a config to expose some ports from it. That seems very much like a dogmatic everything-must-be-a-container position. I'm not arguing for never using containers, I'm arguing for using them where they make sense and not using them where they don't.
From my perspective, what they said (across several comments) was, "I like this workflow, it has some advantages, some of my customers want containers, and in our case it doesn't add much more complexity," which I took to be a mix between a pragmatic and aesthetic position, and in the comment I responded to you said, "it didn't solve all of the problems we can identify, so why bother?" I took that to be a maximalist position.
Sometimes customer wants containers; most of our (ops/orchestration) runs outside of container but containerising it (for same reason, one static blob) is easy
But the code I nicked the example from was actually our internal k8s/docker deployment testing/debug app, so, well, it's in containers by design.
Because I get other levels of isolation such as at the network, filesystem, and syscalls and I can run many instances of the same binary on a host much like a VM, but vastly lighter-weight.
No, containers are not a problem and they don't add any additional problems.
The same reason you'd use one to begin with, primarily isolation of processes. That doesn't go away just because the binary is statically compiled. But you don't have to, of course, plenty of people don't.
I once worked with a scientist who could only replicate their results on a single computer which had been extensively modified over time. Like, hot patching the C library and stuff. They were absolutely stuck on this machine, and never once considered the possibility that their results were due to the local modifications.
In retrospect this is not completely surprising given the incentive system in science.
How do you specify the need for reproducible software installations in an incentive system?
Also, what kind of scientist was this? I'm a physicist. I'm deeply concerned about the reproducibility of my results. I periodically try to rebuild my software systems on a clean computer to make sure it's both possible and well documented.
A biophysicist- a person who built 3d structural models. Most scientists from tyhe era I'm describing use computers as tools and see periodically moving from system to system as an interruption/distraction from them doing science.
Were they paying for their own computer? This is a problem I've seen in academic science. Grants and departments don't want to pay for computers, so there's a tacit expectation that you'll buy one yourself. My spouse had to do this. In turn there's no consistent IT management of these computers.
My wife's attitude is typical: If they want to touch my computer, they can buy one that they own.
The group lead ("principal investigator") raised funding (from NIH and NSF) through grants to pay for computers. That's how it's normally done. Making grad students and postdocs buy their own server (not laptop) is not an act of honor. It's also a huge security issue (as you say, no consistent IT management).
In this case the scientist was a postdoc so they partly used the group lead's funding and partly had their own funding. That money was for science, not buying computers. It paid for NMR machine (a $10M device), solvents, reagents, going to conferences, grad students, color figures in papers... but my professor was fairly well-funded so we had a full-time "professional" sysadmin (not a grad student) and everybody had a $15K SGI Indy on their desk (which was abysmally slow, because it was the lowest of the low end capable of running OpenGL).
One thing I've learned when deploying: Pin absolutely everything. From the exact Docker base image version, to the package installer (e.g. Poetry) version, to all dependencies.
For everyone that went straight into the comments, this article is meant to guide people through _preventing_ situations like "it works in my container."
Although this was always a problem until the mac M1 chips it didn’t matter much. Now it happens almost every week. I would prefer to have at least the same architecture between my local and prod environments.
Yes change the entire production environment to accommodate the development environment… it’s a lot cheaper to just switch developer laptops at next refresh
There are definitely footguns with docker. I still feel like if I build the image I have a lot more control over it than when I'm trying to document build scripts, create the right package files or lock files, etc. I don't buy this argument and stopped reading when it was obvious that these issues could be solved by using a clearly specific build tag for the base image, etc.
Imo, the strength of containers is portability, not reproducible builds. Even portability isn't perfect, since images are sensitive to CPU architectures and containers can still actually rely on host system configurations (mounting part of fs, privileged mode).
It's a nice list of problems and tricks to help, and it is a fun take on issues.
For those focusing on the title, a nice way of seeing a difference is less "my container does X" and more "when I run the production container...". Sure your local build might do one thing but you can get what's built in production and try running it. Maybe you can quickly swap between multiple combinations of service versions to replicate a bug that happened during deployments.
The difference is that it doesn't work on MY container, it works on A container. And I can run the same container byte by byte in my machine, yours, or a production cluster. Most/all of the issues highlighted in this article can be mitigated by making your system more self-contained (less dependencies), or using some layers of abstraction on top (e.g. docker compose).
Despite these hot takes containers are a massive step in simplifying system deployments.
This is something I've been trying to fix with PingQuick[0]. I got tired of spinning up containers, dealing with backends and setups only for it to be broken somewhere along the line, which then turns into me 4 hours deep in googling docker commands. I just want my code somewhere that someone else can ping it with no setup - that's it.
the memory in my s/w development history was a long-lived build host which had masses of install-loop dependencies (things which need earlier versions of themselves to exist, to bootstrap installing the one you want) as well as dependencies hidden inside things. Basically, "it built on <x>" became a stock joke because it was demonstrably true: anything would build on a host with almost all dependencies resolved, if badly.
one thing containers do is provide an audit/reproducible basis to say WHY something builds. it may still be an implicit dependency: something in a specific version of an underlying substrate like the OS or a boot time act which works on that container and not on another. You may not "see" it at first but its a lower bar to understanding it, if the revision control behind the build chain is good enough.
a build chain of :latest is probably not good. reproducible builds drive against that.
I don't build things for a living any more. Probably this is all well known and argued better by others.
Casey Muratory was spot on about using containers was not a solution to any problem, but just more of the same complexity increase. Full presentation at: The Only unbreakable Law
He says the same about virtual machines, package managers, engines and even libraries.
That presentation is a borderline-psychotic rant unless viewed through a very narrow lens where the only thing that matters is maximum-performance systems programming.
He's disregarding every productivity improvement in pursuit of maximum performance. He makes that very clear, and in that context it makes sense, but most people won't have the same priorities because we don't live in a world where we can afford to write our own TCP/IP stack to maximize the throughput on our shitty REST APIs that C/R/U/D our customers' TODO items in our database.
"Easier to use" comes at a cost. I have to wonder whether there's a lower limit to how "easy" reproducibility can be. It is a matter of physics at some level, and the ease of reproducing software can often depend upon what its dependencies are and how easy *they* are to reproduce.
I think that using less software, and better software is part of the solution to making reproducibility easy. But FWIW, I do think that Nix lowers the barrier to making reproducible software.
I'll take it works in my container a billion times before it works in my machine. Then I'll know its just configuration. Yes configuration can be hard, but at the end of the day, docker forces all injection points to be super explicit.
I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways.
But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.