
How did we end up with containers? - mmphosis
https://www.tedinski.com/2018/04/03/why-containers.html
======
wrs
This is mostly a good post, but he doesn’t mention one huge security
_disadvantage_ of containers: When application artifacts all include their own
versions of dependencies locked at build time, it becomes very hard to update
a component that has a security fix. It’s somewhat tractable if you own all
the images — you “just” have to rebuild every application with the vulnerable
component. But once you start accepting prebuilt images from elsewhere, you
are at the mercy of the external source unless you want to fork the
Dockerfiles.

It’s the same lesson that a lot of people learned about static linking, with
the zlib vulnerabilities years ago.

[https://www.linux.com/news/linux-kernel-netscape-affected-
ma...](https://www.linux.com/news/linux-kernel-netscape-affected-major-zlib-
security-vulnerability)

~~~
convolvatron
this is really about static vs dynamic linkage, regardless of the mechanism
you use to effect run or build time binding of dependencies.

there are as always tradeoffs, but a few more arguments against dynamic:

by opening up a path (automatic updates) to a third party, you're providing a
wide vector for those updates to cause new* problems (correctness, security)
for your deployment, as well as fix old ones

transitive api compatibility across these upgrades is more of a statement of
faith and effort of will than any real guarentee. in reality the
compatibilities between the various components are piecewise, and if the graph
gets big enough it can be really hard to find a compatible set of libraries.

runtime binary integration is always strictly worse than build time source
integration, since there is tooling in the latter promoting consistency, and
basically none in the former

since you'd really like to be able to deploy changes in your own* software
with very little effort - dependency updates can ride on the same train for
free.

presumably you have to have some intervention to restart your service against
the new dependencies regardless. at that point its a question of the relative
complexity of a dynamic restart of an existing service vs a build and deploy.
in the limit these are both buttons.

from that perspective, dynamic updates are a helpful crutch for people who
don't have a good handle on their own build, test and deploy. thats pretty far
from being an arguable fundamental good.

~~~
pxc
> this is really about static vs dynamic linkage, regardless of the mechanism
> you use to effect run or build time binding of dependencies

Are you familiar with the functional package management approach of tools like
Nix ([https://nixos.org/nix/](https://nixos.org/nix/)) and Guix
([https://www.gnu.org/software/guix/](https://www.gnu.org/software/guix/))?

They use dynamic linking but store all packages in a content-addressed format,
outside the usual FHS. Packages are free to rely on separate versions of their
dependencies as needed. Packages are defined in a 'source-based' format, but
because of the content-addressable storage strategy, both Nix and Guix can
offer transparent binary caching.

Guix folks in particular are working on verifying the integrity of binary
caches, so you don't have to blindly trust them, and they've developed a
system they call 'grafting' which can safely and quickly replace vulnerable
packages without recompiling.

Both Nix and Guix have some container infrastructure as well. I'm not as
familiar with the Guix side, but Nix has tooling for deterministically
generating Docker images and systemd-nspawn containers, so you can leverage
these packaging tools at the same time as existing container deployment
infrastructure.

I like your analysis of 'build time' vs 'runtime' integration of dependencies,
and I think you might enjoy playing with/critiquing Nix and Guix, because they
address these problems in a strange and powerful way. They use dynamic linking
for deduplication and small upgrades, but try very hard to integrate
everything they can at build time. I'd be interested to see where you think
their approach is strong and weak, if you ever get a chance to try it out.

~~~
convolvatron
i dont really deal with these issues directly anymore, so no, I haven't had a
chance to use any tools in the Nix family. but sure, once you eliminate the
ambiguity around 'which bytes' then it hardly matters whether you distribute
them in one chunk or several chunks.

What makes Nix more interesting than just that is that it actually provides
tools to manage the dependencies and build. As much as I think owning your
whole world is important, it can really consume alot of engineering resource
just due to the state of the world. Since Nix signatures are transitive (as I
understand it) it also captures the versions of the included tools, which is
usually too far for most organizations to go. And while issues having to do
with compiler versions are rare, they can really consume alot of energy trying
to track down.

So, without having any firsthand experience, it seem like its as dependable as
static linkage, and may be quite a bit more convenient.

------
glenjamin
The description at the end of the article about how package management doesn't
have to be done in a way that shares all the files sounds a lot like a
description of [https://nixos.org/nix/](https://nixos.org/nix/)

------
mlosapio
I feel like containers (at-times) are the artifact organizational disfunction,
the inability for teams to collaborate and a lack of engineering quality.

What’s sad is that I believe this to be true for the overhwelming majority.

~~~
jacquesm
Mixed bag: containers applied right can solve some real problems, containers
applied wrong cause some real problems.

------
prewett
I can't seem to find any easily accessible documentation on this, but it
sounds like a container is basically an chroot jail: you copy over all the
libraries and files into a subdirectory, chroot into it, and run. Am I right?
And if so, what took people so long? Chroot has been around for quite a while.

The article sort of makes it sound like a lightweight version of containers
would be putting everything in its own directory, rather than scattering
everything in /usr/lib, /usr/bin, /usr/share, etc. Which is basically macOS'
app bundles. I've never understood why other OSes didn't steal that idea. And
while I hear a lot of griping about iOS and static libraries, basically what
iOS is doing is creating a container, only since they call it an "application
bundle" instead of "container" they get all this hate. But... Grandma Tillie
(or whoever the user story for Linux on the Desktop back in the The Day was
named) can actually figure out how to delete her Mac and iOS apps.

Seems like iOS is kind of containers writ large.

~~~
anonacct37
chroot is a filesystem namespace, and containers are basically just that plus
some other namespaces. Prior to containers getting really popular I was using
debian's schroot project and prior to that I was using solaris zones. There
were other similar technologies prior to that but I wasn't really technical or
even alive at the time. It's not new.

Here's what's made them so popular lately IMO. Docker improved the usability
by adding the container registry and adding some better tooling around running
a container.

Compare installing schroot plus configuring a new container plus running
debootstrap plus potentially configuring lvm/btrfs/directory snapshotting to:

docker run ubuntu:lts /bin/bash

------
exabrial
I like the focus on artifacts and repeatable testing.

Are full blown containers even necessary though? The newest version of systemd
can do filesystem namespaces, and use cgroups to limit access to resources.
[http://0pointer.de/blog/projects/changing-
roots](http://0pointer.de/blog/projects/changing-roots)

~~~
coldacid
Not even every Linux distro uses systemd, to say nothing about other systems
on which one may want to host containers. Remember, there's more to
containerization than just Docker and its consumers and dependencies.

~~~
bamboozled
Which production grade distros don't use systemd?

~~~
zokier
Gentoo!

 _ducks_

~~~
oblio
I think RHEL 6 also doesn’t use it. But then you have bigger problems :)

------
gumby
I have a less charitable explanation, but a container fan on our team
convinced me recently to switch to fully static binaries which did indeed
simplify many things. So there is something to be said for this in an age of
cheap storage.

~~~
jacquesm
Static linking was the norm until someone figured out that all the copies of
glibc were taking up a lot of space in a time when disk space will still
expensive. Today it makes much less sense than it did back then but there is
one more advantage: updating a library (which is trivial because almost all
these systems are online all the time now) will automatically cause all new
instances of the rest of the software on the system to use the new library
without having to re-link all the binaries.

~~~
maxxxxx
"there is one more advantage: updating a library (which is trivial because
almost all these systems are online all the time now) will automatically cause
all new instances of the rest of the software on the system to use the new
library without having to re-link all the binaries."

In Windows this is called "DLL hell". Too often the new version will have some
subtle incompatibilities breaking a lot of stuff.

I am a fan of static linking or putting DLLs into a dedicated folder so these
shared libraries are actually not shared.

~~~
kazinator
Windows does a good job of versioning the actual system DLL's like user32.dll,
kernel32.dll, and so on.

The run-time libraries on top of this aren't part of the system. Each
application brings its own and so of course hell will break loose if you mix
and match.

Windows has no malloc or printf (not for public use anyway).

In the POSIX world, the user-space C run-time support is part of the platform
so it is carefully versioned, just like kernel32.dll on Windows.

User space compiler run-time support isn't part of the platform on Windows; it
is not that carefully versioned. Everyone is supposed to provide their own. If
you have a compiler from Acme, Inc, you use the Acme C library with Acme
malloc and Acme printf.

> _putting DLLs into a dedicated folder_

Which actually _works_ on Windows, that's the thing.

Searching for shared libs in the same directory as the executable is very
nice: like me.c finding its #include "me.h" next to it.

The Unix people didn't pay attention to their own programming language when
they designed shared libs.

~~~
marshray
Most compilers used by app developers don't use this, but...

link /dump /exports c:\windows\system32\msvcrt.dll

    
    
        Dump of file c:\windows\system32\msvcrt.dll
        Section contains the following exports for msvcrt.dll
    
        00000000 characteristics
        57898F96 time date stamp Fri Jul 15 18:36:22 2016
            0.00 version
               1 ordinal base
            1317 number of functions
            1317 number of names
    
        ordinal hint RVA      name
        ....
           1160  487 00019DE0 malloc
        ....
           1183  49E 00048940 printf

~~~
kazinator
Most compilers used by app developers don't use that for good reason:
Microsoft says that it is an unsupported, undocumented library which is off
limits to application developers. Programs that are part of Windows itself use
it.

[https://blogs.msdn.microsoft.com/oldnewthing/20140411-00/?p=...](https://blogs.msdn.microsoft.com/oldnewthing/20140411-00/?p=1273)

------
neuromantik8086
> In short, there’s no reason why apt and dnf or whatever couldn’t work in a
> similar way npm or bundle do, except that Linux distributions seem to be
> disinterested. (And occasionally throw a specious fit about “bundling” when
> features like this are proposed. Or are unwilling to understand why the
> ability to install multiple versions of a package is necessary.)

Wouldn't this just require tighter integration with environment modules
([http://modules.sourceforge.net/](http://modules.sourceforge.net/)) or
similar?

------
excalibur
> How did we end up with containers?

1) You bought stuff. 2) Bonus!

------
EngineerBetter
Containers != Images

