Author here - I'm working on a blog post for this and didn't expect this to be posted on HN so soon, but I guess that's what I should expect for posting a link to it in another HN comment! :)
These slides correspond to a workshop I conducted a week ago demonstrating the internals of Docker and how Docker containers can be run without using any of the Docker tools or runtime.
Docker is a great tool, and I'm glad it's gained so much traction. But containerization is still new to many people, and even then there's still a lot of confusion about the difference between Docker and containerization in general. The goal of this presentation isn't to discourage anyone from using Docker, but to outline the lay of the land for people interested in using containers.
Personally, I run containers both with systemd and with Docker. The good news is that it's really easy to switch from one to the other, so there's very little cost to trying it out both ways.
Thank you so much for the slides - even though I've been following the Docker/Rkt/LXC developments ever since Docker came out (albeit passively), this is the best resource I've come across that explains what happens internally in an easy to grok manner. The fact that you can do nearly everything in systemd is indeed a bonus.
I'm sorry if I jumped the gun before the blogpost was ready. :) Hopefully, the discussion will help even more!
Just a small FYI - whatever slide software that is - it doesn't render properly on mobile devices. On my 6+ half the slides are off the screen and it scrolls horizontally rather than vertically (that may be intended).
Always nice to cut through the hype and see how things really work. I knew Docker used namespacing for some quasi-virtualization, but was wary of using it because I haven't had time to dissect it on my own. You're a good writer, able to get a lot of information across in an engaging way.
Reveal.js is another fairly common alternative, unless you're looking for the exact visual effect (which could probably be achieved somehow). http://lab.hakim.se/reveal-js/
Sorry for slightly offtopic but what do you use to generate slideshows like this from the .slide file? I'm kinda fed up with creating powerpoints every week for meetings, would be great if there's a way to generate these locally. Is it possible to create templates? Would probably be more or less required to keeping the organization profiling.
Looking forward to reading that! I was only introduced to Docker on Monday, my first day at my new job, but I've already enjoyed the adventure of incorporating it into our deployment process.
Unfortunately not, though there's been a lot of interest in this talk, so I'll likely be presenting it again. And even in the last couple of months there's been a lot of development in the containerization world, so there's a even more to include.
If anyone's interested in hosting and/or filming an updated version of this talk, drop me a line! My email is in my profile.
The slides mention that Docker containers have hashes. This is not the case. In fact, it's probably one of the biggest user misunderstandings of Docker. Container IDs and Image IDs are not SHA hashes. Even though it may look like it they are actually just randomly generated 256-bit hex-encoded unique identifiers.
Since the slides also mention that you can use Docker images with a systemd-nspawn/machinectl setup it would be great if they soon supported the v2 Docker Registry and image format which actually does use content-addressable hashes for images.
> Since the slides also mention that you can use Docker images with a systemd-nspawn/machinectl setup it would be great if they soon supported the v2 Docker Registry and image format which actually does use content-addressable hashes for images.
I haven't used the v2 registry, so I don't know if systemd (machinectl) supports this yet, but I imagine they will soon if they don't already.
Use LXC and LXD. It's the best of both worlds. Docker is very limiting and already falling short in features. For instance daemon must run as root which sucks. LXC supports unprivileged namespaces so all users of a system can have their own set of containers. Docker insists on being the parent process, That means when docker dies, so do all your containers.. This is really bad...
How do you get boot2LXC working? Any docs, or does everyone have to DIY it all again?
Is there an ecosystem around LXC that provides things like Flynn?
I can do it all myself, but I can't do it, my development job, and be home for dinner at night. Like most tools today, the value is in the ecosystem, not the tool itself.
LXC is generic container technology like KVM OR Xen are generic virtualization technologies. LXC virtualizes the OS environment and gives you lightweight containers that you can seamlessly transition your VM workloads to, or use as a lightweight portable alternative to VMs, so its use case is general and not a narrow focus on paas or deployment centric technology.
Users can then decide how they want to deploy. Docker takes that base container and adds layers of aufs, constrains the container OS template to single app by modifying the container OS's init, gives you the dockerfile and focuses on deploy centric functionality with immutability idempotency etc, and this makes it much more complex to use than LXC. Its a use case built on Linux containers, not containers itself.
LXC is not 'low level kernel capabilities' [1] as Docker misleadingly refers to it on it's website. This has resulted in a lot of confusion about LXC in the Docker ecosystem with folks thinking its 'difficult to use' or 'just low level stuff'. A tad unfair to LXC given Docker was based on it till 0.9 and knew exactly what it was, and is as accurate as referring to docker or nspawn as low level capabilities.
That would be kernel namespaces and cgroups that LXC uses to give end user containers, like Docker uses post 0.9 directly with libcontainer and systemd-nspawn uses for its containers.
Docker builds on containers to deliver additional functionality. There is an additional cost in complexity but if that is your use case the trade off may be worth it, but for other use cases the complexity may be overkill.
You can simply make a VM image of LXC installed and you have boot2lxc, the vast ecosystem of orchestration technology that works in VMs and systems works in LXC, you don't need specific tools to be designed just for LXC. its not opinionated or exclusive like the tools built around the Docker ecosystem that are finely focussed on a specific use case and typically support Docker only.
So, you're saying it's about a month or six to replicate the conveniences that docker has provided for our organizations?
Docker isn't anything special. There's nothing that is magical or revolutionary there, but what you describe is quite a bit of work to replicate the ecosystem. That's time that could be used creating novel value.
Not sure saying LXC is "generic" is helpful either. It is opinionated that you should be virtualizing the whole OS environment, like Docker is (fairly) opinionated that you should not be.
I think some of the confusion arose as people used to use LXC as a generic term for the container technologies as well as the product, before there were any other products. namespaces+cgroups is not such a nice name. And using "linuxcontainers.org" as the website does not make it clearer.
You may want to mention that the network and disk isolation are not what someone from Docker would expect by default.
It uses the docker equivalent of "net=host" (which provides better performance at the cost of isolation), and the disk is pointing at a shared "changeroot" on disk, instead of at a layered FS.
Both of these can be better isolated with natted interfaces and a `btrfs` (which has its own reliability issues) layered image, but they are not what you expect by default.
"But, you NEED to run the installer on that (bare metal) server!" Nope, I can just boot from knoppix (remember?) and mount the disk and run debootstrap on it.
Every so often I run into programmers and sysadmins that believe these things are a kind of magic. They're not. They're just files on a disk.
Very interesting. Almost got ready to up my systemd-hate on slide 19: "machinectl -H root@example.com:debian-tree"[1] -- but calmed down and had a look around and saw that it just uses ssh, not some pottering pixiedust kerberos abomination...:
With proper handling of access (allowing unprivileged users to start containers) along with --bind for the home directory, this could be a viable alternative to Debian's schroot [s].
There's also a complimentary lwn article from 2013 that's worth reading:
That also contains a quote that explains a bit about systemd (if read maliciously): "As part of the development of systemd, the team looked at various kernel features to see if they were relevant to the project."
At least with this (containers w/log handling etc) we get something for our complexity. Still, having had two seperate machines fail to boot/even come up with a text console with some sensible errors - I'm far from sold on the idea that I want all these features in PID 1.
[1] changed user "foo" to "root" to be a little more clear. Maybe "user1" would work as well - but systemd (unlike lxc etc) requires root?).
Reminds me that I should probably make write-up of how I set up schroot to allow "source"-access for root, and automagic sessions for a standard user backed by lvm -- the documentation is a bit dense.
tl;dr It's basically a tech demo for systemd, and the systemd-nspawn tool in particular (which you might recall recently gained Docker format support).
What they mean is when the slides were made, Debian 8 was about to be released in "stable" with systemd. With debian 8 released the only oddball is ubuntu with their own early fork of systemd called upstart. It means from this day forward the things you see in the demo are already or inevitably available on every install of linux. You don't need to install anything special to do what he's doing there.
1. Upstart is not a fork of systemd, and was started a fair amount of time before systemd.
2. Ubuntu 15.04 just shipped with systemd as the default init system (although installing "upstart-sysv" the system should revert to using upstart).
This is exactly my problem with systemd. It exposes new problems and issues while does not provide the same functionality that software it is replacing.
In this case it would perfectly if it just tried to do the mount, exactly as written in fstab. Instead it tries to be clever and ends up outwitting itself.
This is what normal LXC containers has always done. Systemd nspawn does not yet provide a toolset to wrap these capabilities like the LXC project. Things like userland tools, library of OS templates for containers, networking, features like unprivileged containers that allow non root users to run containers etc.
Lennart Poettering has spoken about containers and btrfs subvolumes and easy snapshots, this could be the direction systemd goes in future for managing the OS with apps in btrfs subvolume containers, with rollback, management etc so this seems like it may mature fairly fast, except unprivileged container support which Lennart does not seem to like.[1]
As a heavy docker user who only has a couple of years of focused linux experience I found this fascinating, even if you weren't quite ready to post it. Thanks for the slides.
This is almost exactly how we run the Ironic provisioning agent that supports Rackspace OnMetal. We export use Docker to build and export an image, then use CoreOS + systemd-nspawn to run it across every unprovisioned machine.
Funny, we're running the rackspace-monitoring-agent on CoreOS like that. however we do docker pull/extract before running systemd-nspawn, if i got it right the systemd-tooling can handle docker images as they are, which would be nice.
that's a good intro to systemd-nspawn and machinectl - which happen to be much nicer to use than docker (yet transparently work with docker images if you want that). its also easier to install since you generally "already have it" and there's no setup.
There's a few things it doesn't do (neither docker, or lxc for that matter) - yet at least - such as mounting fses before container start or manage upgrades.
I'm not sure how you'd manage upgrades - that's an OS function, but for nspawn you'd just write your unit file to Require some *.mount files before boot.
updates generally require a daemon restart (and in this case perhaps a container restart), if you couple it with requisite mounts, the whole machinery has to care for mounts, upgrades and restarts altogether
I suspect it could be hacked somehow with service files, too
This is particularly significant if you use something like btrfs snapshots with a base mount and child mounts, or overlayfs and invalidate inodes during upgrades, instead of a dumb-ish "yum upgrade/apt-get upgrade/etc"
The main difference is that in this case the update is at the mount / container level when propagated from the base image.
Some (most) others also do that with image versions and a full image swap.
Everything that is old is new again. For full disclosure i went from support Linux/Solaris (sysadmin) to mainframe middleware (CICS/WebSphere app server).
Yeah. Back when I started as a sysadmin chroot jails with bind, qmail etc were all the rage.
I've seen them used in good and bad ways, but mostly the former. It's good to see something that actually solves real problems coming back in to use again. Added bonus, it's an extremely mature (from a tech perspective) way of doing things. Chroot jails have been around for decades.
This slideshow is impossible to read on an iPad; the scrolling isn't synchronized with the width of the screen, and attempting to align a page makes it jump to another. Any solutions?
Edit: Apparently, if you don't touch-move but only tap, you can keep it for, getting out of sync.
Somewhat unrelated. Is there a way to ship a container image with just a "diff" of the default image. Say I build a container with ubuntu 14.04 + some packages installed with apt. I would like to be able to "export" the filesystem, but only the files that are not present in the base os image. The reason I would like this is that I would like to use docker (or systemd-nspawn) to run containers on an device that will not always have access to the internet. It would be nice to be able to add a new package to that device, and not be multi 100 MBs. As long as I can guarantee that the base image is present on the device, I should not need to include it in the image.
You can hack this up yourself using the output from 'docker save'. 'save' produces a tarball which contains layers, and you can delete the layers from your 'save' tarball that you know you already have on the destination.
Ya the point is it won't have access to a registry. I would like to be able to ship docker "packages" that have everything they need, but not have a lot of duplicate data (the base os image) in each package. I should be able to have the base image as its own package for example, and the other packages just have the "diff" files that they need.
debootstrap just populate a directory with files from the debian/ubuntu distros, LXC uses it, everyone who wants to create/boot a debian image that they create use it.
One major difference between systemd-nspawn and LXC is how simple and reliable systemd-nspawn - in particular the guest OS/image has to run systemd as well which is what provides the integration.
Eventually, if you wanted, you could mix and match the tools just fine.
Sorry, I must not be that bright, I cannot guess your email from your username :-( (and I don't have a tumbler login, and you haven't enable dm from strangers on twitter), but I wanted to point at that you have a typo on slide 9, s/
journactl/journalctl/.
* Ease of use
* Publicly available image registry
* Dockerfiles
I should stress that last point - I dislike certain aspects of docker, but I love their system for building images. I want to somehow adapt it for building bare metal images.
I'm very sad that Hacker School had to change its name (for entirely valid reasons, although it's sad they're there). "Recurse Center" makes it sound like some . . . creationist think-tank.
chroot's aren't namespaced, which is a big important difference. Being able to kill the entire container by killing it's init process (and keep it's processes out of the host namespace) is a huge deal.
Most of these implementations actually kill all processes in the namespace, not just init. First, there may not be an init (the root process could be the service, which is actually a best practice with respect to containers); and second, the signal may not be propagated to its children, especially if a non-trappable signal (e.g. SIGKILL) is sent.
These slides correspond to a workshop I conducted a week ago demonstrating the internals of Docker and how Docker containers can be run without using any of the Docker tools or runtime.
Docker is a great tool, and I'm glad it's gained so much traction. But containerization is still new to many people, and even then there's still a lot of confusion about the difference between Docker and containerization in general. The goal of this presentation isn't to discourage anyone from using Docker, but to outline the lay of the land for people interested in using containers.
Personally, I run containers both with systemd and with Docker. The good news is that it's really easy to switch from one to the other, so there's very little cost to trying it out both ways.