
Docker without Docker - deepakprakash
https://chimeracoder.github.io/docker-without-docker/#1
======
chimeracoder
Author here - I'm working on a blog post for this and didn't expect this to be
posted on HN so soon, but I guess that's what I should expect for posting a
link to it in another HN comment! :)

These slides correspond to a workshop I conducted a week ago demonstrating the
internals of Docker and how Docker containers can be run without using any of
the Docker tools or runtime.

Docker is a great tool, and I'm glad it's gained so much traction. But
containerization is still new to many people, and even then there's still a
lot of confusion about the difference between Docker and containerization in
general. The goal of this presentation isn't to discourage anyone from using
Docker, but to outline the lay of the land for people interested in using
containers.

Personally, I run containers both with systemd and with Docker. The good news
is that it's really easy to switch from one to the other, so there's very
little cost to trying it out both ways.

~~~
webwanderings
Off topic: how do I create slides like this? What did you use to create 'em?

~~~
dorfsmay
An alternative to this system that I have been enjoying using is slidy, it
makes slick presentation:

[http://www.w3.org/Talks/Tools/Slidy2/](http://www.w3.org/Talks/Tools/Slidy2/)

~~~
Apofis
No scroll wheel support? F that.

~~~
dorfsmay
Scroll wheel on a slide?

------
jlhawn
The slides mention that Docker containers have hashes. This is not the case.
In fact, it's probably one of the biggest user misunderstandings of Docker.
Container IDs and Image IDs are _not_ SHA hashes. Even though it may look like
it they are actually just randomly generated 256-bit hex-encoded unique
identifiers.

Since the slides also mention that you can use Docker images with a systemd-
nspawn/machinectl setup it would be great if they soon supported the v2 Docker
Registry and image format which actually does use content-addressable hashes
for images.

~~~
chimeracoder
> Container IDs and Image IDs are not SHA hashes.

Thanks for catching that - I've updated it.

> Since the slides also mention that you can use Docker images with a systemd-
> nspawn/machinectl setup it would be great if they soon supported the v2
> Docker Registry and image format which actually does use content-addressable
> hashes for images.

I haven't used the v2 registry, so I don't know if systemd (machinectl)
supports this yet, but I imagine they will soon if they don't already.

------
sigmonsays
Use LXC and LXD. It's the best of both worlds. Docker is very limiting and
already falling short in features. For instance daemon must run as root which
sucks. LXC supports unprivileged namespaces so all users of a system can have
their own set of containers. Docker insists on being the parent process, That
means when docker dies, so do all your containers.. This is really bad...

~~~
ebiester
How do you get boot2LXC working? Any docs, or does everyone have to DIY it all
again?

Is there an ecosystem around LXC that provides things like Flynn?

I can do it all myself, but I can't do it, my development job, and be home for
dinner at night. Like most tools today, the value is in the ecosystem, not the
tool itself.

~~~
tobbyb
LXC is generic container technology like KVM OR Xen are generic virtualization
technologies. LXC virtualizes the OS environment and gives you lightweight
containers that you can seamlessly transition your VM workloads to, or use as
a lightweight portable alternative to VMs, so its use case is general and not
a narrow focus on paas or deployment centric technology.

Users can then decide how they want to deploy. Docker takes that base
container and adds layers of aufs, constrains the container OS template to
single app by modifying the container OS's init, gives you the dockerfile and
focuses on deploy centric functionality with immutability idempotency etc, and
this makes it much more complex to use than LXC. Its a use case built on Linux
containers, not containers itself.

LXC is not 'low level kernel capabilities' [1] as Docker misleadingly refers
to it on it's website. This has resulted in a lot of confusion about LXC in
the Docker ecosystem with folks thinking its 'difficult to use' or 'just low
level stuff'. A tad unfair to LXC given Docker was based on it till 0.9 and
knew exactly what it was, and is as accurate as referring to docker or nspawn
as low level capabilities.

That would be kernel namespaces and cgroups that LXC uses to give end user
containers, like Docker uses post 0.9 directly with libcontainer and systemd-
nspawn uses for its containers.

Docker builds on containers to deliver additional functionality. There is an
additional cost in complexity but if that is your use case the trade off may
be worth it, but for other use cases the complexity may be overkill.

You can simply make a VM image of LXC installed and you have boot2lxc, the
vast ecosystem of orchestration technology that works in VMs and systems works
in LXC, you don't need specific tools to be designed just for LXC. its not
opinionated or exclusive like the tools built around the Docker ecosystem that
are finely focussed on a specific use case and typically support Docker only.

[1] [https://linuxcontainers.org](https://linuxcontainers.org)

~~~
rdtsc
> A tad unfair to LXC given Docker was based on it till 0.9 and knew exactly
> what it was,

I had noticed that too initially.

They had to minimize it because otherwise people would just say "why not just
use LXC, what do I need the whale for?".

From the marketing standpoint it had to be "yeah that is complicated low level
bearded guy stuff, you need cool easy slick stuff we provide".

------
falcolas
You may want to mention that the network and disk isolation are not what
someone from Docker would expect by default.

It uses the docker equivalent of "net=host" (which provides better performance
at the cost of isolation), and the disk is pointing at a shared "changeroot"
on disk, instead of at a layered FS.

Both of these can be better isolated with natted interfaces and a `btrfs`
(which has its own reliability issues) layered image, but they are not what
you expect by default.

------
mverwijs
> At its core, your OS is just a bunch of files

"But, you NEED to run the installer on that (bare metal) server!" Nope, I can
just boot from knoppix (remember?) and mount the disk and run debootstrap on
it.

Every so often I run into programmers and sysadmins that believe these things
are a kind of magic. They're not. They're just files on a disk.

Love this presentation. Thank you!

edit: s/mount knoppix/boot from knoppix/

------
e12e
Very interesting. Almost got ready to up my systemd-hate on slide 19:
"machinectl -H root@example.com:debian-tree"[1] -- but calmed down and had a
look around and saw that it just uses ssh, not some pottering pixiedust
kerberos abomination...:

[http://www.freedesktop.org/software/systemd/man/machinectl.h...](http://www.freedesktop.org/software/systemd/man/machinectl.html)

With proper handling of access (allowing unprivileged users to start
containers) along with --bind for the home directory, this could be a viable
alternative to Debian's schroot [s].

There's also a complimentary lwn article from 2013 that's worth reading:

[https://lwn.net/Articles/572957/](https://lwn.net/Articles/572957/)

That also contains a quote that explains a bit about systemd (if read
maliciously): "As part of the development of systemd, the team looked at
various kernel features to see if they were relevant to the project."

At least with this (containers w/log handling etc) we get _something_ for our
complexity. Still, having had two seperate machines fail to boot/even come up
with a text console with some sensible errors - I'm far from sold on the idea
that I want all these features in PID 1.

[1] changed user "foo" to "root" to be a little more clear. Maybe "user1"
would work as well - but systemd (unlike lxc etc) requires root?).

[s] [https://wiki.debian.org/Schroot](https://wiki.debian.org/Schroot)

Reminds me that I should probably make write-up of how I set up schroot to
allow "source"-access for root, and automagic sessions for a standard user
backed by lvm -- the documentation is a bit dense.

------
vezzy-fnord
tl;dr It's basically a tech demo for systemd, and the systemd-nspawn tool in
particular (which you might recall recently gained Docker format support).

~~~
istvan__
Yep, and not particularly interesting. It is a catchy title and some showcase
for Systemd.

~~~
lsc
Much like docker.

~~~
istvan__
I am not sure what you are talking about. I am talking about this:

"Within 9 days, all of the top 10 Linux distributions will use systemd by
default, or descend from one that does."
[https://chimeracoder.github.io/docker-without-
docker/#1](https://chimeracoder.github.io/docker-without-docker/#1)

~~~
mkhpalm
What they mean is when the slides were made, Debian 8 was about to be released
in "stable" with systemd. With debian 8 released the only oddball is ubuntu
with their own early fork of systemd called upstart. It means from this day
forward the things you see in the demo are already or inevitably available on
every install of linux. You don't need to install anything special to do what
he's doing there.

[http://upload.wikimedia.org/wikipedia/commons/5/58/Linux_Dis...](http://upload.wikimedia.org/wikipedia/commons/5/58/Linux_Distribution_Timeline_with_Android.svg)

~~~
baghira
1\. Upstart is not a fork of systemd, and was started a fair amount of time
before systemd. 2\. Ubuntu 15.04 just shipped _with_ systemd as the default
init system (although installing "upstart-sysv" the system should revert to
using upstart).

~~~
rogerbinns
Note that in the Grub boot menu you can also choose whether Ubuntu 15.04 boots
with upstart or systemd. Both are functional.

Thankfully that saved me when systemd couldn't grasp the concept of a btrfs
volume being spread over two disks (raid 0).

~~~
blubbers
really? i have that exact setup except with three striped disks, it works fine

~~~
rogerbinns
Gory details are at
[https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/14478...](https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1447879)

It actually mounts root correctly, but them fails to mount home which has
exactly the same details except a different subvol parameter.

~~~
digi_owl
Systemd seems to have ongoing problems with mounting...

~~~
rogerbinns
In this case it would perfectly if it just tried to do the mount, exactly as
written in fstab. Instead it tries to be clever and ends up outwitting itself.

------
tobbyb
This is what normal LXC containers has always done. Systemd nspawn does not
yet provide a toolset to wrap these capabilities like the LXC project. Things
like userland tools, library of OS templates for containers, networking,
features like unprivileged containers that allow non root users to run
containers etc.

Lennart Poettering has spoken about containers and btrfs subvolumes and easy
snapshots, this could be the direction systemd goes in future for managing the
OS with apps in btrfs subvolume containers, with rollback, management etc so
this seems like it may mature fairly fast, except unprivileged container
support which Lennart does not seem to like.[1]

[1][https://plus.google.com/+LennartPoetteringTheOneAndOnly/post...](https://plus.google.com/+LennartPoetteringTheOneAndOnly/posts/W2itNERXvMh)

------
markbnj
As a heavy docker user who only has a couple of years of focused linux
experience I found this fascinating, even if you weren't quite ready to post
it. Thanks for the slides.

------
russell_h
This is almost exactly how we run the Ironic provisioning agent that supports
Rackspace OnMetal. We export use Docker to build and export an image, then use
CoreOS + systemd-nspawn to run it across every unprovisioned machine.

~~~
mkulke
Funny, we're running the rackspace-monitoring-agent on CoreOS like that.
however we do docker pull/extract before running systemd-nspawn, if i got it
right the systemd-tooling can handle docker images as they are, which would be
nice.

------
monochromatic
Anybody else having issues navigating? I'm on Firefox.

~~~
po1nter
It's works fine here using the right arrow on Firefox Nightly 40.0a1 running
on Ubuntu 15.04.

~~~
monochromatic
Ha, it didn't occur to me to use the keyboard.

~~~
coldtea
What I find incredible is that the same exchange takes place on HN every
single time a web slideshow is posted...

Isn't trying the arrow keys / keyboard navigation an instictive thing to do?

~~~
monochromatic
> Isn't trying the arrow keys / keyboard navigation an instictive thing to do?

Evidently not. My position is that a web page that is impossible to navigate
via mouse is broken.

------
zobzu
that's a good intro to systemd-nspawn and machinectl - which happen to be much
nicer to use than docker (yet transparently work with docker images if you
want that). its also easier to install since you generally "already have it"
and there's no setup.

There's a few things it doesn't do (neither docker, or lxc for that matter) -
yet at least - such as mounting fses before container start or manage
upgrades.

~~~
XorNot
I'm not sure how you'd manage upgrades - that's an OS function, but for nspawn
you'd just write your unit file to Require some *.mount files before boot.

~~~
zobzu
updates generally require a daemon restart (and in this case perhaps a
container restart), if you couple it with requisite mounts, the whole
machinery has to care for mounts, upgrades and restarts altogether

I suspect it could be hacked somehow with service files, too

This is particularly significant if you use something like btrfs snapshots
with a base mount and child mounts, or overlayfs and invalidate inodes during
upgrades, instead of a dumb-ish "yum upgrade/apt-get upgrade/etc"

The main difference is that in this case the update is at the mount /
container level when propagated from the base image.

Some (most) others also do that with image versions and a full image swap.

------
Twirrim
We're back to chroot jails again?

~~~
zmonkeyz
Everything that is old is new again. For full disclosure i went from support
Linux/Solaris (sysadmin) to mainframe middleware (CICS/WebSphere app server).

~~~
Twirrim
Yeah. Back when I started as a sysadmin chroot jails with bind, qmail etc were
all the rage.

I've seen them used in good and bad ways, but mostly the former. It's good to
see something that actually solves real problems coming back in to use again.
Added bonus, it's an extremely mature (from a tech perspective) way of doing
things. Chroot jails have been around for decades.

------
lobster_johnson
This slideshow is impossible to read on an iPad; the scrolling isn't
synchronized with the width of the screen, and attempting to align a page
makes it jump to another. Any solutions?

Edit: Apparently, if you don't touch-move but only tap, you can keep it for,
getting out of sync.

------
seiferteric
Somewhat unrelated. Is there a way to ship a container image with just a
"diff" of the default image. Say I build a container with ubuntu 14.04 + some
packages installed with apt. I would like to be able to "export" the
filesystem, but only the files that are not present in the base os image. The
reason I would like this is that I would like to use docker (or systemd-
nspawn) to run containers on an device that will not always have access to the
internet. It would be nice to be able to add a new package to that device, and
not be multi 100 MBs. As long as I can guarantee that the base image is
present on the device, I should not need to include it in the image.

~~~
cpitman
As long as you are pulling images from a docker registry, this should
automatically happen.

~~~
seiferteric
Ya the point is it won't have access to a registry. I would like to be able to
ship docker "packages" that have everything they need, but not have a lot of
duplicate data (the base os image) in each package. I should be able to have
the base image as its own package for example, and the other packages just
have the "diff" files that they need.

------
gosub
So we should also add Docker to the chimera that systemd is becoming?

~~~
Gigablah
It's already in there...

------
haddr
What is the actual difference here between using debootstrap and LXC?

~~~
zobzu
debootstrap just populate a directory with files from the debian/ubuntu
distros, LXC uses it, everyone who wants to create/boot a debian image that
they create use it.

One major difference between systemd-nspawn and LXC is how simple and reliable
systemd-nspawn - in particular the guest OS/image has to run systemd as well
which is what provides the integration.

Eventually, if you wanted, you could mix and match the tools just fine.

------
dorfsmay
@chimeracoder:

First thanks, this was really interesting.

Sorry, I must not be that bright, I cannot guess your email from your username
:-( (and I don't have a tumbler login, and you haven't enable dm from
strangers on twitter), but I wanted to point at that you have a typo on slide
9, s/ journactl/journalctl/.

------
yesbabyyes
Combining this with IPFS could be pretty cool!

------
mmgutz
I haven't used Docker myself. So what value add does Docker provide?

~~~
ekimekim
* Ease of use * Publicly available image registry * Dockerfiles

I should stress that last point - I dislike certain aspects of docker, but I
love their system for building images. I want to somehow adapt it for building
bare metal images.

~~~
rodgerd
> * Publicly available image registry

I am unconvinced downloading system images from the Internet is a great thing
overall.

~~~
ekimekim
I somewhat agree, but it's a thing that many people see as a feature.

------
epaulson
What about the entries in /dev? Is mknod not a thing anymore, or do I not need
to have /dev entries in my container root filesystem?

~~~
Sanddancer
It's systemd based, so it'll be bringing udev along for the ride.

------
octatoan
I'm very sad that Hacker School had to change its name (for entirely valid
reasons, although it's sad they're there). "Recurse Center" makes it sound
like some . . . creationist think-tank.

------
faizshah
This is interesting, any recommendations for further reading?

------
bandrami
What's old is new. You're describing containers I think.

~~~
miah_
See: User Mode Linux, available since somewhere around 2002 or 2003.

[http://user-mode-linux.sourceforge.net/](http://user-mode-
linux.sourceforge.net/) [https://en.wikipedia.org/wiki/User-
mode_Linux](https://en.wikipedia.org/wiki/User-mode_Linux)

~~~
icebraining
Not really the same, since that runs a full kernel for each "container".
OpenVZ is a better example (and is, in fact, the precursor of LXC).

------
synaesthesisx
I feel like HN really loves docking.

------
anonbanker
So, this is just a systemd commercial pretending to be about docker?

~~~
eugeneionesco
Well it says "without Docker" in the title so not sure what where you
expecting...

~~~
anonbanker
anti-systemd comment buried to the bottom of the page. worth burning karma
for!

