
A Workshop on Linux Containers: Rebuild Docker from Scratch - mastabadtomm
https://github.com/Fewbytes/rubber-docker
======
hardwaresofton
In the same vein:

Building a container from scratch in Go (Liz Rice) @ Container Camp 2016 ->
[https://www.youtube.com/watch?v=Utf-A4rODH8](https://www.youtube.com/watch?v=Utf-A4rODH8)

What's a container really let's write one in go from sctach (Liz Rice) @
Golang UK Conference ->
[https://www.youtube.com/watch?v=HPuvDm8IC-4](https://www.youtube.com/watch?v=HPuvDm8IC-4)

Cgroups, Namespaces and beyond: What are containers made from (Jerome
Petazzoni) @ DockerCon 2015 -
[https://www.youtube.com/watch?v=sK5i-N34im8](https://www.youtube.com/watch?v=sK5i-N34im8)

Building Containers in Pure Bash and C (Jessica Frazelle) @ ContainerSummit
2016:
[https://containersummit.io/events/nyc-2016/videos/building-c...](https://containersummit.io/events/nyc-2016/videos/building-
containers-in-pure-bash-and-c)

First two are basically the same talk, but it doesn't hurt to hear the same
ideas more than once.

~~~
zerr
So basically it is just the usage of particular API the underlying OS
provides.

~~~
tyingq
Yup. It's all namespaces and cgroups. The base "docker" command adds a little
value with os respositories, network overlays, etc, but not a lot.

Personally, I think K8S with their cri-o ([https://github.com/kubernetes-
incubator/cri-o/blob/master/RE...](https://github.com/kubernetes-
incubator/cri-o/blob/master/README.md)) runtime is going to eventually eat
Docker's lunch.

As a company, Docker needs Swarm/Enterprise to take off in order to have a
differentiator. That isn't happening.

Kubernetes, on the other hand, just needs cri-o (mostly done) and an image
repository / builder to kill off Docker.

See this for how small the gap really is:
[https://github.com/p8952/bocker/blob/master/README.md](https://github.com/p8952/bocker/blob/master/README.md)

Basically, Docker has almost no moat. They have mostly only brand recognition.
Not discounting their efforts, but they are playing checkers and Google is
playing chess.

~~~
hardwaresofton
I think this is how I most commonly recognize someone that is familiar with
lxc +/\- docker and containerization in general. A firm grasp of the fact that
they're just beefy processes, isolated wiht namespaces and cgroups is the
best, most succint way to describe docker (without even mentioning the
benefits), but it also requires that the hearer knows what namespaces and
cgroups are.

If I had to rank understanding in explanations of docker:

1\. "Makes your application really portable"/other explanations that only
cover the benefits of docker not how it works

2\. "Lightweight VMs"

3\. "LXC + some other stuff"

4\. "processes + namespaces + cgroups (+/\- image management tools, etc)"

100% agreed on the point you made, pretty sure docker+swarm and other
orchestration efforts have basically lost the competition. Kubernetes just has
the mind share, and even better than that -- it's __actually good __.

Kubernetes also has multiple competing container runtimes all rushing to fit
the CRI (Container Runtime Interface). Just some of the stuff out there:

* cri-containerd - [https://github.com/containerd/cri-containerd](https://github.com/containerd/cri-containerd)

* runv (hypervisor based) - [https://github.com/hyperhq/runv](https://github.com/hyperhq/runv)

* clear containers (hypervisor based) - [https://github.com/clearcontainers/runtime](https://github.com/clearcontainers/runtime)

* kata containers (hypervisor based, collaboration of runv + clear container) - [https://katacontainers.io/](https://katacontainers.io/)

* frakti (combination of runv & docker, enabling switching @ runtime) - [https://github.com/kubernetes/frakti](https://github.com/kubernetes/frakti)

A bunch of the projects are in their infancy, but the CNCF/kubernetes and the
community are doing the right thing investing in lots of them, and letting the
good ones bubble to the top. I don't run a production kubernetes cluster but
I've found cri-containerd to be really easy to use and haven't had any major
problems with it, mostly small configuration things.

The last one, frakti, gets me really excited, because it lets you be flexible
about which containers run in more protected environments and which don't.
Also really exciting is frakti v2
([https://github.com/kubernetes/frakti/tree/containerd-
kata](https://github.com/kubernetes/frakti/tree/containerd-kata)), which is
kata + containerd.

~~~
tyingq
Kinda feels like I'm I a foreign country and run into someone that speaks my
language :). Thanks for the additional info, like frakti, for example...didn't
know about that.

Explaining docker is frustrating for me because I was a Unix admin back in the
90's. So, for me, it's pretty easy to see what it is. And, it isn't new. There
isn't much it does that Solaris zones or BSD jails, didn't do. And those
predate docker by years. Namespaces, cgroups, and pivot_root. That's basically
all of it. Kudos to docker for marketing it better.

Explaining it, on the other hand, to a broad audience...

~~~
zbentley
That's sorta true, but the layered-userspace-filesystem support is a pretty
big feature that Docker pioneered Linux support for.

Did userspace filesystems/layered filesystem-in-a-box implementations exist
before? Sure: from ZFS to squashfs to tar, all the components were around.

But Docker's popularity is due to its integration; not its novelty. By
bundling a filesystem-in-a-box abstraction layer with varying degrees of
native OS support into something like a mountable artifact/image, hiding the
image caching mechanics behind a nice CLI and image-configuration file format
(love 'em or hate 'em, Dockerfiles are a phenomenal example of minimalism and
convenience: add some meta commands, and everything else is just shell), and
integrating all that with the privacy/isolation (namespaces/chroots) and
resource management (cgroups/quotas) stuff, Docker made a really, really
powerful _model of thinking_ about "containers" as a single concept.

The thought model is Docker's big achievement: other competitors may unseat
Docker eventually, but the concept of "container" as a single, unitary thing
that's almost like a VM image will stick around, and the more piecemeal
understanding of how containers work will be largely unnecessary, and thus
save a lot of time for a lot of people.

~~~
cyphar
LXC had most (if not all) of what you're describing, and a long time ago
Docker was just a simple wrapper around LXC. A lot of the inspiration for
containers came from other operating systems (FreeBSD Jails and Solaris Zones)
as well as previous work such as Xen.

I think what made Docker popular was that it was easier for _developers_ to
use, with small bits of information like what ports the container wants to
listen on and so on. From an administration standpoint, LXC was already more
than good enough.

------
derefr
Or rather, rebuild _runc_ from scratch.

These days, Docker isn't really "about" the running of containers (all the
logic for which is encapsulated outside of the Docker project itself, in
runc.) Docker—the thing that Docker Inc produces—is the tooling that gets
_container images_ fed into a daemonized instance of runc.

So: the Docker Registry daemon (the thing you can push/pull images to/from,
such as runs on Docker Hub); the `Dockerfile` format, and the build logic that
uses it, and the CI bots that use that build logic; the local daemon that
holds a mini-registry that builds and pulls write to; and the tooling to move
and create and dump images between all those places.

If there's a tutorial that replicates _that_ stuff, I'd love to read it.

~~~
dozzie
Do you realize that " _that_ stuff" is merely a binary packages system?

~~~
zbentley
Eh, if it's a binary package system it's a very nice one--perhaps an
historically nice one.

Unlike most package systems, most people that develop _in_ docker containers
are writing the equivalent of a package manifest (Dockerfile), and they're
doing it for things that weren't previously packaged. While there are plenty
of exceptions, the standard for deployment of webapps has long been piecemeal
deployments; not RPMs or whatnot. Docker's convenience features changed that.

Also, the layering system/virtual filesystems are integrated into Docker such
that it yields a ton of convenience (in "extending" others images, speeding up
deployments/caching, and making huge/arbitrary changes to the filesystem
highly reversible) without making users manually manage most of it. There are
times when the abstraction leaks (looking at you, hard limit on number of
layers), and all of the component technologies that went into it existed
before, but as I've posted elsewhere in the comments here, the big advantage
of docker is that it integrates those technologies in a way that provides a
_simple mental model_ for reasoning about containers, and a very
convenient/beginner-accessible toolset for packaging really complex
dependencies. Even the nicest "typical" packaging tools out there (rpm/dpkg
are pretty crufty, but stuff like Nix and FPM are getting quite nice) still
have much rougher edges.

~~~
dozzie
> Unlike most package systems, most people that develop in docker containers
> are writing the equivalent of a package manifest (Dockerfile)

Dockerfile is a set of build rules. You need to have build rules for RPMs and
DEBs (unless you use FPM that you somewhat praised later, in which case
there's no good place to put the rules and you get terrible packages with
missing metadata).

Unlike most package systems, Docker was designed by programmers for
programmers, so you don't need to learn those yucky tools for sysadmins and
you can stay oblivious to how the OS actually works (despite that you should
actually know the OS you're dealing with, so it's a dumb idea on its own).

> and they're [writing Dockerfiles] for things that weren't previously
> packaged.

Which is only a progress because programmers are putting the code into
packages. Exactly the same could be achieved with RPMs and DEBs, even if you
needed something that was already shipped by the distribution (you'd just put
your things on the side, just as you are doing now with virtualenv or gems or
whatever).

> While there are plenty of exceptions, the standard for deployment of webapps
> has long been piecemeal deployments; not RPMs or whatnot. Docker's
> convenience features changed that.

But Docker didn't change that because it allowed anything that was previously
impossible or even difficult. Quite the contrary, it was _easy_ , it just
required to actually know how the OS works, which is not a common knowledge
among web programmers. OS packages were ignored by programmers solely because
they were sysadmins' tools, and as such they were boring. I see no other
reason, given that Docker gives virtually no technical benefit beside heavy
brittle magic on network configuration, so all your packaged daemons can
listen on the very same address 0.0.0.0:8888 and still communicate with each
other.

> the big advantage of docker is that it integrates those technologies in a
> way that provides a simple mental model for reasoning about containers

This mental model, it is what exactly? Because there's virtually no mental
model with the packages. A tarball with necessary files, that's all.

~~~
zbentley
> You need to have build rules for RPMs and DEBs.

Yes, but as you said just after that, those in a Dockerfile are much more
popular with a wide range of programmers. That might be because the dockerfile
abstractions/API is better and simpler, or because the programmers like
writing shell scripts but not RPM build rules, or for some other reason.

> you should actually know the OS you're dealing with, so it's a dumb idea on
> its own ... it was easy, it just required to actually know how the OS works,
> which is not a common knowledge among web programmers.

I'm really tired of this attitude. Of course you should know the OS you're
dealing with. What you need to know _about_ it depends on what you're doing
with it. If I want to do kernel work, I don't need to know the best design
principles for an ES7 web framework. If I want to make a website, I don't need
to know how to write Apache 2 from first principles, and nor do I need to know
how to manually chroot/install quotas/set up namespaces and capabilities to
build a container system from scratch.

> This mental model, it is what exactly? Because there's virtually no mental
> model with the packages. A tarball with necessary files, that's all.

If it was just a tarball, it would be less powerful--it's the whole bunch of
technically-unrelated things (resource management, networking, capabilities,
namespaces, tarball-ish features, layering, dockerfile API, nice CLI with
pluggable backends, standardized container interface) all unified under the
abstraction of "this is a single unit, just like an RPM package". That concept
is powerful exactly _because_ it hides the fundamentals of the specific
component technologies from people who don't need to know them--at least not
at first.

Saying Docker is just "package maintenance for stupid people who only play
with Duplo legos", you're being ignorant of the real needs of people at best,
and deliberately elitist at worst. It's like saying "Dropbox is just for
people who don't want to learn how file syncing over the network works--
_real_ programmers will just use curlftps and SVN".

~~~
dozzie
> I'm really tired of this attitude. [...] What you need to know about [the
> OS] depends on what you're doing with it.

You want your system deployed, so you should know _how to deploy_. If you
don't know how most of the things are deployed, you're likely to create a
monstrosity that doesn't fit in any way to what the OS can do sensibly.

It's like arguing that a web programmer doesn't need to understand HTTP
protocol, because he only works with it through half a dozen layers of
abstractions (how currently it's done).

> If I want to make a website, I don't need to know how to write Apache 2 from
> first principles, and nor do I need to know how to manually chroot/install
> quotas/set up namespaces and capabilities to build a container system from
> scratch.

Of course, because that's what sysadmins do. But you should know how to
configure the said Apache. Docker hides that away behind a heavy magic, which
is bound to break apart for non-trivial requirements.

------
markbnj
There is also [http://chimeracoder.github.io/docker-without-
docker/#1](http://chimeracoder.github.io/docker-without-docker/#1)

------
infodroid
A similar project is 'mocker', a "crappy imitation of Docker, written in 100%
Python":
[https://github.com/tonybaloney/mocker](https://github.com/tonybaloney/mocker)

~~~
tzakrajs
> I keep hearing statements like "Docker is basically

> just cgroups", "Docker is just chroot on steroids",

> which is totally incorrect.

But this project basically proves that the characterization is pretty
accurate.

~~~
erikb
go back to the top level of this discussion and ctrl-f for "runc" then it
becomes more clear. Docker is more than we usually think it is, because it
makes all the stuff around containers so easy to use.

~~~
hodgesrm
Agree. I'm working through the rubber docker lab now. The fact that docker (a)
handles all the details of cgroups, overlays, namespaces; (b) is super easy to
use; _and_ (c) runs cross-platform is impressive.

------
simonw
I like how this includes a very simple, readable C extension for Python that
exposes some missing syscalls: [https://github.com/Fewbytes/rubber-
docker/blob/master/linux....](https://github.com/Fewbytes/rubber-
docker/blob/master/linux.c)

------
baconomatic
I've been looking for something like this for a while now. I find building
things from the bottom up is a great way to learn how things work (naturally).

~~~
arca_vorago
Also taking things apart is a great way to know how to start a bottom up
build.

------
acobster
I read the OCI runtime/container specs when I was first starting to move all
my work projects' dev environments to docker (Lando, to be precise - in case
you haven't used it, it's awesome!). That was very illuminating, but there was
a lot to it that I didn't understand because of the relatively dense language.
Wish I'd had this! Thanks for posting.

~~~
cyphar
Yeah, we really could've done a better job with explaining how to write your
own configurations. I tried to make the conversion from images to runtime
specified in some manner[1], but really the trade-off between granularity and
ease-of-use was tipped to the "extreme granularity" side of the scale.

Though if you're writing low-level container configurations you really should
already know enough to write a (simple) container runtime yourself (because
the security pitfalls with a slightly misconfigured container can be pretty
bad -- even though runc does quite a lot of things to keep you safe that are
outside of the spec).

[1]: [https://github.com/opencontainers/image-
spec/blob/v1.0.0/con...](https://github.com/opencontainers/image-
spec/blob/v1.0.0/conversion.md)

------
amq
How would you describe a linux container in one sentence? I'm working with
them for some years already, but I'm still struggling with a consceise
description aimed for e.g. fresh CS undergrads.

~~~
insaneirish
How about: "There is no such thing as a 'Linux container.'"

Followed by: "Now let's talk about namespaces and cgroups."

~~~
Annatar
Correct, you grok it! Only Solaris and illumos kernels have true containers
(resource limits) optionally applied to zones (virtualized OS instances
providing full blown UNIX servers) running at bare metal speed as a bunch of
processes in the global zone. FreeBSD jails come close (they were the
inspiration for zones), but are more akin to chrooted jails than containers.
Nowadays they are conceptually more like zones than they were in the
beginning.

------
kompiuter
Great, something to pass the time tonight. Many thanks!

