
Understanding and Hardening Linux Containers [pdf] - secalex
https://www.nccgroup.trust/globalassets/our-research/us/whitepapers/2016/april/ncc_group_understanding_hardening_linux_containers-10pdf
======
X86BSD
Obviously the author put a lot of effort into this paper. Hard work shows
throughout. Kudos to you sir!

Granted I realize the title is "Understanding and Hardening _Linux_
Containers".

However, personally, this just illustrates my frustration and the frustration
of others with the Linux world.

They exist in their own little echo chamber. Linux was not the first to create
"containers" that are secure. There is no inclusion of other solutions outside
Linux.

The reason other solutions do not have these problems is because the authors
actually thought about the problem. And if the Linux camp would simply look
outside their echo chamber long enough to see how others solved these problems
before them this paper might not have been written.

It's just simply frustrating to watch Linux reinvent some wheels, poorly, time
after time.

Flame on!

~~~
Annatar
Totally agree with you. I fail to see the point of attempting to harden Linux
"containers" (Linux actually does not have container technology, since cgroups
are not containers). Why bother with that, when illumos, and SmartOS on top of
it, provide free, open source Solaris zones technology? Not even installation
is required, since the thing runs from random access memory, and can run Linux
at the speed of bare metal?

Everybody is just flapping on Linux containers. Meanwhile illumos and SmartOS
are eating Linux's lunch with techology which has been working reliably in
Swiss banks since 2006, and it is free open source software. What's not to
love?

There are T-shirts which say "Go away or I will replace you with a very small
shell script." Now it's "Go away or I will virtualize your Linux inside of
SmartOS." The damn thing runs like a bandit, while Linux containers keep on
flapping trying to find a working solution with papers like these. Why bother,
the problem has already been solved, go run SmartOS and enjoy a working
solution!

~~~
z3t4
SmartOS if very nice in theory. I tried to get SmartOS working on a new
Supermicro server and failed, even though I got a lot of help from their
awesome community and people at Joyent.

Then I tried Linux (Ubuntu) and everything just worked! It was also very easy
to find guides on the web for most things Linux.

~~~
Annatar
If you want the JustWorks(SM), you have to get hardware which is on the Joyent
engineering bill of materials' list, or research and buy hardware which
illumos has drivers for.

[http://eng.joyent.com/manufacturing/bom.html](http://eng.joyent.com/manufacturing/bom.html)

SmartOS being descended from Solaris has detailed manual pages built into the
operating system, because when one runs a real UNIX, one is entitled to
comprehensive documentation with lots of examples.

A lot of the Oracle Solaris documentation, which is also very comprehensive,
applies to and can be used on illumos derivatives. The SmartOS wiki and the
smartos-discuss mailing list are good sources of SmartOS specific
enhancements. Did I mention the built-in man pages yet?

------
geggam
This..... this is the thing every architect attempting to roll containers out
to production needs to read and re read

Complexity at scale: Orchestration frameworks (Rancher, MESOS/Aurora, Docker
Swarm, LXD, OpenStack Containers, Kubernetes/Borg, etc) are only recently
catching up to the container craze, there are too many competing models to
list. Many have questionable or unaudited security or leave major requirements
out, such as secret management. While containers may be easy to get working
within a workstation or a few servers, scaling them up to production
deployment is another challenge altogether, even assuming your application
stack can be properly ``containerized''

~~~
kyrra
_The opinions stated here are my own, not necessarily those of Google._

From my understanding Google was one of the main contributors of the core
features that have allowed containers on Linux. Specifically cgroups and LXC.
And they have have been running containers for 10 years:
[https://research.google.com/pubs/pub43438.html](https://research.google.com/pubs/pub43438.html)

I'm not sure what you consider new, but it's definitely not super new to
Google. I'm guessing your issue is that it's fairly new to being used in
production across many companies, so security researchers are just starting to
work on finding holes in it.

~~~
jethro_tell
This is true but containers for mortals have lagged behind. Especially in
management tools. There was cloudvz. Then cgmanage that couldn't run au pid 1
which has security concerns of its own. It wasn't until systemd that a small
team could stand up containers with any type of scale. Basically everything
before, more or less, took a proprietary config management setup.

~~~
ak217
We stood up containers, on very large scale, in a PaaS, with a very small
team, in 2012 (long before systemd, or Docker for that matter, became a major
force).

This was made possible by the work done by Serge Hallyn, Stephane Graber, and
others at Canonical. They delivered LXC and its integration with AppArmor,
which in turn stood on top of the cgroups/namespaces work that Google
contributed to the kernel. Ubuntu 12.04 was the release where the ability to
securely run containers at scale became available to mere mortals.

------
tptacek
Exposing /dev/random in containers does not put the system entropy pool at
risk. That claim is repeated twice in the paper, and is false.

~~~
josh-wrale
Could you link to supporting reference? Is there some smarts in the kernel to
namespace, block or limit draining?

~~~
zuzun
I assume the original author meant to say: draining /dev/random does not
affect the cryptographic quality of the random numbers produced by both
devices.

But yes, since there's only one entropy pool, attackers can drain /dev/random,
causing other programs that rely on /dev/random to block.

All I can say is: on newer kernels, attackers can still drain the pool by
using the getrandom syscall, so unless you block that syscall, not mounting
/dev/random does not increase the security.

~~~
dyn
Hi author here.

Yes, that is what I was trying to say. I'll clarify it in a future version.

That's a good point, I'll include that! Thanks.

------
nicolast
> and a number of other systems such as Mirage OS (a reimplementation of
> Solaris Zones).

I'd say that's not entirely correct. At all.

~~~
amirmc
Yeah, there are a number of statements about unikernels that are ... somewhat
inaccurate.

~~~
dyn
Hi author here. I expected some folks who know a lot more about unikernels
might have some feedback. I wanted to cover them because I think they're an
interesting from a security perspective/they relate to containers but will
admit I don't have much experiance with them. If you have time, I'm happy to
fix the inaccuracies, feel free to DM me on twitter / email me?
(first.last@nccgroup.trust) or (twitter.com/@dyn___). Thanks.

------
molecule
@ 100+ pages, it would be great to get this in mobi or epub format.

------
ishtu
"Containers are great. It's a shame Linux doesn't have any."

------
tyingq
_" While Linux Container systems (LXC, Docker, CoreOS Rocket, etc) have
undergone fast deployment and development, security knowledge has lagged
behind. The number of people focused on container security...seems
disproportionately small"_

I agree with this part. Most containers aren't running as an unprivileged
user. Those environments that do support it only support it in a very limited
set of os/kernel/whatever versions. Somewhat concerning since containers are
getting traction almost everywhere.

~~~
cyphar
Oddly enough, quite a few people in the runC community (including myself) are
working on implementing the ability to start containers without root. If we
can get this to work, it will be brought to Docker and you'll be able to start
containers without even needing a daemon running as root (although you'd lose
some functionality due to deficiencies in some of the kernel interactions with
user namespaces -- but it should be more secure than it is now). It does
bother me that the Linux kernel community entirely ignored other container
implementations.

~~~
tyingq
Yes, I've been following the progress, and I see the disconnects across the
space. Like this issue:
[https://github.com/systemd/systemd/issues/321](https://github.com/systemd/systemd/issues/321)

Appreciate your efforts. It is good to know that there are people pushing to
get this working.

~~~
cyphar
If you're interested, I've got a WIP branch of runC that actually implements
working rootless containers. This is really exciting. I'll be writing a blog
post soon.

[https://github.com/opencontainers/runc/pull/774](https://github.com/opencontainers/runc/pull/774)

------
kazinator
Harden _this_ container!

[http://regex.info/i/pic/2005-09-06_16:03.28__00002.jpg](http://regex.info/i/pic/2005-09-06_16:03.28__00002.jpg)

Ideas: Cross-linked polymer? Switch to glass? Too heavy. Carbon fiber, epoxy
resin composite?

~~~
peterwwillis
A unix hacker would wrap it tightly in plastic wrap. Once you're done with the
application you can re-use the wrapper.

~~~
kazinator
I'm with you: plus the plastic wrap is a small self-contained tool that sort
of does one job half-well.

------
stcredzero
Has there been a writeup of the history of Container technology? How did these
features find their way into the Linux kernel in the first place? What was the
original motivation? (Sandboxing, I would guess.) Who were the stakeholders
who pushed those features, and why?

It's weird and wonderful that such technology infrastructure arose and such
innovation happens. I'm curious how.

------
fpoling
The article contains the best explanation of Linux capabilities that I have
seen.

Too bad that Docker and other container solutions defaults to rather broad set
of capabilities requiring to use things like

    
    
        --cap-drop=ALL --cap-add=NET_BIND_SERVICE ...
    

with typical container invocations to minimize a chance of container escape.

------
xori
And now I wait for a poor soul to summarize those 100 pages in a paragraph or
two.

~~~
nisa
TL;DR: Pay attention to arcane details or don't use containers for security
related stuff. Also grsecurity.

Honest question: Are illumos zones or FreeBSD jails better designed or is just
nobody looking for kernel bugs there? From a usability perspective both win
(IMHO) against this Linux mess.

This document is really great through.

~~~
dyn
Author of the paper here. Thanks. As for Jails and Illumos, I would be willing
to bet it's people not looking, but I haven't looked, so I can't really say. I
agree it's kind of a mess, but it's getting better!

~~~
Annatar
You would lose that bet so fast your head would be both spinning and smoking:
zones have been hardened and worked on for enterprises since 2006, and in ten
years have had three known vulnerabilities, the last two having already been
fixed in illumos (and not exploitable without being able to be explicitly run
by a user inside of a hypervisor).

As Bryan Catrill has said in one of his talks:

"we walked the trail if tears since our customers were very large companies;
if they had a problem, we had a problem!"

The illumos and smartos mailing lists are hyperactive, with bugs being fixed,
and new functionality added, which even Oracle Solaris doesn't have -- just
subscribe to those two mailing lists and see for yourself. I warn you in
advance: be prepared to be buried under the vortex of e-mails.

~~~
droopybuns
Invoking the Trail of Tears to describe development hardship is, I think,
inappropriate. Wish that guy would be a bit more cautious with his metaphors.

