
Unprivileged Docker Builds – A Proof of Concept - kiyanwang
https://zwischenzugs.com/2018/04/23/unprivileged-docker-builds-a-proof-of-concept/
======
jacques_chester
I think the key distinction here is unprivileged builds outside a container.
Kaniko was aimed at an actually-quite-large problem, which is performing
unprivileged building inside a container in a shared cluster.

This matters a lot, because otherwise CI/CD is a serious weak point in your
overall security model. _Especially_ if you will run PRs that folks send you
-- ie, run sight-unseen 3rd-party code in a privileged container.

In terms of the OpenShift vs local development experience, we heard something
very similar about Cloud Foundry too, so now there's `cf local`[0].

Essentially, Stephen Levine extracted buildpack staging into standalone
container images (which now use Kaniko) along with a little CLI tool to help
you make the local development experience more like the `cf push` one.

As it happens there is a lot of cooperation happening in this space right now
-- folks from Google, Pivotal and Red Hat have been comparing notes about
container building in the past few months. It's something we share a common
interest in improving.

At Pivotal we have also begun closer cooperation with Heroku on buildpacks,
which is really exciting. The buildpacks model lends itself to some really
cool optimisations around image rebasing[1].

Folks heading to Kubecon this week who are interested in container builds
should make sure to see the joint presentations by Ben Parees (Red Hat), Steve
Speicher (Red Hat) and Matt Moore (Google).

Disclosure: I work for Pivotal.

[0] [https://github.com/cloudfoundry-
incubator/cflocal](https://github.com/cloudfoundry-incubator/cflocal)

[1]
[https://www.youtube.com/watch?v=jsNY4OP3IrE](https://www.youtube.com/watch?v=jsNY4OP3IrE)

~~~
dvlsg
Heh. I wonder if anyone has ever opened a bunch of PRs with bitcoin miners in
them, just to see if they could get them to run on someone else's CI.

~~~
jacques_chester
Yep: [https://motherboard.vice.com/en_us/article/nzkxgm/bitcoin-
mi...](https://motherboard.vice.com/en_us/article/nzkxgm/bitcoin-mining-
github-open-source-bots)

------
scarface74
Not getting root on your own machine as a developer? That's ridiculous. I left
a company where I was the lead developer because I got tired of waiting two
weeks for the outside networking consultants to get resources provisioned on
AWS. Our own internal network people had to wait on them.

Now I work for a much smaller company and they gave me full admin access to
the AWS console from day one and I am "just" a senior developer.

~~~
staticassertion
You know what developers do with root?

They install bitcoin miners.

They install malware.

They install all sorts of shit.

It's unfortunate but it's true, I've seen it 100x.

We let our devs have local admin at my company but please don't misunderstand
the serious risks involved in that decision. We take great measures to ensure
that we can do that while still keeping our customers safe.

We would never allow admin AWS access, that's absurd. An attacker on your box
would be able to own prod. Sorry that it's an inconvenience! When you manage
to solve infosec, hit your ops/sec team up, they'll be happy to hand you the
keys after that point.

~~~
scarface74
Trust but verify. AWS has plenty of logging capabilities. I'm not saying that
_all_ developers should have unfettered access. But someone has to be trusted.

~~~
ntnn
Logging doesn't help you when you business has to shut down because someone
took over you account and deleted everything.

Separation of access is important and _required_. Developers don't need access
to prod, admins maintaining the infrastructure don't need access to the
directory, IDM doesn't need access to either QA or prod.

Developers do need full access in an environment to properly test - but that
environment should be basically hermetically sealed from the rest of the
companies infrastructure. So even if they do screw up the whole business won't
be affected.

~~~
andrewstuart2
If someone took over your account and deleted _everything_ , and you couldn't
get any of it back, you weren't taking care of the "availability" third of
security. I agree that developers don't need access to everything, but I
completely disagree that they should have no access to prod.

The games of phone tag and "try typing this" that happens during prod issues
is a waste of everybody's time, and I fully believe that the people who write
the code should be the ones with both the responsibility of the pagers and the
ability to fix the code they've deployed. Everybody is happier, and the job
gets done more quickly, when the job gets done by the people most qualified to
do it (because they wrote it), and when they bear the consequences of writing
bad code.

The environment needs to be set up to be forgiving of mistakes, yes, but
that's easily done these days and should never result in loss of data if the
infrastructure is properly automated. If giving production access means your
developers can screw something up, then your admins can just as easily screw
something up. Create environments that forgive these failures because they'll
happen one way or another.

~~~
georgebarnett
There are already examples companies which have folded overnight after losing
creds and having everything deleted.

Removing root is not a trust issue - it’s a security surface area issue. You
increase the number of audit points and attack options by at least an order of
magnitude (1 admin : 10 devs).

In a small shop this might be acceptable, however in a large org it’s plain
old insane.

If you believe that devs require root then that’s an indicator that your
build/test/deploy/monitor pipeline is not operating correctly.

~~~
andrewstuart2
> If you believe that devs require root then that’s an indicator that your
> build/test/deploy/monitor pipeline is not operating correctly.

For one, I never said anything about root. I'm not sure _anybody_ should have
root in production, depending on the threat model. What I _am_ saying is that
the people who wrote the proprietary software being operated should be the
ones on the hook for supporting it, and should be given the tools to do so,
since they're the most aware of its quirks, design trade-offs, etc.

That means not just CI/CD and monitoring output, but machine access, network
access, anything that would be necessary to diagnose and rapidly respond to
incidents. That almost never requires root.

~~~
georgebarnett
I completely agree. Very well said.

------
ggm
Maybe its me, but I don't see what the restrictions would be to make an
__image __which had a root FS inside it, without privileges. The "bits" which
make it be a root FS don't need you to "mount" it as a root process and run
chflags, you can dd into the image, change things, do math, change checksums.
Sure, its a lot _harder_ but the principle of what an image is, is: its a
stream of bits. If you can modify the stream of bits, you don't need root to
mark it as a region which has magic root properties be it a chflags FS state,
or a setuid bit of whatever.

Also, again it may just be me, but if you are running a hypervisor limited VM
image on a stream of bits, and you can modify those bits outside that state,
restricting this VM not to have root runtime is slightly odd.

This reads like a proscriptive "no root" rule has been metasploited into "we
will wake you at 3am to check if your dream, you are running as root" type
extremism.

~~~
jacques_chester
It's less about "could I write a program to modify bits directly?" and more
about "what can I get that I won't have to support myself for the rest of
time?".

Nothing stops anyone from writing code to interpret Dockerfiles or to directly
fiddle with image layers. But taking the cost:value ratio proportional to
everything else you need to be doing, it's probably a poor investment of time.

Google has economies of scale around this exact problem, which is why they've
been pumping out work in this area -- Kaniko, Skaffold, image-rebase, FTL,
rules_docker, Jib etc.

~~~
ggm
So his story reduces down to the real problem: what tooling can I find, which
runs without root, but makes images which include root outcomes, for the set
of things I need in an image which can't run un-privileged, and those tools
need to work without running setuid() or seteuid() to root.

Thats a good story btw. I have people working near me who probably want the
same thing from a lower driver, but nonetheless interest in non-root required
builds.

------
verst
Related to this, so pardon my pitch: We [Azure] literally just launched Azure
Container Registry Build. [1]

With one command you can build your container in the cloud and have it stored
in your registry. Either push the source code from local or have it use a
remote git repository.

> $ az acr build --registry $ACR_NAME --image $IMAGE_NAME:$TAG_NAME --context
> .

[1] [https://docs.microsoft.com/azure/container-
registry/containe...](https://docs.microsoft.com/azure/container-
registry/container-registry-tutorial-quick-build)

~~~
dayglo
Yes! now I can ditch my full-fat-vm vsts build agents! Thanks to your team.

------
freedomben
This is interesting, but I'm not sure if I'd stay at a company that refused me
root access to my own machine. A culture where such a lack of trust exists is
not one that sounds attractive to me.

~~~
jcims
Fair enough, but there are cases where allowing that while also meeting the
web of regulatory requirements that beset some industries is difficult if not
practically impossible.

~~~
user5994461
In layman terms, expect banks, defense, government and healthcare to have a
dedicated operation team with root access. Developers do not have it.

~~~
vbezhenar
Can you run VirtualBox and use it as your working machine instead?

~~~
phs318u
If the VM is on the corporate network then it’s the same as connecting an
unmanaged device - defeats the purpose of locking down machines. Developer VMs
should be on their own VLAN.

------
tome
The big unanswered question in my mind is "why on earth does being in the
docker group give root access?". The relevant section from the Docker manual
is

> First of all, only trusted users should be allowed to control your Docker
> daemon. This is a direct consequence of some powerful Docker features.
> Specifically, Docker allows you to share a directory between the Docker host
> and a guest container; and it allows you to do so without limiting the
> access rights of the container. This means that you can start a container
> where the /host directory is the / directory on your host; and the container
> can alter your host filesystem without any restriction.

[https://docs.docker.com/engine/security/security/#control-
gr...](https://docs.docker.com/engine/security/security/#control-groups)

This does not seem like a necessary consequence of setting up a daemon for
building disk images. What am I missing here? Is this an engineering oversight
on the part of Docker or is there some technical reason that forces it to be
like this?

~~~
zenlikethat
Basically docker group exists because it's a lot easier to get people to just
add themselves to that group than to type 'sudo' over and over again. Sad but
true.

It's not mandatory to use the docker group though. That's completely optional
and you could definitely just 'sudo' whenever you do a 'docker' command. The
docker _daemon_ needs to run as root because it needs to be able to do all
kinds of privileged system calls to actually set up the containers. But if
everyone who interacts with the docker daemon is a sudoer, that's not a
problem.

~~~
tome
Thanks for the reply. The docs say "only trusted users should be allowed to
control your Docker daemon". Presumably this means that the Docker daemon can
be coerced into doing all sorts of nasty stuff. Is that right? If so, doesn't
that imply that it's badly written?

------
marcosnils
[https://github.com/genuinetools/img](https://github.com/genuinetools/img)

~~~
peterwwillis
_" Building Container Images Securely on Kubernetes:"_ _" Standalone, daemon-
less, unprivileged Dockerfile and OCI compatible container image builder."_ _"
TLDR; will work unprivileged on your host not in a container til we work out
some kinks."_ [https://blog.jessfraz.com/post/building-container-images-
sec...](https://blog.jessfraz.com/post/building-container-images-securely-on-
kubernetes/)

------
zwischenzug
Author here. I worked at a place that had full root for everyone everywhere
and introduced docker back in 2014. It was fun. Then I moved to finance where
regulatory requirements mean that root is unacceptable. Whatever we think of
that (and I now build openshift clusters at home to test, which I'll blog on
soonish) it is a reality for a lot of people. Don't get me started on trying
to use git on Windows either...

Also, I have no admin access to uat or production envs or codebase. It's a
challenge.

~~~
zwischenzug
PS I work in infra now, no longer as a dev. It's the same for us. The sec
team, on the other hand...

------
fulafel
Unpaid advert: LXD supports unprivileged containers quite well.

~~~
cyphar
Yup. I implemented the rootless containers support in runc and I love working
with the LXC folks when it comes to topics related to low-level container
implementation.

It should be noted though that by default LXC does require some privileged
(suid) helpers (for networking and cgroups) -- though you can disable them as
well. runc by default doesn't, though that's just a matter of defaults and
what use-cases we were targeting.

------
alexellisuk
A PoC was shown of how to do this with buildkit several weeks ago in [0], but
in your words - it's also not for the faint of heart (involving patching the
Kernel). This is the way of the future - Docker image builds should not need
to be privileged (they often are for mounting filesystems)

[https://github.com/genuinetools/img](https://github.com/genuinetools/img) [0]
Similar:
[https://github.com/AkihiroSuda/runrootless](https://github.com/AkihiroSuda/runrootless)

~~~
cyphar
> A PoC was shown of how to do this with buildkit several weeks ago in [0],
> but in your words - it's also not for the faint of heart (involving patching
> the Kernel).

Rootless builds work without kernel patches (the "rawproc" stuff mentioned in
issues is not going to be merged into the kernel and there are other ways of
fixing the issue -- like mounting an empty /proc into the container). I can do
builds right now on my machine with orca-build.

The main reason it's for the faint of heart is that we don't really have nice
wrappers around all of these technologies (runc works perfectly fine, as does
umoci, as does orca-build, as does proot, as does ...). Jess's project is
quite nice because it takes advantage of the cache stuff BuildKit has, though
I personally prefer umoci's way of doing storage and unprivileged building
(though I am quite biased ofc). I'm going to port orca-build to Go and
probably will take some inspiration from "img" as to how to use Docker's
internal Dockerfile parsing library.

For more information you can take a look at
[https://rootlesscontaine.rs/](https://rootlesscontaine.rs/).

------
nebulous1
devs can't get root on their own machine? Is this common?

~~~
spc476
I may depend upon the company. I work at a company what writes software for
smart phones [1] and we don't have such restrictions. Yet our parent company
(we were bought out several years ago) does deal with financial transactions
and has all sorts of regulatory restrictions what can and can't be done with
computers.

Unfortunately, for some obscure reason, it's only _now_ that we might have to
conform to the regulatory compliance issues (no one has been able to answer me
"Why now? Why not when we first got bought years ago?") We're trying to fight
it but the fix seems to be in.

[1] Enhanced caller ID. Part of our software lives on the cell phone. Part
lives on the phone network.

------
arbie
Access to large compute farms in most enterprises requires that users do not
have root access to the nodes. Being able to run Docker containers on the farm
would be fantastic. This is a great article that I will investigate further.
Thanks!

~~~
CaliforniaKarl
Not just on compute farms in enterprises; in academia as well.

~~~
cyphar
I was actually part of a university research group when I discovered the need
for this, and started working on rootless containers as a result -- I gave a
talk about this last year[1] (though quite a few things have changed since
then).

[1]:
[https://www.youtube.com/watch?v=r6EcUyamu94](https://www.youtube.com/watch?v=r6EcUyamu94)

------
_muff1nman_
This is hopefully something that buildah can achieve:
[https://github.com/projectatomic/buildah/issues/171](https://github.com/projectatomic/buildah/issues/171)

------
humbleMouse
What does this approach have that openshift doesn't provide?

~~~
cyphar
Doesn't require running anything as root. Almost all of OpenShift runs as
root.

------
maltalex
According to the author's Linkedin [0] he's "Lead OpenShift Architect at
Barclays"

I get the security risks and the fear of malicious insiders at a company such
as this. But having expensive, fairly high level employees work around not
having root access on their own machines strikes me as odd. The guy does
docker for a living and can't run docker on his own machine.

Also, recall that he has physical access to said machine (I hope) so if he
really was a malicious insider, he could already pretty much own it. Then
again, maybe he doesn't.

[0]: [https://www.linkedin.com/in/ian-
miell-694496/](https://www.linkedin.com/in/ian-miell-694496/)

~~~
ajackfox
Unfortunately I think these types of restrictions are fairly common in large
organizations, even non-financial.

Speaking as someone in a Fortune 500, sometimes the bureaucratic hoops one
must jump through to develop in a VM or with docker on ones own machine isn't
even worth it.

~~~
eat_veggies
At a company I interned at, even getting _Linux_ on a laptop was a multi-week
process. I ended up running a Linux VM on virtualbox on Windows, used cmder to
ssh into it, and used a shared folder to edit code natively while running it
on Linux.

~~~
user5994461
To be fair, it makes no sense for the company to allow any desktop on Linux.
The desktop wouldn't have the centralized authentication, the system updates,
office or the drivers for the printer that are well integrated only on their
windows machines.

~~~
mbrumlow
Did you mean Linux on a desktop?

Umm, you can do all of those things with Linux... I can't remember the last
time I had any issues with printing on Linux at all.

~~~
lifeisstillgood
So take a major financial insititue you probably have an account with -
300,000 employees each of whom has at least one PC age ranged upto a decade.
Including servers and routers they probably have as many machines as AWS, but
in several _thousand_ "data centres" some of which get unplugged by the
cleaners vacuum.

The management headaches for this kind of distributed computing is off the
scale, and banning Linux and locking down senior devs workstations is just
table stakes. Everyone is heading to a thin client running citrix to a data
centre for pretty much this reason

~~~
mbrumlow
None of it has to be complicated, it is because people want to make it
complicated, and make computers some magical box.

Lets firs talk about removing root privileges on personal workstations or
laptops. This is pointless. Anything that might be bad for root to do on a
single user system is going to be just as bad running as a user. The second
you allow any custom code to run as a user on any system you should treat it
as potentially compromised -- adding root in to the mix really does not change
things -- on a single user system. Worried about root user getting access to
some customer data on the system? To bad, if the data was on the system it is
more than likely the user level account (windows or linux for that matter) had
access to the data therefor any intruder will also have access, just at the
user level. The same goes for just about any other issue you can run in to. Am
I suggesting running things as root? No, because there really is no need for
most things -- at the same time if your developer needs to have root level
access so they can test or work with technologies that require it when
deployed to production then it really should be a non issue for sysadmins. The
problem is sysadmins are mostly scared to be outed for doing nothing for most
organizations these days. These sort of power sweeps are often used to justify
big budgets and teams of people who tell you to "reboot" when it does not work
right. There is also a bit if power hungry attitude associated with it too.

You state that Linux makes it harder, I can't see how and you did not show me
anything convincing. Bold statements without any details into facts can just
be tossed into the trash can as far as I am concerned.

Now lets talk about citrix. How does that help? All that does is move any real
or perceived problem to a different system. If any of these VMs get accessed
by bad actors they will still be able to own any of the information on them
that the user had access too.

In any case I did not really come here to argue any of this, your comment is
just sort of out of place with relation to what I said.

If you can't trust your employees don't hire them, or just pay them and tell
them to sit in a dark room so they can't hurt anything.

~~~
lifeisstillgood
hmmmmm. I may have drunk too much corporate kool-aid over the recent years.
You have many valid points.

~~~
user5994461
Your first comment was on point. It's a massive hassle to manage the many
environments that come with a hundred thousand computers.

The last poster has zero argument and is just ignoring the problem. Go setup a
thousand printers for ten thousand employees in a hundred locations. They all
have to work flawlessly and on all OS.

------
tannhaeuser
The zoo of build tools and scripts on top of Docker and "orchestration" tools
sure reaks of incidental complexity. Could someone explain to me what material
problem Docker is solving that couldn't be solved using a statically linked
binary as demon?

~~~
empath75
Great now deploy that demon and updates across an entire fleet of servers a
few times a day. And make sure you have metrics for it, and that you’re
efficiently using cpu and memory resources across your cloud footprint.

~~~
tannhaeuser
Docker doesn't do these things neither for you.

~~~
orf
It does 1 and 2 well, and 3 can be built using it

