
How We Built SELinux Support for Kubernetes - twakefield
https://gravitational.com/blog/how-we-built-selinux-support-for-kubernetes-in-gravity-7-dot-0/
======
rwmj
I've written a bunch of SELinux policy, also without being especially expert.
The tooling is definitely getting better, but still there are many annoying
things from the obscure file layout, the giant all-in-one everything-is-a-
global policy[1], to the small annoyances like m4 in the policy files.

Of course you need to buy into the whole premise of SELinux - that in addition
to simple Unix permissions you can use mandatory access control to fine-grain
control everything that a process can do and get access to. Obviously this is
a giant boil-the-ocean project that has consumed many good people for decades.
For SVirt (secure virtualization) this was pretty successful, allowing many
qemu instances to run on a single server, and confining them with SELinux so
each can only access its own files even when they are all running as the same
Unix user.

The linked article is a good intro.

[1] But they're fixing this, yay! I can't find a link now but there's a
project to decentralise SELinux policy. Perhaps it's still an internal-only
Red Hat project.

~~~
yrro
Worth mentioning that SVirt is also used for running containers on
RHEL/Fedora/CentOS etc.

~~~
totony
I don't think that's the case, svirt is specific to libvirt + selinux-virt,
whereas containers use selinux-container which are integrated into docker &
al. Same principle tho

~~~
yrro
Originally all the domains had 'svirt' in the name:

[https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_con...](https://www.projectatomic.io/blog/2016/03/dwalsh_selinux_containers/)

These days containers have their own domains:

[https://www.mankier.com/8/container_selinux](https://www.mankier.com/8/container_selinux)

As you say, the principle is the same.

[edit] container_t is (or was?) an alias for svirt_lxc_net_t:

[https://danwalsh.livejournal.com/79191.html](https://danwalsh.livejournal.com/79191.html)

------
marbu
While the article shows how they build SELinux policy from scrath, they also
mention udica tool[1] which generates SELinux profile based on container
inspection:

# podman inspect my_container_id | udica my_container

I wonder how much would the result be different in their case or how much time
would they save by having the first version of the policy being generated this
way instead of writing it from scratch.

[1] [https://github.com/containers/udica](https://github.com/containers/udica)

------
lima
What's the point of SELinux support in a k8s environment, where services are
already sandboxed by the container runtime?

It's unsafe to run untrusted workloads in the host kernel using only
namespaces - SELinux does not change this, no matter how much Red Hat wants it
to be true. The kernel has a massive attack surface. For untrusted workloads,
you need virtualization like gVisor or Kata/Firecracker which isolates
workloads from the host kernel.

SELinux can mitigate some logic bugs in the container runtime, yes, but it
doesn't make your containers secure.

Also, the title of this submission on HN is inaccurate - Gravitational did not
build SELinux support for k8s. Red Hat did, a long time ago. They added
support for SELinux to their own product, Gravity.

~~~
VWWHFSfQ
Defense in depth. Running untrusted code in a kubernetes cluster is risky even
with the sandbox. SElinux adds another layer of protection

~~~
Spivak
I would go further and say that SELinux (and AppArmor) aren't defense in depth
but actually a necessary part of the tooling to implement sandboxing of
containers.

------
heywire
An off-topic but somewhat related question. What is a good way to whitelist
which applications can access the network on my desktop linux system?
(Currently running Arch, but I distro-hop quite often). Years ago when I had a
MacBook, I used a program called Little Snitch. I see that there is a similar
application for Linux called OpenSnitch, but I’m hoping there is a better way
to utilize the built in features of Linux rather than introducing a 3rd party
kernel module. Are SELinux / AppArmor the answer? Cgroups? Something else?

~~~
totony
Selinux can add labels to network packets which can be matched with nftables &
al. Its a bit complicated tho. Since you need to add and create labels.

You can also match by owner in iptables. So you can have a network restricted
user and sudo to it.

------
tedk-42
SELinux was a good way of securing linux machines with many services sharing
things like file systems and binaries.

k8s solves this in a different way. You can write applications that are
completely stateless when spun up on a k8s node and can be limited to a read
only file system. Your container can be a scratch container with no other
executables binaries.

I don't know what their product is, but i seriously doubt mixing these two
concepts is the right thing to do.

~~~
kevin_nisbet
I'm not sure it's fair to say the advantages of selinux are solely around
sharing services and filesystems on a host. I believe selinux also allows a
more expressive policy on how a running piece of software can interact with
the kernel than say seccomp can, which is probably what you're container
runtime is using to limit kernel access. And unless you're configuring
seccomp, you're probably picking up some default policy designed to work on as
many systems as possible, which sort of violated least privilege.

I'm also not sure I would consider this a totally solved problem within
kubernetes. In supporting gravity, our kubernetes distribution, we used to
hard-disable all privileged containers. This become a common point of
friction, since many pieces of common kubernetes software seemed to just use
privileged containers instead of properly listing and setting needed
capabilities. Although this is a separate issue from selinux, I'm just
pointing out that many kubernetes clusters may not be as isolated between
running processes as expected.

And to answer what the product is, gravity is a kubernetes distribution, that
in this use case allows for packaging a kubernetes based application together
with the runtime, for a unified installer for the application + runtime. Think
something like packaging up a SAAS application, and being able to install a
copy within a banks network who doesn't want their systems talking to the
internet while abstracting away the kubernetes portions.

The problem we were running into, is that end user would keep trying to
install the application onto selinux enabled hosts, which would break the
install or runtime. And whether you believe in selinux or not, it causes
significant friction to try and tell your customers we got this, just turn off
that security feature in production.

~~~
pm90
Given the additional context “we need out k8s deployment to work for customers
who use selinux on their nodes”, the motivation to do this makes more sense.
The majority of k8s users likely don’t share their clusters with untrusted/low
trust users so it seems unlikely to help them much.

Not to dunk on the work you’ve done: it’s still awesome. Just want to point
out the disconnect from users wondering how this would help them.

~~~
jandrewrogers
I think this misrepresents the use case. Most customers that require k8s on
SELinux don't share their clusters with untrusted users. It provides much
stronger isolation between containers in the case that one is compromised,
reducing the scope of damage when a system is successfully attacked.

In other words, they don't assume that a trusted container/user stays
"trusted" operationally.

------
ape4
NSA made SELinux. My theory is that they made it so hard to use so nobody
would use it /s

~~~
LinuxBender
Redhat helped the NSA make SELinux substantially easier for more people to
use. Disclaimer: It's still not super easy to use. The implementation at the
NSA did not have the concept of "unconfined" anything and they use MLS
policies. At least, that was the intent. This may have changed with time and
perhaps by now they are using Redhat's version of SELinux.

------
kapilvt
Tools write selinux policies, humans write apparmor policies .

~~~
VWWHFSfQ
apparmor is dead at this point right. selinux is the standard

~~~
nisa
could you explain why you think so? We are are Debian/Ubuntu shop and it looks
like it's all AppArmor.

~~~
totony
AppArmor is used by debian/ubuntu, while selinux is used by Redhat and
Android. Don't think any of them is more dead than the other. The refpol used
by redhat isn't very restrictive/complete and no one except for big corps are
using it. Apparmor is barely looked at too in my experience.

