
Using the IOMMU for Safe and Secure User Space Drivers [pdf] - emmericp
https://www.net.in.tum.de/fileadmin/bibtex/publications/theses/2019-ixy-iommu.pdf
======
drewg123
I actually went straight to one of their references
([https://gianniantichi.github.io/files/papers/pciebench.pdf](https://gianniantichi.github.io/files/papers/pciebench.pdf))
which was an amazing find. I should keep up more with the academic
conferences, I guess.

The most interesting / surprising thing about this paper is how damning it is
for the Intel IOMMU.

~~~
emmericp
Yeah, this paper is amazing; anyone considering using the IOMMU or interested
in performance at the PCIe level should read it.

We can also confirm their result about the IOMMU TLB size of only 64 on the
CPUs we have here. (Yes, the TLB size is completely undocumented and there are
no performance counters...)

------
snvzz
This is best complemented with a multiserver, microkernel architecture. With
an IOMMU, and with near everything including drivers running in userspace, all
of this can be achieved:

[https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies](https://en.wikipedia.org/wiki/MINIX_3#Reliability_policies)

With a monolithic kernel such as Linux, the reliability benefits of running
some drivers in userspace are negligible, as the TCB is still millions of LoCs
with no proofs.

Fuchsia, Genode and HelenOS are relatively active, promising open source
operating systems with a microkernel, multi-server architecture and drivers
running as unprivileged processes.

~~~
0815test
If comprehensive static proofs are in the picture, there's little benefit to
running trusted code in user mode. It's just adding a protection boundary to
something that could already be proven as safe via static analysis. IO-MMU
however could still be useful since it somewhat obviates the need to trust the
device itself, and devices are much harder to characterize properly.

~~~
snvzz
>If comprehensive static proofs are in the picture, there's little benefit to
running trusted code in user mode. It's just adding a protection boundary to
something that could already be proven as safe via static analysis.

Try seeing it the other way around. It minimizes what needs proving (it isn't
cheap) by having the microkernel (small and proven) enforce separation.

Capabilities (not to be confused with POSIX capabilities, which are something
else) do help massively here. Refer to the Genode Book[0] introduction chapter
for an introduction to capabilities.

In the driver example, running the driver unprivileged means the damage a
faulty driver can be contained. In combination with a iommu, it can be made so
that the hardware can only talk to the driver, to complete the protection.
This is only possible if the kernel itself can be trusted, which would mean
the kernel has to be proven, and that's unrealistic unless it's a microkernel,
as it is a very costly process that doesn't even scale linearly with LoC.

The alternative is to make all drivers part of the TCB, which isn't realistic
for the same reason. Unfortunately, on top of this, driver code quality is
known to be particularly bad.

[0] [https://genode.org/](https://genode.org/)

------
sitkack
I love where this work is going and I really enjoyed your ixy project.

I see no mention of Dune [0] in the bibliography. Given the narrow focus of
this research I think it is required.

Once processors are tuned for cloud workloads, they will have the affordances
that will make full blown operating systems optional.

[0]
[http://dune.scs.stanford.edu/belay:dune.pdf](http://dune.scs.stanford.edu/belay:dune.pdf)

------
ilovecaching
This paper doesn't cover BPF or XDP, which is the future of high speed
networking in the kernel. If you need to do user space processing and XDP
can't meet your requirements, you can now use AF_XDP. DPDK was always a way of
circumventing the kernel, now we have kernel approved methods for line speed
network processing.

------
ilovecaching
This paper doesn't cover BPF or XDP, which is the future of high speed
networking in the kernel. If you need to do user space processing and XDP
can't meet your requirements, you can now use AF_XDP. DPDK and other user
space networking was always a way of circumventing the kernel, now we have
kernel approved methods for line speed network processing. XDP programs are
actually safer than using Rust, because XDP programs are statically analyzed
byte code that is guaranteed to finish executing, among other checks.

~~~
emmericp
XDP is completely orthogonal to this work and would not have been a useful
performance comparison. "Using AF_XDP" vs. "using a user space driver with
IOMMU" is just apples vs. oranges.

The goal we are trying to achieve in this project here is to show that drivers
can (and should) be written in better languages to improve security and
safety. Note that one of the drivers in the thesis is written in Rust.

And regarding XDP being safer than Rust: Yes of course. But it's also very
limiting; you can't write a driver in eBPF. (It currently just prohibits jumps
with negative offsets but there's some ongoing work to allow for at least some
bounded loops). We are interesting in making drivers themselves safer, not
applications building on top of them.

(As advisor of the thesis I however agree that XDP should have been mentioned;
I'm not super happy with the length of the thesis but quite happy with the
implementations)

~~~
ilovecaching
XDP is already being used to replace DPDK and IPVS use cases, so I don't see
how its Apples to Oranges. It's generally not a good idea to circumvent the
kernel, and in this case the kernel developers are providing better and better
tooling for high speed networking to the point that user space driver
implementations are no longer attractive compared to using the XDP ingress
hook points. In industry I see rapid adoption of XDP (BPF in general is THE
topic in Linux right now), and it's enabling all sorts of fascinating new use
cases. I would be very interested to see a paper on iommu perf from the XDP
perspective.

~~~
emmericp
This isn't about performance of user space drivers but about safer/better
drivers. Whether that driver offers an XDP interface or something else is
irrelevant. Also, your kernel driver running XDP should also use the IOMMU for
both safety and security (e.g., Thunderclap).

(The thesis features a performance evaluation to show that it doesn't get
slower when used properly: hugepages are absolutely required; this is very
different from non-IOMMU drivers where hugepages only boost performance by
~1-3% in the best case. Also, performance is simple to quantify so it makes
for a great evaluation.)

Some context for this thesis/what I'm working on at the moment:

C as a programming language for drivers has failed from a security-
perspective. Cutler et al. [1] analyzed 65 security bugs found in Linux in
2017 allowing for arbitrary code execution, 40 of them could have been
prevented if the driver was written in a safer language. I've looked at these
40 bugs and found that 39 of them are in drivers.

I disagree with their conclusion that you should therefore consider to write
the whole operating system in Go (that's just unrealistic in the near future).
But we can at least start to write drivers in better languages as they seem to
be the weakest point gaining 97% of the improvement for ~40% of the work.
Getting a high-level language driver upstreamed in Linux is of course
unrealistic, so user space drivers it is.

Network drivers are particularly interesting because user space drivers are
already quite common there and there's a trend towards having network stacks
in the user space anyways: QUIC is user space only, iOS runs a user space TCP
stack. (Somewhat related: the future API between applications and the network
stacks is TAPS instead of sockets, TAPS is also more friendly towards user
space stacks as it's finally a modern abstraction for the stack.)

[1]
[https://www.usenix.org/system/files/osdi18-cutler.pdf](https://www.usenix.org/system/files/osdi18-cutler.pdf)

~~~
0815test
> Getting a high-level language driver upstreamed in Linux is of course
> unrealistic, so user space drivers it is.

There's work being done on source-level translation from Rust code to C
(mrustc is part of this of course, but nowhere near complete), so I'm not sure
why this should be seen as unrealistic.

~~~
jononor
I don't see why using mrustc to generate C code would make Rust drivers more
upstream-able in the Linux kernel? As far as I understand it, the main thing
is that the team (Linus at least) does not want to maintain source code in
multiple languages.

I do like the mrustc effort though.

