
Userland PCI drivers - wean_irdeh
https://wiki.netbsd.org/projects/project/userland_pci/
======
djsumdog
I remember seeing the PCI userspace option in the Linux kernel menuconfig and
wondered why anyone would do that, and then a few years ago at Kiwicon I saw
my first use case. A presenter was trying to hack a Cisco router.

Older Cisco routers ran IOS directly on proprietary hardware. At some point,
Cisco decided to switch to Intel hardware but didn't port their kernel. They
use a Linux kernel and ran IOS as a huge 50MB+ binary. The guy doing the talk
got shell access and only found one ethernet device when running ifconfig. The
actual switching hardware was being handled in userspace by the large binary.

I'm guessing they probably just wrote some shim layers to connect their PCI
drivers up to the userspace PCI Linux API.

~~~
q3k
Well, most of the magic of hardware routers also comes in the form of hardware
acceleration of the actual data plane - ie. L3 switching - on the silicon
itself. That's what makes them fast and so expensive. That sort of mechanism
doesn't map nicely into the Linux interface paradigm (unless you do things
like exporting kernel routes into the hardware, but that's borderline absurd).

I think even if the driver were to be implemented in kernelspace, it would
still probably not expose any of it's physical interfaces to userspace as
plain ethernet devices, maybe apart from virtual/mgmt ones to run SSH on, and
perhaps one so that the kernel can handle packets that the router doesn't have
flows programmed for (like in OpenFlow).

~~~
kijiki
> That sort of mechanism doesn't map nicely into the Linux interface paradigm
> (unless you do things like exporting kernel routes into the hardware, but
> that's borderline absurd).

Not absurd at all. Cumulus (which I cofounded) does exactly that. There are
>1000 customers, including several of the largest cloud operators in the
world.

It works really well in practice, since you can just fall back to the kernel
for non-fast-path stuff like ARP. IOS/NXOS implement ARP (and everything else)
themselves. We can just use the kernel's implementation.

The idea is essentially to use the lightning fast forwarding ASIC as a
hardware accelerator for the networking functionality the kernel already has.

------
kuwze
I really like NetBSD. It supports Xen[0], it has the awesome multithreaded
NPF[1], the CHFS filesystem tailored for SSDs[2], and support for rump
kernels[3].

[0]: [https://wiki.netbsd.org/ports/xen/](https://wiki.netbsd.org/ports/xen/)

[1]:
[https://en.wikipedia.org/wiki/NPF_(firewall)](https://en.wikipedia.org/wiki/NPF_\(firewall\))

[2]: [https://en.wikipedia.org/wiki/CHFS](https://en.wikipedia.org/wiki/CHFS)

[3]: [http://rumpkernel.org/](http://rumpkernel.org/)

------
ttflee
Reminds me of this 34C3 talk:

[https://media.ccc.de/v/34c3-9159-demystifying_network_cards](https://media.ccc.de/v/34c3-9159-demystifying_network_cards)

~~~
ehntoo
I heartily recommend this talk. It and the corresponding proof-of-concept
Intel userland packet processing driver[1] went a long way for me in removing
a lot of the magic from low-level network packet handling and device
management in Linux.

[1] [https://github.com/emmericp/ixy](https://github.com/emmericp/ixy)

------
baruch
NVMe and networking are now used from use space to reduce latency in storage
systems. I'm doing that in my current job. There are libraries to help you
with it such as DPDK and SPDK which are good starting points.

------
snvzz
Liedtke was right.

If it can be done outside the kernel, it shouldn't be in the kernel.

~~~
tinus_hn
That’s a great idea if you don’t care about performance.

~~~
snvzz
Common misconception. I suggest this article.

[https://blog.darknedgy.net/technology/2016/01/01/0/](https://blog.darknedgy.net/technology/2016/01/01/0/)

------
mrpippy
IRIX had a userland interface for PCI access: pciba(7)

I don’t know of anything that used it, but I’m sure there were custom PCI
cards for data acquisition, hardware control, etc etc. that used it.

[http://nixdoc.net/man-pages/irix/man7/pciba.7.html](http://nixdoc.net/man-
pages/irix/man7/pciba.7.html)

------
mey
Isn't the advantage of running hardware drivers in userspace to limit the
attack surface of a driver being exploited?

~~~
nickpsecurity
See Tannenbaum et al's paper on reliability/security mechanisms for a nice
intro:

[http://cs.furman.edu/~chealy/cs75/important%20papers/secure%...](http://cs.furman.edu/~chealy/cs75/important%20papers/secure%20computer-2006a.pdf)

The main benefit is reliability. Driver code is usually lower-quality than
other code that runs in kernels. The hardware itself can act weird in a way
that messed the drivers up. The infamous Blue Screen of Death on Windows was
usually driver errors. Isolating them in their own address space prevents
errors from taking the system down. One might also use safe coding, static
analysis, model-checking, etc when developing drivers themselves. Microsoft
eliminated most of their blue screens with SLAM toolkit for model-checking
drivers. Of the two, isolation with restarts is the easiest given you can use
it on unmodified or lightly-modified drivers in many cases.

Far as security, it really depends on the design of the system and hardware.
The basic, isolation mechanisms like MMU's might restrict the rogue driver
enough if the attack just lets them go for other memory addresses. If it uses
DMA, then they might control the DMA to indirectly hit other memory or even go
for peripheral firmware. If the DMA is restricted, then maybe not. It all
depends as I said on what the hardware offers you plus how the system uses it.

All these possibilities are why high-assurance security pushed in the
1980's-1990's to have formal specifications of every component, hardware and
software, that map every interaction of state or flow of information. That
didn't happen for most mainstream stuff. Without precise models, there's
probably more attacks to come involving drivers interacting with underlying
hardware that's complex. It's why I recommend simple, RISC CPU's with verified
drivers for high-security applications. Quite a few folks from the old guard
even use 8-16-bit microcontrollers with no DMA specifically to reduce these
risks.

Far as verifying drivers, here's a sample of approaches I've seen that weren't
as heavy as something like seL4:

[http://spinroot.com/spin/whatispin.html](http://spinroot.com/spin/whatispin.html)

[https://www.cs.dartmouth.edu/~trdata/reports/TR2004-526.pdf](https://www.cs.dartmouth.edu/~trdata/reports/TR2004-526.pdf)

[https://www.microsoft.com/en-
us/research/blog/p-programming-...](https://www.microsoft.com/en-
us/research/blog/p-programming-language-asynchrony/)

[http://etd.library.vanderbilt.edu/available/etd-11172015-221...](http://etd.library.vanderbilt.edu/available/etd-11172015-221551/unrestricted/YiLi.pdf)

[https://lirias.kuleuven.be/bitstream/123456789/591994/1/phd-...](https://lirias.kuleuven.be/bitstream/123456789/591994/1/phd-
willem-penninckx.pdf)

Note: Including that last one specifically for the I/O verification part.

~~~
tinus_hn
It’s nice if drivers are not running in the kernel but even if your graphics
drivers are running in userspace, if they crash you can’t use your pc anymore.

The main advantage is that you don’t have to deal with all the limitations of
kernel mode programming.

~~~
wean_irdeh
You forget that if the graphic drivers crash in the userspace, it can be
automatically restarted, while if it crash in kernel space, nothing you can do
except hard reboot

------
digi_owl
Reminds me that Intel didn't include PCI on their Atom variants aimed at
mobile devices, supposedly because it was too power hungry.

This in turn lead to Microsoft balking at supporting said hardware as Windows
is deeply reliant on PCI (even the ARM SOCs powering the Windows RT products
support PCI).

In turn Intel developed Moblin, that later merged efforts with Nokia's Maemo
to become Meego. Later still foisted onto the Linux Foundation.

------
Retr0spectrum
I guess monolithic kernels have gone full-circle now.

~~~
digi_owl
Most real life implementations of "microkernels" end up being hybrids. NT
started out as a micro, but Microsoft have been moving things (the graphics
subsystem in particular) in and out of kernel space in the hunt for the
optimal tradeoff between stability and performance.

Similarly i think the Mach kernel powering Apple's OSs are a "fat micro" where
various things that should be in userspace, if one followed the microkernel
orthodoxy, resides in kernel space.

Perhaps the only orthodox microkernel OS out there is QNX, these days
languishing in the bowels of Blackberry's holdings.

~~~
dfox
Mac OS/XNU is in fact derivate of DEC's OSF/1 (later called Tru64 Unix). It
has very weird hybrid design where essentially anything that would be in
monolitic kernel runs as one big Mach process.

Edit: it is somewhat ironic that Alpha's memory protection model is designed
such way that the natural way to implement any OS would be to write your own
microkernel as OS-specific PALcode (something between firmware and microcode,
written in extended Alpha ISA and the only thing that the CPU hardware sees as
privileged code), but none of the Alpha OSes is implemented this way. In OSF/1
you thus get limited microkerne-ish thing that runs two process-ish things,
one of which is Mach kernel and the other currently running Mach task, which
in turn is either the essentially monolithic Unix kernel or Unix userspace
process.

~~~
TickleSteve
Sorry, but MacOS/OSX is a deriviative of NeXTStep, not Tru64.

There is no relationship to Tru64 except that HP did also support OpenSTEP at
one point.

~~~
lukeh
From memory, they did swap out Mach from the version 2.5 used in NS/OS for
version 3 from MkLinux (which was from OSF). This was at the time of the
Rhapsody to OS X transition. My memory is hazy on the BSD kernel mode
component history.

------
nbsd4life
if you are interested in this or other projects, GSoC is now and the deadline
for student applications is 27 March (tomorrow).

------
zombieprocesses
Good to see the redheaded step child of the BSD world finally getting the
limelight. FreeBSD gets all the attention, while OpenBSD gets all the praise.

~~~
feelin_googley
The "redheaded step child" quote reminded me of this:

"BSD is Dying"

[https://www.youtube.com/watch?v=6l1HghEDJf4](https://www.youtube.com/watch?v=6l1HghEDJf4)

As I recall, in that presentation, the redheaded stepchild is _OSX_.

