
The Jury Is In: Monolithic OS Design Is Flawed [pdf] - ingve
http://ts.data61.csiro.au/publications/csiro_full_text/Biggs_LH_18.pdf
======
pweissbrod
I wonder if there exists a parallel dimension where linux is microkernel
design and folks are pushing for monolothic citing the driver friendliness and
performance

~~~
andrewstuart2
If parallel dimensions exist, then it's most certainly one of the closest
dimensions to ours.

If I'm sure of one thing, it's that as soon as we decided to build everything
as Microkernels, we'd have the same squeaky wheels touting the massive
benefits of Monotlithic OS design. We're hilariously cyclical in our
preferences.

"Think of all the runtime efficiencies of the shared memory, and how much
easier it would be to develop on the shared codebase!"

~~~
KMag
The dirty secret is that we take a microkernel, rename the syscalls as
"upcalls", throw in some hardware emulation code as a less efficient and
uglier API for software that can't be bothered to be ported, and call it a
hypervisor.

Also, your phone's baseband processor almost certainly runs a microkernel, and
likely so does your car.

~~~
pjmlp
And what many seem to be unaware, is that project Treble made Android Linux
into a microkernel where drivers use Android IPC to talk with the kernel.

[https://source.android.com/devices/architecture/kernel/modul...](https://source.android.com/devices/architecture/kernel/modular-
kernels)

[https://source.android.com/devices/architecture/hidl/](https://source.android.com/devices/architecture/hidl/)

So everyone running Android Oreo or newer on their phones, not only has a
microkernel on their basebase radio, they also have a Linux tamed into a
microkernel on their main ARM SOC.

~~~
ysleepy
I don't see a microkernel architecture here. Requiring drivers to be loadable
modules is just arriving in the 1990s of monolithic kernels. Also, only new
devices with SoCs introduced for Android 9 are required to use kernel modules
and kernels newer than 3.18. [1]

This means most (updated) Android 8 devices in the field are not Project
Treble and are running old-style module-less kernels and don't have A/B system
partitions.

HALs only seem to be there to abstract over the APIs of different hardware
implementations and access to them. Not necessarily their drivers.

I would love to be wrong about this, so I would really appreciate evidence to
the contrary here.

[1]
[https://source.android.com/devices/architecture/kernel/modul...](https://source.android.com/devices/architecture/kernel/modular-
kernels#kernel-version)

~~~
pjmlp
Because you just stopped at the kernel modules part and haven't spent time
reading how HAL in Android actually works, starting at the HIDL link.

There are two HALs in Android.

The old HAL, previous to Treble, which was hardly used by OEMs.

And the new HAL, which is enforced by Treble.

On the new HAL, drivers are implemented as Android services using Android IPC
via interfaces defined in HIDL, or by using the new shared memory APIs
introduced in Oreo as well.

[https://source.android.com/devices/architecture/hidl/service...](https://source.android.com/devices/architecture/hidl/services)

[https://source.android.com/devices/architecture/hidl/binder-...](https://source.android.com/devices/architecture/hidl/binder-
ipc)

[https://source.android.com/devices/architecture/hidl/memoryb...](https://source.android.com/devices/architecture/hidl/memoryblock)

[https://source.android.com/devices/architecture/configstore/](https://source.android.com/devices/architecture/configstore/)

~~~
scintill76
I'll admit to not being steeped in the terminology, so feel free to educate
me.. but some quick Googling suggests nobody else has called Treble a
"microkernel." There are hits about Fuchsia (a totally separate OS), and some
Android forks on microkernels.

I'm guessing that the Treble modules expose the traditional /dev and /sys
interfaces like before, and these new HALs talk to devices through those,
right? Is that not still a runtime-linked monolith, with fairly-thin userspace
services between the kernel and the framework?

The bulk of driver code logic (e.g. interfacing with a touchscreen controller)
is probably still running in the kernel - show me if I'm wrong. I would think
this would be how you define a microkernel. One exception may be the GPU --
those drivers are often very thin in the kernel because of GPL, with a fat
userspace library that they can legally keep closed-source.

~~~
pjmlp
They use Android IPC, which is a kind of RPC between processes.

[https://source.android.com/devices/architecture/hal-
types](https://source.android.com/devices/architecture/hal-types)

------
jacquesm
The jury was in in the mid 1990's, but Linus Torvalds doesn't know when he's
wrong and to listen to his betters. Linux succeeded because of its community,
_not_ because of its architecture. QnX has shown the strength of microkernels
for decades, they are far more stable and much easier to work on than
monoliths. The (small) speed penalty should be well worth the price of
admission.

~~~
iamcreasy
> QnX has shown the strength of microkernels for decades

Could you elaborate on this one please?

~~~
jacquesm
QnX is so common that if you removed it from the industrial world the world
would _literally_ grind to a halt. Machinery and vehicles would stop moving,
factories would stop producing, chemical plants would (hopefully!) shut down
and airplanes would no longer take off (or would have a much harder time
trying to land without the usual guidance systems), boats would drift and
messages would stop being sent through many systems that you probably would
never have heard of.

For one example of such a system: A computerized way to communicate the
availability of cargo and space to brokers all over the world, on a good
Monday morning several million such messages are sent with a guaranteed
maximum time between the first and the last such message sent (to ensure a
fair market).

QnX is extremely pervasive and it - and other OS's like it - are so reliable
that people tend to forget the systems it powers.

Blackberry made a pretty smart move with their acquisition of QnX, pity that
it did not end with QnX being open sourced, that would have been very nice.

~~~
alexandercrohde
Just as a casual reader of this comment, a few tips:

\- The rhetoric about the world grinding to a halt isn't really pertinent and
makes you look unreliable. Also, no, not "literally." I think what you're
getting at is that it's chosen for high-reliability systems, with the
implication being that it's chosen for its superior reliability(?)

\- But I say this because it sounds like you have some knowledge as to how
this obscure microkernal is used and sharing that is valuable to the
community. It's just a shame if the message gets lost in the delivery.

~~~
jacquesm
The world really would literally grind to a halt, you are severely under-
estimating how common this particular OS is. It has been the embedded system
of choice for a very large variety of applications, it is anything but obscure
it is just obscure to those who spend their time working on web stuff but in
industry you would come across it (and RTOS:
[https://rtos.com/](https://rtos.com/)) very often.

Neither of those names will ring a bell for people who spend their days with
your usual web toolsets.

~~~
nickpsecurity
It really wouldnt. They'd keep using the OS and tools to sustain their systems
until they got a replacement vendor supporting QNX or replaced the systems
with non-QNX software. They might also sue original supplier.

So, rather grind to a halt, it would all continue running in legacy mode,
users would spend more money, and it gets supported or replaced. Lots of
precedents for that sort of thing.

Between acquisition price and QNX's revenues, it's unlikely support will be
terminated any time soon. We wont get to see the hypothetical tested unless
RIM bankrupts with nobody acquiring and supporting their assets. That a losing
company acquired QNX did worry me a bit, though.

------
enitihas
Reminds me of the famous Torvalds Tanenbaum debate.
[https://groups.google.com/forum/m/#!topic/comp.os.minix/wlhw...](https://groups.google.com/forum/m/#!topic/comp.os.minix/wlhw16QWltI)

~~~
thsowers
Thanks so much for the share, great gems in here, I still found myself
surprised at this:

> True, linux is monolithic, and I agree that microkernels are nicer.

It's hard to believe that he was only 23 when he wrote this

~~~
tinco
I think thinking about operating systems is precisely the sort of thing 23
year old college students do. I know a few in my class did, I was hyped up by
MS's Singularity OS back then.

Unfortunately they were not aware yet that Windows was going to lose the
Server OS war and they never pulled Singularity out of research.

Operating Systems is a 2nd year course in most universities.

------
acd
There is a reason why kernel code run in privileged mode, speed! If you run
more kernel code in privileged mode then you do not need to copy as much data
between the kernel and user space. Vs a micro kernel you will have to copy
more data up to user space. Copying data to user space causes context switches
and gives less performance.

Larger mono kernels: Speed

Micro kernels have advantages such as: smaller privileged attack surface and
thus more secure, more crash proof as you can restart user land processes for
example device drivers

[https://en.wikipedia.org/wiki/Microkernel](https://en.wikipedia.org/wiki/Microkernel)

~~~
pradn
Do you really need to do a full copy? What if you had a shared page and
notified the user process when the data in the page was available at a certain
offset.

Take a look at this paper (FlexSC: Flexible System Call Scheduling with
Exception-Less System Calls):

[https://www.usenix.org/legacy/event/osdi10/tech/full_papers/...](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Soares.pdf)

~~~
tlb
Sharing pages between user processes and kernel is extremely bug-prone,
because threads in the user process can mutate the data while the kernel is
reading it, leading to all kinds of race conditions. You can't depend on user
processes respecting mutexes for security.

You can make this work by removing the page from the user process before
making it available to the kernel, but the synchronization overhead of doing
this (especially on a NUMA system) is probably worse than copying moderate
amounts of data.

~~~
pradn
Hmm. Perhaps temporarily switching the TLB entry to read-only mode for that
page during kernel accesses? There might be some interesting software-hardware
co-design solutions here.

------
imglorp
The title is missing "from a security standpoint". Of course, everything is a
tradeoff. TLDR:

> We have presented what is, to the best of our knowledge, the first
> quantitative empirical assessment of the security implications of operating
> system structure, i.e. monolithic vs microkernel-based design.

> Our results provide very strong evidence that operating- system structure
> has a strong effect on security. 96% of crit- ical Linux exploits would not
> reach critical severity in a microkernel-based system, 57% would be reduced
> to low severity, the majority of which would be eliminated alto- gether if
> the system was based on a verified microkernel. Even without verification, a
> microkernel-based design alone would completely prevent 29% of exploits.

> Given the limited number of documented exploits, we have to assume our
> results to have a statistical uncertainty of about nine percentage points.
> Taking this into account, the results remain strong. The conclusion is
> inevitable:

> From the security point of view, the monolithic OS design is flawed and a
> root cause of the majority of compromises. It is time for the world to move
> to an OS structure appropriate for 21st century security requirements

~~~
gjm11
So, they've looked at a sample of exploits that _were_ critical on Linux and
established that most _wouldn 't_ have been critical on a hypothetical
otherwise-similar microkernel system.

But they haven't looked at a sample of exploits that were critical on an
actual microkernel OS and seen how many would have been less serious (or not
arisen) on a hypothetical otherwise-similar monolithic-kernel system.

It reminds me of a nice observation in "Surely you're joking, Mr Feynman".
Feynman developed some nonstandard ways of solving mathematical problems.
Other people came to him and he repeatedly solved problems they'd been stuck
on. "He must be much smarter than us!" But the problems they brought to him
were selected as ones they couldn't do, so of course he'd look better than
them on _those_. They never bothered asking him problems he couldn't do but
they could, because they'd already done them.

Now, maybe the authors of the paper are confident that there's no way that a
microkernel design could encourage or exacerbate vulnerabilities. But so far
as I can see they don't offer any actual argument for that proposition.

~~~
nickpsecurity
"But they haven't looked at a sample of exploits that were critical on an
actual microkernel OS and seen how many would have been less serious (or not
arisen) on a hypothetical otherwise-similar monolithic-kernel system."

Cuz they don't exist so far. Microkernel-based systems will have same kinds of
bugs as monoliths if coded in same language for same hardware. From there, the
microkernel architecture leads less bugs in number (less code), less in
severity (more isolation), and sometimes less difficulty in patching or
recovery. If looking for microkernel-based bugs, I'd look for errors in
concurrency and passing data over IPC. Monolithic system are using more
concurrency and middleware than ever now, though. Even more than microkernels
from what I see if talking about all the strategies and their implementations
vs a few, standardized primitives. So, even those areas central to microkernel
design seem like problems shared with modern monoliths.

So, the status quo is that the monoliths mostly add problems and increase
their severity. Vice versa, the microkernels mostly subtract them in number
and/or severity. The field evidence shows this with most of the data on bugs
and vulnerability coming from monolith users. From there, someone might want
to try to see if the opposite is true. Burden of proof is on them, though,
with the status quo being quite reasonable. And that investigation, as I said,
might find "microkernel" problems that hold in how modern monoliths are used
(esp service and web architectures). Still worth attempting since they might
surprise us with what they find. :)

To be clear, that's all about the architectural patterns. I think combining
all the potential benefits of microkernels in a system vs a simple monolith
could lead to more bugs in microkernel. Most of the problems in software will
come from complexity and QA level regardless of architectural style. So, my
post is written with assumption that we're talking about large, complex
systems done with one style or the other.

------
nasoieu
That chart with the growth of the Linux kernel discredits everything. The
Linux kernel continues to grow because they are obsessed with keeping all
drivers in mainline instead of having a stable API for them as any sane
project would.

~~~
rhencke
I believe one of the reasons Linux continues to be successful today is
_because_ they keep the drivers in mainline _without_ a stable API.

It is a key motivator to ensure drivers remain available and supported well
into the future of Linux.

~~~
pjmlp
Which given the way legacy AMD drivers story has been handled on Ubuntu, it is
quite far from actually being so.

My Asus Netbook no longer gets all the acceleration options that it used to
have pre 16.04.

Linux is successful because it is a free beer UNIX clone.

~~~
monocasa
You still have the source to the old drivers. It's just not a priority for
others to support anymore, but you have the means to go add that support
yourself, or pay a third party to do it for you.

Contrast with when windows changed the video driver architecture in Vista, and
the only option you had was to pound sand.

~~~
bunderbunder
Seems like a pretty easy cost/benefit analysis to me:

I can either spend thousands of dollars worth of my time on updating the
driver, or I can pay someone else thousands of dollars to do it. Or I can
spend a fraction of that on a new video card.

Seems like the only solution for Vista users is the only solution I'd ever
have gone with, anyway.

~~~
monocasa
Sure the cost/benefit works out the same for you (which is why no one has done
the work), but that's different from the full set of options available to you
in both cases.

~~~
bunderbunder
How expansive do we want to be about what the full set of options is, though?

I tend to fall on the side of the root poster - if having a stable driver API
means Linux gets more commercial driver support, then that's the one that has
the highest cost/benefit in my book.

In practical terms, being able to hack on open-source drivers isn't
particularly useful to me, but having an easier user experience with Linux on
the desktop (and, more generally, having Linux on the desktop be something
more than an also-ran) would benefit me _immensely_.

------
gnufx
Contrary to what's written about microkernel speed: Systems I used in the '80s
to great effect used at least the moral equivalent of a microkernel
("Nucleus"). They were fast (compared with VAXen etc.) for interactive use and
supported real-time processes. (Some visitors thought context switching was a
bit slow, assuming "microseconds" meant "milliseconds".) The filesystem was
fast enough to dispel the assumption a "database" was always required for
speedy experimental data access rather than a file per spectrum.

[https://en.wikipedia.org/wiki/OS4000](https://en.wikipedia.org/wiki/OS4000)

The performance wasn't just because Nucleus initially was in hard/firmware;
two later software implementations were performant (on faster hardware). Also,
as the article is about security: at least the original Nucleus also supported
an A1/B3-level secure OS.

------
mikkergp
Well, the obvious solution is to design our Kernel's on Kubernetes.

~~~
steveklabnik
[https://www.qubes-os.org/](https://www.qubes-os.org/)

~~~
snug
A reasonably secure os?

I think that's one thing I don't want just reasonable

~~~
steveklabnik
It's kinda self-deprecating humor; the (now ex) Tor developers I know use it.
Nothing is ever 100% secure.

~~~
yjftsjthsd-h
It's like the lesswrong folks; step one was accepting that there are too many
flaws to fix everything, and then step two was to do our best anyways.

~~~
qu4z-2
Also mutt, with "All mail clients suck. This one just sucks less."

------
nwmcsween
IMO a microkernel isn't a design worth pursuing as there will always be
overhead. Instead an exokernel with a simple monolithic 'multiplexing' kernel
or a language that has 100% safety (not really possible).

------
77pt77
Still waiting for GNU hurd.

~~~
xj9
hurd isn't even interesting. seL4 is faster, has production applications, and
is formally verified. you can even use seL4 as a base platform in genode[0].

[0]:
[https://genode.org/documentation/platforms/index](https://genode.org/documentation/platforms/index)

------
swiley
In a perfect wrold microkernel OSes would be perfect but then it's all
pointless.

In real life there are certain parts of the OS that _have_ to work or the
whole device stops working. Furthermore: the isolation of dynamic and less
tested application code from these parts is generally a good idea, that's why
monolithic OSes are so popular; they're simply less demanding.

~~~
ysleepy
Nobody claims faults of core functions must be survivable in microkernel
designs. The restarting of driver processes is a nice trick, but in the end
the isolation and clean interfacing are particularly useful in an imperfect
world.

The goal is to minimize the trusted computing base (TCB) so that it is, at
least to some degree, verifiably correct. Then tack on features using isolated
components.

------
Sporktacular
Some super low quality commentary here. Straight to the Monolithic/Microkernal
tribalism, talk of parallel dimensions, as long as we ignore the content of
the paper - which remains far more empirical and convincing than any rebuttal
seen here.

------
nine_k
I'd hazard to say that _every_ design is "flawed" in some regards: there's no
way to achieve all desirable qualities and none of undesirable qualities. For
one, some desirable qualities contradict each other.

So "${thing} is flawed" is not precise enough; an interesting statement would
be "${thing} is not the best choice for ${conditions}". A monolithic OS is not
the best choice for a high-reliability system on unreliable hardware. A
microkernel OS that widely uses hardware memory protection is not the best
choice for a controller with 4KB of RAM. A unikernel setup is not the best
choice for a desktop system where the user is expected to constantly install
new software. Etc, etc.

In other words, the ancient concept of "right tool for the job" still applies.

~~~
amelius
> In other words, the ancient concept of "right tool for the job" still
> applies.

Or use a multi-tool ;)

~~~
SteveNuts
The trade-off with a multi-tool is additional complexity, more difficult
maintenance, it's heavier, takes up more space, etc.

------
renox
OK so seL4 is safer than Linux, that's not really news.. I have questions
about seL4 though: is it able to manage several multicore CPU efficiently?
What about power management, does it work?

~~~
pjmlp
Given that it is responsible for powering many embedded systems deployed into
production, yes.

~~~
monocasa
They only added multicore support in the past year, and it contains no power
management code beyond sleeping in the idle loop.

~~~
pjmlp
So I wonder how do those devices manage themselves.

~~~
monocasa
They dont. You don't need power management for a lot of embedded systems that
are high assurance. Power is generally plentiful in high assurance situations
for a lot of reasons.

------
matachuan
Of course it's about Gernot Heiser's verified ukernel...

------
justicezyx
Now it's a good time to revive microkernel for prime as server less offers the
abstraction for making transition transparent to users and application
developers.

Anyone tried that?

~~~
nostrademons
Much of the cloud/virtualization industry is actually built on exokernels; a
hypervisor is basically the commercially-viable version of an exokernel. The
unikernel movement is an attempt to revive the LibOS and use exokernels as
they were originally designed (i.e. move OS abstractions into user-space and
not have a whole OS running on top of an OS), but there've been several
pragmatic barriers that've kept them from gaining all that much adoption.

~~~
justicezyx
I think I am mostly want to say to the advocate of a certain on machine first-
level software infrastructure above hardware to feel more free to try to prove
their technology is the best.

I am not too much about particular alternatives.

------
brewski
This is like saying Linux is more secure than Windows because not all of
Window's critical vulnerabilities appear in Linux.

~~~
Jweb_Guru
What classes of vulnerabilities are you referring to that are likely to show
up in a microkernel-based design but not in a monolithic one? Do you think
adding those classes of vulnerabilities to the picture will result in more,
about the same, or fewer critical security issues between the systems overall,
and why? Please be specific.

------
throw7
The Jury Is In: Microkernel OS Design Is Flawed.

See I can make clickbait titled papers also.

~~~
Jweb_Guru
Great, now all you have to do is build a secure (preferably formally verified)
monolithic kernel, exhaustively, thoroughly research all the vulnerabilities
in popular micokernels, find evidence to support your thesis, write up the
paper, and get it peer reviewed, and you're all set!

------
nkkollaw
Linux is the most popular OS in the world (of course, counting Android), so I
guess this means it doesn't really matter is monolithic is flawed, or it is
not flawed enough.

~~~
pjmlp
Sure it matters, because Android Oreo has actually forked Linux into a
microkernel like design, where drivers run as separate processes and use
Android IPC to talk with the kernel.

~~~
Symmetry
And Fuchsia, if it ends up replacing Android, is a microkernel design from the
bottom up.

------
0xdeadbeefbabe
> We find that almost all exploits are at least mitigated to less than
> critical severity, and 40% completely eliminated

This probably sounds trollish, but security isn't _that_ important. An OS
architecture has to take into account other things too.

~~~
guy98238710
It's kind of pointless to have a faster server at the cost of seeing that
server owned by someone else.

