
Is it time to rewrite the operating system in Rust? [slides] - masklinn
https://www.slideshare.net/bcantrill/is-it-time-to-rewrite-the-operating-system-in-rust
======
jillesvangurp
I think the question is wrong. People are already writing new operating
systems in Rust just because they can. The question is whether we will end up
using their efforts or whether we keep on going back to the same monolith
kernels we have been using for the last thirty years or so. Whether it is NT,
Linux, or any of the BSD kernels, they each have decades of history behind
them.

These will of course still be around for the foreseeable future (i.e.
decades). At the same time, we are seeing Rust pop up in a lot of places that
used to be the exclusive domain of C. People are already re-implementing
libraries, popular command line tools, etc. A lot of these implementations
have clear merit in the sense that they are faster, safer to use, easier to
scale, maintain, etc.

IMHO it is just a matter of time before vendors start providing e.g. drivers
written in Rust for their hardware. I could see linux evolve to a point where
integrating drivers like that is both possible and common. Once that happens,
it will be a hybrid kernel effectively. Rust is not the only thing moving that
direction; wasm is also going that direction. People are talking about running
that in the kernel. And of course Rust runs on top of that as well. So one
outcome would be a lot of Rust code running in a wasm sandbox on top of legacy
kernels.

~~~
twtw
> wasm is also going that direction

Genuine question: why would WASM in the kernel make any sense at all?

~~~
nicoburns
Security. WASM is sandboxed by design.

~~~
twtw
Many things are "sandboxed by design." Few things are true sandboxes.

I'm very skeptical of the idea of throwing out virtual memory and process
isolation to depend instead only on software sandboxes.

------
tom_mellior
Bit of a mixed bag.

Slide 12:

> Go etc. are garbage collected, making interacting with C either impossible
> or excruciatingly slow.

"Impossible" is, of course, a lie. "Excruciatingly slow" may be the case,
though it depends on the kind of interaction you are looking at. Without any
context, this statement is overly general.

Slide 14 can be taken out of context and misrepresented as showing that "Rust
can be faster than C", although the programs being compared use different data
structures, so it's inconclusive. (It's not clear to me whether the author
wanted to make a broad claim like this. The title suggests so, but again, the
difference in data structures suggests otherwise.)

Slide 16:

> every operating system retains some assembly for reasons of performance

Is this really the case? I would be interested in examples. My impression was
that assembly is needed for hardware interaction for which there are simply no
C constructs available (setting timers, interrupts, system registers, and
whatnot), and possibly highly space-constrained bootloader stuff, but
performance? C compilers are not stupid.

Slide 19:

> in-kernel C tends to be de facto safe

This is a bit of a stretch. I mean, yes, _most_ lines of the Linux kernel have
never been the cause of a CVE. But it's hard to tell which lines are the
critical ones. Properly encapsulating safe/unsafe regions does seem like an
improvement. (My understanding is that Rust's "unsafe" is not properly
encapsulated: You might mess something up that will cause a segfault later on,
in ostensibly "safe" code. Still, it's a step forward.)

~~~
arcticbull
> Slide 12:

IMO the big GC related issue is lack of determinism especially in the
management of external resources. If you need that in a managed language
you're forced into grafting reference counting on top which is particularly
error-prone and unpleasant. Interacting with other languages can be slow, and
bridging uncomfortable.

> Slide 14:

Rust can be faster than C in the general case because of the optimizations
permitted by, among other things, strict aliasing rules. [1] The Rust language
invariants are stronger and therefore the compiler can be more aggressive.

> Slide 16:

IMO you can do all that stuff in C with a combination of volatile pointer I/O
and if necessary compiler intrinsics. I've done a lot of AVR programming and
never dropped down to assembler because something wasn't available in C. I'd
imagine the answer is really neither, just that some people are set in their
ways and view ASM as 'faster' even though in reality LLVM would probably
generate faster code by better taking into account instruction issue and data
dependencies.

> Slide 19:

Unsafe Rust is unsafe because you can't express what you want in safe Rust. To
your point, you can absolutely break things. The implicit contract is when
writing unsafe sections that you are to maintain the invariants of the safe
language at entry and exit of the block.

What makes safe Rust safer than C, though, is all the flagship features. You
can detect data races across threads (and interrupts) statically. You can't
write to and read from data at the same time. You have fully deterministic
dynamic collection of resources. etc.

[1] [https://robert.ocallahan.org/2017/04/rust-optimizations-
that...](https://robert.ocallahan.org/2017/04/rust-optimizations-that-c-cant-
do_5.html)

~~~
jonathanstrange
Any dynamic memory allocation lacks determinism, which is why in high
integrity systems it often forbidden to dynamically allocate memory.

As for Rust can be faster than C, well, people said the same about Ada but it
rarely is the case so I wouldn't hold my breath for Rust either. After all,
those languages use the same backends - GCC in the case of C & Ada, LLVM in
the case of C & Rust. But it's enough to as fast as optimized C and have all
the safety features.

~~~
jcranmer
> After all, those languages use the same backends - GCC in the case of C &
> Ada, LLVM in the case of C & Rust.

The killer flaw of C for performance is the pointer aliasing problem. C only
has the very coarse-grained tools of strict aliasing (often disabled in large
applications because it breaks code!) and restrict (which relies heavily on
manual programmer annotation) to control these aliasing issues. Since Rust has
the rule that you can only have one mutable reference to an object at once
(and read-read aliasing issues don't matter), it can effectively automatically
add in these annotations without relying on programmer annotation.

~~~
arcticbull
I think it's fair to say that "average" Rust code will be faster than
"average" C code due to lack of pointer aliasing issues as the average
developer doesn't care about/understand this enough to make the annotations in
any codebase I've worked in. Can you make both equally fast? I don't see why
not, they're both LLVM.

~~~
setr
Given enough time, effort and smarts, you can probably make any lang as fast
as any other :-)

------
Animats
I'd really like to see an open-source QNX-type microkernel in Rust. Everybody
is re-implementing early Unix in Rust, but that's not a good model. The great
thing about the QNX model is that the tiny kernel barely changes from year to
year, because everything is outside it and not very trusted. So it approaches
being totally debugged.

L4 is too low-level - you have to run another OS on top of it, usually Linux.
QNX offers a POSIX interface.

~~~
telotortium
What is the state of documentation on QNX? Is there sufficient documentation
that it would be possible to write a QNX-compatible kernel from the
documentation (plus access to a QNX machine, which has to be available somehow
outside of industry, right?)?

~~~
Animats
Until Blackberry bought it, it was open source. There were free downloads. Now
it's proprietary. But there's a lot of info out there from the open source
era. The manuals are available.

Here's the original QNX paper.[1]

[1] [https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-
paper92.p...](https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.pdf)

~~~
dman
I am amazed that I cannot find a copy of the source code from the time it was
freely available now.

~~~
Animats
Here's the 2007 press release announcing QNX going open source.[1]

On April 9, 2010, the day the acquisition closed, Research and Motion took the
source code offline, with no notice. All user open source projects on QNX were
abandoned shortly thereafter.[2]

[1]
[http://www.qnx.com/news/pr_2471_1.html](http://www.qnx.com/news/pr_2471_1.html)

[2]
[https://community.qnx.com/sf/sfmain/do/listProjects](https://community.qnx.com/sf/sfmain/do/listProjects)

------
BruceM
It'll be interesting to see how much use Rust gets within Fuchsia atop the
Zircon microkernel.

~~~
steveklabnik
The current state is, a few hundred thousands of lines, and Google hiring more
Rust engineers quite aggressively.

I'm also excited to see how it turns out :)

~~~
agumonkey
Any other high profile company/project using Rust ? (beside mozilla of course)

~~~
steveklabnik
Facebook, Dropbox, cloudflare, fastly, tons of startups you’ve heard of...

------
exabrial
Presentation by the Joyent CTO btw, who hired a lot of the old Sun Solaris
team.

~~~
qaq
Among other things because he is also was part of the old Solaris team?

~~~
exabrial
Hilarious example of why English is a terrible language: I meant for the the
pronoun to refer to Joyent, not [necessarily] the CTO.

~~~
magduf
No, that's an example of someone not knowing the language very well. You
should have written something like, "which is the company that...".

I can't think offhand of any language that's really better here in a sense of
being more efficient, except probably Classical Latin which is horribly
complicated and of course a dead language.

~~~
TomMarius
Czech allows you to specify that with a change of one letter depending on who
you mean (the CTO - který, the company - která) - this depends on different
genders of the two words, but is commonly used.

~~~
magduf
Are they officially calling themselves "Czechia" these days? I saw that on
Google Maps recently, I think. It certainly is shorter and simpler than "Czech
Republic".

~~~
TomMarius
It's the oficial short name (the full name is still Czech Republic), local
people don't like it much but they're coming around. However I wasn't
referring to the country (Czechia/Czech Republic) but to the language (Czech).

------
orbifold
A bit surprising that he doesn't mention any other C++ based kernels such as
[https://github.com/l4ka/pistachio/](https://github.com/l4ka/pistachio/) and
most of the fuchsia code base except for the lk micro kernel. In my opinion
fuchsia points towards where things are going multi-language implementations
(mostly c++) with an interface description language for interfacing
components. From a point of view of postmodern c++, rust has little to no
advantages left. It has a much more mature ecosystem and set of libraries, is
just as fast and can be very safe if used correctly.

~~~
fredmorcos
> From a point of view of postmodern c++, rust has little to no advantages
> left.

My entire adult life I've been writing C, I resisted Rust for a while but took
the plunge when a C++ project came along that was in dire need of being
reworked. I liked it, the language and the compiler and tooling around it
helped me tremendously.

I assume you're aware of all the things that Rust statically guarantees for
you, and I'll also assume you know the difference between language complexity
and language implementation complexity (eg, the compiler or some other
tooling):

I've been skimming through the C++ section of cppreference.com, and oh boy,
there's heaps and heaps of... stuff... just stuff. The interplay between all
that stuff is complex and it's hard to keep everything in mind. Keeping things
in mind is important for correctness, or at least for having a reasonable
amount of confidence in what you're writing. Information locality is also
important for correctness, and C++ lacks both of those. I'll even go further
and say that C++ is a write-only language, like a garbage bin of features and
exceptions to each of those features. Rust has many many many clear advantages
(and some disadvantages) compared to postmodern C++.

~~~
orbifold
Maybe just to elaborate: In contrast to rust, C++ is a standard defined
language, with >3 mature and competitive implementations, two of which (clang
/ microsofts compiler) provide excellent tooling and support for refactoring
(clang in particular) beyond any other language in existence (maybe except
Java). Instead of looking at cppreference.com, maybe consider browsing
[https://github.com/isocpp/CppCoreGuidelines/blob/master/CppC...](https://github.com/isocpp/CppCoreGuidelines/blob/master/CppCoreGuidelines.md)
the Cpp Core Guidelines, this outlines a forward looking vision of how good
c++ code should look like.

A lot of the static guarantees that rust provides can be modelled in C++ as
well, see for example "[https://clang.llvm.org/extra/clang-
tidy/checks/cppcoreguidel...](https://clang.llvm.org/extra/clang-
tidy/checks/cppcoreguidelines-owning-memory.html"). Eventually someone will
write a rust-style borrow checker as a clang analysis pass. Already now it is
possible to express a lot of compile time properties cleanly in c++, with
features such as constexpr lambdas, static_assert etc. Additional features
such as Concepts / meta-classes / modules will improve C++ expressivity even
further.

In short, yes C++ has a lot of stuff and everyone knows that it is hard to get
a grip on all its features and misfeatures. But there has been a continuous
effort to rectify and improve on its short comings. See for example
[https://clang.llvm.org/extra/clang-
tidy/checks/list.html](https://clang.llvm.org/extra/clang-
tidy/checks/list.html) for a list of clang based code transformations that are
able to automatically improve your c++ code.

The way I see it, just because of legacy reasons (I can interface with the
majority of commercial software developed in the last 30 years (CAD, EDA
software, Houdini, Abelton, ...)), tooling reasons (the rust compiler is a
slow hot mess, compared to the state of the art c++ compilers) and momentum
(there is a large incentive to continuously improve c++, because it is the
foundation of a large fraction of commercial software out there) it is
unlikely that rust will outcompete c++ in a significant way in the long run.

To close of with an example the lean theorem prover
[https://github.com/leanprover/lean](https://github.com/leanprover/lean),
implemented in C++ handily beats the 20+ year old Coq prover. One of its key
features is multicore support. If you look into its implementation details,
the equivalent rust code would have to work around a lot of rust "safety"
features, essentially because in many cases it is hard to convince a hardcoded
heuristic like a borrow checker that whatever you are doing is safe. The
static guarantees that rust gives wouldn't help you at all in correctly
implementing some of the very involved algorithms a theorem prover kernel has
to implement, while standing in the way of the implementation being
straightforward. This leaves aside the obvious tooling (you can use Visual
Studio, good profilers and debuggers) and integration (SAT checkers like Z3,
LLVM backend) advantages.

~~~
red75prime
> the lean theorem prover

Which has its share of segmentation faults (around 50 closed issues) and
unknown number of corner cases with silent memory corruption/data races.

------
mepian
The slide about the operating systems of the 1980s ignores Lisp machines.
Genera was not a "research system" and it had good performance. There was also
Xerox Star's OS written in Mesa.

------
jorangreef
I think Zig would be better suited.

See: [https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-
Al...](https://github.com/ziglang/zig/wiki/Why-Zig-When-There-is-Already-
CPP%2C-D%2C-and-Rust%3F)

~~~
ansible
I took a brief look at Zig.

The top slogan for Rust is "fearless concurrency", which ties back into the
memory model, mutability and sharing references. This helps you prevent (at
compile-time) data races and other interesting bugs.

I didn't see any support for that in Zig at all.

~~~
jungler
Last I checked Zig is more concerned about overflow conditions than
Rust(numeric overflow, stack overflow, OOM etc.) and boasts features in that
vein - so it competes, but not in the same territory.

If you are writing a kernel it's highly likely that you'll end up in a lot of
scenarios where you have to use unsafe Rust, in which case Zig would be the
better language for the task. Admittedly, it would be nice to have some of
both, and they both have room to get better. But I see Rust as more of a
"libraries and applications" centric setup, where it's the right thing for a
web browser and the wrong thing for a device driver.

------
EatYourGreens
These slides are quite high-level, so I've got to ask: what's the expected
benefit of using Rust to implement a kernel? I somehow thought that nearly all
interesting concurrency there would not fit into the paradigm of exclusive
ownership. And if we are preaching for programming in unsafe Rust, then
doesn't the message become less compelling?

~~~
eximius
The purported benefit would be to wrap all of the unsafe bits in safe wrappers
and then have a less bug prone, more secure OS, I suppose.

This would be, to put it mildly, quite difficult. Even then, unsafe code is
sometimes not written correctly which brings the whole thing down.

I, personally, still think it's worth it. I think efforts like Redox OS can
teach us a lot about what we're doing and offer a chance to collapse some of
the layers of cruft existing OSes have accumulated.

~~~
OoooooooO
Having only a small part of the code with unsafe means you only have to check
only a small % of the code for UB/security holes and not the whole code base
like in C.

It just limits the places shit can happen and which need to be closely
reviewed which alone is a big help.

~~~
cyphar
It should be noted that you can have correctness bugs can occur in safe code
if a Rust guarantee was violated in an unsafe block. This might seem obvious,
but it does mean that when you hit one of these bugs you might start with
debugging safe code and thus it's not _as_ clean a separation as some Rust
evangelists might imply.

------
twoodfin
Is there a video of this presentation? bcantrill’s always a fun / interesting
watch.

~~~
justincormack
There will be it was recorded. From Qcon SF this week.

------
mitchtbaum
Another important and related question to ask is what role Lua should play in
writing drivers and applications on top of a new kernel. NetBSD started on
this direction, but exactly what an operating system looks like when it
leverages both systems-level and scripting-level programmability waits to be
seen.

[https://www.netbsd.org/~lneto/dls14.pdf](https://www.netbsd.org/~lneto/dls14.pdf)

[https://www.lua.org/wshop13/Cormack.pdf](https://www.lua.org/wshop13/Cormack.pdf)

~~~
z3t4
I played with the idea of setting up a runtime for a scripting language such
as JS, then I was thinking, what is a OS anyway ? If it can be written in Rust
or any other high level language, it can also be written in a scripting
language !...?

------
stefs
i think there is a case for a rewrite, apart from the memory safety
guarantees. two things come to my mind:

1\. elimination of backwards compatibility. this is true for all new operating
systems, independently of the language. clean slate and throw away all the
baggage that was reasonable 20 years ago, doesn't hold true today anymore but
still is baked into the old architecture. this might be less relevant for all-
purpose OSes, but might be a viable option for specialized systems (IoT,
Network Appliances, ...). you remember what linus says when a kernel patch
breaks buggy userland code? we now get a second chance to implement a new
system that avoids whole classes for buggy code (while the underlying problem
doesn't go away, it may get greatly reduced).

2\. reduction of complexity. this _is_ a rust thing, and the story to back
this up is [stylo]([https://blog.rust-lang.org/2017/11/14/Fearless-
Concurrency-I...](https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-
In-Firefox-Quantum.html)), the parallel css engine used in firefox. i remember
reading that google tried to implement concurrent styling in chrome but
failed, because it got too complex in c++. human intellect is pretty much
finite and doesn't scale well; we rely on better tools. so, rust might enable
us to do things that were considered impossible before.

the emergence of such an OS would be gradual of course, niche at first, then
slowly growing until one day it's the new de-facto standard.

lets say, game engines. you rarely write your own game engine, because that's
too complex and ties up all the resources you need to actually implement your
game. so you buy one and it's in C++, of course. because they all are, and
thus your developers are fluent in C++. there are no game engines in rust and
few developers who know the language. but over the years, more and more game
devs will try rust in their spare time and like it and write game engines in
rust for their side or indie projects, where it's still feasible. a few of
those will grow and get more features and tooling and at _some point_,
suddenly, rust will be a viable alternative. even though rusts strengths
aren't really that important in game engines. performance happens on the
graphic cards anyway. security is not as big of a deal. parallelism is quite
constrained anyway. etc, etc. but if all else is equal, rust might be more
productive.

so, is it time to rewrite the OS in rust? sure, why not. it's just that we
wont all switch to redox overnight.

------
Leace
I'm surprised there are no mentions to Microsoft Research's Singularity, that
used only software barriers, and the languages used there (Sing#, Spec#) had
ownership concepts ([Claims] attribute).

------
discoball
Nim is realtime garbage collected and compiles to optimized C and it can use
the C ecosystem of libraries and tooling. It also produces smaller binaries
than Rust and is faster for most tasks.

~~~
m0th87
Despite what the Nim documentation says, realtime and garbage collection are
mutually exclusive (unless you count non-deferred reference counting and give
up on collecting cycles.) Setting maximum pause times doesn't change the fact
that the garbage will be collected nondeterministically, which is a problem
for kernels.

~~~
lucozade
Those are 3 orthogonal(ish) things.

Realtime just means that actions triggered by event occur in an explicitly
bounded time. It doesn't usually mean that they occur in the same time. So a
GC with bounded pause _could_ be used in a realtime system assuming you could
guarantee the explicit bounds.

I can't comment on whether or not this is true of Nim and pause time probably
isn't sufficient (depends on how it's defined) but they're not mutually
exclusive.

And most kernels don't need to be realtime unless they are realtime kernels.

Having said that, of course, the vast majority of GC systems are most
definitely not realtime, or anywhere close. But that's just actually true, not
theoretically true.

~~~
m0th87
> So a GC with bounded pause could be used in a realtime system assuming you
> could guarantee the explicit bounds.

You _can 't_ guarantee explicit bounds on when a given resource will be
collected with GC. Bounded times can provide upper bounds on GC pass runtime,
but they do that by potentially deferring further collection. Those deferrals
make it nondeterministic.

> And most kernels don't need to be realtime unless they are realtime kernels.

That's not true. Most (all?) kernels have internal realtime needs, e.g.
responding to interrupts.

There is a way to write a kernel with a GC'd language, as Niklaus Wirth has
demonstrated, by circumventing the GC where needed. But the critique I was
bringing up was simply that there's no such thing as a realtime GC.

------
adamnemecek
Rust allows for new architectures. Eg beos style os would be a match made in
heaven as rust really shines when it comes to async code.

Also maybe you could have a less complicated memory manager as you can deal
with allocations statically.

~~~
qaq
"rust really shines" they need to land async/await

~~~
steveklabnik
We really do. It’s in nightly; likely to hit stable early next year. One of
its precursors is being stabilized as we speak.

------
Animats
_in-kernel C tends to be de facto safe_

Huh?

------
jbb67
no.

~~~
keir-rex
I was expecting to see this type of response. I don't program at low levels
but it seems like this would be the responding answer. Kernel programmers
would be using something else if that's what they wanted to use. Decades of
experience and hard learnt lessons down the drain.

~~~
keir-rex
Also, if you can abstract away at a sufficiently low level why not just do it
in C where there's already a generation of experience and familiarity?

~~~
RhodesianHunter
Sure, why not just continue writing C for the rest of time?

------
eeZah7Ux
NO, and I wish HN stopped upvoting this sort of propaganda. There are hundreds
of languages around and the slides did not even attempt to compare pros and
cons across languages that would be suitable for kernels.

------
otabdeveloper1
> Is it time to rewrite the operating system in Rust?

No. Next question, please.

~~~
albru123
How constructive of you...

------
bryanrasmussen
[http://www.erights.org/elang/intro/index.html](http://www.erights.org/elang/intro/index.html)

~~~
tux1968
Hadn't heard about this before, and it does look interesting. But not sure why
you're linking to it in this context without comment? Especially since it
seems to be a moribund project.

~~~
bryanrasmussen
E had a pretty interesting security model and there were in various E writings
expressions of pipe dreams that operating systems should be rewritten using
it. I should probably have linked to one of those to make the connection more
explicit, but really it was just a momentary bit of snarkiness that didn't
deserve more than what I gave it.

