Hacker News new | comments | show | ask | jobs | submit login
Is it time to rewrite the operating system in Rust? [slides] (slideshare.net)
149 points by masklinn 6 days ago | hide | past | web | favorite | 139 comments





I think the question is wrong. People are already writing new operating systems in Rust just because they can. The question is whether we will end up using their efforts or whether we keep on going back to the same monolith kernels we have been using for the last thirty years or so. Whether it is NT, Linux, or any of the BSD kernels, they each have decades of history behind them.

These will of course still be around for the foreseeable future (i.e. decades). At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

IMHO it is just a matter of time before vendors start providing e.g. drivers written in Rust for their hardware. I could see linux evolve to a point where integrating drivers like that is both possible and common. Once that happens, it will be a hybrid kernel effectively. Rust is not the only thing moving that direction; wasm is also going that direction. People are talking about running that in the kernel. And of course Rust runs on top of that as well. So one outcome would be a lot of Rust code running in a wasm sandbox on top of legacy kernels.


>The question is whether we will end up using their efforts or whether we keep on going back to the same monolith kernels we have been using for the last thirty years or so. Whether it is NT, Linux, or any of the BSD kernels, they each have decades of history behind them.

Please just... stop. You are using terms that you clearly don't understand the meaning of. What is a "monolith" from your point of view? Because I can tell you that Windows NT is most certainly not a monolithic kernel. Its architecture is very much that of an impure hybrid kernel. You're using the term "monolith" as some sort of buzzword in a realm where that exact term already has a highly specific meaning.

>At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools,

Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite-the-same thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

>I could see linux evolve to a point where integrating drivers like that is both possible and common. Once that happens, it will be a hybrid kernel effectively.

Hybrid kernels have absolutely nothing to do with the language they are written in beyond the fact that they're written in it. Monolithic, hybrid, microkernels and exokernels are all terms to describe the architecture of a kernel.

>So one outcome would be a lot of Rust code running in a wasm sandbox on top of legacy kernels.

So then Rust and wasm isn't a part of the kernel is what you're saying.


>> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools,

> Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite-the-same thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

That ripgrep isn't GNU or POSIX compliant is indeed a distinction worth making, but certainly not in this context. You're taking on a very pedantic interpretation of "re-implementation" here that doesn't really matter to the general point. Specifically:

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

None of this requires any of the "re-implementations" to be 100% strict bug-for-bug compatible tools.

This general confusion comes up so often that I have a FAQ item for it: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#pos...


> a distinction worth making, but certainly not in this context.

It is a worthwhile distinction, because the "re-implementation" talked about is often materially different, to the point that it no longer adheres to what little standardization there is.

So realistically we are talking about a new product, or a radical evolution of an existing one. It and everything above it also have to change.


I commented because ripgrep was specifically brought up as a counter-example to the notion "At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C." What specifically, about ripgrep, removes it from consideration as evidence that Rust programs can operate in the same domain as C?

Consider this. If instead of

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing libraries, popular command line tools, etc. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

they wrote

> At the same time, we are seeing Rust pop up in a lot of places that used to be the exclusive domain of C. People are already re-implementing popular command line tools, albeit with different interfaces that lack POSIX compatibility. A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

Other that the words being more precise for pedants, does the central point of the statement change? No, it doesn't.


> Yes, but some (e.g. ripgrep) don't actually do the same thing as the programs they are supposed to "re-implement". Most Rust "re-implementations" don't actually re-implement programs, they are new programs doing a similar-but-not-quite thing to the older C implementations. For example, ripgrep is neither GNU or POSIX compliant, and therefore it is not a re-implementation.

Is there a reason that command line tools cannot be re-implemented in Rust and maintain GNU and POSIX compliance? If not then the fact that ripgrep is not compliant is irrelevant to the larger point that Rust command line tools could eventually replace existing C command line tools.


>Is there a reason that command line tools cannot be re-implemented in Rust and maintain GNU and POSIX compliance?

No, there isn't, but what "re-implementations" that I know of don't even try to do that.

>If not then the fact that ripgrep is not compliant is irrelevant to the larger point that Rust command line tools could eventually replace existing C command line tools.

Of course it's possible. You can go ahead and re-implement them in BASIC even. The point is that it hasn't been done, and until it has been done then such a replacement will not happen.

At that point there are further considerations. The C linker is ubiquitous, and remains a strong, underlying presence even on Windows, the one platform where C stands the weakest. The reason that the C linker is so widespread is that while C has no standard ABI, the platforms have very strongly defined ABIs (e.g. systemv abi), and those ABIs are extremely stable and haven't had a single change in over a decade. Those ABIs are also very, very simple and the result is that writing any type of software towards a C library in any language is dead simple.

C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that, nor does it have a stable ABI. C/C++ are the two most interoperable language out there, with an extremely mature and reliable toolchain. It's simply not justifiable to replace C/C++ with Rust as a systems programming language until this serious problem is fixed.

Then furthermore, a vast amount of effort has been put into these tools over the last years to ensure that they run on any *nix and any architecture, and even so they can be ported with relative ease. For example, how do you suggest implementing glibc in Rust? You might argue "why would we need the C standard library when we're porting things to Rust", and the answer is that if you want to make a move towards rust, having a C library (particularly the GNU one, a lot of software depends on GNU extensions) is of utmost importance until that goal has been achieved.


I don't want to be defending Rust but your post has so many misconceptions. May I ask where you got all that information and what your background is?

> C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that, nor does it have a stable ABI.

Rust has #[no_mangle] and extern "C" and those two guarantee that ABI stability you're looking for.

> Then furthermore, a vast amount of effort has been put into these tools over the last years to ensure that they run on any *nix and any architecture, and even so they can be ported with relative ease.

That's actually a breeze in Rust, including cross-compilation (which is relatively painful in C/C++).

> For example, how do you suggest implementing glibc in Rust? You might argue "why would we need the C standard library when we're porting things to Rust", and the answer is that if you want to make a move towards rust, having a C library (particularly the GNU one, a lot of software depends on GNU extensions) is of utmost importance until that goal has been achieved.

There's work towards that: https://gitlab.redox-os.org/redox-os/relibc


>I don't want to be defending Rust but your post has so many misconceptions. May I ask where you got all that information and what your background is?

I am primarily a systems programmer and write software for the petroleum industry in my country. I work with C++ and in some (rare) cases C, and I've been doing that for the past 4 years. I have my own WIP hobby unix-like microkernel project written in C as well.

>Rust has #[no_mangle] and extern "C" and those two guarantee that ABI stability you're looking for.

I admittedly did not know this before it was pointed out to me.

>That's actually a breeze in Rust, including cross-compilation (which is relatively painful in C/C++).

It really isn't. Have you looked at rustc's compiler targets? It's not a very long list. Support is improving, but there's still a lot of key areas that are completely missing. To my knowledge it supports x86, ARM, MIPS and POWER. That's not a long list.

>There's work towards that: https://gitlab.redox-os.org/redox-os/relibc

That's good, and is genuinely what's necessary to put Rust in key components.


Based on [1], also full support for s390x (IBM mainframes), as well as support for SPARC, WebAssembly, and asm.js as targets only (you can't run rustc itself on them, but can cross-compile to them). RISC-V support was recently merged into master (not sure why it's not on that page). AVR and m68k (seriously) both have actively developed ports which are living in forks until they're ready to be merged.

Anyway, most of the work of porting Rust to new platforms is not in rustc itself, but rather porting LLVM to target that platform. This is bad in a way and good in a way. It's true that LLVM doesn't have as extensive a list of supported targets as GCC. However, for people developing new architectures and OSes, LLVM is often the go-to choice as the first compiler to port. For example, WebAssembly, eBPF, and AMDGPU are all supported in upstream LLVM at present, but not in GCC. On the other hand, RISC-V got GCC support first – but lowRISC also actively developed an LLVM backend, which is now upstream. [2]

Of course, C benefits from having multiple implementations; even on platforms that only LLVM has a backend for, you can just use LLVM's C compiler (Clang), whereas there's (currently) no Rust compiler with a GCC backend. I'd love to see that change in the future. Still, I'd say Rust target support is already pretty strong when it comes to the (embedded) platforms most people are doing new development for, and rapidly getting stronger.

[1] https://forge.rust-lang.org/platform-support.html

[2] https://www.lowrisc.org/llvm/status/


It’s not on that page for the simplest reason in open source: nobody sent in a PR.

I’ll look into it tomorrow, thanks.


> C++ at least has extern "C" going for it, which disables name mangling. Rust doesn't have that

It does, it's called "#[no_mangle]", and together with an extern "C" fn declaration you get a non-mangled function directly callable from C.


I see, I admittedly didn't know that. Thanks.

I think it's more that part od the attraction of rewriting the software is being able to upfate and improve the interface. If you want the old interface then the existing tools are usually perfecrly good.

I believe seeing people re-implement libraries or popular command line tools is a behavior so common in rising in popularity programming languages that we might almost describe it as the default.

That's what nebulet is doing!

> wasm is also going that direction

Genuine question: why would WASM in the kernel make any sense at all?


It's sometimes useful to run "arbitrary" code in the context of the kernel. For a concrete example look at the Berkley Packet Filter[1]. In it's original usecase it allows packet filter expressions to get compiled into a bytecode, which in turn get verified & executed in a kernel-space virtual machine. This is useful because it allows the user to implement arbitrary expressions, in this case firewall rules, but have them executed in the context of the kernel. When you're trying to filter millions of packets per second, or instrument operations that are measured in sub-microseconds, avoiding that context switch matters.

[1]: http://www.brendangregg.com/ebpf.html


It is incapsulation of code into a sandbox. You can do it by moving code into user-space and use hardware implemented sandbox, or you can do it by making your own sandbox in software.

Hardware implementation runs code faster (kind of "executes more instructions per second"), but switching from user-space into kernel-space and back are costly. The costs are very high, high enough to projects like "user-space TCP/IP stack" start popping up: it is all about avoiding switching to and from kernel-mode.

Software implementation of sandbox in the kernel allow costless (or near costless) switching of context, but runs slower. It is tradeoffs, and not every one piece of software would benefit from moving into kernel-WASM, but some would do.


Security. WASM is sandboxed by design.

Many things are "sandboxed by design." Few things are true sandboxes.

I'm very skeptical of the idea of throwing out virtual memory and process isolation to depend instead only on software sandboxes.


That and performance. System calls have a lot of overhead when running in user space. You can dodge that when running as part of the kernel.

> it will be a hybrid kernel effectively.

What do you mean by this exactly?


> A lot of these implementations have clear merit in the sense that they are faster, safer to use, easier to scale, maintain, etc.

Depends on who you ask. Hipp on Rust in 2016:

Rewriting SQLite in Rust, or some other trendy “safe” language, would not help. In fact it might hurt.

https://blog.regehr.org/archives/1292#comment-18452

> drivers written in Rust

If you have to write inline assembly, and generally bypass Rust's security features and use "unsafe" code, why wouldn't you just use C?


Regarding SQLite, here's the official tune (https://www.sqlite.org/whyc.html#why_isn_t_sqlite_coded_in_a...):

----

All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include:

A Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.

B Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.

C Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.

D Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.

E Rust needs a mechanism to recover gracefully from OOM errors.

F Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.

If you are a "rustacean" and feel that Rust already meets the preconditions listed above, and that SQLite should be recoded in Rust, then you are welcomed and encouraged to contact the SQLite developers privately and argue your case.

----

I follow the Rust story from afar, so I don't know the status of every one of these points.

B and F (FFI and speed) are covered.

C and E (embedded story and OOM recovery) is being actively worked on.

A will improve with time (for example, I suppose that the Rust 2015 dialect will settle at some point, feature-wise, once Rust 2018 is out).

D (branch coverage of compiled binaries) is on the radar, but not addressed yet.


Because C has big flaws (signed/unsigned automatic conversion, arrays without length) that Rust doesn't have. That said, you're right that I find Rust "story" that you should wrap unsafe operation behind safe APIs as something easier said than done.

Because it is a big difference to have 100% of the code unsafe or just a set of visible code regions marked as such.

Hyperbole, or do really believe every line of C is unsafe, in the literal sense, by default? Pray tell me what is the difference between

  // safe, tested, C
  ...
  // potentially dangerous C deemed thus for whatever arbitrary reason
And

  // "safe" Rust
  ...
  // "unsafe" Rust

Life experience since I touched C for the first time in 1992.

So yes I do belive, given the quality of the code bases I have seen in production.

And as discussed at Linux Kernel Security Summit 2018, which videos are freely available for you to watch as well, even with the rigorous process to accept patchs into the kernel, CVEs keep increasing.

68% of them were caused by memory corruption, something mostly unique to C and languages that are copy-paste compatible with it.

Then another good chunk was related to UB, something that even Linus already ranted a few times.

So yeah, Safe C is an oxymoron, unless we are talking about a new variant with safety turned on by default, where warnings are dealt as errors and everyone uses static analysers that break their CI builds on error.


Static analysis.

Bit of a mixed bag.

Slide 12:

> Go etc. are garbage collected, making interacting with C either impossible or excruciatingly slow.

"Impossible" is, of course, a lie. "Excruciatingly slow" may be the case, though it depends on the kind of interaction you are looking at. Without any context, this statement is overly general.

Slide 14 can be taken out of context and misrepresented as showing that "Rust can be faster than C", although the programs being compared use different data structures, so it's inconclusive. (It's not clear to me whether the author wanted to make a broad claim like this. The title suggests so, but again, the difference in data structures suggests otherwise.)

Slide 16:

> every operating system retains some assembly for reasons of performance

Is this really the case? I would be interested in examples. My impression was that assembly is needed for hardware interaction for which there are simply no C constructs available (setting timers, interrupts, system registers, and whatnot), and possibly highly space-constrained bootloader stuff, but performance? C compilers are not stupid.

Slide 19:

> in-kernel C tends to be de facto safe

This is a bit of a stretch. I mean, yes, most lines of the Linux kernel have never been the cause of a CVE. But it's hard to tell which lines are the critical ones. Properly encapsulating safe/unsafe regions does seem like an improvement. (My understanding is that Rust's "unsafe" is not properly encapsulated: You might mess something up that will cause a segfault later on, in ostensibly "safe" code. Still, it's a step forward.)


>> in-kernel C tends to be de facto safe

> This is a bit of a stretch.

Not "a bit;" quite a lot of a stretch. What makes C problematic is that the language is specified in terms of an abstract machine that looks absolutely nothing like real hardware, and executing instructions that make no sense in the abstract machine but are very well-defined in the real hardware is liable to cause problems for the optimizer in the compiler. The assumption that it's safe is built on the principle that a) compiler vendors aren't going to break the kernel build and b) it's possible to jam enough flags onto the compiler to make the abstract machine adhere to the real semantics. Both of these are dangerous assumptions.

One of the more infamous compiler-writers-are-breaking-code-for-no-reason rants is the optimization that assumes that a dereferenced pointer cannot be null, so it deletes null checks. Of course, the actual code that caused that rant was this:

    static unsigned int tun_chr_poll(struct file *file, poll_table * wait)
    {
	struct tun_file *tfile = file->private_data;
	struct tun_struct *tun = __tun_get(tfile);
	struct sock *sk = tun->sk;
	unsigned int mask = 0;

	if (!tun)
	    return POLLERR;
This isn't some advanced compiler optimization breaking your code for being an idiot. That code would only work in the first place if you had compiler optimizations that pushed the dereference of tun below the check, which means there's no simple abstract machine interpretation that would make it work.

What makes C problematic is that the language is specified in terms of an abstract machine that looks absolutely nothing like real hardware, and executing instructions that make no sense in the abstract machine but are very well-defined in the real hardware is liable to cause problems for the optimizer in the compiler.

Is there a language (other than assembly) that would have a closer model to the hardware?


To my knowledge, there isn't any language that really tries to be the kind of "portable assembler" that people try to shoehorn C into. There just really isn't the demand for it, since trying to be a portable assembler requires trading off fidelity to hardware for optimization, and most people would rather have faster code than more accurate hardware representation.

The internal IRs of compilers (such as LLVM IR or GCC GIMPLE) is likely to be much more accurate to hardware (and the ones I called out are indeed so). It's debatable if you want to actually call these non-assembly languages, though.


The claim about compiler IRs is strange. For one thing, hardware is different. Which hardware are you claiming LLVM IR is "accurate" to?

More concretely, LLVM IR has integer types for any bit size, while actual hardware architectures have only a few. Many real CPUs have condition code registers set as side effects of operations, while LLVM requires explicit compares, etc.


The big thing that you see in LLVM IR that you don't see in C is vector register support. There is also support for multiple return values, as well as some better coverage of hardware instructions via intrinsics (e.g., fma). It is somewhat poor in support for things like exposing hardware exceptions and the condition code support, and its support for vectors, floating point, and bit sizes is definitely in the realm of supporting union of architectural possibilities instead of the intersection.

I definitely would not call LLVM IR accurate to hardware, but it is more accurate than C is.


Maybe something generic that can infer interfaces and algebras from VLSI. (I sound like fluff but I had a few ideas in the past about a way to map hardware closer to avoid ambiguity pain and inefficiencies, I just can't remember)

What if the implementation handles dereferences of invalid pointers by accessing the value at the corresponding address, and all possible addresses are all valid?

(I have worked on a system that worked liked this. There was some kind of process information block at address 0, and NULL at runtime was all bits reset. The system compiler was not gcc.)


Primarily because it's written in C, the Linux kernel is currently in a full-blown security crisis. Nobody has credibly challenged the facts that Dmitry has laid bare, but at the same time, few people are motivated to fix the problem.

https://www.youtube.com/watch?v=qrBVXxZDVQY


We have tons of production OCaml code interacting with C and it's not "excruciatingly slow" at all, in fact it's very fast indeed.

The problem with many of these "very fast indeed" claims, is that they usually compare to not interfacing with C at all.

That is, how much is the overhead of the call from OCaml to C? Not just how much faster did the entire system get.

My hunch is that it is still quite favorable.


There are two ways to call into C code from OCaml depending on whether the C code needs to allocate on the OCaml heap or not. If not (the so-called "noalloc" method) then it's just a call instruction. Otherwise it goes through a trampoline which IIRC uses a computed branch which probably kills branch prediction.

One other problem is that you cannot inline C into OCaml functions (which is generally true when mixing languages) so there's always some overhead even in the noalloc case, although it will be fairly small on modern CPUs.

Edit: https://camltastic.blogspot.com/2008/08/tip-calling-c-functi...


> Slide 12:

IMO the big GC related issue is lack of determinism especially in the management of external resources. If you need that in a managed language you're forced into grafting reference counting on top which is particularly error-prone and unpleasant. Interacting with other languages can be slow, and bridging uncomfortable.

> Slide 14:

Rust can be faster than C in the general case because of the optimizations permitted by, among other things, strict aliasing rules. [1] The Rust language invariants are stronger and therefore the compiler can be more aggressive.

> Slide 16:

IMO you can do all that stuff in C with a combination of volatile pointer I/O and if necessary compiler intrinsics. I've done a lot of AVR programming and never dropped down to assembler because something wasn't available in C. I'd imagine the answer is really neither, just that some people are set in their ways and view ASM as 'faster' even though in reality LLVM would probably generate faster code by better taking into account instruction issue and data dependencies.

> Slide 19:

Unsafe Rust is unsafe because you can't express what you want in safe Rust. To your point, you can absolutely break things. The implicit contract is when writing unsafe sections that you are to maintain the invariants of the safe language at entry and exit of the block.

What makes safe Rust safer than C, though, is all the flagship features. You can detect data races across threads (and interrupts) statically. You can't write to and read from data at the same time. You have fully deterministic dynamic collection of resources. etc.

[1] https://robert.ocallahan.org/2017/04/rust-optimizations-that...


Any dynamic memory allocation lacks determinism, which is why in high integrity systems it often forbidden to dynamically allocate memory.

As for Rust can be faster than C, well, people said the same about Ada but it rarely is the case so I wouldn't hold my breath for Rust either. After all, those languages use the same backends - GCC in the case of C & Ada, LLVM in the case of C & Rust. But it's enough to as fast as optimized C and have all the safety features.


> After all, those languages use the same backends - GCC in the case of C & Ada, LLVM in the case of C & Rust.

The killer flaw of C for performance is the pointer aliasing problem. C only has the very coarse-grained tools of strict aliasing (often disabled in large applications because it breaks code!) and restrict (which relies heavily on manual programmer annotation) to control these aliasing issues. Since Rust has the rule that you can only have one mutable reference to an object at once (and read-read aliasing issues don't matter), it can effectively automatically add in these annotations without relying on programmer annotation.


I think it's fair to say that "average" Rust code will be faster than "average" C code due to lack of pointer aliasing issues as the average developer doesn't care about/understand this enough to make the annotations in any codebase I've worked in. Can you make both equally fast? I don't see why not, they're both LLVM.

Given enough time, effort and smarts, you can probably make any lang as fast as any other :-)

Rust doesn't require a heap, you can use it fully statically and still get the deterministic destructors when your stack object is destroyed.

The same for Ada, and it also has strict aliasing rules, but that hasn't made it faster than C historically. It's in the same ballpark as C, though.

> MO the big GC related issue is lack of determinism especially in the management of external resources.

This isn't a property of GCs per se; it's just that most GCs are optimized for throughput. Go's GC's pause times are on the order of 1ms, which might not be appropriate for every application, but it's probably fine for soft-realtime systems. There are a lot of other levers one could imagine as well, like semantics for demarcating critical sections.


Lack of determinism is absolutely a property of GCs, by definition. You’re never 100% sure when a resource is going to get deallocated/finalized with absolute certainty. This is a dealbreaker for kernel type applications. For instance think of a network socket wrapped in a structure, you can’t rely on a finalizer invocation to close your file descriptor, you don’t know when if ever it will happen so you have to wrap it with your own brand of ref counting.

I’m not criticizing pause times, but rather that you can’t rely on code inspection to identify the point of resource deallocation, which is pretty much the flagship feature of a GC :)


Sorry but this is nonsense. There's a whole field of study of hard realtime garbage collection, and several implementations. Also it's easy even in ordinary GCs to determine when a resource will be freed in the cases when you need that, eg by providing a socket.close() method.

I think we probably agree. I'm not arguing that you can't have hard realtime constraints on a GC language, just that when you're interfacing GC memory with underlying resources you have to graft your own ownership model on top (socket.close()) which is not enforced by the compiler (in any language I've seen) introducing a new vector for error. Rust binds the two deterministically and statically validates both.

> I’m not criticizing pause times, but rather that you can’t rely on code inspection to identify the point of resource deallocation, which is pretty much the flagship feature of a GC :)

This is completely wrong. The flagship feature of a GC is memory management, __not__ filehandle management or management of other resources. While many GCs support finalizer hooks, most languages with GCs (Go, Python, etc) explicitly discourage using these to close file handles for specifically the reasons you cite. Automatic resource management is useful and desirable, but it's out of scope for GCs.


In re-reading my statement, I understand the confusion, when I said 'not being able to rely on code inspection is the flagship feature of a GC' I meant, glibly, that non-deterministic deallocation of memory in a way the programmer does not have to (alternatively, cannot) reason about meaningfully.

I suppose in reality it depends. Finalizer hooks are absolutely bad, I agree. Which means that resource management has to be grafted on top of a GC memory model. Since the compiler won't be helping you enforce the management of resources since, as you point out, it's out of scope, it introduces a new vector for error. Rust allows you to bind the two and handle them together, both in a checked/enforced way. I've yet to see a GC'd language that does the same, but I can't think of why one couldn't exist.


> Since the compiler won't be helping you enforce the management of resources since, as you point out, it's out of scope, it introduces a new vector for error

It's not a new vector of error; it's the same vector that C has to deal with.

> resource management has to be grafted on top of a GC memory model

Only if this means "don't bind resource lifetimes to memory lifetimes", which is the solution that every tracing GC language implementation (that I'm aware of) uses. There's nothing stopping you from refcounting file handles just like you would in C. The compiler could even automatically add in the refcounting instructions for you, or the language runtime could have a special GC just for file handles.


Sure but we're comparing it with Rust not C.

Why should the GC not be able to inform the programmer/language designer when memory is discarded? This is a non-issue.

Besides, a kernel has to deal with plenty of forms of nondeterminism anyway.


How would the GC in this case inform you when an object is deallocated? The whole point is scope can be extended arbitrarily by anything and that finalization may happen, or it may not, at some point in the future.

In the same way as a kernel deals with e.g. with the question whether an I/O interaction has succeeded or failed? But there is a difference, a GC can guarantee that finalization will happen in the future, unless someone switches off the machine by pressing the power button. I/O cannot make this guarantee.

Do you maybe have hard real-time systems in mind when you say "deterministic"? I agree that a GC would be a bad idea for those, but these require validation of the hardware in combination with the software anyway.


No GC I've ever encountered can guarantee a given object will be collected or finalized.

> Rust can be faster than C in the general case because of the optimizations permitted by, among other things, strict aliasing rules.

Sure. The same is true for Fortran, for example. It's just that the few benchmarks of Rust vs. C I have seen so far all seem rigged in one way or another.


Definitely, and Rust isn't the only language that can be faster than C. All benchmarks are rigged in some way :)

> (setting timers, interrupts, system registers, and whatnot)

Yes, you're right. Some of those can occasionally be accessed via memory mapped registers, but at least some inline assembly should be expected/required for most architectures. So it's not merely performance. Though it also makes sense for some critical tasks to be tuned via intrinsics or assembly.


I'd really like to see an open-source QNX-type microkernel in Rust. Everybody is re-implementing early Unix in Rust, but that's not a good model. The great thing about the QNX model is that the tiny kernel barely changes from year to year, because everything is outside it and not very trusted. So it approaches being totally debugged.

L4 is too low-level - you have to run another OS on top of it, usually Linux. QNX offers a POSIX interface.


> QNX offers a POSIX interface.

You say that like its valuable. POSIX has a lot of problems, if you're going to go through the trouble of throwing away a mature kernel like Linux with all the hardware support that brings, you may as well also ditch your POSIX baggage.


What is the state of documentation on QNX? Is there sufficient documentation that it would be possible to write a QNX-compatible kernel from the documentation (plus access to a QNX machine, which has to be available somehow outside of industry, right?)?

Until Blackberry bought it, it was open source. There were free downloads. Now it's proprietary. But there's a lot of info out there from the open source era. The manuals are available.

Here's the original QNX paper.[1]

[1] https://cseweb.ucsd.edu/~voelker/cse221/papers/qnx-paper92.p...


I am amazed that I cannot find a copy of the source code from the time it was freely available now.

Here's the 2007 press release announcing QNX going open source.[1]

On April 9, 2010, the day the acquisition closed, Research and Motion took the source code offline, with no notice. All user open source projects on QNX were abandoned shortly thereafter.[2]

[1] http://www.qnx.com/news/pr_2471_1.html

[2] https://community.qnx.com/sf/sfmain/do/listProjects


Would it be possible to use rump kernels with this to get support for the myriad hardware out there from the get-go? Or are QNX-esque microkernels and rump kernels mutually exclusive for some reason?

Using the drivers from rump kernels with a QNX kernel might work. QNX runs drivers, file systems, and networking in userland. The rump kernel people have those components. The QNX kernel itself does only memory allocation, process management, CPU dispatching, timers, and inter-process messaging. So it's tiny, relatively trouble-free, and has a small attack surface.

So if someone implemented a QNX-style microkernel in Rust, file systems and networking could be adapted from rump kernels. Drivers would require more porting. That would be a nice way to get a usable but small OS for embedded and IoT devices. Linux has more than you want in an embedded device.


It'll be interesting to see how much use Rust gets within Fuchsia atop the Zircon microkernel.

The current state is, a few hundred thousands of lines, and Google hiring more Rust engineers quite aggressively.

I'm also excited to see how it turns out :)


Any other high profile company/project using Rust ? (beside mozilla of course)

Facebook, Dropbox, cloudflare, fastly, tons of startups you’ve heard of...

Presentation by the Joyent CTO btw, who hired a lot of the old Sun Solaris team.

Among other things because he is also was part of the old Solaris team?

Hilarious example of why English is a terrible language: I meant for the the pronoun to refer to Joyent, not [necessarily] the CTO.

No, that's an example of someone not knowing the language very well. You should have written something like, "which is the company that...".

I can't think offhand of any language that's really better here in a sense of being more efficient, except probably Classical Latin which is horribly complicated and of course a dead language.


>> Presentation by the Joyent CTO btw, who hired a lot of the old Sun Solaris team.

> No, that's an example of someone not knowing the language very well.

I disagree. The sentence is clear and unambiguous as written. The subject of the first part of the sentence is the CTO himself. If the continuation of the sentence is intended to switch subjects, then it must be written so. For example, '... by the Joyent CTO, whose company ...', or '... by Joyent, who/which ...' thus 'who' refers to 'CTO'.


You don't understand. The writer of that responded in another post, saying explicitly that he did intend to switch subjects, and then complained about the English language. You're absolutely right: the sentence is clear and unambiguous as written, but it is not what the writer intended. And you're absolutely right later about how it would be changed to switch subjects.

Czech allows you to specify that with a change of one letter depending on who you mean (the CTO - který, the company - která) - this depends on different genders of the two words, but is commonly used.

Are they officially calling themselves "Czechia" these days? I saw that on Google Maps recently, I think. It certainly is shorter and simpler than "Czech Republic".

It's the oficial short name (the full name is still Czech Republic), local people don't like it much but they're coming around. However I wasn't referring to the country (Czechia/Czech Republic) but to the language (Czech).

Yea, but the desirable language feature is a differentiation of Joyent and its CTO, based on Joyent being an adjective and CTO a noun, not on noun gender. (Did I set up the problem correctly?)

Yeah I understand, just wanted to point out that some languages have a hack for that. I can't really imagine how would your ideal solution work/look like.

I think using `which` instead of `who` would have been sufficient in this case; but this is a non-general fix for similar reasons

Operator errors do not indicate the quality of a language.

Though they are interesting in the context of a discussion of the relative safety of kernels written in C and Rust.

Funny, when it comes to programming languages like C, operator errors are very much blamed on the quality of the language.

use "which hired"

*whom

RUSTful English officially needs to be thing

A bit surprising that he doesn't mention any other C++ based kernels such as https://github.com/l4ka/pistachio/ and most of the fuchsia code base except for the lk micro kernel. In my opinion fuchsia points towards where things are going multi-language implementations (mostly c++) with an interface description language for interfacing components. From a point of view of postmodern c++, rust has little to no advantages left. It has a much more mature ecosystem and set of libraries, is just as fast and can be very safe if used correctly.

> From a point of view of postmodern c++, rust has little to no advantages left.

My entire adult life I've been writing C, I resisted Rust for a while but took the plunge when a C++ project came along that was in dire need of being reworked. I liked it, the language and the compiler and tooling around it helped me tremendously.

I assume you're aware of all the things that Rust statically guarantees for you, and I'll also assume you know the difference between language complexity and language implementation complexity (eg, the compiler or some other tooling):

I've been skimming through the C++ section of cppreference.com, and oh boy, there's heaps and heaps of... stuff... just stuff. The interplay between all that stuff is complex and it's hard to keep everything in mind. Keeping things in mind is important for correctness, or at least for having a reasonable amount of confidence in what you're writing. Information locality is also important for correctness, and C++ lacks both of those. I'll even go further and say that C++ is a write-only language, like a garbage bin of features and exceptions to each of those features. Rust has many many many clear advantages (and some disadvantages) compared to postmodern C++.


Maybe just to elaborate: In contrast to rust, C++ is a standard defined language, with >3 mature and competitive implementations, two of which (clang / microsofts compiler) provide excellent tooling and support for refactoring (clang in particular) beyond any other language in existence (maybe except Java). Instead of looking at cppreference.com, maybe consider browsing https://github.com/isocpp/CppCoreGuidelines/blob/master/CppC... the Cpp Core Guidelines, this outlines a forward looking vision of how good c++ code should look like.

A lot of the static guarantees that rust provides can be modelled in C++ as well, see for example "https://clang.llvm.org/extra/clang-tidy/checks/cppcoreguidel.... Eventually someone will write a rust-style borrow checker as a clang analysis pass. Already now it is possible to express a lot of compile time properties cleanly in c++, with features such as constexpr lambdas, static_assert etc. Additional features such as Concepts / meta-classes / modules will improve C++ expressivity even further.

In short, yes C++ has a lot of stuff and everyone knows that it is hard to get a grip on all its features and misfeatures. But there has been a continuous effort to rectify and improve on its short comings. See for example https://clang.llvm.org/extra/clang-tidy/checks/list.html for a list of clang based code transformations that are able to automatically improve your c++ code.

The way I see it, just because of legacy reasons (I can interface with the majority of commercial software developed in the last 30 years (CAD, EDA software, Houdini, Abelton, ...)), tooling reasons (the rust compiler is a slow hot mess, compared to the state of the art c++ compilers) and momentum (there is a large incentive to continuously improve c++, because it is the foundation of a large fraction of commercial software out there) it is unlikely that rust will outcompete c++ in a significant way in the long run.

To close of with an example the lean theorem prover https://github.com/leanprover/lean, implemented in C++ handily beats the 20+ year old Coq prover. One of its key features is multicore support. If you look into its implementation details, the equivalent rust code would have to work around a lot of rust "safety" features, essentially because in many cases it is hard to convince a hardcoded heuristic like a borrow checker that whatever you are doing is safe. The static guarantees that rust gives wouldn't help you at all in correctly implementing some of the very involved algorithms a theorem prover kernel has to implement, while standing in the way of the implementation being straightforward. This leaves aside the obvious tooling (you can use Visual Studio, good profilers and debuggers) and integration (SAT checkers like Z3, LLVM backend) advantages.


> the lean theorem prover

Which has its share of segmentation faults (around 50 closed issues) and unknown number of corner cases with silent memory corruption/data races.


"can be very safe if used correctly"

the same bit applies to C, and we all know how it's working out


What we don't know, is how it would have worked out if we didn't have C.

There is also a bit that most of the bad security failures are not as sophisticated as the ones we mostly talk about. (Ironically, of course, the widest vulnerability lately was at the CPU level...)


The slide about the operating systems of the 1980s ignores Lisp machines. Genera was not a "research system" and it had good performance. There was also Xerox Star's OS written in Mesa.


This isn't quite what I was going to say, but it's close enough.

I think parts of an OS can and should be written in something rust like, however the whole thing should probably not be written in rust.

My personal thought is that C and C++ 98 era are bad, but they were what we needed at the time. Part of this badness was that you could do anything but exact facilities for specialized tasks had to be rolled from scratch. Libraries are possible, but not ideal (no namespaces for C and crazy header + recompile issues for both).

In order to replace C and C++ we need more than just one language, we need an entire family of languages that all have the specific features implemented to support their domain. Rust can probably be used in game programming, but Zig (or in theory eventually Jai) is being specifically made with that goal in mind so they will probably be better. Rust itself will be much more useful for applications that cant stomach managed languages (ie web browsers, etc) because that's what they are specifically building it to do. It's got high level features in a form factor that you would expect from C#/Java with low runtime and no GC.

We can probably write OSes in Rust or Zig, but ideally someone who really likes thinking about OSes will create a language specifically geared towards writing OSes that will be able to replace C.


I took a brief look at Zig.

The top slogan for Rust is "fearless concurrency", which ties back into the memory model, mutability and sharing references. This helps you prevent (at compile-time) data races and other interesting bugs.

I didn't see any support for that in Zig at all.


Last I checked Zig is more concerned about overflow conditions than Rust(numeric overflow, stack overflow, OOM etc.) and boasts features in that vein - so it competes, but not in the same territory.

If you are writing a kernel it's highly likely that you'll end up in a lot of scenarios where you have to use unsafe Rust, in which case Zig would be the better language for the task. Admittedly, it would be nice to have some of both, and they both have room to get better. But I see Rust as more of a "libraries and applications" centric setup, where it's the right thing for a web browser and the wrong thing for a device driver.


These slides are quite high-level, so I've got to ask: what's the expected benefit of using Rust to implement a kernel? I somehow thought that nearly all interesting concurrency there would not fit into the paradigm of exclusive ownership. And if we are preaching for programming in unsafe Rust, then doesn't the message become less compelling?

The purported benefit would be to wrap all of the unsafe bits in safe wrappers and then have a less bug prone, more secure OS, I suppose.

This would be, to put it mildly, quite difficult. Even then, unsafe code is sometimes not written correctly which brings the whole thing down.

I, personally, still think it's worth it. I think efforts like Redox OS can teach us a lot about what we're doing and offer a chance to collapse some of the layers of cruft existing OSes have accumulated.


Having only a small part of the code with unsafe means you only have to check only a small % of the code for UB/security holes and not the whole code base like in C.

It just limits the places shit can happen and which need to be closely reviewed which alone is a big help.


It should be noted that you can have correctness bugs can occur in safe code if a Rust guarantee was violated in an unsafe block. This might seem obvious, but it does mean that when you hit one of these bugs you might start with debugging safe code and thus it's not as clean a separation as some Rust evangelists might imply.

Exclusive ownership is only half of Rust's ownership story. The other half is borrows. Borrows actually fit very well with concurrency.

For example, if you have some data structure synchronized with a mutex, the mutex would be the owner of that data structure. Everything else would just get borrowed access to the data structure when it locks the mutex. Rust's borrow checker can make sure that you don't keep any references to that data structure after you have released the mutex so you can't access the data structure again without locking the mutex again. The mutex itself would need unsafe code, but everything using the mutex wouldn't.


In practice, even in redox there’s not as much unsafe code as you might think.

Is there a video of this presentation? bcantrill’s always a fun / interesting watch.

There will be it was recorded. From Qcon SF this week.

Another important and related question to ask is what role Lua should play in writing drivers and applications on top of a new kernel. NetBSD started on this direction, but exactly what an operating system looks like when it leverages both systems-level and scripting-level programmability waits to be seen.

https://www.netbsd.org/~lneto/dls14.pdf

https://www.lua.org/wshop13/Cormack.pdf


I played with the idea of setting up a runtime for a scripting language such as JS, then I was thinking, what is a OS anyway ? If it can be written in Rust or any other high level language, it can also be written in a scripting language !...?

i think there is a case for a rewrite, apart from the memory safety guarantees. two things come to my mind:

1. elimination of backwards compatibility. this is true for all new operating systems, independently of the language. clean slate and throw away all the baggage that was reasonable 20 years ago, doesn't hold true today anymore but still is baked into the old architecture. this might be less relevant for all-purpose OSes, but might be a viable option for specialized systems (IoT, Network Appliances, ...). you remember what linus says when a kernel patch breaks buggy userland code? we now get a second chance to implement a new system that avoids whole classes for buggy code (while the underlying problem doesn't go away, it may get greatly reduced).

2. reduction of complexity. this _is_ a rust thing, and the story to back this up is [stylo](https://blog.rust-lang.org/2017/11/14/Fearless-Concurrency-I...), the parallel css engine used in firefox. i remember reading that google tried to implement concurrent styling in chrome but failed, because it got too complex in c++. human intellect is pretty much finite and doesn't scale well; we rely on better tools. so, rust might enable us to do things that were considered impossible before.

the emergence of such an OS would be gradual of course, niche at first, then slowly growing until one day it's the new de-facto standard.

lets say, game engines. you rarely write your own game engine, because that's too complex and ties up all the resources you need to actually implement your game. so you buy one and it's in C++, of course. because they all are, and thus your developers are fluent in C++. there are no game engines in rust and few developers who know the language. but over the years, more and more game devs will try rust in their spare time and like it and write game engines in rust for their side or indie projects, where it's still feasible. a few of those will grow and get more features and tooling and at _some point_, suddenly, rust will be a viable alternative. even though rusts strengths aren't really that important in game engines. performance happens on the graphic cards anyway. security is not as big of a deal. parallelism is quite constrained anyway. etc, etc. but if all else is equal, rust might be more productive.

so, is it time to rewrite the OS in rust? sure, why not. it's just that we wont all switch to redox overnight.


I'm surprised there are no mentions to Microsoft Research's Singularity, that used only software barriers, and the languages used there (Sing#, Spec#) had ownership concepts ([Claims] attribute).

Nim is realtime garbage collected and compiles to optimized C and it can use the C ecosystem of libraries and tooling. It also produces smaller binaries than Rust and is faster for most tasks.

Despite what the Nim documentation says, realtime and garbage collection are mutually exclusive (unless you count non-deferred reference counting and give up on collecting cycles.) Setting maximum pause times doesn't change the fact that the garbage will be collected nondeterministically, which is a problem for kernels.

Those are 3 orthogonal(ish) things.

Realtime just means that actions triggered by event occur in an explicitly bounded time. It doesn't usually mean that they occur in the same time. So a GC with bounded pause could be used in a realtime system assuming you could guarantee the explicit bounds.

I can't comment on whether or not this is true of Nim and pause time probably isn't sufficient (depends on how it's defined) but they're not mutually exclusive.

And most kernels don't need to be realtime unless they are realtime kernels.

Having said that, of course, the vast majority of GC systems are most definitely not realtime, or anywhere close. But that's just actually true, not theoretically true.


> So a GC with bounded pause could be used in a realtime system assuming you could guarantee the explicit bounds.

You can't guarantee explicit bounds on when a given resource will be collected with GC. Bounded times can provide upper bounds on GC pass runtime, but they do that by potentially deferring further collection. Those deferrals make it nondeterministic.

> And most kernels don't need to be realtime unless they are realtime kernels.

That's not true. Most (all?) kernels have internal realtime needs, e.g. responding to interrupts.

There is a way to write a kernel with a GC'd language, as Niklaus Wirth has demonstrated, by circumventing the GC where needed. But the critique I was bringing up was simply that there's no such thing as a realtime GC.


Rust allows for new architectures. Eg beos style os would be a match made in heaven as rust really shines when it comes to async code.

Also maybe you could have a less complicated memory manager as you can deal with allocations statically.


"rust really shines" they need to land async/await

We really do. It’s in nightly; likely to hit stable early next year. One of its precursors is being stabilized as we speak.

in-kernel C tends to be de facto safe

Huh?

jbb67 5 days ago [flagged]

no.

The slide deck comes to this conclusion, at least for now. Rather, he discusses a hybrid approach of writing certain modules / drivers of a C-based kernel in Rust so it can be done a little bit at a time.

The reason for "no" given: "why completely replace what works already" paired with "fighting second system syndrome". It's not a real "no", but reasons why it's challenging. Aren't those always the case? It shouldn't stop something new from appearing.

I was expecting to see this type of response. I don't program at low levels but it seems like this would be the responding answer. Kernel programmers would be using something else if that's what they wanted to use. Decades of experience and hard learnt lessons down the drain.

The entire point of the presentation is to discuss these issues, and recommend rust for more ancillary tasks (modules/drivers, firmware, utilities) than the kernel itself.

> Kernel programmers would be using something else if that's what they wanted to use.

The vast majority of the languages having arisen in the last 40 or so years are completely unsuitable for kernel development, so not necessarily.


On the other hand, the design of Rust is directly and deeply influenced by those very lessons.

Less “down the drain,” possibly more close to “baked directly into the tools we use.”


Also, if you can abstract away at a sufficiently low level why not just do it in C where there's already a generation of experience and familiarity?

Sure, why not just continue writing C for the rest of time?

NO, and I wish HN stopped upvoting this sort of propaganda. There are hundreds of languages around and the slides did not even attempt to compare pros and cons across languages that would be suitable for kernels.

> Is it time to rewrite the operating system in Rust?

No. Next question, please.


How constructive of you...


Hadn't heard about this before, and it does look interesting. But not sure why you're linking to it in this context without comment? Especially since it seems to be a moribund project.

E had a pretty interesting security model and there were in various E writings expressions of pipe dreams that operating systems should be rewritten using it. I should probably have linked to one of those to make the connection more explicit, but really it was just a momentary bit of snarkiness that didn't deserve more than what I gave it.



Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: