
A reimplementation of NetBSD using a Microkernel [video] - agumonkey
https://www.youtube.com/watch?v=0pebP891V0c
======
Animats
That's nice, but late. QNX had that 10-15 years ago. With hard real time
scheduling, too.

All you really need in a practical microkernel is process management, memory
management, timer management, and message passing. (It's possible to have even
less in the kernel; L4 moved the copying of messages out of the kernel. Then
you have to have shared memory between processes to pass messages, which means
the kernel is safe but processes aren't.)

The amusing thing is that Linux, after several decades, now has support for
all that. But it also has all the legacy stuff which doesn't use those
features. That's why the Linux kernel is insanely huge. The big advantage of a
microkernel is that, if you do it right, you don't change it much, if at all.
It can even be in ROM. That's quite common with QNX embedded systems.

(If QNX, the company, weren't such a pain... They went from closed source to
partially open source (not free, but you could look at some code) to closed
source to open source (you could look at the kernel) to closed source. Most of
the developers got fed up and quit using it. It's still used; Boston Dynamics'
robots use it. If you need hard real time and the problem is too big for
something like VxWorks, QNX is still the way to go.)

~~~
unethical_ban
I'm watching the video now, but are you suggesting that QNX, which is not Free
and Open Source, has already accomplished MINIX's stated goals of OS
reliability?

I would like to hear Mr. Tanenbaum's answer to the less provocative form of
the sentiment: "What design decisions were made with MINIX3 that other RTOS
with microkernels didn't consider?"

~~~
jacquesm
QnX achieved Minix's stated goals of OS reliability 20 years ago.

And Minix isn't a micro kernel in the same way that QnX is.

~~~
rbanffy
QNX achieved it years before MINIX existed.

~~~
betaby
Interesting how reliability is measured in the context of OS kernel? Also I'm
sure someone took a look to the QNX source code to understand which technics
were used to achieve that. Since code is available but not opensource, perhaps
someone even re-implemented those ideas. Are there any good up to date source
on the subject ?

~~~
jacquesm
Reliability of an OS kernel is measured exactly the same way as any other
complex piece of technology: you note how often it does not do what it should
do. In the case of a large QnX deployment over all the time that I worked for
the company involved (several years), _0_ incidents that we could attribute to
the QnX kernel (or even any part of QnX).

If something didn't work it was either hardware or our own code, the way it
should be. It's not magic but it is very good at what it does. And it is a way
of building things that just works, message passing is a very powerful
technique for making reliable distributed systems.

~~~
ptaipale
> Reliability of an OS kernel is measured exactly the same way as any other
> complex piece of technology: you note how often it does not do what it
> should do.

To me this sounds a bit funny - isn't the whole point of microkernel
architecture that you achieve the reliability by _simplifying_ the kernel -
it's a less complex piece of technology, so it can be more reliable, and the
complexity is offloaded into user space processes.

Yes, it does not do what it should not do, because it does less.

~~~
jacquesm
A microkernel is simple but the whole OS is not and when you have all the
lines counted a micro kernel + associated programs will be about the same
complexity as a macro kernel + associated programs.

It's just that with the microkernel more of the code will be in stand-alone
programs. Every driver, file system, network layer and so on will be a program
all by itself.

~~~
ptaipale
Exactly my point. Now, the interesting question is whether you have a smaller
total number of bugs, and amount of impact, with this architecture or that
architecture. I don't know about that, but if you are measuring the
reliability of microkernels "exactly the same way as any other complex piece
of technology", you must keep in mind that you are probably measuring the
behaviour of a smaller and simpler thing, and the risk for bugs has been
offloaded somewhere else, it has not disappeared.

~~~
jacquesm
You have a much smaller number of bugs because (a) each component is much
simpler (b) runs as a separate process and so can be debugged and worked on by
mere mortals and (c) works using a well defined interface (message passing)
which makes testing and debugging a much simpler affair.

------
agumonkey
Youtube video description:

Based on the MINIX 3 microkernel, we have constructed a system that to the
user looks a great deal like NetBSD. It uses pkgsrc, NetBSD headers and
libraries, and passes over 80% of the KYUA tests). However, inside, the system
is completely different. At the bottom is a small (about 13,000 lines of code)
microkernel that handles interrupts, message passing, low-level scheduling,
and hardware related details. Nearly all of the actual operating system,
including memory management, the file system(s), paging, and all the device
drivers run as user-mode processes protected by the MMU. As a consequence,
failures or security issues in one component cannot spread to other ones. In
some cases a failed component can be replaced automatically and on the fly,
while the system is running, and without user processes noticing it. The talk
will discuss the history, goals, technology, and status of the project.

Research at the Vrije Universiteit has resulted in a reimplementation of
NetBSD using a microkernel instead of the traditional monolithic kernel. To
the user, the system looks a great deal like NetBSD (it passes over 80% of the
KYUA tests). However, inside, the system is completely different. At the
bottom is a small (about 13,000 lines of code) microkernel that handles
interrupts, message passing, low-level scheduling, and hardware related
details. Nearly all of the actual operating system, including memory
management, the file system(s), paging, and all the device drivers run as
user-mode processes protected by the MMU. As a consequence, failures or
security issues in one component cannot spread to other ones. In some cases a
failed component can be replaced automatically and on the fly, while the
system is running.

The latest work has been adding live update, making it possible to upgrade to
a new version of the operating system WITHOUT a reboot and without running
processes even noticing. No other operating system can do this.

The system is built on MINIX 3, a derivative of the original MINIX system,
which was intended for education. However, after the original author, Andrew
Tanenbaum, received a 2 million euro grant from the Royal Netherlands Academy
of Arts and Sciences and a 2.5 million euro grant from the European Research
Council, the focus changed to building a highly reliable, secure, fault
tolerant operating system, with an emphasis on embedded systems. The code is
open source and can be downloaded from www.minix3.org. It runs on the x86 and
ARM Cortex V8 (e.g., BeagleBones). Since 2007, the Website has been visited
over 3 million times and the bootable image file has been downloaded over
600,000 times. The talk will discuss the history, goals, technology, and
status of the project.

------
fmstephe
That is very exciting. I am glad to hear about fairly substantial amounts of
money being granted for this kind of project. I wish them well, but I won't be
jumping on board this bus for a while.

~~~
carussell
I took a serious look at MINIX over the winter, and digested several of
Tanenbaum's talks around that time. (For anyone wondering if this talk
contains anything substantially different from past ones, the answer is no.)

Here are some things to add:

\- Nowadays x86 is built with LLVM by default, and ARM is using GCC.

\- X11 is mentioned in the video, but the 3.3.0 release from last fall didn't
ship with a working X server, although past releases have. There is a message
on the mailing list from someone who writes that they've got it working on a
subsequent snapshot release.

\- You may have heard something in the past about 10 minute build times. That
info is out of date as of the switch to NetBSD userspace for 3.3.0. MINIX
itself (i.e., all the interesting parts) still only takes about 10-15 minutes
to build, but there's no way to just slurp down the sources for its
kernel/drivers/servers to build a MINIX "core" and then supplement it with the
prebuilt binaries for userspace (at least, not without doing some significant
work on your end to allow for that). The initial build for x86 on my modest
machine takes about 3 hours, almost all of it spent building LLVM twice.

\- For anyone looking for serious collaborators, MINIX is seriously lacking in
infrastructure from a project/community standpoint. E.g., a fair bit of
documentation is missing and much of what you will find on the wiki is out of
date. Development processes are neither documented nor easily discoverable
because there are effectively no development processes in place. Until about
six months or so ago, MINIX was without a bugtracker.
Organizationally/project-wise, the whole thing is pretty sparse.

\- If you have watched previous talks on MINIX, e.g. FOSDEM 2010, you will be
familiar with the open calls for those interested in working for pay, using
the money from the two grants mentioned in this video. That money is now gone.
During that time, MINIX was basically a research project run by grad students
who were working ~full time on MINIX, with the funding from those grants. It's
the same now as far as the student-run aspect goes, but with drastically fewer
contributions. Not much of the paid man hours seem to have gone towards
scaffolding out project infrastructure as I mentioned before, or the sorts of
drudge work that volunteers are unlikely to take up.

\- The code quality is... I dunno, fair? As I mentioned before, there are/were
effectively no development processes in place; no code review, etc. So there's
a fair bit of nastiness that got checked in directly, like copy and paste,
especially among the arch-specific boot time stuff; the comments are fairly
sparse; and you can find dead code and references to routines/fields that
either no longer exist or have been renamed, even in the relatively short
(~500 line) main.c.

For anyone looking in to maybe starting to work with MINIX, I'd suggest
assessing whether or not you would be comfortable striking out and doing
things on your own, and then being prepared to do so. With MINIX, you aren't
going to find a thriving community that you can just add your piece to, so as
to contribute to the effort. You might run into a certain level of that sort
of old-guard, paralyzing stop energy, so in a way it's got a lot of the
downsides of a greenfield project except with few, if any, of the upsides.

~~~
fmstephe
Thanks for this post. That is very interesting. It seems like a shame, because
a project like this is a long road. If they haven't built a place where work
can continue it will likely fade away.

I would love to see a sustainable, non-vapourware, micro-kernel. I've _heard_
such great things about them :)

------
cyber
This is pretty cool. It would be neat to seem some of this technology folded
back into NetBSD (potentially with the already existent modules
infrastructure.)

------
luckydude
He lost me when he said "start a new window" as the work around to not having
job control.

Neat idea but seems nowhere near done.

And as others have said, this was nicely handled by QNX way more than 15 years
ago, I was running multiple users on a 80286 around 1986 or so. Really neat
system.

~~~
sigzero
I loved QNX.

------
nickysielicki
I am 100% on the side of Linus Torvalds when it comes to microkernels.[0]

I will concede that in some instances a microkernel may outperform a
monolithic kernel in stability or performance or both. I am not the least bit
excited about any progress made in microkernels, I feel that it can only
result in much more closed systems that are easier to implement in ways that
make them harder to modify. This is why I wish for Hurd to continue to fail.

[0]:
[http://www.oreilly.com/openbook/opensources/book/appa.html](http://www.oreilly.com/openbook/opensources/book/appa.html)

~~~
vezzy-fnord
Microkernels have been running the world behind the scenes for a while now,
but most people don't seem to have gotten the memo and are still stuck with
associating the Mach server as representative of u-kernels in general.

~~~
JoshTriplett
There have been wildly successful microkernels. One of Xen's greatest
successes was demonstrating that to encourage widespread adoption of a
microkernel, you rebrand it as a hypervisor. More recently, some people have
started running software directly under Xen without a full OS, including
language runtimes, all without ever calling it a microkernel.

~~~
cbd1984
Microkernels and hypervisors are not the same thing.

~~~
JoshTriplett
What's the difference between Xen and a microkernel? They both manage memory,
do CPU scheduling, provide efficient message-passing, protect "processes"
running underneath them from each other, and leave almost everything else to
the "processes" running underneath them.

~~~
cbd1984
> What's the difference between Xen and a microkernel?

Xen allows full OSes to be guests and run on top of it. Microkernels only
allow servers to run on top of it, and those servers have to be purpose-
written and cannot meaningfully be ported.

Xen doesn't provide hardware abstraction and is fully invisible (except to the
extent it advertises itself); microkernels are neither.

Paravirtualization (what you did before VT-x and similar) was an oddity, and
blurs these lines a tiny bit, but the distinction is fairly clear otherwise.

~~~
JoshTriplett
> Xen allows full OSes to be guests and run on top of it. Microkernels only
> allow servers to run on top of it, and those servers have to be purpose-
> written and cannot meaningfully be ported.

L4 is an archetypal microkernel, and people often run full OSes or other
ported software under it, including Linux.

> Xen doesn't provide hardware abstraction and is fully invisible (except to
> the extent it advertises itself); microkernels are neither.

Microkernels typically don't abstract any hardware other than CPU and memory;
any other drivers would run _under_ the microkernel.

And Xen is only "invisible" if you run full hardware virtualization and no
paravirtualized drivers.

> Paravirtualization (what you did before VT-x and similar)

People still use "paravirtualization" today; see the "virtio" drivers
typically used with KVM.

------
vezzy-fnord
MINIX 3 has been based on the NetBSD userland since the beginning, I think.
That said, always interesting to hear Tanenbaum talk and the dynamic
upgrade/checkpointing features sound interesting.

~~~
istvan__
Weren't they using FreeBSD?

~~~
carussell
No.

~~~
istvan__
It seems they were using some of the tools:

[https://2011.eurobsdcon.org/papers/gras/minix-
bsd.pdf](https://2011.eurobsdcon.org/papers/gras/minix-bsd.pdf)

"ABSTRACT MINIX 3 has imported a significant amount of userland BSD code. The
trend began several years ago, but the pace has quickened markedly. We have
already imported NetBSD’s buildsystem, NetBSD’s C library, the pkgsrc package
management infrastructure, and various userland utilities from NetBSD and
FreeBSD."

~~~
carussell
I thought we were talking about whether or not MINIX was using FreeBSD
userland _before_ moving to NetBSD for 3.2.0.

~~~
istvan__
I was just generally asking. But thanks.

------
boardwaalk
I was trying to install minix 3.3 for fun and ran into a bug in the e1000
driver that caused VirtualBox to throw up. It's fixed already, but not in 3.3:

[https://github.com/Stichting-MINIX-Research-
Foundation/minix...](https://github.com/Stichting-MINIX-Research-
Foundation/minix/commit/92293fafd3877cfbb7bde350a225b2e07e737de1)

virtio seems to be working.

------
codezero

      The latest work has been adding live update, making it possible to upgrade 
      to a new version of the operating system WITHOUT a reboot and without running
      processes even noticing. No other operating system can
      do this.
    

Can anyone correct me if I'm wrong – but can't Linux do this with Ksplice, and
the more recent live kernel patching by Red Hat?

~~~
nickpsecurity
An AIX admin claimed it could do a live update of even kernel code without a
reboot. The solution for most others is clustering, needed to deal with
hardware risk anyway. OpenVMS clusters have reportedly gone 20+ years without
system downtime. Individual instances needed reboots so little that admins
occasionally forgot how to do that. So, yes, there's precedents in other
operating systems according to their users.

Note: And as far as OpenVMS, I believe it because those designers built it
like their job depended on it not failing. A cluster of that OS shouldn't
experience any significant downtime given about everything I've ever read on
it.

~~~
wumbernang
We had 11 years of uptime on a VAX cluster at a company I worked for in the
late 1990s.

They took it down in 2001 to replace it with something that took up 2U of rack
space, about 2kw less power and ran windows 2000.

I turned up in 2012 to replace it again with something cloudy and it had 11
years of uptime (well done NT!) again[1] so YMMV.

The cloud based version has gone down about 10 times (thanks Azure!).

[1] not a great position but this was on an isolated network with locked down
everything so less of a problem than a normally networked system.

~~~
nickpsecurity
That's funny. I swear I've considered just buying up a boatload of used Alpha
and Itanium machines to keep a VMS cluster going another decade. Put a guard
in front of it to block any attacks due to its age or protocols. People might
laugh but my stuff would stay running no matter what. Example below:

[http://h71000.www7.hp.com/openvms/brochures/commerzbank/comm...](http://h71000.www7.hp.com/openvms/brochures/commerzbank/commerzbank.pdf)

Notice how the Intel hardware all failed when things heated up a bit. The
AlphaServers running VMS just kept chugging along. The eventual fail-over
didn't loose a single transaction. Aggravates me that I can't easily obtain
such reliable IT hardware/software anymore outside eBay. I mean, HP NonStop
sure as hell doesn't have a hobbyist program with used servers for $130. ;)

~~~
wumbernang
The mid-range HP stuff is pretty reliable. We had a DL380p Gen7 survive the
switch underneath it catching fire. Had zero chassis failures on about 500
nodes in the last 12 years as well. Lose disks and power supplies all the time
and the odd Ethernet interface but nothing else.

Agree with ebay. I still look around for Sun Ultra kit now and then but the
wife has other ideas because it's noisy and expensive to run.

~~~
nickpsecurity
Wow! That is impressive. I appreciate the tip on those.

" I still look around for Sun Ultra kit now and then but the wife has other
ideas because it's noisy and expensive to run."

The battle that never goes away. Haha. This is why you need a basement or
soundproofed room for that stuff. That's on my todo list for next house.

------
fithisux
I believe microkernels are a very good abstraction/concept.

For me it is very interesting. I would like to see some day darwin running on
a real micro kernel (e.g. an updated mach).

I also like the fact that Hurd makes some progress. when ready, I will
definitely switch.

In any case hardware vendors must release more documentation on their hardware
to revive the OS scene. Blobs do not do much good.

------
mzs
Oh wow, mklinux is still around...
[http://www.mklinux.org/](http://www.mklinux.org/)

Sorry this really has nothing to do with the video, just that the tangental
thought of linux on mach made me wonder and I was pleasantly surprised.

------
nickpsecurity
The thing I like best about Tannenbaum's videos is that the dude is a great
presenter. Funny too. You just don't get bored easily. We might get more
outreach on some of these topics if presenters in those areas dress it up
better.

------
brian_smith
I actually just bought the 3rd edition of Operating Systems: Design and
Implementation (and Design of the Unix Operating System) when I watched this.
Now I'm quite excited. Any recommendations for other books on this topic?

------
regularfry
The live update and reincarnation server in general remind me a _lot_ of the
ideas in Erlang.

