
Unikernels: No Longer an Academic Exercise? - rumcajz
http://250bpm.com/blog:138
======
kev009
I'd like to point out that there is a unikernel called IBM's TPF which is
something everybody is interacting with daily. It's used in the payment card
network by Visa, as well as the airline and hotel industry for reservations.
This is typically run under the hypervisors z/VM or PRSM.
[https://en.wikipedia.org/wiki/Transaction_Processing_Facilit...](https://en.wikipedia.org/wiki/Transaction_Processing_Facility)

A few years ago someone trying to spin container technology did a lot of
damage to other attempts at unikernels with marketing dogma, not reality.
Claims of non-debugability and other FUD. It's long been standard operating
procedure to trace and debug systems from outside the system's view. This is
how people do bringup of new chips as well as OS ports. On hardware there is
usually a dedicated debug and trace facility as part of the CPU or board
support package or a firmware monitor. In a virtualized environment like a
unikernel this is way easier because you can run code above the guest's ring 0
supervisor privilege against its RAM/pagetable root. Modern systems like POWER
even allow debugging from the BMC, with a plan to allow full gdb sessions over
that out of band interface [https://github.com/open-
power/pdbg](https://github.com/open-power/pdbg).

There's nothing implicitly wrong with unikernels, or other systems software
ideas like microkernels, just because they are less popular technology at the
moment. I'd encourage people to continue exploring this space.

~~~
im_down_w_otp
That FUD was from Bryan Cantrill
([https://news.ycombinator.com/item?id=10953766](https://news.ycombinator.com/item?id=10953766))
making aggressively resolute, but also easily falsifiable claims about the
production-worthiness of unikernels.

From the perspective of someone who'd debugged and traced unikernels from both
inside the runtime (LING) and outside the runtime (xentrace), and who worked
in the domain of the very z/TPF system you referenced above, the indictment
seemed at-best strangely misguided and at-worst intentionally duplicitous.

As you can imagine the indictment wasn't at all persuasive to me, and thus I
keep exploring the space in bits and pieces where applicable.

~~~
phs318u
While acknowledging that your claims about his claims may well be 100%
correct, I find it hard to believe that Bryan Cantrill would be "intentionally
duplicitous". People, even very smart people, can occasionally be very wrong
(btw, I'm not making a claim either way here), and that doesn't mean there is
an agenda/intent that is malicious in nature.

~~~
perbu
Cantrill is a smart guy. But he has always had a big mouth. And the fact is
his post was written with ZERO experience with Unikernels. Zero. He was just
assuming they act the way he imagined and attacked that image.

Which is fine if you're upfront about it. He wasn't.

~~~
bcantrill
I appreciate the kind words, but let's give me a tad more credit in the
experience department, please: I have been doing production OS kernel
development for over two decades, and have done non-trivial work in
essentially every privileged subsystem across several different microprocessor
and OS architectures. If you want to say that I have gobs of experience in
kernel development (and more generally at the hardware/software interface),
but no experience with unikernels per se, then fine, I guess -- but at the
same time, let's acknowledge that you are the CEO of a unikernel company who
very much has a dog in the fight?

~~~
perbu
Absolutely. I would rank you in the very top of developers with experience in
development of traditional operating systems. In particular DTrace stands out
as an excellent piece of work. It's one of those fundamental advances that
serves as inspiration for others.

Now, I'm not sure I agree that I have a horse in the race. I don't necessary
believe that there is a race. I've never really been a proponent of the schism
between Unikernels and Containers. I struggle to see how Unikernels can offer
the same flexibility and ease of deployment as containers. We're likely won't
be able to support the vast amounts of runtimes and infrastructure needed to
replace something like Docker. Perhaps there could be very specific uses where
something like the paper described could be used, but I'm not betting on it.

As a software project IncludeOS has a much narrower target than what people
traditionally have thought when thinking of Unikernels. And as a result of of
this we're not in the business of replacing neither containers not general
purpose operating systems(GPOS). We're aiming to carve out a few niches where
we are confident that a GPOS isn't the answer. We're only going to address
those needs where we're pretty certain we can actually add some value.
Basically we're think we can improve on security in addition to adding real
time capability whilst still remaining source-code compatible with Linux
(mostly thanks to musl).

My grief is singularly with the myths you helped create that Unikernels are
something where you are forces to work with stone age tools and hardly without
any tools, except printf, for debugging. We've had to spend a lot of time
dispelling these. There are a few other things I believe you where wrong about
at the time but I'll spare you the details. Better suited discussions over a
beer of coffee.

------
bunnycorn
It's a nice exercise, but I don't see t going anywhere.

1st. I don't see that many bugs in the TCP/IP, filesystems, etc. specially
those exploitable by black hat hackers, specially remote exploits. We usually
see wide-spread vulnerabilities in the middleware like OpenSSH, which is in
the userspace, putting everything in the app is not going to solve that, quite
the contrary, we need to wait for the library maintainer to fix the bug, for
the applications to integrate with the newer version of the library, and then,
update all the applications.

2nd. Side-channel attacks. I'm not really confident in letting applications
have that low level of hardware access. I mean, it's really easy for one
application to steal the resources from another. We need authority if we are
sharing peripherals. Well, we have one peripheral already that is shared by
processes, that's memory and look at the mess that is a modern MMU and all the
attacks on those that we have seen throughout the decades.

3rd. If we are going to implement that as a shared library, the result is
exactly the same as today's micro and hybrid kernels, if we are not going to
save memory and implement that as a static library, it's exactly the save as
virtualization.

Correct me if I'm wrong.

~~~
mato
Co-author of the paper referenced in the blog post here.

1\. Regarding bugs: There is a lot of code in your monolithic OS kernel. There
have been, and will be, a lot of vulnerabilities in that code. Various
sandboxing mechanisms notwithstanding, the vast majority of processes running
on your system can potentially use all of that code as an attack vector.
Unikernels let you switch off access to most of that "host" code, and, through
the use of a library OS, contain only the minimal set of libraries needed to
run your application in the "guest". Regarding updates: Valid point, but no
different from any modern application of substantial complexity, except that
now you also have to update (e.g.) the library providing its TCP stack.

2\. Unikernels don't make that problem any worse. In fact, if you run on
something like Muen ([https://muen.sk/](https://muen.sk/)), it'll mitigate a
bunch of these attacks by giving you a less precise RDTSC at the subject
("VM") level.

3\. I don't follow. Implement what?

~~~
nostrademons
If the unikernel is running as a normal process, all that potentially
vulnerable code is still on the box, and still potentially executable in the
event of a vulnerability. The security selling point of a standard unikernel
is that the all the potentially-vulnerable code _doesn 't even exist_, because
only the libraries that are actually used get compiled and linked into the
app, and it has no code relevant to functionality that it's not actually
using.

The paper suggests using system-call filtering via seccomp or similar to block
off this attack surface, but if you trust seccomp to have zero bugs, you might
as well do traditional process sandboxing a la Google Chrome.

The problem unikernels solve is that traditionally, a vulnerability in C code
means "game over" and the box is completely pwned with all memory and storage
directly accessible, while with a unikernel, a vulnerability means only that
an attacker can access other functionality within the app, and can't rely on
eg. a shell or a debugger being available to arbitrarily move bytes around.

~~~
mato
> If the unikernel is running as a normal process, all that potentially
> vulnerable code is still on the box [...]

Correct. Conceptually the same applies to all Type 2 hypervisors. Type 1 less
so, but you could still potentially exploit Xen and you have all of the dom0
to play around in.

> The paper suggests using system-call filtering via seccomp or similar to
> block off this attack surface, but if you trust seccomp to have zero bugs,
> you might as well do traditional process sandboxing a la Google Chrome.

If you do that, you will have to do a line-by-line analysis of the code you
want to sandbox in order to determine exactly which syscalls it's using. With
the approach presented in our paper the developer does not need to care about
this. For example, she can develop her MirageOS unikernel as a normal UNIX
process and switch to a guaranteed-to-work-minimal-seccomp sandbox with a
simple change of target (build-time configuration option).

But yes, you are now trusting seccomp instead of KVM. I believe in giving
people the ability to easily make that choice.

> The problem unikernels solve is that traditionally [...] can't rely on eg. a
> shell or a debugger being available to arbitrarily move bytes around.

That part does not change with the sandboxing mechanism changing. The stuff
that's inside is still only your unikernel.

~~~
mycall
> you are now trusting seccomp instead of KVM.

Which do you trust more?

~~~
mato
That's a very good question. The answer is, "it depends".

For the 80% case, on x86_64, I consider them more or less equivalent. KVM is
used daily in anger to provide isolation (e.g. GCE, and now ChromeOS) and has
been around much longer _but_ you need to trust hardware virtualization which
is a large attack surface on the CPU itself. Given what we've learned about
CPU vulnerabilities over the last year, I wouldn't be surprised to find some
lurking in the VT-x/SVM implementations.

Seccomp OTOH is difficult to use correctly for arbitrary/existing applications
but exposes less of the kernel (depending on your metric, see our paper) and
does not need hardware virtualization.

For the 20% case, where the stakes are higher (e.g. High Assurance), I would
use something like Muen or SeL4 and run a disaggregated system on top of that.

------
TheRealPomax
I don't quite understand the title of this article. Where is the part that
explains why it's no longer an academic exercise? Nothing has been shown to
now exist at a scale that makes it any more than another toy (in the academic
computer science meaning of the word).

~~~
eyberg
I was personally an unikernel conference in Beijing in May. Baidu, Alibaba,
VMWare, ARM and many other multi-billion dollar companies were talking about
how they are rolling in support into things like their serverless
infrastructure.

Also - there _are_ production deployments out there.

~~~
TheRealPomax
VM support for unikernels is literally a prerequisite for unikernel uptake, so
that's hardly surprising, but coming from academia: if a highly niche market
using a thing still keeps it a mostly academic exercise, and this article does
not contain anything that talks about how it has finally escaped the "mostly
interesting to academics, and that handful of people who use it in their
highly specialized or possibly instead just plain old 'someone once set it up
and it's too critical to change now' systems".

~~~
pjmlp
Microsoft used their unikernel research (Drawbridge) to bring SQL Server into
Linux.

[https://arstechnica.com/information-
technology/2016/12/how-a...](https://arstechnica.com/information-
technology/2016/12/how-an-old-drawbridge-helped-microsoft-bring-sql-server-to-
linux/)

It is also how they kind of implement secure kernel in the recent versions of
Windows 10.

------
zamadatix
I don't see the value in unikernel-as-a-process. Most of the syscalls save
time by e.g. sending a bulk payload and having the kernel space TCP/IP
implementation break that up into fragments and handle sending them to the
interface. Moving the barrier halfway down now means you did all of the work
and got negative performance benefits - seemingly under the guise of
security... but isn't the whole point of unikernels to run hardware assisted
VMs as your segmentation barrier instead of relying on software ferrying calls
and data between hardware trust levels?

~~~
perbu
The value lies in security in this context. You also get a bit of performance
as the "unikernels" are self-contained. So you can do most things, including
the IP stack without switching into kernel mode.

I'm guessing this is mostly interesting for FaaS platforms or similar. You get
isolation similar to hardware-assisted vms but with a lot less overhead and
with phenomenal boot-times.

------
hvidgaard
While it is possible to do, there is an important cost benefit analysis to
make. All of the benefits the article lists can be a drawback as well. Say
your specific network needs tweaking to get good performance, with a unikernel
you must do this with all applications, while for a "traditional" monokernel
you can do it once and it works for all applications.

That is not to say the approach is not interesting, but it'll probably be for
in some kind virtualization, where you abstract away the OS entirely, and run
applications directly on the hypervisor.

~~~
justicezyx
I am not sure the example makes sense at all.

Default settings always not working for all apps.

If you need to tune network, if that's a tuning for all apps, development can
be done inside net lib, all apps rebuilds and release, that's not more
expensive than tuning a kernel config.

If you need to tune on per app basis, then unikernel is wildly safer because
of the isolation.

~~~
hvidgaard
Not everything is open source.

~~~
justicezyx
Isn't that make my statement more true?

------
en4bz
Single Root IO Virtualization (SR-IOV) has the potential to enable this. If
the number of virtual functions climbs high enough for the number of process,
or at least the number of critical processes, then you can just map a virtual
function into all of the necessary processes. DPDK and SPDK already partially
enable this but the number of virtual functions or complete lack there of for
NVME devices is still a limiting factor.

------
galois198
What are the best tools for experimenting with unikernels?

~~~
perbu
I believe it comes down to your programing language preference. If you are
into OCaml the Mirage Unikernel is rather complete. If C or C++ is your thing
there is IncludeOS (potentially adding other language runtimes next year) and
for Haskell there is HalVM.

There are others as well, but AFAIK these are the ones with active development
happening. Please correct me if there are active ones I've forgotten.

All three support compiling your application as a Linux binary, meaning you
can do most of the development and debugging using the tools you're used to,
IDE with visual debuggers and the whole shebang.

Once you want to run in a separate VM you can still debug, but it becomes a
tad bit harder. I know how to debug IncludeOS application when they are
running under Qemu or KVM. Qemu can act as a gdb remote, you just need to get
it working, which is a bit of pain.

And now you have the option of running it the way it is describes in the
paper. I've never touched this so I don't have a feeling for how hard it is
getting it to run.

~~~
ghkbrew
Rumprun[0] is another big one in the C camp. Though I think you can use
basically any software that conforms to posix.

[0]
[https://github.com/rumpkernel/rumprun/](https://github.com/rumpkernel/rumprun/)

~~~
perbu
No commits since April. I think Antti has left the project and started a
brewery or something.

------
gvb
For the counter argument, see "Unikernels are unfit for production" January
22, 2016 - by Bryan Cantrill

[https://www.joyent.com/blog/unikernels-are-unfit-for-
product...](https://www.joyent.com/blog/unikernels-are-unfit-for-production)

~~~
toast0
I like this essay, but I think it's really pointing to a narrow space, rather
than a lack of space.

Bryan points out many weaknesses of unikernels, but assumes everybody needs
multi-tenancy and the ability to spawn processes. And debugability is weak,
but that probably depends on the application / environment you run. A lot of
people want to run unikernels in a VM environment, but for me, it seems like
the right application of unikernels is where you have one application that you
want to expand to fill a single machine -- bare metal, boot to the
application, save all the layers; there's no multi-tenancy, but if I'm running
on hundreds/thousands of machines, I don't need multi-tenancy.

Debugability is important, but lots of people run without a kernel debugger,
so it might not be that important to everyone. If you want it, you'll have to
build it, but DTrace and friends had to be built too -- and it's easier to
build it a second time, since you know it's possible and what it should look
like.

~~~
mato
And indeed, we have built some tooling for debugging, since we wanted it:
[https://github.com/Solo5/solo5/blob/master/docs/debugging.md](https://github.com/Solo5/solo5/blob/master/docs/debugging.md).
It wasn't that hard.

------
lixtra
Can a in kernel Webserver be understood as unikernel?

I.e.
[https://en.m.wikipedia.org/wiki/TUX_web_server](https://en.m.wikipedia.org/wiki/TUX_web_server)

~~~
perbu
Sure. As long as you disable init on the server you'll have a unikernel setup.
Round hole, square peg, though. Linux isn't meant to work like that.

------
mitchpron
I still think it is. There's still been no push in the open source community
to make tooling around Unikernels to make it user to a broader audience and a
broader set of use cases.

------
rayiner
[https://pdos.csail.mit.edu/archive/exo/](https://pdos.csail.mit.edu/archive/exo/)

------
akavel
What advantages do unikernels have over microkernels?

~~~
msla
Unikernels are older than microkernels, in terms of design philosophy, and owe
more to VM (the IBM project from the 1960s) than later OS design.

Microkernels are about splitting a single OS into what are sometimes called
'servers', whereas unikernels are individual systems running on a hypervisor;
ultimately, the biggest advantage is that, in a microkernel, if one 'server'
goes down (the disk server, for example) the whole system is down, whereas a
single unikernel can fail without impacting any other unikernel.

[https://jdebp.eu/FGA/microkernel-conceptual-
problems.html](https://jdebp.eu/FGA/microkernel-conceptual-problems.html)

[https://utcc.utoronto.ca/~cks/space/blog/tech/HypervisorVsMi...](https://utcc.utoronto.ca/~cks/space/blog/tech/HypervisorVsMicrokernel)

~~~
Fnoord
> the biggest advantage is that, in a microkernel, if one 'server' goes down
> (the disk server, for example) the whole system is down

Rather, if the disk server goes down, the whole system _might_ go down. It
doesn't necessarily _will_ go down. You could, for example, simply restart it.
If your Graphics server goes down, you might still be able to SSH to the
machine, and in the example of yours which I quote you could run a remote
emergency SSH server in full RAM.

Unikernel and microkernel are good examples of (fine-grained) principle of
least privilege. Doing that right is _difficult_. Just ask the NSA or Intel.

~~~
msla
Realistically, if a disk server goes down, there are two possible reasons:

1\. The disk hardware is bad. By all means, take everything down until you
replace the disk so you don't get garbage in important files!

2\. The disk driver software is bad. By all means, take everything down until
you replace the software so you don't get garbage in important files!

In neither case is simply bouncing the disk server an acceptable answer.
Debugging and fixing the underlying problem is essential, and that requires
taking the whole system down.

~~~
toast0
The disk server could conceivably fail because a disk took an absurdly long
time to respond to a command --- by all means, update the firmware, but in the
mean time, restarting the disk server (and replaying any pending writes,
somehow) will get you back on track. At least restarting the disk server in
read only mode allows reads to continue to function -- maybe even allows for
an orderly evacuation of the data.

~~~
Fnoord
Good thinking. Read-only mode is an interesting suggestion. Filesystems allow
this as well, and it can allow for the live extraction of data having to
resort to professional recovery means (ie. physical and expensive).

I've had times with bad sectors (Deathstar at the very least) where reads
would result in complete OS lock up or very laggy situation. That can be
solved by only trying so many times to read (though that was on Windows 9x
with PATA).

------
TimJYoung
We're talking about pretty significant improvements in network I/O performance
(among other things) with a unikernel, correct ? My understanding is that
there have been several projects that have revolved around putting Linux
network drivers into userspace so as to improve the throughput and avoid the
kernel mode switch.

~~~
dweekly
Here are some presentations from Google touching on reducing kernel overhead
for high performance networking:
[https://netdevconf.org/1.2/slides/oct6/03_jerry_chu_usTCP_LK...](https://netdevconf.org/1.2/slides/oct6/03_jerry_chu_usTCP_LKL.pdf)
[https://www.usenix.org/sites/default/files/conference/protec...](https://www.usenix.org/sites/default/files/conference/protected-
files/nsdi16_slides_eisenbud.pdf)

~~~
TimJYoung
Beautiful, thanks for the links.

------
m0llusk
It might be interesting, at least as an exercise, to combine unikernels with
the split kernels from distributed "lego os". Having a slimmed down only what
is used footprint spread out across the network could lead to multiplied
advantages.

------
Sohcahtoa82
After a presentation I saw at ToorCon earlier this year, I don't trust
unikernels in general. Several of them are beyond bad from a security
standpoint with little to no protections from malicious software.

