
IncludeOS – A minimal, resource efficient unikernel for cloud services - unixhero
http://www.includeos.org/
======
ArtWomb
Worth watching is Justin Cormack's recent presentation on unikernels,
LinuxKit, eBPF and databases such as SeaStar:

The Modern Operating System In 2018

[https://www.youtube.com/watch?v=dR2FH8z7L04](https://www.youtube.com/watch?v=dR2FH8z7L04)

~~~
ksec
How could Windows + Linux be 99.9% of the marketshare?

I know not a lot of people are using FreeBSD, but combined with OpenBSD I am
pretty sure there are more than 1%.

Edit: Turns out it wasn't a lie.... at least in terms of Web Server usage...
FreeBSD has less than 1%.

Sigh.

~~~
2trill2spill
FreeBSD may have less then 1% of server market share but FreeBSD drives a
large amount of network traffic. For example FreeBSD runs the open connect CDN
that Netflix uses and that accounts for almost 32% of North American
downstream traffic [1]. Limelights edge CDN nodes run FreeBSD and Limelight is
one of the Bigger CDN companies out there [2]. Juniper routers are FreeBSD
based and are the second most used routers after Cisco. There's plenty of
other examples but FreeBSD may not be widely deployed but it serves a
significant amount of the worlds internet traffic.

[1]: [http://testinternetspeed.org/blog/half-of-all-internet-
traff...](http://testinternetspeed.org/blog/half-of-all-internet-traffic-goes-
to-netflix-and-youtube/)

[2]: [https://www.freebsdfoundation.org/testimonial/limelight-
netw...](https://www.freebsdfoundation.org/testimonial/limelight-networks/)

[3]:
[https://en.wikipedia.org/wiki/Junos_OS](https://en.wikipedia.org/wiki/Junos_OS)

------
tinktank
I have never really got unikernels, maybe someone could educate me?

\- There is the argument of reduced attack surface/typechecked safety. I get
that, but I don't see how this is different from compiling custom Linux/NetBSD
kernels per VM?

\- There is the argument for increased performance. I don't get that at all --
I've done experiments on unikernel system for MirageOS vs IncludeOS vs Kernel
on a web and memcache server and the performance results were worse (15%),
0.4% better respectively for MirageOS respectively but for a tonne more effort
and really painful debugability.

\- There is the argument for isolation but I don't see how it's different
because the biggest cause of churn is the hypervisor? My results with Xen 4.x
didn't show any difference in isolation performance with both Include and
Mirage.

\- There is the argument that OSes don't map to existing hardware as well,
e.g. in the InfoQ talk in the comments the speaker talks about PCIe VFs for
example. All of these VFs map to physical resources, and you end up doing a
much worse job when you can't centrally manage these -- that's one of the main
reasons SR-IOV hasn't been adopted (including the fact that migration is non-
existent). Similarly with CPU hardware threads or NVMe device drivers -- they
are abstractions provided for convenience and reducing the burden of porting
existing drivers to work in hypervisor environments but surely the same issues
that appear with VM contexts (hardware scheduling and isolation) still exist
in unikernel environments? I feel I'm missing something in particular about
this point as I see it made often and it doesn't make sense to me.

\- How do unikernels work in a world of containers? This has always puzzled me
as I thought people moved away from VMs due to the heavyweight nature but of
course the flip side is the underlying isolation abstractions are more
lightweight -- so in essence are unikernels in containers user-level processes
or are they replacing the kernel bits too?

~~~
fwsgonzo
I am a kernel developer working on IncludeOS. Take everything I say with a
grain of salt as this is just my thoughts at 9 in the morning in written form.

1\. I can build a bootable binary without the full TCP and UDP, shaving off
60% of the whole IP stack. LTO removes a large amount of code as well. It
makes all the difference.

2\. Performance comes from building a fixed binary (all memory addresses are
constant), as well as using LTO and C++. IncludeOS is not at its most
performant because we haven't been working on performance. Project is still
young. We should be faster overall compared to just about anything that isn't
compiling to a fixed binary layout.

3\. Isolation can come in many forms. On traditional hypervisors you have the
same old same old. While on a newer type of hypervisor you can do just about
anything from avoiding VMEXITs altogether to removing the ability to even talk
to the hypervisor after configuration stage. If the hypervisor is just a thin
configuration program...

4\. Yes doubly so if your OS uses threads as well, and the schedulers are
almost always beating differently on the hypervisor and the guest. This causes
even more trashing than normal. In any case, IncludeOS is not using threads at
all. It is fully async and matches the hardware really well. The bottleneck is
the multiplexing ability of the cloud platform itself. We can live with that.

5\. I don't know much about containers other than that I avoid them.
Containers share kernel. Sorry, I can't answer this.

~~~
bullen
So how can you build a parallel application on top of this that can share all
cores of the CPU?

~~~
fwsgonzo
I wrote an SMP API once, and while its fully implemented still in the OS, it
won't work right because we changed C API to MUSL and it enables/disables
locks internally by counting up/down number of threads. So, that means you
cant use it at the moment.

It looks like this: [https://github.com/hioa-
cs/IncludeOS/blob/dev/api/smp](https://github.com/hioa-
cs/IncludeOS/blob/dev/api/smp)

With that you can schedule work to CPUs directly.

------
perbu
FWIW the most interesting tidbit I've learned after working with IncludeOS for
couple of years is this; The operating system cannot reconfigure itself.

All Unix and Windows-derived systems are meant to have the ability to
reconfigure themselves. For a lot of systems there isn't really a good reason
why the system itself should have this capability.

My Wifi access point is a good example. The fact that it runs Linux means that
a security flaw gives the attacker the tools to fundamentally modify the
devices' behavior. The attacker can install sniffers, blockchain miners or
spambots, without vendor approval. This makes any RCE bug critical and in
general means the device is dangerous to use if the vendor stops patching it.

~~~
tinktank
Your response confuses me so I must be misunderstanding it. Code is code, all
I need is an indirect jump to a pointer somewhere in memory to run the
arbitrary code I've installed to compromise the system, so why are unikernels
not susceptible to this attack?

~~~
nutjob2
"arbitrary code I've installed"

How do you do that if there is no facility to load and run code? That's what
he's getting at, the binary installed has fixed functionality and has no
facilities to do anything but what it's programmed to do. It may have no file
system nor almost any other feature that an OS has. It likely supports network
ports, but they are unlikely to allow arbitrary code execution becuase again,
they will have a fixed function.

The only way to do what you're talking about is replacing or modifying the
binary, which means you have physical access or there is a serious breakdown
in security which nothing could withstand.

~~~
tinktank
Surely the moment I have a vulnerability that allows me to execute code (via a
buffer overflow or similar) I have the ability to run the code that adds that
feature?

In my understanding even if I have a piece of code that just loops printing
"hello world" and nothing else if it's got a vulnerability that allows me to
execute code all bets are off.

------
jazzyjackson
Would love to dig into this and try some small C servers.

FYI if anybody @ IncludeOS is here, the page at [http://www.includeos.org/get-
started.html](http://www.includeos.org/get-started.html) automatically made a
link out of the file path `./seed/service` as well as
`your_service/service.cpp` and 404'd on me, which was confusing for a moment.
Looks like putting them in code blocks would prevent it.

------
bigbrooklyn
OSv is another, compatible with some unix apps [http://osv.io](http://osv.io)

~~~
ofrzeta
It seemed interesting back then but it's kind of dead since the core
developers (Avi Kivity, Nadav Har'El and others) have moved on to another
startup called ScyllaDB that is a re-implementation of Apache Cassandra in
C++.

~~~
nyh
The OSv project is indeed now moving much slower than it used to when we had
an entire startup company devoted to it, but is not dead - it still has three
committers from three companies - myself (Nadav Har'El), Waldek Kozaczuk and
Timmons Player. It still works, and you are welcome to try it. OSv is not as
minimal as includeOS, which of course depending on your viewpoint and use
case, is either a pro or a con. OSv supports more types of hypervisors, and
has support for a larger subset of the Linux ABI, can run pre-compiled,
unmodified, Linux shared libraries, and supports multi-core VMs.

~~~
nutjob2
OSv is a much friendlier entree to unikernals which makes it much more
productive for most existing code. Glad to hear development is continuing.

------
asien
This is potentially a big step forward for the cloud, this is also the missing
component to push Serverless.

Currently if you want to use containers and Serverless you'll need to spin up
200+MB of linux bistro + tons of dependancies that have nothing to do with
what you originally planned, that's to say running a container with a virtual
machine on it.

This kernel would allow to replace Unix for a much more lightweight (1MB) and
efficient kernel to run programs on it with an hypervisor.

This is promising.

~~~
perbu
You'd need a small and efficient hypervisor. Ukvm and Solo5 provides such a
system and if I remember correctly IncludeOS can boot and shutdown in less
than 10ms on those systems.

So yeah, it would be interesting to see this on a serverless platform. Hit me
up if you have a serverless platform. :-)

------
a012
Do these unikernel OSs tend to be language specific? Like if this app is
written in x language, use y OS?

~~~
chubot
Yes, because they rely on the compiler (for a specific language) to build and
deploy only what the application depends on.

The main examples I know of are Mirage (OCaml compiler) and IncludeOS (C++
compiler).

Though, I think there are experimental / research systems that try to
understand a program composed of multiple languages that each compile to LLVM,
or anything that compiles to JVM bytecode. So those might not be limited to a
single language.

------
emersonrsantos
Can I use gdb to debug my applications?

~~~
fwsgonzo
Yes, build for userspace linux and run it as a normal program. Then use all
your favorite tools. You can also attach GDB to a qemu process that is waiting
for a connection (abit harder). In a production environment on a public cloud
it gets even harder. For that I can only recommend UBsan (undefined-
sanitizer), logging and memory dumps.

------
kunthar
finally the end of lx* containers on the way. not the actual end of course,
who knows docker guys could buy this one as they did to some other unikernel
companies.

~~~
ckocagil
I don't understand all the hatred towards containers. It's simply an OS
abstraction to facilitate isolation. What's wrong with that? In fact I'd love
to see more containers, especially on the Linux desktop: for example a
container aware compositor that could render windows from container processes
and color code their borders, like Qubes does.

I have nothing against VMs either. But even when they're tiny (as in the case
of unikernels), do VMs scale as much as processes? I'm skeptical. Maybe
someone has data.

~~~
tannhaeuser
Docker-style containers only shield you from mixed-library situations eg.
where one "service" needs version A and another needs version B. It's painful
because this situation is entirely accidental and could be trivially solved by
linking everything statically. Then once you have two or more containers up
and running, these libraries still need security updates and other fixes which
is the entire point of shared libs in the first place. Basically, Docker is
not a solution but part of the problem. But with Docker comes a ton of
constraints; for example, basic file access and anything related to
permissions is a PITA. And you need overreaching orchestration software such
as mesos/marathon, k8s, or openshift. It's not that these tools are bad, but
rather that they're nuclear weapons for relative little benefit, requiring
expensive devops experts. All so you can pack multiple services on a single
physical host (which would be much easier using classical service runtimes),
and so your HTTP services (because everything has to be HTTP because of the
firewalls) can all listen on virtual port 80/8080.

~~~
AnIdiotOnTheNet
> It's painful because this situation is entirely accidental and could be
> trivially solved by linking everything statically.

It could and historically has been solved even without linking everything
statically, it's just that Linux people have some shared delusion that a
stable and consistent base system is bad, there should be no distinction
between system and application, and having two copies of a library is an
unforgivable sin. Consequently we get the nightmare complexity and
inflexibility of the package management scheme to do things we did in DOS on a
286 with no special software at all.

------
presscast
A few silly questions from someone who comes from containers (Docker,
specifically).

1\. Can I "run" and "manage" unikernels on my development machine like I would
a Docker image? (Use case: keeping build and deploy environments identical)

2\. What's the workflow for inspecting running containers?

3\. Is it possible to work in languages other than C++?

------
jokoon
I just found about OpenRISC
[https://fr.wikipedia.org/wiki/OpenRISC](https://fr.wikipedia.org/wiki/OpenRISC)

I'm wondering how this is performing against the raspberry SOC chip. I'm not
sure what is the most expensive part, the licensing or fabricating the chip.

------
tobiaswk
Every time I see this project pop up I think about how awesome an idea it is.
Not sure about the practical side... but the project is very cool indeed.

------
chewzerita
Cool! Can this be used with languages other than C++?

~~~
richdougherty
Appears to be C++ only based on my reading.

Here's a nice list of unikernel projects, some with more portability in mind
than others: [https://github.com/cetic/unikernels#existing-
projects](https://github.com/cetic/unikernels#existing-projects)

~~~
scns
Looks nice, thank. There is github.com/solo-io/unik as well. Had no time to
check it out yet.

~~~
joshumax
Unik is pretty nice, but it's basically a wrapper for other unikernels. I use
it for microservice deployments every now and then. It does make packaging
Java for OSv a lot simpler, however. For a more language-agnostic unikernel
that can run on bare hardware, rump/rumprun has been the de facto baseline for
me.

------
exabrial
It seems that hypervisors have developed a whole new level of OS API/ABI, sort
of like the POSIX specification.

~~~
tannhaeuser
Except there is no common specification, and there's not going to be one
either because "standards are for loosers". No meaningful standard has been
developed in this decade. It's almost as if the situation with two or more
super-stable, compatible POSIX O/Ss, compiler suites, databases, etc. we had
was too good to be true.

~~~
perbu
IncludeOS can support most of POSIX. It uses Musl, which provides a full POSIX
implementation for the system calls that are implemented.

Obviously things like fork() won't work.

~~~
tannhaeuser
> _Obviously things like fork() won 't work._

And that's a pretty tough constraint in non-GCd environments, as now you can't
rely on a host O/Ss process resource management. Manual memory management in C
and C++ apps and long-running services make all the difference in terms of
development complexity because of memory fragmentation, async (where memory
can be required/released at any time), etc.

~~~
perbu
fork() requires support for multiple processes. IncludeOS only has one
process, so a fork() doesn't make sense there.

Also, I should point out that relying on fork/exec to clean up long running
processes sound like something of a hack. Would you really like to have a
system that is required to restart itself ever to often as a strategy to
manage memory?

We've just spent quite a bit of time implementing a buddy allocator to combat
memory fragmentation on long running systems. It seems to do a good job and I
believe it is more or less a solved problem.

~~~
tannhaeuser
I don't agree multiple processes are hackish at all. CPUs/MMUs have dedicated
hardware for managing memory, O/Ss have tools to limit and monitor memory
usage, and separate memory regions/segments can be very effective and simple
means for security isolation (save for meltdown/rowhammer-style attacks). I
know this could be used as a general argument against any innovation, but
single-process-per-requests have worked well since inetd (4.3BSD in 1977).
Considering hardware has improved orders-of-magnitudes since, what is the
advantage of doing the same with very restricted programming models,
especially when developer costs are dominating new service developments?

~~~
perbu
I'm not saying multiple processes are a hack, just relying on fork/exec to
clean up memory. Safeguarding against memory fragmentation is imho a solved
problem - or at least solvable if you apply known methods.

Other than that I think you are right. Unless IncludeOS can prove that it can
offer something akin to the same level of convenience as current Linux-based
systems have I think it'll be hard for the OS to get traction.

The upside would be stronger isolation and a system that would be a lot harder
to break into. The absence of system calls makes most attacks very, very
inconvenient and the absence of self-modification functionality is
strengthening it further.

