

A unikernel experiment: A VM for every URL - amirmc
http://www.skjegstad.com/blog/2015/03/25/mirageos-vm-per-url-experiment/

======
rdtsc
Already exists for Erlang. Here is a demo:

[http://erlangonxen.org/](http://erlangonxen.org/)

Site is running on such a VM as well, press escape for a console to see stats.

Here is an example output from running their demo:

\---

300 sec : is how long it takes to launch a Linux instance to availability on
Amazon EC2.

50 sec : is the time between power on and the lock screen for an Android
phone.

0.4 sec : ago is when we received your request. Within this time we managed to
create a new Xen instance, boot it, and run the application that rendered the
page you are viewing. By the time you are done reading this, the instance will
be gone.

\---

~~~
avsm
Note that both of the setups are quite different in how they boot unikernels.
The EoX Zerg demo boots one unikernel per connection by having an HTTP proxy
in the frontend spin up the VM.

Magnus' version builds a static set of unikernels whose total resource usage
is known ahead of time, and boots them based on DNS lookups as necessary.
Because there is a race between a DNS response, the first TCP SYN from the
client, and the VM booting up to receive it, there is a helper unikernel
called "Synjitsu" which terminates TCP connections and replays them to the
MirageOS unikernel as soon as it boots up. This is very easy due to the OCaml
TCP/IP stack in Synjitsu sharing its state via serialising it into an
s-expression. More gory details in [http://anil.recoil.org/papers/2015-nsdi-
jitsu.pdf](http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf)

Now, imagine a version of Synjitsu that did this handover with TLS connections
rather than just TCP. The recipient unikernel wouldn't ever even have to know
the private key, just the ephemeral session key...

~~~
rdtsc
That is just a demo to show startup times. Presumably (just by looking at it
from a high level) a proxy instance for handling initial connection + a pre-
allocated, or maybe booted but paused, set of worker images would work in
either architecture.

------
mwcampbell
I suppose work like this is demonstrating that, contrary to what e.g. the
creators of Docker say, a virtual machine image is _not_ too heavyweight to be
our industry's equivalent of a shipping container. What is too heavyweight is
a VM image of a conventional operating system distribution with all of the
baggage that it brings.

Would these fine-grained unikernels also run well on KVM, VMware ESXi,
Hyper-V, or bhyve (FreeBSD's new VMM)? Or is Xen uniquely suited to this kind
of thing?

~~~
amirmc
> _" What is too heavyweight is a VM image of a conventional operating system
> distribution with all of the baggage that it brings."_

Indeed. I didn't realise that Docker/Container folks had ever said VMs were
too heavyweight. Is this linked anywhere?

> _" Would these fine-grained unikernels also run well on KVM, VMware ESXi,
> Hyper-V, or bhyve (FreeBSD's new VMM)?"_

We've been asked this before and for KVM at least, it just boils down to
someone suitably motivated to write the boot and VirtIO code [1, 2]. If
anyone's interested in this, we'd love to hear about it [3]!

[1] [http://lists.xenproject.org/archives/html/mirageos-
devel/201...](http://lists.xenproject.org/archives/html/mirageos-
devel/2014-08/msg00103.html)

[2] [http://lists.xenproject.org/archives/html/mirageos-
devel/201...](http://lists.xenproject.org/archives/html/mirageos-
devel/2015-01/msg00034.html)

[3] [http://lists.xenproject.org/cgi-
bin/mailman/listinfo/mirageo...](http://lists.xenproject.org/cgi-
bin/mailman/listinfo/mirageos-devel)

~~~
mwcampbell
> Indeed. I didn't realise that Docker/Container folks had ever said VMs were
> too heavyweight. Is this linked anywhere?

[https://github.com/docker/docker#user-content-better-than-
vm...](https://github.com/docker/docker#user-content-better-than-vms)

~~~
amirmc
Thanks. In hindsight that should have been an obvious place for me to look.

------
sp332
The next step is to compile it all to JS and just serve static files that boot
in the browser! :) Something like
[http://bellard.org/jslinux/](http://bellard.org/jslinux/)

~~~
cookrn
See: [https://www.destroyallsoftware.com/talks/the-birth-and-
death...](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-
javascript)

------
NhanH
I've always thought that something really cool could be done with unikernel
for SaaS: you let each of your customer have a VM on their own, spinning the
unikernel up when the requests came in, and let it process any request come
in, after a while, if the specific user stop requesting then you kill the VM.
There seems to be a lot of benefits from that: single user authentication,
horizontally scaling is simple ("vertical" scaling also get easier as you can
just move VM away), app design will easily let it become an on premise
installation. Code upgrade is also easier as it won't disturb everyone at
once, and you can upgrade when the VM is not running for a specific user (no
hotswap needed). There is definitely operational overhead in a DNS server, but
besides that it seems like the benefits would outweight the cons.

I know that's what DabbleDB was doing back then (in Squeak Smalltalk, each of
their users has a smalltalk VM serving them). Is there anyone else doing
anything similar nowadays?

~~~
avsm
That sort of design is what motivated Magnus' post that has been linked here.
We've been laying the uniSaaS groundwork with a new Xen toolstack daemon
called Jitsu ("Just in Time Summoning of Unikernels").

Here's the camera ready paper that we'll be presenting at USENIX NSDI in May:
[http://anil.recoil.org/papers/2015-nsdi-
jitsu.pdf](http://anil.recoil.org/papers/2015-nsdi-jitsu.pdf)

There's a lot more besides just the spawning to build a robust SaaS of course,
but you can see the direction we're heading in...

~~~
jacques_chester
I think the obstacle is that most folk are not prepared to give up their
languages for this, so containerising PaaSes are still likely to dominate the
scene.

Disclaimer: I worked on Cloud Foundry.

~~~
avsm
> I think the obstacle is that most folk are not prepared to give up their
> languages

s/languages/frameworks, and I agree. One step at a time -- this work is a
foundational block that we're building on, not the ultimate solution.

~~~
jacques_chester
Hm. Got me thinking about how to fit unikernels into the CF architecture,
which is largely taking containers (Garden or Docker, take your pick) as the
unit of currency at the low level.

It looks like the place to make the change would be in Diego in the
Executor[0][1] component, which currently implements behaviour in terms of
containers (using Linux-Garden as the container system).

So long as you can conceptually fit into the Receptor Actions API[2] it looks
as though unikernels would be a plausible addition to Executor. Once you have
that, you can either wire into Lattice[3] or a full Cloud Foundry installation
as a first-class citizen.

Disclaimer: I'm not as familiar with Diego as other CFers are, so it's
possible I'm just making stuff up.

[0] [https://github.com/cloudfoundry-
incubator/executor](https://github.com/cloudfoundry-incubator/executor)

[1] [https://github.com/cloudfoundry-incubator/diego-design-
notes](https://github.com/cloudfoundry-incubator/diego-design-notes)

[2] [https://github.com/cloudfoundry-
incubator/receptor/blob/mast...](https://github.com/cloudfoundry-
incubator/receptor/blob/master/doc/actions.md)

[3] [http://lattice.cf/](http://lattice.cf/)

~~~
avsm
Thanks, this are all _really_ helpful pointers! It's particularly useful to
find Lattice, which seems to be a simpler entry point than a full CF setup.

~~~
jacques_chester
The shiny website was new to me. I wonder if we've officially announced it
yet?

In any case, we started the Lattice effort last year to make it easier for
people to play with Diego without having to stand up a whole Cloud Foundry
installation.

My only Lattice-related claim to fame is not knowing what is was:
[https://github.com/pivotal-cf-experimental/lattice-
app/issue...](https://github.com/pivotal-cf-experimental/lattice-app/issues/1)

------
amirmc
The fun thing about this is that you can try it out locally and apparently it
took (about) an afternoon to put together (most of the work is in rewriting
URLs).

This kind of fast experimentation is going to lead to some really cool things
happening with unikernels.

------
thom_nic
I seem to remember something similar that was an erlang VM running (I think)
directly on Xen. The underlying idea in that case was each _request_ causes a
new VM to boot, service that request, then terminate.

~~~
tobz
Mentioned in another top-level comment:
[http://erlangonxen.org/](http://erlangonxen.org/)

------
caligastia
This is an interesting experiment in moving the computation to the endpoints
of a namespace. I would suggest a variant where instead of a VM you place a
Fossil file

[https://www.fossil-
scm.org/index.html/doc/trunk/www/fileform...](https://www.fossil-
scm.org/index.html/doc/trunk/www/fileformat.wiki)

Using that you can store a whole filesystem at the URL instead of one file,
and you also get synchronization mechanisms built in, so your endpoints can
communicate with each other in a structured way that you control.. vs the
undefined communication you will get between VM's if you try this with the VM
architecture.

------
UhUhUhUh
I'm not sure what the applications/market for this is or could be, but I love
the idea. Everything that goes "micro" I love dearly.

~~~
ccvannorman
I'm no DNS or security expert, but couldn't one application be increased
security of connections? E.g. it's harder to do that 0day kernel hack with a
buffer overflow if every URL request is sandboxed in a VM?

I probably got it wrong but I'd love to hear a more expert opinion on why/why
not this could be used for security..

------
lbotos
" I have not been able to find a hosting provider that has a minimum
configuration which would be appropriate for large or small unikernels (e.g.
8MB vs 64MB RAM)." This is the biggest problem with unikernels right now.

~~~
sp332
Looks like an arbitrage opportunity! Buy 64MB server, sell 8MB servers.

------
biot
Similar to ZeroVM [0], where a VM is spawned and destroyed for each individual
request.

[0] [http://www.zerovm.org/](http://www.zerovm.org/)

------
reilly3000
This is something I've been _dreaming_ of. It is lovely to see it coming into
the world. Death to the website! Long live the UX in a container!

~~~
agentultra
This is better than a "container." A Mirage unikernel literally compiles in
the pieces of an "OS" as libraries. There is no monolithic operating system
with code sitting around waiting to be exploited. The web server in this
example _is_ the kernel.

------
hunnypot
This is cool, especially when you think of the opportunities it offers for
DDoS prevention.

~~~
Nilef
Can you tell us more about these opportunities?

------
cjg
When do we upgrade from IPv6?

