
Zerg: Boot a VM instance per request, all in under 250 ms - tylerhou
http://zerg.erlangonxen.org/
======
vog
This is a very interesting development!

But doesn't that mean we are merely reimplementing old concepts on a new
level? Almost everything we do on VM level we already did on process level:

\- Read-only VM images are essentially like statically linked binaries, just
better encapsulated when they run.

\- Tiny, fast loading VM images are essentially stripped static binaries with
dead code elimination.

\- Starting those by incoming network request is just like old "inetd"
versions started custom handler programs on each request.

\- Same with CGI, just on HTTP rather than TCP level.

\- Manual maintaining the lifecycle of those VMs like FastCGI in nginx.

\- Maintaining the lifecycle of such a started VM is essentially like FastCGI
in Apache (or nginx + auto-spawner).

The only real difference is the improved encapsulation (controlled network
access and resource consumption) of the running code, isn't it?

What might come next?

\- Multiple VMs could share the same images to save memory, essentially
reinventing shared libraries and dynamic linking.

\- Network traffic into and out of a VM is controlled more tightly,
reinventing SELinux. (Perhaps followed by noticing that these profiles are
hard to maintain, rediscovering the maintenance issues of SELinux systems.)

\- Providing the network traffic shape directly with the image, so these can
be maintained at a single place, by the original developers. This would
reinvent Pledge.

\- Introducing permissions for network traffic, e.g. such that just one group
of application VMs can communicate with the database VMs. Ideally with an ACL
system as used for managing file permissions. This would reinvent Unix
Sockets.

~~~
catern
Yes, all this "container per request" or "unikernel per request" stuff is
essentially the same as what everyone was doing 20 years ago with basic CGI.
But this stuff also has far worse performance than CGI's "process per
request", since processes can be started very fast, in microseconds.

It would be more interesting if we just made more sandboxing powers available
for traditional processes, though there is already quite a lot of sandboxing
available for Linux processes...

~~~
user5994461
A process takes multiple milliseconds to start, not microseconds. Creating a
process is really expensive.

CGI starts a process per request. fastCGI can reuse processes. Problem already
found and solved 20 years ago.

~~~
catern

        catern@home ~ $ { timeout 1 sh -c 'while :; do /bin/echo foo; done'; echo; } | wc -l
        1542
    

So at least sub-millisecond. And this is less efficient than possible because
it's also writing to a pipe.

Also, I quite agree that CGI is a bad way to do things, but that's what the
original post is proposing. :)

------
jeswin
The demo might be down for now, but their vision
[http://erlangonxen.org/case/a-personal-
facebook](http://erlangonxen.org/case/a-personal-facebook) is something I
strongly believe in.

Basically, the cost of spinning VMs is coming down drastically (with Hyper.sh
and Intel ClearContainers offering 200ms boot) and it is now feasible to run
your apps on a personal cloud sandbox. You data resides in that box and runs
code you have approved; away from the prying eyes of FB, Google and the
government. Everyone's facebook will be an individually hosted app, talking to
each other over agreed upon protocols; as opposed to exchanging data within
the facebook infra.

Of course, a lot of UX and engineering problems need to be solved along the
way. Provisioning a personal sandbox cannot be any harder than creating an
account on a Social Network. Beiber's sandbox will need to distribute data to
105M followers efficiently. You'll need to patch vulnerabilities at scale, so
there needs to be some trust model for applying patches. You need relays to
overcome censorship etc.

~~~
amitport
A personal facebook still needs to talk with others' personal facebooks.

I tried to solve this problem using derivative regular expressions as graph
queries

paper:
[http://onlinelibrary.wiley.com/doi/10.1002/spe.2226/abstract](http://onlinelibrary.wiley.com/doi/10.1002/spe.2226/abstract)

source code:
[https://github.com/amitport/graphpack](https://github.com/amitport/graphpack)

(there are other solutions, I admit this is just a plug, but it will be great
if it could gain traction)

~~~
abstractbeliefs
What you - and the personal facebook case - are really describing is IndieWeb,
and already developing movement of home-building your social media and linking
with others implementing open protocols.

The main issue just now is that writing (or deploying) IndieWeb platforms and
hosting them is the preserve of geeks, and it will not take off and displace
current platforms until we can make it trivial for Mom and Dad to join us.

------
sairamkunala
Here's the google cached version -
[http://webcache.googleusercontent.com/search?q=cache:zerg.er...](http://webcache.googleusercontent.com/search?q=cache:zerg.erlangonxen.org&gws_rd=cr&dcr=0&ei=nnREWvr7BMHrvgTj_KMg)

------
Improvotter
Wouldn't this be good for shutting down unused VMs in order to save money when
renting infrastructure? Most website can go "down" during the night and when a
user tries to connect to it, it would take 0.2s extra, but after that it would
stay on.

Now that I think of it, spiders/bots might activate it beforehand. You could
use robots.txt, but not all spiders/bots seem to follow those "rules".

Food for thought!

~~~
rplnt
What does a website really do if it does nothing? What money does it consume?
What else can you do with the machine it's on? Wouldn't moving VMs around
accomplish the same goal?

Sidenote: some customers might use monitoring :)

~~~
JoeAltmaier
Cloud providers charge for time VMs are active. So it saves money to de-
commission them when idle.

------
htgb
It's definitely impressive, kinda like an own serverless implementation.

However, is there a point in shutting down the VM right away, other than being
a cool demo? It seems to me that you could quite easily keep it around if
there are more requests queued, and quickly spawn more workers if the queue
gets too long. So a regular worker pool, just that the number of workers can
be adjusted relatively quickly.

~~~
pjc50
> own serverless implementation

Isn't this just cgi-bin, right down to the long startup time and lack of
persistence?

~~~
jxf
It's similar in intention, but "serverless is just `cgi-bin`" misses a lot of
nuance. Zerg and cgi-bin are solving different problems and with different
implementations. It's extremely difficult, if not impossible in some
environments, to get the same kind of isolation with cgi-bin that you do out
of a VM.

~~~
catern
You gain on isolation with Zerg (and other "VM per request" things) but you
lose on everything else compared to CGI with processes.

But you can reach comparable levels of isolation with processes: just use
Linux namespaces.

------
MBCook
It seems I managed to get through right before it went down. Sort of a neat
idea. But is it really worth starting up a new VM for each request? That seems
to be the majority (380 ms?) of the time spent.

Slightly off-topic: the page lists be time for an android phone to boot as 50
seconds. Is that correct? I remember iPhones used to take forever but I
recently realized that they sing to boot extremely fast. I have no idea when
that change happened.

~~~
joombaga
Just tested my Pixel 1. It takes 24 seconds from button press to lock screen,
and another 18 seconds for me to type my password and display everything on
the home screen, but I'm a slow mobile typer and my password is 16 characters.

~~~
modeless
Just tested a Pixel 2 XL. 12 seconds to lock screen and 4 more to the home
screen including my pattern entry. I think this is the fastest booting phone
I've ever had.

Actually I had to do it twice because the first time it installed a system
update, but even then it only took 26 seconds to install, finish booting, and
show the lock screen. System updates have been seriously optimized in the last
few Android releases. They used to take like 5 minutes.

~~~
rplnt
Is this from off -> on or a reboot? I think I've noticed huge difference
between the two on my Samsung in the past.

~~~
modeless
From off.

------
murukesh_s
Can anyone explain what does this mean? How is it different from regular VMs?
Is it not booting an OS? What does it have to do with erlang?

~~~
rdtsc
It's an implementation of the Erlang VM on top of Xen. Xen is a virtualization
hypervisor. Erlang VM is a virtual machine which runs Erlang bytecode. The
default implementation is called BEAM and it runs on *nix, Windows and other
architectures. This implementation is more lightweight and only runs on top of
Xen. One of its main features is a fast boot time.

So to recap instead of the typical stack of:

    
    
        Erlang Code | BEAM VM | Linux or other OS | VM or Hardware
    

You get:

    
    
        Erlang Code | Ling | Xen | Hardware
    
    

The use cases could be isolation, scaling or what authors calls 0-footprint
cloud. That is, only spawn instances as you process requests. When requests
are not coming, you don't have any instances running. In more modern terms
maybe it could be used for building a serverless architecture used to run
cloud "functions"

Yes there is obvious irony here with the demo not handling requests well after
the HN frontage hug of death, but I think it is still a beta / experimental
product, it has been around for a good number of years and is somewhat
abandoned. If anyone knows more please feel free to correct me.

~~~
djhworld
I enjoyed your post, it made me think, could you do something similar with say
the JVM?

I know loading JVM based applications have horrific startup times on cold
boot, but I was just wondering could you run the JVM directly on top of Xen?

~~~
RainaRelanah
You might want to look at OSv[0], an extremely lightweight OS for running a
singular JVM process on cloud vms. There's a decent list of unikernels
(including OSv and LING) here[1].

0\. [http://osv.io/](http://osv.io/) 1\.
[http://unikernel.org/projects/](http://unikernel.org/projects/)

------
tylerhou
It seems like the demo is down from high traffic; here's an archive link for
those who can't see it:
[https://web.archive.org/web/20171227080115/http://zerg.erlan...](https://web.archive.org/web/20171227080115/http://zerg.erlangonxen.org/)

------
trevorriles
How would this compare to mirageos? It sounds similar to a project called
Jitsu.

[http://www.skjegstad.com/blog/2015/03/25/mirageos-vm-per-
url...](http://www.skjegstad.com/blog/2015/03/25/mirageos-vm-per-url-
experiment/)

~~~
trevorriles
This post is actually more relevant.
[http://www.skjegstad.com/blog/2015/08/17/jitsu-v02/](http://www.skjegstad.com/blog/2015/08/17/jitsu-v02/)

------
AntiRush
I always thought ZeroVM was a cool idea in this space. The NaCL runtime makes
a lot of sense as a base for servers that can take advantage of existing
libraries.

[http://www.zerovm.org](http://www.zerovm.org)

~~~
Smerity
After reading about ZeroVM I was incredibly excited - but then saw the last
Git commit was in January 2015 =[

Is it considered "stable" or is it simply abandoned? Are there any successors?
Near zero overhead VM acting on data storage is something I'll need for a
planned project of my own.

~~~
justincormack
It is abandoned, Rackspace laid them off. NaCl is still stable, although the
team is working on Web Assembly now which is a replacement (but less mature).
ZeroVM does still work but is a pain to build.

------
upbeatlinux
Didn't Rackspace acquire ZeroVM during it's 2010-2013 SaaS buying spree? I'd
anticipated seeing ZeroVM as some part of their "cloud" offering but it looks
like it was rolled into OpenStack Swift. Has Cloud Files performance improved
at all in the past 4 years and how much of this can be attributed to ZeroVM (a
lot of ignorance in this question I know).

------
anilgulecha
This is the future of computing/software -- pure commodity of CPU+memory.
Applications that build on top of these granular interfaces will largely the
next wave of successful architectures.

I'm thinking of building something here for static sites as an open source
project/spec. If anyone is interested in bouncing ideas on this, please ping
me via email (in profile).

~~~
dcow
I agree with your first paragraph. But why would this provide any benefit for
a static site where no compute is needed (or all compute is client side)?

~~~
anilgulecha
My bad -- I meant static sites with one additional functionality that all
static sites try to outsource: comments.

(it would no longer be strictly static, but the idea is to decouple site from
hosting/compute, so it would be easy to port a static/dynamic site between
providers at the switch of a DNS entry).

------
jesperlang
is Zerg related to Ling? It seems like this project has (sadly) died out.
Anyone know the current status?

[https://github.com/cloudozer/ling/issues/141](https://github.com/cloudozer/ling/issues/141)

~~~
jhgg
Yes, Zerg is powered by Ling. If you scroll down you can see how it's involved
in the request summary. A cached copy of the page is in a sibling comment

------
exikyut
> _Over capacity_

> _The demo is limited to 16 concurrent instances and 2 libvirt connections.
> Due to these limitations we were unable to spawn a new instance to service
> your request. Please try again later._

Kool.

~~~
AdamJacobMuller
Did they just inadvertently point out why this is a bad idea for anything
general-purpose?

~~~
derefr
Not really; those are exactly the sort of config settings you'd sensibly use
to prevent someone from DDoSing your untrusted prototype-level infrastructure.

If your VMs are truly cheap, and you've ironed out the kinks from your proof-
of-concept, you'd obviously set up your production system to run more than 16
of them.

~~~
StreamBright
Right, and who decides that a request coming from a customer ot an attacker?
If the attacker can generate enough requests non of the customers have a
chance to use your service. This is absolutely not the way to handle DDoS.

~~~
derefr
My point was that it's a sensible strategy for the demo of a technology that's
still under development (like this project), where you don't actually have any
"customers." In production, you'd let it scale unboundedly, and combine it
with some sort of anti-DDoS technology like Cloudflare.

------
voiper1
If it's down, just hit reload a few times.

I see phase A from 126.3 ms to 187ms.

------
maxpert
Over capacity error

~~~
skocznymroczny
Need more overlords

~~~
berbec
You must construct additional Pylons

------
waits
> Server over capacity

Ah, there’s the rub.

