
Nginx on Wasmjit - wofo
https://www.wasmjit.org/blog/nginx-on-wasmjit.html
======
gok
Sounds like the big idea from Singularity (run everything in the same hardware
security ring, use static analysis for memory safety) is going mainstream,
incrementally. Unfortunate that this is happening right as it becomes clear
that such a design is fundamentally unsound on modern hardware.

~~~
zimbatm
+1 for pointing at the irony but it's not really true.

1\. Hardware design flaws are not irremediable and will be fixed in time.

2\. It's still very useful to run software that is trusted to not be malicious
and/but not trusted to be void of memory bugs.

~~~
kllrnohj
> Hardware design flaws are not irremediable and will be fixed in time.

There's no particular reason to believe this. Neither Intel nor AMD have
committed to any form of side-effect-free speculative execution or similar.
That'd required a rather large chunk of transistors and die space, and if
nobody is footing the bill for it it's not going to happen.

Preventing side-effects from leaking between _processes_ (and ring levels)
entirely at the hardware level is definitely going to happen. Within a
process, though? That's not going to happen without multiple major players
demanding it. Since nearly all the major CPU consumers are currently happy
with process boundaries being the security enforcement zones, there's no
particular reason to believe that in-process sandboxing will ever have
hardware fixes to prevent spectre attacks.

~~~
monocasa
> That's not going to happen without multiple major players demanding it.
> Since nearly all the major CPU consumers are currently happy with process
> boundaries being the security enforcement zones, there's no particular
> reason to believe that in-process sandboxing will ever have hardware fixes
> to prevent spectre attacks.

The amount of untrusted code running in browser is only increasing over time,
and the CPU vendors have absolutely noticed this. That's why ARMv8.3A added an
instruction that literally has JavaScript in it's name (FJCVTZS) for instance.

Once you have mitigations for colocating trusted and untrusted code in ring 3,
doing the same in ring 0 almost certainly isn't a big deal.

~~~
kllrnohj
Browser vendors have already rolled out process isolation to handle this. So
why would CPU vendors spend silicon on something userspace has already solved
with the building blocks that are already supported?

In-process sandboxing by all appearances is simply dead.

As for "doing the same in ring 0 almost certainly isn't a big deal" no, very
extremely no. If you let code run in ring 0 it has everything ring 0 can do,
period. You cannot put restrictions on it, that's what spectre proved. Give
code access to a process and it has _entire_ access to that process. Similarly
give something ring 0, and it has the _entirety_ of ring 0.

Untrusted code goes in ring 3 in an isolated process. That's the security
model of x86, and it's the only model that CPU vendors have any pressure to
fix.

~~~
monocasa
It's worse than what you're stating. Spectre is remotely exploitable with
network packets (NetSpectre). The current state of the world isn't just fine
with process level security, and the chip vendors are going to need to fix it,
because ultimately you can exploit then if you control the data, not just the
code. They have several options available to them, all of which also let's you
run sandboxed untrusted code in the same MMU context as trusted code.

~~~
kllrnohj
It is unclear if NetSpectre works in a real environment, but either way it's
fundamentally similar to an IPC call and can be treated as such. As in, that's
where you'd insert hardening boundaries.

Chip vendors have & will fix process boundaries. But _nobody_ is talking about
any sort of protection of any kind that would let in-process sandboxing work
again. It's just not even on the table at this point.

------
rusbus
The intention here is to use wasm to allow you safely run user code _within_
the kernel. Their primary targets are nginx and FUSE. Conceivably, avoiding
the context switch into and out of the kernel will have significant
performance implications, but there aren't any numbers out yet for nginx
specifically.

~~~
simmons
That's certainly a fascinating idea. My initial thought was "Wait, doesn't the
kernel already provide a sandboxed execution environment -- called userspace?"
This would still have scheduling overhead, but I assume the idea is to avoid a
lot of the other context switching steps such as switching page tables. And
instead rely on Wasm/JIT checks to ensure ahead of time that memory violations
won't happen.

~~~
wahern
Once upon a time syscalls were slow, but architectures now provide features
like syscall/sysenter for switching privilege levels, with costs comparable to
userspace function calls.

Once upon a time switching page tables was slow, but now we have features like
PCID that allow preserving buffers.

Soon, if not already, the principle cost to context switching will be the
necessity to flush prediction and data buffers. In-kernel solutions like
Wasmjit must incur the same costs. Quite possibly they may turn out to be
slower overall: 1) they won't be able to take advantage of the same hardware
optimized privilege management facilities (existing and future ones--imagine
tagged prediction buffers much like PCID), and 2) they still incur the extra
runtime overhead of running in a VM which, JIT-optimized or not, eats into
limited resources like those prediction and data buffers that have become so
critical to maximizing performance.

Granted, if it's going to work well at all than Nginx seems like a good bet,
especially because of I/O. But there are many other solutions to that problem.
Obsession with DPDK may be waning, but zero-copy AIO is still a thing and
there are more ergonomic userspace alternatives (existing and in the pipeline)
that let you leverage the in-kernel network stack without having to incur
copying costs. And then there are solutions like QUIC that redefine the
problem and which should work extremely well with existing zero-copy
interfaces.

CPUs are incredibly complex precisely because so much of the security heavy-
lifting once performed in the OS is being accomplished in the CPU or dedicated
controllers. And these newer optimizations were designed to be integrated
within the context of the traditional userspace/kernel split.

Wasmjit looks like an extremely cool project and I don't doubt its utility.
There's plenty of room for alternative approaches, I just don't think the
value-add is all that obvious.[1] Probably less to do with performance and
more to do with providing a clear, stable, well-supported environment for
solving (and subsequently maintaining!) difficult integration problems.

[1] I just want to reiterate that by saying the value-add isn't obvious I'm
not implying anything about the potential magnitude of that value-add. I've
been around long enough to understand that most pain points are invisible and
just because I can't see them or people can't articulate them doesn't mean
they don't exist or that the potential for serious disruption isn't there.

~~~
brobdingnagians
Just an honest question, could you elaborate on what methods are the: "there
are more ergonomic userspace alternatives (existing and in the pipeline) that
let you leverage the in-kernel network stack without having to incur copying
costs". I've been curious about DPDK, FStack, Seastar, IncludeOS/MirageOS,
etc. but wondering if there are easier ways to get the zero-copy IO.

~~~
wahern
Off the top of my head:

Netmap - DPDK-like packet munging performance but with interfaces and
semantics that behave more like traditional APIs. Signaling occurs through a
pollable descriptor, meaning you can handle synchronization and work queueing
problems much more like you would normally.

vmsplice - IIRC it recently became possible to be able to reliably detect when
a page loan can be reclaimed, which is (or hopefully was) the biggest
impediment to convenient use of vmsplice.

peeking - Until recently Linux poll/epoll didn't obey SO_RCVLOWAT, which made
it problematic to peek at data before using splice() to shuttle data or
dequeueing a connection request. I have a strong suspicion that before this
fix many apps like SSL sniffers simply burnt CPU cycles without anybody
realizing. Though in the Cloud age we seem much more tolerant of spurious,
unreproducible latency and connectivity "glitches".

AIO - There's always activity around Linux's AIO interfaces. I don't keep
track but there may have been a ring-buffer patch merged which allows
dequeueing newly arrived events or data without having to poll for readiness
first.

Device Passthru - CPU VM monitor extensions make it easier to work with
devices directly. Not quite the same thing as traditional userspace/kernel
interfaces, but it seems like people are increasingly running what otherwise
look like (and implemented like) regular userpace apps within VM monitor
frameworks. Like with Netmap all you really need is a singular notification
primitive (possibly synthesized yourself) that allows you apply whatever model
of concurrency you want--asynchronous, synchronous, or some combination--and
in a way that is composable and friendly to regular userspace frameworks. VM
monitor APIs and device pass thru permit arranging the burdens between
userspace/VM and the kernel more optimally.

------
sargun
I don’t understand. How is this possible? The POSIX API is not implemented for
WASM. Non web embedding have not yet been standardized, nor threads. How have
they implemented this? Are they implementing a non standard embedding and
pthreads?

[https://webassembly.org/docs/non-web/](https://webassembly.org/docs/non-web/)

------
drzaiusx11
It seems like wasm will finally enable the "write once, run everywhere"
promise that java made but never truly executed by starting with the premise
that you don't need "one true language" (java), but rather just the VM.

Yeah, I know the JVM supports several languages these days but most require
non-superficial similarities to Java (garbage collected, etc.)

~~~
DannyBee
This ignores the tons and tons and tons of work that would really have changed
that. I don't think it really has anything to do with whether you have one
true language or not.

There were plenty of well-funded efforts to have "write once, run everywhere"
in the past that were just VM's and formats (ANDF, etc).

In practice, a lot of things have changed since Java that have made this kind
of approach feasible. As a simple example: good compiler infrastructure to
build on top of is _much_ more available than it was then. These days you
pretty much just have to write a frontend.

Even though GCC existed then, it was still compiling statement at a time!

~~~
drzaiusx11
I know there have been many efforts to write a universal VM. Two things I
think will make wasm more successful than prior attempts is:

1\. it comes on with platforms already via a browser or node.js

2\. like you said, the tools are here now (really LLVM made most of this
possible)

~~~
DannyBee
Sure, i'm just saying your comment seemed to be saying "java's write-once run
everywhere failing was due to trying to have one true language", and i think
that part is fairly orthogonal.

~~~
drzaiusx11
I don't believe it's failure is entirely based on it shipping with a
prescribed language--but I wouldn't say it's orthogonal. Java, the language,
promised "write once, run everywhere" whereas the JVM in the beginning was
just an implementation detail. The JVM slowly evolved into a universal vm
concept, mostly at the hands of the community and not those (Sun, Oracle) that
had the most control over it.

------
jeltz
How do you avoid having to copy buffers from the kernel into WebAssembly? As
far as I know WebAssembly does not provide a way to access memory outside the
linear memory block, but maybe in this specific case (nginx) it is possible to
have all syscalls write to a buffer allocated by WebAssembly.

------
lachlan-sneff
Wasmer just got nginx working on their wasm runtime as well!

------
coverband
I might be getting too old, because I truly don't get the benefit of doing
this... Nobody I know really cares about portability, and I don't see how
running nginx with WASM is in any way better than running it directly on the
system. Does anyone care to ELI5 for me?

~~~
lwb
This is satire but still very interesting:
[https://www.destroyallsoftware.com/talks/the-birth-and-
death...](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-
javascript)

------
zymhan
Does anyone else look at this title and see a jumble of letters?

I miss the days of pronounceable acronyms

~~~
idle_zealot
"engine-ecks on whaz'm-jit" It's certainly pronounceable, though perhaps that
pronunciation is non-obvious.

------
traverseda
Obligatory birth-and-death-of-javascript

[https://www.destroyallsoftware.com/talks/the-birth-and-
death...](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-
javascript)

I suppose calling it METAL would have been too on-the-nose.

~~~
molf
Not sure why you're downvoted, this is a very interesting talk.

~~~
traverseda
It's pretty relevant to the discussion of "why would you want to run wasm in
the kernel", but I'm not too worried about the votes.

~~~
reitzensteinm
It's not, though... WebAssembly doesn't really have all that much to do with
js, any more than Flash or Java plugins would if they ended up being
standardized instead. Every time there's a wasm thread it gets posted, but it
misses the point of the talk to suggest that wasm is the prediction bearing
fruit.

The talk is great, but I'd suggest that's the reason for the downvotes.

~~~
earenndil
> it misses the point of the talk to suggest that wasm is the prediction
> bearing fruit

Does it? It seems that the talk has two main points:

1\. Javascript succeeded because it was (at least initially) just good enough
not to be completely unbearable, but bad enough that people ended up using it
primarily as a target for other languages.

2\. Ring 0 JIT can be 4% faster that normal binaries.

WASM is primarily a target for other languages, and qualifies as a language
that can theoretically be JITted 4% faster than native code can be run.

~~~
reitzensteinm
Point number one isn't applicable to wasm.

The execution inside the kernel is related, but nobody replies to a lua in
kernel post with a link to the talk.

~~~
pjmlp
Because Lua isn't related to Web development, while JavaScript and WASM are.

~~~
reitzensteinm
Right, but that's my point. Wasm is tenuously connected with JavaScript
because both are web technologies, so people link the talk.

But they couldn't be more different technically, and if wasm does indeed
become the lingua franca of future computing it will be much more boring than
the craziness of js doing the same.

The talk was great because it was about an insane yet plausible future. We now
have a boring and probable future.

~~~
pjmlp
I don't believe in that, too much experience to believe in a miracle bytecode
format that will magically succeeded where others failed.

It will just be yet another VM platform.

~~~
reitzensteinm
Even less reason to link to the talk then :-)

------
syrusakbary
Hi HN!

I'm Syrus, from the Wasmer team. We have been working in something similar,
but with a special focus on maintainability and with bigger goals in mind:

[https://wasmer.io/](https://wasmer.io/)

Here is the article about our journey on Running Nginx (which funnily enough
we actually accomplished just before wasmjit):

[https://medium.com/@syrusakbary/running-nginx-with-
webassemb...](https://medium.com/@syrusakbary/running-nginx-with-
webassembly-6353c02c08ac)

~~~
kodablah
> but much more maintainable.

> we actually accomplished just before wasmjit

> Wasmer is the first native WebAssembly runtime [...]

Whoa there. I like both projects and respect competition as much as the next
guy, but maintainability is subjective and being first is of little
importance.

~~~
syrusakbary
True! This statements without analysis are wet paper.

In the article I've linked there is a better analysis on why:

1\. in wasmjit, the machine instructions are hardcoded into the runtime (this
is like creating your own LLVM, by hand... and only available for x86)

2\. it doesn't have a single test

I was talking about my own experience here, because I tried to contribute to
wasmjit before creating Wasmer... and was quite challenging!

It might be useful to check how many people interacted with the code in each
of this projects! ;)

Also note the timeline that took for this projects to accomplish the same:
wasmer (<2 months) wasmjit (6 months)

~~~
hangonhn
You may be correct on all those points but still come off as rude. And I think
you're just trying to be helpful and steer people in the direction that you
think is right. However, given the context of where you're doing it, it feels
like someone is trying to ruin someone else's parade.

~~~
glibgil
They are competing and think theirs is better. They are trying to get people
to look at the competition. That’s not ruining anything for anyone

~~~
zapzupnz
Then the discussion should have been a post unto itself, not a comment piggy-
backing on another project's thread. I dunno, seems like basic netiquette to
me.

But I agree that if you think your solution is better, there's really nowhere
better to put it out there than in front of eyes that are looking at something
similar.

~~~
cryptoplot
Piggy backing rubs the wrong way, but sometimes there is interesting content.

Personally, I would challenge the piggybacker to show me something.

Talk is cheap.

