
Wasmjit: Kernel Mode WebAssembly Runtime for Linux - varenc
https://github.com/rianhunter/wasmjit
======
ncmncm
The actual problem with this, after you have satisfied yourself about the
numerous non-problems, is that practically any real app in wasm has calls to
library code implemented in C or C++ and compiled to real asm to implement the
extended Javascript runtime, that you would not want running in ring 0. To
those are added all the kernel-mode functions that must be used correctly. (If
the app can be provoked to use them wrong, boom.)

So, the plausible use is an implicitly trusted app not exposed to untrusted
input, that is system-call bound, and that you want to distribute cross-
platform without recompiling to targets that have installed this module in a
known kernel version. It seems easier just to distribute source for a kernel
module; or kernel modules built for the (2? 3 tops) architectures required.

[edit: spelling error.]

~~~
_wmd
I'm not sure why linking third party code is a concern, it all must live
inside the sandbox provided by the runtime. This reduces the trust
requirements from depending on a (software | hardware) solution where the
hardware portion cannot be audited, and by 2018 are no longer to be considered
trustworthy by default, to only depending on access and bounds checks
implemented by a single body of code visible to everyone.

The 'system call interface' available to those buggy libraries need be no more
capable than existed previously, that system calls are serviced without
switching protection mode need not extend the potency of any possible attacks.

A future implementation of this style of design will have much better security
and auditability properties than anything that has ever popularly existed in
the past. It may have originated with the web, but I welcome wasm and all the
market punch it promises to bear on ancient and long-unquestioned corners of
our software stack like this. (Naturally I was also a Microsoft Singularity
fan)

~~~
deathanatos
How does WebAssembly fix hardware issues? For example, take Rowhammer. I don't
see how running your code in a VM fixes Rowhammer; surely there would still be
some mapping of data in wasm to RAM, and I don't see why a sufficiently
motivated individual couldn't determine and subsequently exploit that mapping.

Or what does WASM do about the concerns that major CPU vendors embed a second
co-processor running untrusted, unauditable code with ring -1 access alongside
the main CPU?¹

¹which may or may not be active, depending on CPU model and configuration.
I've never really fully understood when it is, or isn't, but I think it's
somewhat moot.

Further, this assumes the runtime (which is not _just_ the WASM runtime:
system calls and other functionality will inevitably need access to the
actual, underlying hardware in order to do their job) from being buggy, and
since the code is now in ring 0, you get all the consequences of that. (Full,
unrestricted access to _everything_. At least previously, you'd need to find
some root exploit to get that.)

(Also, as another poster hints below, since everything is running inside the
same protection layer from the CPU's point of view, what prevents you from
just running the same attack, in spirit, as Spectre? Only now, there is no
protection — everything is ring 0 — so the CPU is "correct" to speculate a
read. Sure, the VM will deny it to the code inside the VM, but that didn't
matter in the case of Spectre?)

~~~
candiodari
> since the code is now in ring 0, you get all the consequences of that

No you don't. Because WASM does not provide many of the facilities that real
assembly provides. So, for instance, there is no way to stack-smash using WASM
instructions, regardless of the any security problems in the code itself, or
even outright malicious code. It just can't do it.

More generally for VMs, there are secure VMs which provide a mathematical
proof that the code will, for instance, observe memory safety. Such a proof is
much better than ring-level protection, because :

1) You can verify the proof. Good luck verifying the hardware implementation
of ring-level security in processors

2) It doesn't take any resources at runtime

3) the proof can be verified at code load time, so insecure code (accidentally
insecure or otherwise) just doesn't start executing, ever

4) It doesn't allow for manufacturers to hide "secure coprocessors" or any
other bullshit like that

------
saurik
I have seen at least three projects to run WebAssembly in the kernel, but not
a single strong attempt at a good userland binary format for WebAssembly: I
don't want to run WebAssembly in my kernel, I just want an architecture
agnostic binary format for Linux executables.

~~~
JoshTriplett
> I just want an architecture agnostic binary format for Linux executables.

What would you use that for, if you had it?

~~~
pjmlp
I guess like IBM i, IBM z and Unisys ClearPath, have a computer system that
you can change whatever you feel like at CPU level and have only the kernel
take care of the underlying changes.

Universities should teach more about mainframe architectures.

------
kettlecorn
Increasingly I’m becoming convinced that something like this may become the
backbone of future computing.

Imagine truly cross platform code that performs well, and can be written in
whatever language you care for.

The web illustrated certain advantages to developing software that can be
easily delivered and run on other systems. However, the learnings of the web
were always coupled to the browser.

WebAssembly, through projects like this, may soon be able to empower cross
platform code that delivers small, well performing programs on whatever
platform a user desires to use.

~~~
zimablue
Genuinely ignorant question, what's the difference with C from this
perspective? I thought the whole point of C was to be a common language.

~~~
naasking
Binary compatibility across architectures, no undefined behaviours are the
primary ones.

~~~
quickben
Yes of course, here is an example of a well defined, no ub, == overloads.

[https://slikts.github.io/js-equality-game/](https://slikts.github.io/js-
equality-game/)

It is all bound to make JS kernel programing a Pilar of OS Stability.

~~~
rictic
You're confusing JS with wasm and undefined behavior with confusing behavior.
JS and wasm are strongly defined and the specifications consider undefined
behavior a bug.

------
joshumax
I stumbled across WasmJIT about a year ago and actually used many of the
concepts in the project for my own kernelspace implementation of the portable
native client (pNaCl). Projects like these are extremely interesting to me
because they offer the possibility of a completely architecture-neutral
userspace that can be used across multiple devices without recompilation.

~~~
infogulch
This goes a step further and could offer an architecture-neutral _kernel
space_. If part of loading the module includes validation and external memory
access is checked as in browser implementations this could even lead to
safer/less exploitable kernel modules with more well-defined failure modes.

~~~
ChickeNES
The problem I see with this is that there are still no 8bit or 16bit data
types (or bitfields for that matter) in the wasm spec, which are kind of
useful for interfacing with hardware and for protocols (Ethernet, TCP/IP,
etc). Sure, you can pack and unpack them with masks and shifts (which I assume
clang,LLVM,gcc,etc do), but it seems awfully inefficient when the resulting
code ends up running on x86 or other architectures with native 8/16 bit
instructions.

~~~
josephg
Unless I'm missing something, this should be pretty easy for the webassembly
working group to fix. At a guess they probably just need some clear decisions
around how to handle endian issues. But right now they're just focussing on
the browser use case.

If kernel module developers become a vocal part of the wasm ecosystem, I'd
expect 8- and 16- bit data types to get added before long.

~~~
stepik777
> they probably just need some clear decisions around how to handle endian
> issues

They are handling endian issues by simply requiring little-endian.

------
merlincorey
My mind boggles at the meta-ness of using this software to (direct from the
docs) speed up the operation of a webserver compiled to webassembly.

The paranoid security engineer in me screams "Ring0 is what I always wanted my
browser to have access to".

Interesting project!

~~~
quickben
Do not worry, it will be secure. Signed, with verifiable checksums, and gdrp
friendly.

Just, erm, load these binaries provided by FB, Google, Microsoft and Netflix.
:P

------
muricula
This neuters a bunch of the new spectre and meltdown mitigations which are
required if you want ring 0 to be a security boundary.

~~~
comesee
How so? WebAssembly can't address memory outside of its sandbox

~~~
BeeOnRope
The whole point of Spectre is that you can indirectly access memory outside of
your "sandbox" even if you don't have permissions to those pages, using
various side channels.

~~~
comesee
Aren't those side channels dependent on addressing kernel memory that you
don't have permissions for? You can't address kernel memory in WebAssembly,
therefore you can't provoke the speculator into caching those pages.

~~~
dbaupp
I believe Spectre is driven by processor speculation into a bounds-checked
array access, even for when the condition fails (that is, the index is out of
bounds). One can get to arbitrary memory in this way using the right index on
any array.

~~~
vardump
Why would you need traditional bounds checking with Wasm?

Just use MMU hardware to insert 2GB [0] no man's land below and above Wasm
memory and only allow signed 32-bit indexing (-2^31 — 2^31-1).

This way the attacker can only read sandbox memory (and the useless 2 GB no
man's land, mapped to pages full of zeroes or whatever).

When there's no speculation involved or the data is simply out of "speculative
range", Spectre is toothless.

[0]: 2GB is just a basic example. More may be required if for example
something like x86 SIB (Scale Index Base) is used for multiplying the index by
2, 4 or 8.

~~~
dbaupp
"Just" is never a good word in a technical discussion, especially around
security vulnerabilities like Spectre.

That's a good idea, but there's also several reasons that may not be
appropriate:

\- WASM explicitly says that it may be extended to 64-bit indexing (more than
4GB of addressable memory is definitely useful for some things)

\- Spending 4GB of (hopefully, virtual) memory on every WASM instance may be
undesirable or impossible (e.g. 32-bit processor)

That said, it's very reasonable to impose restrictions on things running in
ring-0, and wasmjit could well require a 64-bit machine with 32-bit WASM
indices (which I imagine would be okay assumptions for things one would do
with it anyway).

~~~
vardump
> \- WASM explicitly says that it may be extended to 64-bit indexing (more
> than 4GB of addressable memory is definitely useful for some things)

In that case, just fall back to bitwise AND index clamping. A small
performance penalty, but nothing major.

> \- Spending 4GB of (hopefully, virtual) memory on every WASM instance may be
> undesirable or impossible (e.g. 32-bit processor)

Just page table entries. Wasting physical memory for that would be pointless.
If the entries need to be mapped, on x86-64 it'd incur 4 kB, 2 MB or 1 GB
total "wasted" memory, depending on which page size granularity you want to
use. Of course, you could also simultaneously use this "wasted" memory for any
non-sensitive data.

Well, mapping 2x 2GB memory using 4kB pages does take up hmm... 8 MB of RAM
for the PTEs. So perhaps 2 MB pages would be optimal.

~~~
dbaupp
_> In that case, just fall back to bitwise AND index clamping. A small
performance penalty, but nothing major._

Masking the index will break code that is actually using the larger address
space: running true 64-bit WASM code (as in, using >4GB of space) won't work,
which is what I was referring to.

 _> page table entries_

Indeed, hence the reference to virtual memory. In any case, because both
x86-64 and ARM64 only have 48 bits of actually addressable space, that 4GB of
overhead (plus, up to 4GB of actual addressable memory) only allows for 65536
(or half that) WASM instances. That's definitely a large number, but not one
that is out of reach.

~~~
vardump
> Masking the index will break code that is actually using the larger address
> space: running true 64-bit WASM code (as in, using >4GB of space) won't
> work, which is what I was referring to.

You can also clamp for example at 33-37 bits, giving 8-128 GB array range.

~~~
comesee
You can do masking in the same way Linux does it. It prevents "branch code
bypass" without using an explicit size:

    
    
        cmp %bound, %ptr
        jae bad_ptr
        sbb %mask, %mask
        and %mask, %ptr
    

Just two extra instructions. No need to memory map or hard code the size of
bounds.

See `array_index_mask_nospec` in
[https://github.com/torvalds/linux/blob/master/arch/x86/inclu...](https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/barrier.h)

~~~
vardump
> Just two extra instructions. No need to memory map or hard code the size of
> bounds.

Pretty neat idea! [Although the (register) dependency chain looks a bit nasty.
'and' will need 'sbb' to commit and 'sbb' will need to wait for 'cmp' to
commit (flags register). But I guess the few/rare cases where this latency is
really an issue can be dealt one-by-one basis.]

> No need to memory map

Well, using MMU can have performance benefits. Less repetitive bounds checking
code and better performance in most scenarios. Both solutions have their
strengths and issues, there are no silver bullets.

~~~
comesee
Good point on the MMU performance advantage and trade offs involved. When
everyone's heads were on fire, made sense to indiscriminately mask off user
controlled pointers. Now that the dust has settled a bit I imagine we'll see
more usage of memory mapping tricks in performance critical sections.

------
justinclift
Related to this, the wasm generated by Go is currently tied to being executed
in the browser.

There doesn't yet seem to be a implementable standard to define what a non-
browser execution environment looks like. The beginnings of a common spec are
here though, which some places have started working with:

[https://github.com/CommonWA/cwa-spec](https://github.com/CommonWA/cwa-spec)

That'll likely need to make it's way into the official specs:

* [https://github.com/WebAssembly/design](https://github.com/WebAssembly/design)

* [https://github.com/WebAssembly/spec](https://github.com/WebAssembly/spec)

At which point implementations will have something to focus on. :)

For the Go, this is the (recent) matching issue:

[https://github.com/golang/go/issues/27766](https://github.com/golang/go/issues/27766)

Kind of guessing it'll turn into a tracking issue to get it done.

------
yissp
Prescient [https://www.destroyallsoftware.com/talks/the-birth-and-
death...](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-
javascript)

~~~
wcrichton
Except, WebAssembly isn't even close to Javascript. They're completely
different languages. WebAssembly is closer to C than to Javascript.

~~~
gtremper
WebAssembly is basically just a binary encoding of asmjs, which is the subset
of javascript discussed in the talk.

~~~
callahad
While asm.js was basically just a textual encoding of C in JavaScript... round
and round we go! :)

~~~
gtremper
I'd say it more a textual encoding of LLVM IR. Which makes the s-expression
text format of WebAssembly a text encoding of a binary encoding of a
javascript encoding of a compiler intermediate representation of your program.
Round and round indeed.

------
ArtWomb
>>> results in a significant performance increase for syscall-bound programs
like web servers

Reminds me of "Why we use the kernel's TCP stack"? And eternal debate over the
performance benefits of "kernel bypass" and "zero-copy" technologies such as
DPDK versus full userspace TCP implementations like OpenOnload.

[https://blog.cloudflare.com/why-we-use-the-linux-kernels-
tcp...](https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/)

Conclusion is that the untapped fruit in kernel TCP gains comes from tuning
performance of the network stack itself: optimizing overhead a packet incurs
on its path through the stack. A very active area of development.

Netdev 0x13, THE Technical Conference on Linux Networking

[https://netdevconf.org/0x13/](https://netdevconf.org/0x13/)

~~~
snaky
The recent work on AF_XDP seems like it could became established middleground
sweet spot almost as efficient as kernel bypass and userspace networking.

------
snaky
"Lua support in the NetBSD kernel" (2013)
[https://news.ycombinator.com/item?id=6562611](https://news.ycombinator.com/item?id=6562611)

~~~
gnufx
Long ago someone stuffed a Scheme(-ish?) interpreter into Linux, but I don't
think it went anywhere. I forget the details, but it may have used the SIOD
implementation. I'd prefer it in a microkernel's userspace, though...

------
TimTheTinker
Another step towards Metal[0] becoming a reality.

[0] A (half) joke from Gary Bernhardt's excellent _The Birth and Death of
JavaScript_ (pronounced "yavascript"):
[https://www.destroyallsoftware.com/talks/the-birth-and-
death...](https://www.destroyallsoftware.com/talks/the-birth-and-death-of-
javascript)

------
lachlan-sneff
nebulet is an implementation of a similar idea, but on a different scale.

[https://github.com/nebulet/nebulet](https://github.com/nebulet/nebulet)

~~~
moosingin3space
How has the work on Nebulet's IPC layer progressed? In theory, a kernel-mode
wasm interpreter could make IPC as fast as calling a function in a shared
library through a vtable. Of course, this has to be mediated properly to avoid
letting processes mess with each others' memory!

~~~
lachlan-sneff
Slowly. I've been quite busy with school. What little time I have spent on
nebulet has been on the fledgling network stack.

~~~
moosingin3space
I see. I'm excited about nebulet and would be interested in participating in
the design process for the IPC layer -- is there a place that discussion is
taking place?

~~~
lachlan-sneff
Yeah, totally! The gitter is where most conversations about this take place:
[https://gitter.im/nebulet/nebulet](https://gitter.im/nebulet/nebulet)

------
JepZ
Does anybody know if that kernel module can execute WASM generated from Go or
Rust too?

~~~
steveklabnik
Unless they’re doing something non-standard, wasm doesn’t care what language
it came from.

That said, this claims it relies on emscripten, I think? You _can_ do that
with Rust, though it’s not the preferred way. I sure about Go.

~~~
comesee
From looking at high_level.h it can instantiate any wasm file, but it seems to
need the host code specially integrated. There is a function to instantiate
the emscripten runtime, probably code can be added to instantiate the go or
rust runtimes.

~~~
steveklabnik
Rust doesn’t have a runtime. Well, at least not in the sense of other
languages; it’s comparable to C :)

Emscripten provides a libc; I wondered if it required it. I guess not.

~~~
lloeki
> it’s comparable to C

I think you may want to reassess your definition of what a _runtime_ is ;)

> "crt" stands for "C runtime", and the zero stands for "the very beginning".

[ [https://en.wikipedia.org/wiki/Crt0](https://en.wikipedia.org/wiki/Crt0) |
[https://en.wikipedia.org/wiki/Runtime_library](https://en.wikipedia.org/wiki/Runtime_library)
]

Yup. C definitely has a runtime!

~~~
steveklabnik
That’s why I said that Rust’s is comparable to C. Both have very, very small
ones. Which is what most people mean when they say “no runtime”, since every
non-assembly language has some amount of runtime.

------
elcritch
Fascinating concept! Wouldn’t this pause/block the kernel while the module
runs? Can’t find anything in the readme. I always assumed kernel modules had
to behave politely in terms of returning and not running indefinitely.

------
gigatexal
I’ll keep all my apps if possible in user space please. Call me paranoid.

------
uluyol
Serious question: what's the advantage of something like this over eBPF? As I
understand it, you can generate eBPF bytecode from C and Linux already has a
JIT compiler for it.

~~~
kbumsik
Many other modern languages such as Rust and Go are aiming to _officially_
support WASM. I don't know much about eBPF but it looks like it only support C
for now.

~~~
steveklabnik
A company just announced some cool eBPF and Rust stuff today:
[https://blog.redsift.com/labs/ebpf-ingrained-in-
rust/](https://blog.redsift.com/labs/ebpf-ingrained-in-rust/)

------
_pmf_
Finally, full stack engineers live up to their title.

