Hacker News new | past | comments | ask | show | jobs | submit login
Lunatic is an Erlang-inspired runtime for WebAssembly (lunatic.solutions)
131 points by juancampa 5 months ago | hide | past | favorite | 40 comments



After reading the Elixir implementation of X-Plane MMO backend (which appeared here yesterday, 1), I came across this: https://github.com/lumen/lumen. It is an alternative BEAM implementation, designed for WebAssembly. I am personally learning Rust and I find it interesting that there are multiple BEAM (Erlang runtime) inspired implementations in Rust gaining traction.

1. https://news.ycombinator.com/item?id=27998323


Thanks for posting this on a Friday, this is basically gonna be my weekend now. I love the idea of meshing a lightweight process framework with webassembly.


> Lunatic builds on WebAssembly's security. We all use unaudited third-party libs that get deployed with our code, Lunatic can use capability based security to limit them.

If they happen to load all dependencies into a giant blob of linear memory, C written extensions will take care of corrupting their runtime's memory, regardless of WebAssembly "security".


Yes, that's the point. If you need to parse a JPEG, you spawn a VM to do so. It is allowed to use n seconds of CPU, k bytes of memory, and send a bitmap back to the caller.

(Sure, you could screw it up if the caller accepts a negative-size bitmap and corrupts its heap. Lets assume that process is written in a memory-safe language.)


Assumptions is how exploits get triggered.


It would be unlikely to incorrectly assume that the language you are using is not in fact memory safe no?


The big question that I did not see answered is does it support location transparency. One of the huge benefits of Erlang is that it is easy to take a process and move it to another machine or even another data center.

If Lunatic can do that, it would be a game changer for distributed computing.


I don't know about Lunatic, but there's a bunch of Rust actor frameworks that run on many machines. I think I saw one that supported moving an actors between machines

For example, Kay https://github.com/aeplay/kay (used in https://aeplay.org/citybound)

I'm less sure about Axiom https://docs.rs/axiom/0.2.1/axiom/

And then there is Bastion https://crates.io/crates/bastion that I think allows moving an actor to another machine


kay ended single machine. bastion and actix has examples of multi node cluster. wasmcloud is wasm also. riker and some other has no examples of multinode cluster


Thanks for those suggestions! Do you know if some of those multi-node runtimes can move an actor from one node to another?

I think I saw some Rust library do this but I can't find it.


i am on search myself. seeking orleans like solution. bastion has cluster support. so may be it should be checked. cluster is needed for multi node and move. so for now i am coding raw redis, k8s pods, in mem cache and websockets...


Hi! Author of lunatic here.

We are working on a distributed lunatic implementation built on top of nats.io.


An interesting capability of webassembly is that you can pass programs across VMs. So if you have service discovery, you basically can have RCE, which I imagine makes the "run anywhere" kinda easy to build yourself.


As a complete layman in WebAssembly I always struggled to understand what are the use cases. I think what I don't understand is what the _Web_ part exactly means, is a browse thing? Or could be used to more general use cases?


Webassembly is a language-agnostic, non-proprietary bytecode, intended both to be used online and natively.

Like Flash, Silverlight and Java, it allows the embedding and running of binary applications in the browser (although currently this also requires a Javascript shim to allow access to the DOM.) Unlike those, however, Webassembly is not intended to be used only by a single language. You can compile C, C++, Rust, and many other languages to Webassembly and run them both in the browser and as native applications.


Web "means" it's portable and sandboxed, so theoretically you can run untrusted from the web and it only has access to what you expose.

This stacks with DOM sandboxed APIs so you get same level of isolation in the browser as JS but better perf and a memory model more suitable to other languages.

Outside of browser you still get sandboxed low level VM suitable for running C and other low level languages. Eg. you could compile a C module for say node and ship it compiled as WASM


"Web" means that WebAssembly was originally designed for web browsers.

That's a high bar. It needs to be portable to everywhere web browsers run, and it needs to be sandboxed so that it's safe to run downloaded code.

WebAssembly is now widely used outside of web browsers as well. The article linked here is not about web browsers; Lunatic runs on a server.


WebAssmebly is not web specific. And it's not assembly.

I think the best way to think of it is that "WebAssembly" is a great marketing name, but what it does is be a modern-day JVM: a byte code language that can run anywhere.


WebAssembly is a game changer. Why? Because it removes the need for JS, and is more performant. Couple that with a secure sand box to run your apps and you have a new distribution model, somewhere between a full native app and a completely online web app.

You have also removed the shackles of JS and provided an environment to which you can compile and you get the ability to port a lot of libraries and code to the new platform. How does this help? One, you can use other languages to write apps for the web. Two, you can use existing libraries to help develop your app. Three, by having a permission system to access system resources in a secure way, you can incorporate native application features and performance into your web app. Think about stuff like direct printer access, USB access, etc.


Now can we have the non-hyped up version?

As someone who doesn't mind the "shackles" of JS, from what I have seen so far, apps written in WebAssembly are up to two times slower than their JS counterparts. DOM access is abysmal, and from what I've seen in the wild, no one has written a serious business critical application using it yet.

What does webassembly do today, and what experience does it provide over javascript? I get the sandbox and distribution, but again, those advantages mostly apply to JS as well.


> from what I've seen in the wild, no one has written a serious business critical application using it yet.

There are lots of such applications, including:

* Figma

* Unity games on the Web

* Google Earth

* AutoCAD

* Aside from entire applications, crucial features in things like Zoom and Google Meet (filters, backgrounds, etc.).

WebAssembly won't replace JavaScript - it's for different things. Wasm lets you port native code to the Web, and it lets that type of code run very fast. That's even without SIMD and multithreading - with those things, wasm is even faster.


I rewrote the core part in C, invoked one command line operation, and now I can call the C like it was JS and the core part is 10x faster.

I'm not even a JS dev but it is certainly literally mind-blowing once you see it.


you can crosscompile from other languages to wasm. so you could have a codebase written in C for example, you crosscompile + bind to a minimal JS part. now your app works in a browser.


Tangential, but can WASM be used to run massive legacy C/C++ codebases in a memory safe manner without slowing down too much? One example is iOS and Android messaging apps previously could parse messages that exploited the OS.


similar, so different angle https://fluence.network/ these have state full actors for public computing in wasm. if actor service dies, your deploy new one via pi calculus topology script


This sounds awesome!

How does messaging work between processes? What sort of overhead is involved?


Thanks! Author of lunatic here.

Processes don't share any memory, so the message will need to be copied from one process to another. This should be the biggest overhead involved when sending them.

Erlang uses a trick for big binary data structures, they are allocated inside a common memory space and reference counted. That way big messages can be sent just by reference. I was thinking of doing something similar, the wasm spec allows you to import multiple memories and one of the memories could be a commonly shared one. The only issue is that most higher level languages, like Rust, don't have a concept of multiple memories. So it would be really hard to use this trick inside them.

Once we can run lunatic in a distributed way, some messages will need to be sent over the network and we can't cheat there. From this perspective it's even better if we can force architectures where the messages stay small.


Aside from some amount of inspiration, this seems to have nothing to do with Erlang. Kind of a gratuitous use of the name IMO.


> Lunatic's design is all about super lightweight processes. Processes are fast to create, have a small memory footprint and a low scheduling overhead. They are designed for massive concurrency.

> All processes running on Lunatic are preemptively scheduled and executed by a work stealing async executor.

Sounds like Erlang to me.


Isn't go that way too?

Erlang is special for other reasons.


Sure, lots of languages are like Erlang. Actors and CSP are pretty fundamental patterns.

Some things that would make it more like Erlang than Go (but that I won't bother to verify):

1. Go isn't really pre-emptive, it just has a lot of hidden yield points afaik

2. Go doesn't enforce message passing, Erlang obviously does

3. When you send a message to a goroutine that has panicked you hang and, potentially at some point, panic. In Erlang an actor dying can send out a final exit message that other actors (likely a supervisor) can subscribe to.

So yeah IDK. To me if it's got lightweight processes, pre-emption, and message passing, it's got everything that makes Erlang special. If it says it's inspired by Erlang I'm going to assume that, along with those things, it also shares goals like fault tolerance.


> Sure, lots of languages are like Erlang. Actors and CSP are pretty fundamental patterns.

Actors do not an Erlang make. What makes an Erlang:

- Isolation. A crash in an actor cannot bring down other actors. Cannot bring down the runtime

In Go `panic` crashes the program. In Erlang "panic" crashes an actor, and... that's it.

- Monitoring. The above makes an important property of the system: a process can be monitored, and when it dies you can be guaranteed to receive a message that it died, and why

This lets you build things like supervision trees that are impossible/hard/ineffecient(chose two) in other languages.

- Everything in the VM is aware of processes, concurrency and parallelism

Every process gets its fair share of time: each process gets X "reductions". A reduction is a function call, or a message pass. Each function call or message pass reduces the counter. Once it reaches zero, the process is put on hold, and the next process is run.

This means that almost everything in the system, including the VM itself is re-entrant. Even Erlang's regexp implementation is re-entrant. You never even have to think about "wait, if I call this function, and the process is put on hold, what happens". The process will be re-awakened and continued.

----

The first two are what makes Erlang special. The rest is gravy, and there is quite a lot of it on top.


Sure. I'm pretty familiar with Erlang's historic roots and motivations and I would agree that isolation is one of those fundamental components.


"Inspired" is also a highly subjective/pretty broad statement to make. You can have the idea for something while thinking about another thing and not be wrong to say it was inspired by that other thing. I don't see any point saying "you need to justify your assertion that you were inspired by Erlang when working on this project".


Honestly erlang isn't really preemptive either, it just has a lot of hidden yield points. In earlier iterations (iirc) of the vm, the list length() function was very obnoxiously blocking and would not yield.

Erlang doesn't enforce message passing. You can use ets tables to coordinate processes (but don't).

A lot of pls claim erlang inspiration, e.g. pony, but miss some very important points and focus on "actors". Erlang isn't even really an actor system at heart, it just looks like it superficially. Actor systems at their core are message and coordination focused; when you write an erlang or elixir program you are typically minimizing the message passing because it is imperformant and hard to reason about. The raison d'etre of erlang processes is not really message passing concurrency, it's failure domain definition and failure domain bundling (if x crashes also crash y, because now the state of y can't be trusted).


> Honestly erlang isn't really preemptive either, it just has a lot of hidden yield points.

At some level this is true of all systems - you can't yield in the middle of a `mov`, for example. But as Erlang is interpreted it doesn't rely on yield points. The caveat is that it does if you use CFFI.

> You can use ets tables to coordinate processes (but don't).

Yeah I'm sure there are endless ways to bypass the intended use of Erlang, I don't think it's really important. This is way different from Go where data is trivially shared across a channel.

> The raison d'etre of erlang processes is not really message passing concurrency, it's failure domain definition and failure domain bundling

Yes, this is true. The foundation of Erlang is based on earlier research that Armstrong cites in his thesis paper, such as the persistent process and transaction as the basis of system resiliency. But it's not a coincidence he landed on actors to express those things.


Erlang is compiled to bytecode, and the yield points are between each instruction sort of.

> The caveat is that it does if you use CFFI.

No shit, I wrote a huge FFI library for the BEAM (that even implements a yielding system.that interops with erlang's yields), that's a huge part of my point about the beam not really being preemptive.

> But it's not a coincidence he landed on actors to express those things.

Yes, that's my point. In erlang the conceptual arrow goes (resiliency => not-really-but-kinda-like-actors). In these other systems, or at least the ones I've peeked at, like pony and bastion the logic goes (actors => underpants gnomes => something like erlang). It's completely backwards.


> Erlang is compiled to bytecode, and the yield points are between each instruction sort of.

This is basically what I'm saying... at some point nothing is preemptive ie: at the instruction level.

> that's a huge part of my point about the beam not really being preemptive.

OK but I think that's really semantics, it doesn't really matter though. You are right that BEAM can't yield within an instruction or during FFI. If you think that means it isn't preemptive, ok, but as I said I think at some point you end up with "nothing is preemptive" and then "why do we have the word preemptive" and then "because it's useful to describe systems like BEAM".

> Yes, that's my point. In erlang the conceptual arrow goes (resiliency => not-really-but-kinda-like-actors). In these other systems, or at least the ones I've peeked at, like pony and bastion the logic goes (actors => underpants gnomes => something like erlang). It's completely backwards.

Sure, actors aren't "the point". They just satisfy important properties that are necessary for resiliency. Pony takes a very very different approach from Erlang. Bastion seems closer.

I don't think "distance from Erlang" is really important or interesting though. Inspiration from Erlang is present in many of these systems, whether they re-implement the exact concepts or not.


However, by separating out into Erlang's lightweight processes using actors, you enable your code to make use of massive concurrency and thus of what Erlang is good at. From a certain level of concurrency onwards in a big enough problem to solve, the imperformance of message passing becomes less relevant in comparison to being able to process concurrently in many lightweight processes. So one could argue, that indeed some focus on actor systems is not entirely missing the point.


Kind of a toxic comment imo.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: