
Lumen: An alternative BEAM implementation, designed for WebAssembly - seeekr
https://github.com/lumen/lumen
======
mrdoops
My favorite part about the presentation was the correlations between the DOM
tree and a supervision tree. If you've looked into the Scenic project
([https://github.com/boydm/scenic](https://github.com/boydm/scenic)) you know
how this Process & Supervision model shows real promise for client-side UI
concurrency. As your states are changing you don't want one bad state to crash
and cascade elsewhere - supervision trees lets you contain and recover from
failures gracefully.

I'll be honest I didn't think it would be possible to implement something like
the BEAM's preemptive scheduler in WebAssembly until I saw the presentation.
Turns out it's very doable with continuations. The timing should be right in
the next couple years as both Lumen and some of the bigger components of
WebAssembly maturity kick into gear. Similar high level language/runtime ->
WebAssembly projects like Blazor(C#) are also pending features like more
native control over the DOM without going through JS interfaces (see here:
[https://github.com/WebAssembly/interface-
types/blob/master/p...](https://github.com/WebAssembly/interface-
types/blob/master/proposals/interface-types/Explainer.md) for more technical
details).

Even if client-side UI frameworks don't pan out for Lumen, I can see Lumen
being valuable for edge-computation especially where concurrency and fault-
tolerance are important.

~~~
bcardarella
Thanks, we are not entirely convinced if a 1:1 relationship of process to DOM
element is going to be where we land ultimately but it most definitely is a
fun and interesting experiment. And it does align well with many of the OTP
concepts. It really comes down to performance numbers and further research to
know what the best direction a rendering engine should take. Personally,
unless a rendering engine in Lumen is doing something OTPish (i.e. supervisor
trees) I don't see the reason why people would choose a Lumen solution over
existing works. I think I mentioned it in the keynote but the performance is
secondary to the architectural perspective that Lumen can bring to client side
development.

As Lumen continues to develop we'll continue to experiment with ideas and see
what works best. I actually have a dbmon demo built but Hans and Luke were
hand compiling the source and the DOM ast was just too much which is why the
presentation had a very simple div/p tag. Once the compiler is ready for use I
hope to write more about the dbmon demo and show of some crashing table rows.
Should be cool!

~~~
truth_seeker
How many bytes each process consumes when mapped to a single DOM element ?

~~~
bcardarella
We've done some benchmarking to help understand the differences:
[https://github.com/lumen/threading-
benchmarks](https://github.com/lumen/threading-benchmarks)

The tldr is that with Web Workers as the threading model that browsers are
forcing WASM to use the memory footprint for BEAM processes is going to be
larger. This is of course a concern for us. We've been talking about
distributing a many processes across a few WWs but ultimately we'd like to see
the vendors give up on WWs are being the implementation of the WASM spec.

------
tiborsaas
This is the gist of it:

"The primary motivator for Lumen's development was the ability to compile
Elixir applications that could target WebAssembly, enabling use of Elixir as a
language for frontend development."

I had to dig to lean what BEAM is :)

"Bogumil Hausman next machine was called BEAM (Bogdan’s Erlang Abstract
Machine). It was a hybrid machine that could execute both native code and
threaded code with an interpreter. That allowed customers to compile their
time-critial modules to native code and all other modules to threaded BEAM
code. The threaded BEAM in itself was faster than JAM code."

[http://blog.erlang.org/beam-compiler-history/](http://blog.erlang.org/beam-
compiler-history/)

I now feel a little less bad that I couldn't figure out the acronym :)

~~~
conradfr
Well if you've never worked with Erlang or Elixir it's quite normal to not
know that.

------
pron
While WASM makes for a convenient compilation target for languages like
C/C++/Rust/Zig, isn't JS a better target for languages that can benefit from a
JIT and/or a GC? You'd get better performance for less effort and a smaller
deliverable. Why compile to WASM when JS seems, at least to me, an obviously
superior target? What great advantage does WASM have that it makes it
worthwhile to lose on three important fronts?

~~~
anthony_doan
Wait what?

WASM is a vm and JS is a language.

Are you asking about the pro and con of a transpiler over a vm solution?

For one thing BEAM is the Erlang VM which Elixir and many other languages run
on. Why would anybody want to implement it in Javascript? BEAM does a lot of
crazy things, including a preemptive scheduler. Also BEAM GC isn't some
generic GC, there's tons of crazy work going on with it to work with the Actor
concurrency model.

~~~
pron
> WASM is a vm and JS is a language.

I'm talking about compiling to WASM vs. compiling to JS. Calling something a
VM or a language is mostly a matter of convention. x86 machine code can also
be viewed as a language, and compilers certainly don't care. A compiler
translates code in one language into code in another (there is no precise
definition for "transpile", which is why compiler people hate that word;
colloquially "transpile" just means that the compilation's target language is
considered to be at a similar abstraction level to the source).

> Why would anybody want to implement it in Javascript?

Why would anybody want to implement it in WASM? If it's so that it could run
in the browser, then compiling BEAM bytecode to JS would achieve the same
goal, be easier to write and would likely result in a smaller _and_ faster
deliverable.

> Also BEAM GC isn't some generic GC, there's tons of crazy work going on with
> it to work with the Actor concurrency model.

There's nothing crazy going on in BEAM. As far as VMs go, it's near the bottom
in terms of sophistication.

~~~
ohyes
Exactly true! C, for example is a VM!

------
bcardarella
Hi I'm Brian, one of the member of the Lumen team. I'll try to address some of
the questions I've seen in this thread:

1\. Why?

We've seen every growing complexity in building, maintaining, extending, and
debugging client side applications. There are several reasons why this is but
in our opinion how JS handles concurrency (the Event Loop) is a primary
factor. Erlang's concurrency model is superior (also our opinion) in how a
developer reasons about concurrency. We want to bring this power to building
client side web apps.

2\. I'm afraid of [it] being over-engineered, and hard to contribute to.

Lumen itself is just the beginning of what we hope is an eco system of tooling
and potential framework pieces for building web applications. When building
Elixir or Erlang applications developers are not expected to be contributing
back to the BEAM or to even understand how it works. Lumen will be in a
similar position. When we finally get to `mix install lumen` you will likely
only be including Lumen as a dependency in your apps. Higher level
functionality will emerge that use Lumen to compile to WASM. However, we
highly encourage people with experience in Rust, compiler design, runtime
design, Erlang/BEAM, or just general curiosity to get involved in developing
Lumen :)

3\. even Chris McCord said that in all his years of consulting, he's only
gotten a handful of requests for these types of apps

Chris works here at DockYard, and this is because we don't put Chris on our
PWA/SPA projects. While there is an obvious demand for LiveView to relieve
people from needing a client side framework there are most definitely many
clients and plenty of projects where client side apps make sense.

4.an alternative BEAM implementation that behaves like a regular VM rather
than BEAM?

Not entirely, although some concessions need to be made when working within
the security model of the web. I would very much like it is the browser
vendors would give up the argument that Web Workers fulfill the threading
requirement of WASM as I personally feel that is disingenuous. I'm hoping that
as Lumen develops and (hopefully) picks up steam/influence we can start to see
some of these features land for real.

5\. Due to size issues I probably wouldn't want to ship the entire BEAM to my
users

Correct, and we aren't advocating for this. Lumen is currently in a very early
stage of development and we haven't even implemented any compilation
optimizations like dead code elimination. There are many other WASM specific
optimizations that could cut the footprint size by 50% and this is before we
even gzip, which WASM is designed to respond well to despite being a binary.

I'll keep a running list of responses in this thread if more questions come
up.

~~~
truth_seeker
I would love to know a deep comparison of Elixir(LiveView or other libraries)
Vs AngularJS or VueJS or ReactJS for building modern progressive SPA and how
much productivity is achieved.

~~~
bcardarella
LiveView isn't meant to be a replacement for Angular/Vue/React. It is meant to
address the many companies/projects that were once doing some light client
side functionality with jQuery or prototype and have since gone full framework
and are now finding themselves with a much longer time to implement what was
once simple. The overhead that frameworks impose are meant to manage
complexity but if the interactions are simple then that overhead is a
liability.

Some people also just don't like JS so there is that.

~~~
truth_seeker
Thanks for reply.

------
lxrbst
Cool idea, cool technology, cool project.

I really just don't get the "why", even after watching the ElixirConf
presentation [1] about it. Like what is the use-case for this, specifically?

[1]
[https://www.youtube.com/watch?v=uMgTIlgYB-U](https://www.youtube.com/watch?v=uMgTIlgYB-U)

~~~
nickjj
Same. There was a quick slide joking about how this solves the fully offline
client side browser app problem but even Chris McCord said that in all his
years of consulting, he's only gotten a handful of requests for these types of
apps.

I'm sure they have their reasons for taking on such a project but as an
outsider I would have liked to see more announcements that apply to what the
majority of people are using Elixir for today (web apps). There's Nerves too
which I'm not into personally, but there were things at the conf for that so
I'm sure that crowd was happy.

For example, telemetry was announced over a year ago with a hint that it was
being worked on next and will be implemented into Phoenix as a web dashboard
in the future but there was no hint of that mentioned this year -- at least
not in any of the Phoenix related talks and Chris' keynote. I also didn't see
it in any of the other talk's titles. But telemetry is one of those things
where if it existed in the form they talked about last year it would be one of
the biggest killer features of any web framework available today (not a single
one does what they proposed).

~~~
josevalim
Telemetry's core is done for a while and it has been announced at previous
events. That's why we didn't put much focus into it this year.

You can learn more at Arkadiusz Gil's talk on ElixirConf EU
([https://www.youtube.com/watch?v=cOuyOmcDV1U](https://www.youtube.com/watch?v=cOuyOmcDV1U)).
I also talked about it in my keynote (someone posted a video below) and Marlus
also gave a demo at this ElixirConf of how we plan to use Telemetry in
Broadway and how a dashboard may look like
([https://www.youtube.com/watch?v=tPu-P97-cbE](https://www.youtube.com/watch?v=tPu-P97-cbE)).

At the same time, please remember that whatever we announce is not a promise,
and the best way to make it happen is to get involved. :) For example,
Telemetry is out there, Phoenix, Ecto and Plug already use it, so there is
nothing stopping anyone from implementing the dashboard right now!

~~~
nickjj
> For example, Telemetry is out there, Phoenix, Ecto and Plug already use it,
> so there is nothing stopping anyone from implementing the dashboard right
> now!

That is very good to hear.

Is it still planned for Phoenix 1.x to have that /metrics dashboard as
something that comes out of the box with little to no configuration or having
to set up external tools?

The one mentioned in the 2018 talk you linked here:
[https://youtu.be/cOuyOmcDV1U?t=2147](https://youtu.be/cOuyOmcDV1U?t=2147)

------
juhatl
I watched the ElixirConf 2019 keynote [1] as soon as it came up. While I was
left excited about Lumen in general, I wasn't entirely sure what kinds of
things Lumen would be most useful for.

Being able to ship a single binary (like with Go or Rust) sounds handy, but
shipping a tarball hasn't been that much of a challenge either. Being able to
run BEAM code in the browser also sounds handy, but due to size issues I
probably wouldn't want to ship the entire BEAM to my users either, right?

Can someone with a more thorough understanding of Lumen share some thoughts on
this?

[1]
[https://www.youtube.com/watch?v=uMgTIlgYB-U](https://www.youtube.com/watch?v=uMgTIlgYB-U)

~~~
rehemiau
I think they mentioned this in the keynote, but bytecode size is on their
radar. To allow dead code elimination, they are not planning to support hot
upgrades. This means that you wouldn't be shipping the whole OTP - only the
parts that you're using.

~~~
johnisgood
What would hot upgrades entail in the context of Web and for example Lumen?
What would its benefits or drawbacks be? Would it be useful? Would it raise
security concerns? Just want to play with the idea. :)

~~~
sansnomme
If Wasm does become as pervasive as JVM i.e. a universal VM of sorts, there
will be plenty of IoT equipment using it since the trend now for anything
"embedded" that doesn't demand ultra low power consumption is to use a full
Linux system à la Raspberry Pi. A company optimizing for a tight
memory/performance budget and whose purpose does not require high uptime
across the entire fleet can simply use traditional blue-green deployments or
something.

------
namelosw
Interesting project.

Although it's viable to transpile Erlang or Elixir to JavaScript, without the
BEAM runtime everything seems pointless.

However if an BEAM is viable we could do interesting things like React fiber
equivalent could just be processes.

------
moomin
So, an alternative BEAM implementation that behaves like a regular VM rather
than BEAM? Not saying it’s useless, but it’s sure idiosyncratic.

------
artfulhippo
This is some of the best news this summer!

Anyone that wants to reduce the influence that Google and Facebook have on the
Web should look into the Erlang/Elixir ecosystem.

~~~
moosingin3space
I'm excited about this project too, but how does it "reduce the influence that
Google and Facebook have on the Web"? It seems pretty orthogonal if you ask
me.

~~~
bcardarella
Considering how much sway Google has over the WASM spec group I would say that
Google still has considerable influence upon this project.

