
Concurrency Among Strangers: Programming in E as Plan Coordination (2005) [pdf] - BuuQu9hu
http://www.erights.org/talks/promises/paper/tgc05.pdf
======
BuuQu9hu
The Monte language is based on E:
[http://monte.readthedocs.io/en/latest/](http://monte.readthedocs.io/en/latest/)

------
Denzel
That's serendipitous. I've been reading Mark's dissertation [1] for the past
week, as well as Rees's paper [2] on W7 (his full O-Cap Scheme language).

A lot of great ideas. The ideas behind Mark's E represent a critical part of
half the solution. I think Google et al. are going in the wrong direction:
churning hundreds of millions of lines of code and yet, they're not reducing
the complication. (Talk to anyone who has tried to deploy a production-ready
Kubernetes cluster.)

There's a wide gap in the market for a product that makes building distributed
systems straightforward. Using Alan Kay's "Is it really 'Complex'? Or did we
just make it 'Complicated'?" [3], I feel it can be done in <100k LOC.
Therefore, a single developer could grok the whole system in under a month.

I'm still working through the details and building prototypes. Allow me to
mind-dump the basic ideas and guiding principles.

It's a fully distributed, orthogonal, concurrent, object-capability, operating
system built upon Scheme. Guided by the principle of isomorphism that code is
data and data is code, and therefore both can be sharded and replicated as
demand dictates.

So, let me break it down piece by piece. First, it's an operating system,
which means this is what you boot. You boot directly into a Scheme runtime.
Second, it's fully distributed at the operating system level, which means you
can spin up new machines, in a cluster, on demand, and they'll join the rest
of the group autonomously. Third, it's an orthogonal operating system, which
eliminates the need for a file system; the "files" are your objects, i.e.,
structured data, and the operating system is responsible for serializing the
machine's runtime. Fourth, it's an object-capability system with real object-
oriented programming based upon messaging. Finally, the concurrency is
achieved similarly to E, Erlang, Go, etc.

Now, one of the more interesting parts comes in the sharding and replication
of data and code, which is built into the runtime. Some parts will require
declarative user specification (think Kubernetes, replicating a process that
represents a web server to all machines in the cluster), and other parts will
be handled autonomously by the runtime (think sharding a list that's become
too big to fit on disk) using split/concatenation plans described either by
the runtime or the user. For example, a user may have a custom data structure
that requires special splitting.

This is all glued together by an optimizer/planner/scheduler that's meant to
maximize the performance of processes running on the entire distributed
system. Of course, this must be solved by an online decision algorithm similar
to demand paging/the ski rental problem.

A lot of properties that you find in a lot of different distributed systems
fall out of this system for "free". And by eliminating a lot of the
complications that have been built up and papered over in Linux and containers
and orchestration and data management, etc. you arrive at the essential
complexity of what's needed to build safe, secure, distributed systems.

Heck, I'm building a web crawler right now, and I salivate at the idea of
having a system like this that would allow me to just build the damn web
crawler and stop messing around in the complications.

P.S. I've glossed over A LOT of the details and it probably all sounds
confusing. But that's cause I'm shoulder-deep in the details and I have yet to
come up and think about the communication/messaging side of the product. I'd
love to talk in more detail with anyone that's interested. Really excited
about this space.

[1]: [http://erights.org/talks/thesis/markm-
thesis.pdf](http://erights.org/talks/thesis/markm-thesis.pdf)

[2]: ftp://publications.ai.mit.edu/ai-publications/pdf/AIM-1564.pdf

[3]:
[https://www.youtube.com/watch?v=ubaX1Smg6pY](https://www.youtube.com/watch?v=ubaX1Smg6pY)

