
Message Passing and the Actor Model - amzans
http://dist-prog-book.com/chapter/3/message-passing.html
======
bdavisx
The article doesn't mention [Vert.x]([http://vertx.io/](http://vertx.io/)),
another JVM library/framework (also works outside of the JVM with Javascript
for example) that is loosely based on the actor model. And now with Kotlin
coroutine support, they've managed to abstract away a lot of the nodejs like
callbacks that clutter up your code - you still need to be aware of what's
happening wrt callbacks and such, but it makes your code a lot nicer:
[http://vertx.io/docs/vertx-lang-kotlin-
coroutines/kotlin/#_g...](http://vertx.io/docs/vertx-lang-kotlin-
coroutines/kotlin/#_getting_one_shot_asynchronous_results)

~~~
natdempk
Thanks for pointing this out. I've never heard of Vert.x, but I will check it
out and see if it seems worth including when I revisit this chapter in the
future.

------
srazzaque
I have coded one significantly large system using Akka. I think conceptually,
actors are quite simple and powerful. But in practice with Akka, I hated the
fact that I couldn't just set a breakpoint and just follow a message all the
way through the system via multiple F7's. Every time I got to a "send" call, I
would need to find out which of the other 200 actors in the code actually
dealt with that message, set another breakpoint, and keep going. Because all
you have at runtime is an ActorRef which could really be anything, and all the
sends are asynchronous. This was perhaps in large part due to the fact that
there was a lot of Akka 101 learning going on during early development so
people went a bit crazy with actorifying everything. If in doubt, create a new
actor. Because erlang, resilience, Telcos use it, supervisor hierarchies, self
healing, and a blog I read last week. Questioning it was a case of the
emperor's new clothes. Interestingly none of the big ticket things that Akka
was sold to the dev team on ever eventuated.

A while into the project, my team and I were responsible for delivering a
subcomponent within the system. We did so with extreme prejudice to only using
actors at the very edges and entry points where a bit of asynchronicity was
required. But everything within the service boundary was just plain old Java.
This was the easiest to understand and maintain part of the system, as
attested by another team who we handed over the entire codebase to. Guess
which part had the least bugs? Most extensible?

I don't know if akka has improved since then, but the debug workflow was such
a departure from the normal way of doing things on the JVM it was enough to
put me off and then steer clear of it in general. There are definitely other
patterns and libraries I would reach for first.

~~~
justanotheratom
I have come to realize that breakpoint style debugging should be avoided.
Atleast for people working on Services. Prefer logging, or other variants like
Event History, State Change history, etc.

~~~
twic
Why?

~~~
mayank
Makes it easier to jump to distributed systems that span multiple processes or
machines, where you can’t just set a breakpoint. Also makes it easier to debug
production systems, where you may have logs but can’t jump back in time to
attach a debugger.

------
fafhrd91
I'd like to mention actix, it is an actor framework for Rust language. it is
single process at the moment, but distributed version is in development.

[https://github.com/actix/actix/](https://github.com/actix/actix/)

------
krylon
Is this a good time to mention concurrentlua, a reimplementation of Erlang's
concurrency model in Lua, using coroutines to model processes?
[https://github.com/lefcha/concurrentlua](https://github.com/lefcha/concurrentlua)

It has not seen a lot development over the past ... couple of years, but it
worked quite nicely when I last tried it.

------
deckarep
I was hoping to see a mention of Pony which looks like a promising language
based on the actor model.

~~~
mcguire
Relating Pony to some of the languages mentioned, two that seem immediately
comparable are Akka and Erlang.

" _Akka’s receive operation defines a global message handler which doesn’t
block on the receipt of no matching messages, and is instead only triggered
when a matching message can be processed. It also will not leave a message in
an actor’s mailbox if there is no matching pattern to handle the message. The
message will simply be discarded and an event will be published to the
system._ "

" _[In Erlang,] receive statements have some notion of defining acceptable
messages, usually based on patterns, conditionals or types. If a message is
matched, corresponding code is evaluated, but otherwise the actor simply
blocks until it gets a message that it knows how to handle._ "

Pony's message passing is syntactically similar to its methods, with a "be"
(for "behavior") keyword introducing a handler rather than "fun". As a result,
the message receive operation is global, like Akka, but the type system
ensures that an actor cannot be sent a message it does not recognize. These
are both different from Erlang, which has a local receive operation. An Erlang
process, upon receiving an A message, can call another receive statement and
block only looking for B messages. At that point, any incoming A messages will
queue up. That has always seemed to me to be a good way to write deadlocks
without realizing it.

Pony is otherwise similar to a typed Erlang that compiles to machine code and
doesn't (currently) have process monitoring. [:-(]

There was some discussion about distributed Pony, but it has rather strong
semantics for message passing, which would require a rather complex underlying
protocol. [I'd rather have simple distributed messages and handle failures at
the program level.]

[I only have experience with Erlang and Pony; I'm hoping my description of
Akka is true from the discussion in the post.]

~~~
rurban
Pony is a bit different to a typed Erlang though:

* Pony has two message passing options: copy or share (by ref). Both are safe, Erlang does only copy.

* Pony is non-blocking only, there are no blocking calls in any stdlib function.

* Erlang supports no native threads, only green threads. So it favors IO over CPU, distributed computing on different cores is only via distributed nodes.

* Pony supports native threads, but no distributed actors on different hosts yet.

* Pony is massively faster, uses much less memory, and has better cleanup via a clever GC protocol.

------
zmmmmm
From a quick skim of it I don't see any mention of the real difficulty I have
encountered with the Actor model in practice: managing imbalanced actor
workloads.

If you consider a simple chain of actors that do some work and then send on a
result to a next actor that then does some more work:

    
    
        A -> B -> C -> D
    

Suppose something (anything) happens that make messages to B take slightly
longer to process than the rate at which A is sending them. The result is that
B's message inbox expands indefinitely. One of two things will happen
depending on the queuing model: either A will block on send because B refuses
to accept another message, or B will simply run out of memory from too many
pending messages.

This of course, is just the linear scenario. Consider a complex network where
there may even be cycles (something D does could somehow indirectly result in
a message coming back to A). You've essentially got a completely unstable,
unpredictable system. The only way to avoid it is to ensure that every actor
is massively over-resourced so that it can never fall behind on processing its
messages.

Of course, the second stage of this is that you start designing scalable actor
"pools" where more than one actor can service messages and you can scale the
size of the pool up and down. But this is at the cost of adding even more
dynamic, unpredictable behaviour, and to some extent you just lost the
simplicity you were trying to gain by using Actors in the first place.

I'm curious if any thoughts from more experienced folks about how to handle
these issues!

~~~
ramchip
There’s a lot of literature on this in the Erlang world. This post is a very
thorough summary of the various solutions available:

[https://ferd.ca/handling-overload.html](https://ferd.ca/handling-
overload.html)

I don’t find that kind of fully async processing pipeline very idiomatic TBH,
at least in Erlang - more typically, every request would be its own actor, and
do blocking requests to other actors that own shared resources.

~~~
zmmmmm
Thank you for the insightful comments. That link looks perfect (down to the
point of having a diagram with a->b->c->d just like mine!).

The idea of each request being its own actor being more idiomatic is
intriguing. Naively though does it really solve the problem? Don't you just
end up with an overflow of actors now instead of an overflow of one actor's
inbox?

~~~
lliamander
In Erlang the actors are lightweight processes are much smaller than OS
threads. I wouldn't worry.

~~~
zmmmmm
Right, but in my scenario (and I appreciate perhaps this means I'm "doing it
wrong"), the messages themselves have a fairly heavy payload (around 800
bytes). Given that, whether I'm spinning up an actor or a message, if it can't
be processed quickly it's consuming a non trivial chunk of memory.

~~~
lliamander
I wouldn't say you are doing anything wrong (or, at least I couldn't say that
for certain without knowing more about your problem domain :) ).

However, I also don't think you have to worry about the message payloads being
too large. In Erlang, normally objects are copied between processes. However,
for large objects (>64 bytes) they are put in a shared heap, and only the
pointers are copied between processes[0].

[0]
[https://hamidreza-s.github.io/erlang%20garbage%20collection%...](https://hamidreza-s.github.io/erlang%20garbage%20collection%20memory%20layout%20soft%20realtime/2015/08/24/erlang-
garbage-collection-details-and-why-it-matters.html)

------
dominotw
I have been wondering about this past couple of days.

Does kafka streams make Actor Model obsolete? What can you do with actors that
you can't with kafka streams if you were starting a project from ground up.

~~~
bb01100100
They are different worlds.. each actor would be analogous to an independent
stream app; or one stream app with lots of KStream/KTable “components”, which
feels like an anti-pattern.

------
ninjakeyboard
I live and breathe this stuff. Started working with Elixir more recently but
have worked with Akka for years. IMO BEAMVM is a great piece of technology and
is worth looking at.

------
baconomatic
Is there an index to this book I'm not seeing? I see the directories, but not
seeing an overall index. A bit hard to navigate.

~~~
inetsee
If you click on "Chapters" it takes you to a directory listing of the
chapters, click on a chapter link and it takes you to another directory
listing of pages. Click on a page link and it takes you to an actual chapter.

It appears that the "book" still needs a fair amount of polishing to be
navigable.

~~~
haggy
See my reply to OP but here is the GH: [https://github.com/heathermiller/dist-
prog-book](https://github.com/heathermiller/dist-prog-book)

------
bringtheaction
> 59 minutes read

Is this actually correct? I am inclined to think not but it is difficult to
judge just from scrolling through it quickly.

~~~
haggy
Im firmly against these "X minutes to read" estimates because the amount of
time it takes to read something is highly dependent on many variables (on both
the reader side and article side). I would much rather have a "Short, Medium,
Long, Novel" kind of scale which is simply a scaled word count of some kind.

------
smoyer
The root of your book's domain (/) has the test page for tufte.css. The
content itself is great!

------
louwrentius
I recall that Microsoft Azure Service Fabric SDK is also based on the Actor
Model.

~~~
keithnz
MS has [https://dotnet.github.io/orleans/](https://dotnet.github.io/orleans/)
which is an interesting take on the actor model.

~~~
socrates666
This became Service Fabric

------
always_good
I haven't given myself the opportunity to sit down and play with an actor
model. But in the meantime, despite reading literature such as this, I have
trouble envisioning what it actually entails day-to-day.

I think of my experience with channel-based communication in Clojure, Go, and
even Rust's mpsc. And every time I feel an instant feeling of debt because I
know I'm just one or two more channels away from misunderstanding the
execution of my own application. Having to take out pencil and paper to reason
about why messages are bottlenecking in this component or why I was wrong when
I thought I knew how my application worked.

For example, we ripped out all of our channels in our Go application and
replaced them with a couple mutexes.

When I read about actors, I get flashbacks to some of my worst struggles with
channels. Where our application is split into disparate agents that everyone
has to eventually map out on a whiteboard to understand even basic runtime
behavior.

I owe it to myself to try it out, but I can't help but be discouraged by what
I think they are. Can someone clarify? What does the actor model actually mean
for your day-to-day code compared to futures, for example?

~~~
bitwalker
I would spend some time experimenting with Erlang or Elixir. The thing that
really made the actor model "click" for me was how each actor in Erlang is
just a process (green thread basically). The only way to communicate between
processes/actors is message passing, and the way you end up constructing
software is by modeling individual tasks and responsibilities of your
application in terms of processes. The way processes are constructed in
Erlang/Elixir allow you to pattern match on messages to selectively receive or
prioritize messages, and discard others. A process can be written in such a
way that it changes from one role to another just by being sent a message, or
from one state to another (in the case of finite state machines and the like).

A typical example of how it gets applied might be a web server, where each
HTTP request is handled by its own process, and potentially delegating
multiple tasks concurrently to other processes in the background which handle
queuing and executing work associated with that request. You might have
pipelines of processes, each responsible for some specific task/transformation
before handing off work to the next stage in the pipeline. On top of that
Erlang/Elixir provide supervision, where components of the system are
started/monitored by a special process type called a supervisor, which will
restart its children when failure occurs, and if a set of conditions fail,
will itself be restarted by its parent supervisor. So your application ends up
structured as a tree of supervisors and worker processes, where components of
the application are branches of the tree, and can be isolated from the failure
of other components.

Hopefully that helps give you an idea of how the model plays out in practice.
I would definitely recommend playing around with either Erlang or Elixir, they
are a lot of fun, and it really changes the way you think - at least it did
for me.

~~~
fpoling
Why it is necessary to discard messages? Isn't it a sign of bad design when
resources have to be spend to create and pass a message that is ultimately
discarded?

~~~
ramchip
I think it’s just a bit badly worded. You don’t normally throw them away, you
leave them in the queue for later.

For instance if you have a process that gets requests and writes stuff to DB,
while processing a request you can send a message to the DB, then use a
selective receive to match on the response from the DB while ignoring all
other messages (you’ll deal with them later i.e. when you fetch the next
request to process).

~~~
fpoling
So "discard" means to post messages to own message queue? If so it seems like
an awkward pattern that essentially implements a priority queue.

~~~
ramchip
No, messages that don’t match are not dequeued in the first place. I don’t
know why the other poster used the word discard.

