

Multithreading Magic - Why Everything You Thought You Knew about Concurrency... - adulau
http://www.slideshare.net/pieterh/fosdem-2011-0mq

======
phintjens
It's great to get feedback on the talk. Impossible to even introduce 0MQ
properly in 10 minutes, so this presentation was really just to drive
discussion. I'd apologise for the polemical title but that's just how I talk.

To some of the comments here...

\- Parallel programming may be taught in CompSci courses, for sure, and it's a
popular use for 0MQ, but (a) it's not applied to mainstream application
development, and (b) it is not designed to scale to networks of any size.

\- The Actor model is not key to building a successful message-based
concurrent application, but it's a good example, and helps people understand
that there are alternatives to shared-state concurrency.

\- It's IMO useless to ask people to learn Erlang or Scala to get message
based concurrency. People use Java, C, C#, PHP, Python, and probably still
COBOL somewhere. So the challenge is how to give this mainstream a toolset
that lets them build large parallel applications.

Perhaps next year at FOSDEM we can do a devroom and take the time to see a
real application evolve from a simple stand-alone process into a real
distributed one. 0MQ is very hard to grasp as a theory, one needs to actually
use it for a few days before the beauty a cheap, universal, fast, intelligent,
easy to use, and asynchronous queuing messaging fabric hits home.

Cheers!

~~~
nuxi
I've glanced at the 0mq sources a bit, and I'm curious about the error
handling. For example, zmq::tcp_socket_t::read() handles following errnos from
recv(): EAGAIN, EWOULDBLOCK, EINTR, ECONNRESET and EPIPE. If recv() fails for
any other reason (e.g. ENOBUFS, ETIMEDOUT) and returns -1, the following is
performed:

    
    
      errno_assert (nbytes != -1);
    

which translates to

    
    
      #define errno_assert(x) \
        do {\
            if (unlikely (!(x))) {\
                perror (NULL);\
                fprintf (stderr, "%s (%s:%d)\n", #x, __FILE__, __LINE__);\
                abort ();\
            }\
        } while (false)
    
    

The first thing I noticed is the use of perror() and fprintf() for error
reporting. Any daemon that closed its stdin/out/err and _not_ had them
redirected to /dev/null will most likely output errors to other file
descriptors (sockets, other open files etc.). Is this by design or something
that was missed?

Secondly, the use of abort() in an API is probably also not the best choice.
Say I'm writing a (robust) database server and an error happens on _one_
connection - the error will bring the whole server down, which kind of defeats
the purpose. Is this also by design?

[Edit: formatting]

------
po
I like ZeroMQ but this slideshare doesn't deliver any substance at all. For an
overview, skip it and check out the white paper (if you can call it that)
cowritten by the same guy (Pieter Hintjens) instead:

<http://www.zeromq.org/whitepapers:multithreading-magic>

~~~
tbrownaw
_The most widely used languages, C and C++, do not offer any support for
concurrency. Programmers roll their own by using threading APIs. Languages
that do support concurrency, such as Java, Python, .NET, and Ruby, do it in a
brute-force manner. Depending on the implementation - there are over a dozen
Ruby interpreters, for example - they may offer "green threading" or true
multithreading. Neither approach scales, due to reliance on locks. It is left
to exotic languages like Erlang to do it right._

You know how I wrote my first concurrent programs? Create threads with
pthread_create(3), and pass ownership of objects by sending pointers over
connections created with pipe(3). Which sounds an awful lot like this cure-all
actor model (which I hadn't heard of yet) that requires languages almost
nobody's ever heard of.

 _Of all the approaches taken to multithreading, only one is known to work
properly. That means, it scales to any number of cores, ..._

clone(2) and socket(7)

 _... avoids all locks,..._

Your threads/actors still block waiting for messages. I've seen an example of
someone using this to write a mutex in Erlang.

 _... costs little more than conventional single-threaded programming,..._

Probably depends how much you have to twist your code to fit this
functional+actor model.

 _... is easy to learn,..._

If you discount having to learn a whole new language and turn your code
inside-out.

 _... and does not crash in strange ways._

Instead of crashing, it hangs because you accidentally wrote a mutex.

~~~
KirinDave
> Your threads/actors still block waiting for messages. I've seen an example
> of someone using this to write a mutex in Erlang.

This statement suggests to me that you've got some fundamental
misunderstandings about the nature of actor based programming. Your _actor_
may block on its message queue (and an infinite computation would mean you'd
never consult the queue again), but during a message wait it doesn't
necessarily block a thread. That's sort of the point of a good Actor model
implementation.

If you write a series of symmetric calls that create a dependency, then yes,
you will deadlock; the Actor model can't work its magic when you explicitly
model blocking calls.

Perhaps you don't know this, but every lock&block system has an equivalently
performant Actor-based system, and vice-versa. (See:
<http://www.sics.se/~adam/pt/duality78.pdf>) What this means, for programmers,
is that whichever model fits our current problem best is the one to use. We
can make either one performant.

P.S., Just because you haven't heard of these languages doesn't mean they are
not widely used. What does your personal experience have to do with the
viability of a technique?

~~~
tbrownaw
> Your actor may block on its message queue (and an infinite computation would
> mean you'd never consult the queue again), but during a message wait it
> doesn't necessarily block a thread. That's sort of the point of a good Actor
> model implementation.

A thread can block on a syscall or a "while(true);", but won't necessarily
idle one of your CPU cores. That's likewise the point of a good task
scheduler.

Actors _are_ threads, just they tend to be implemented in userland rather than
as OS threads.

> Perhaps you don't know this, but every lock&block system has an equivalently
> performant Actor-based system, and vice-versa.

Knowing that they're equivalent (up to "frictionless surface" assumptions for
performance, ignoring what primitives the hardware provides) is a significant
part of why I don't take claims of the actor model's magicalness seriously.

> P.S., Just because you haven't heard of these languages doesn't mean they
> are not widely used. What does your personal experience have to do with the
> viability of a technique?

I have heard of them, always in terms of "this is awesome, what's wrong with
people that almost nobody actually uses it?".

------
slavak
What I got out of this presentation: "You thought the only way to do
concurrency was by using synchronization primitives? Surprise! You can also do
it by message-passing between processes!"

What I think of this: Say whaaaaaaaaaaat? Message-passing as a concurrency
model? That's crazy talk! Brilliant crazy talk! I wish this was such common
knowledge that it was being taught in undergraduate "Parallel Programming"
courses everywhere. Oh wait... IT TOTALLY IS!

~~~
KirinDave
Is that really true? I meet _very few people_ , even fresh out of good
schools, who have any practical knowledge of the Actor model. Many have heard
of it, but most don't know that about even basic concepts surrounding it like
Needham's Duality.

~~~
slavak
While my CDP class didn't explicitly discuss the Actor Model, message-passing
as a concurrency model was studied and we did implement algorithms using it.

How basic is the Actor Model for implementing concurrency using message-
passing, besides the obvious advantage of formally defining a common language
of terms and proven theorems for what you're doing?

------
newobj
Whaa?? "Concurrency is hard... Actors... ZeroMQ... End scene!"

This is far from enlightening.

~~~
phintjens
:-) If you hope for enlightenment from watching a 10-minute talk, you're going
to be disappointed. I do like your summary of the presentation... "Concurrency
is hard... Actors... ZeroMQ... End scene!". Nice.

Enlightenment requires that you change in some way. For me, like many people
who have taken the step of downloading and learning 0MQ, the feeling of "wow,
this is too easy, where's the catch?" comes only after a few days writing
code, and then your brain makes a slight adjustment, and it's all obvious, and
that is a small but real enlightenment.

------
antirez
For what it is worth this is the model Redis, and Redis cluster, are beating
on, and why we are not going multi threaded.

~~~
lylejohnson
You meant "betting on", right? Not trying to be that guy, just wanted to make
sure I understood your comment.

~~~
antirez
Yes, thanks and sorry for the error. Unfortunately it is no longer possible to
edit the comment (but I agree with the edit timeout FWIW).

------
xtacy
CSP (Communicating Sequential Processes) also seem similar:

[http://en.wikipedia.org/wiki/Communicating_sequential_proces...](http://en.wikipedia.org/wiki/Communicating_sequential_processes)

There's a patch for Python that does CSP!

<http://code.google.com/p/python-csp/>

~~~
arethuza
Once upon a time there were even dedicated CPUs that had their own programming
language based on CSP:

<http://en.wikipedia.org/wiki/Transputer>

[http://en.wikipedia.org/wiki/Occam_%28programming_language%2...](http://en.wikipedia.org/wiki/Occam_%28programming_language%29)

------
nc
ZeroMQ is fantastic for low latency communication between processes, but
probably not the best choice for building 'actor' based systems. The slides
don't really provide much basis for the argument.

Erlang & Scala (for a more object-oriented feel) are both excellent for
building actor based systems.

------
vilda
Message passing is not a golden bullet. It is a synchronization primitive for
distributed systems. Basically on the same level as mutexes, semaphores, and
monitors.

Badly designed you are in the same concurrency hell.

~~~
phintjens
Badly designed, anything can become hell. Crazy has no limits. The point of
the talk was that message passing is fundamentally not at the same level as
mutexes, semaphores, and monitors. Shared state does not scale linearly, no
matter how well you use it, whereas message passing does, given cheap, fast,
universal messaging.

~~~
HenryR
A message queue is shared state. It may be more convenient shared state, and
since it only needs to be got right once it might be more bulletproof shared
state, but you have multiple threads that are serialising on access to a queue
no matter how it is implemented.

~~~
phintjens
You're right that a message queue is shared state and has to be gotten right
but that can be done (and ZeroMQ does it) without locks. It's thus invisible
to application developers. You're wrong to say that multiple threads serialize
on a queue no matter how it's implemented. ZeroMQ lets threads write full
speed to a queue while another thread reads full speed from the queue, without
wait states.

------
tommi
I wish Pieter had a bit less sensational title than "Multithreading Magic -
Why Everything You Thought You Knew about Concurrency is Bogus, if not
actually Totally Insane" considering that the slides contain only common
knowledge.

------
slavak
Having read up a bit more on ZeroMQ, I have to ask: How is this, if at all,
superior to, say, MPI[1]?

[1]<http://en.wikipedia.org/wiki/Message_Passing_Interface>

~~~
phintjens
Significantly simpler, faster, and more general. MPI is like one of the
patterns ZeroMQ provides (the pipeline pattern) with a lot on top. That makes
it great for parallel programming, but not so great for (e.g.) data
distribution (pubsub) and other forms of message queuing.

------
philjackson
...the bar man said "we don't serve event loops here"

Node.js walks into a bar...

------
xentronium
Naming Magic: Why Every Other Link Including This One Is A Bait

------
farmer_ted
Whenever I need an Actor, I go with Nathan Fillion.

