
Distributed Systems with ZeroMQ and gevent - llambda
https://raw.github.com/strangeloop/2011-slides/master/Lindsay-DistributedGeventZmq.pdf
======
plainOldText
For anyone interested in this subject, the presentation has clean, easy to
understand drawings which make the reading so much pleasant. I read all 154
pages mainly because of that, and I must admit it gave me a cleaner picture of
how ZeroMQ can be integrated in distributed systems. Thanks for sharing

------
j2labs
People interested in these slides might like to check out the Brubeck
framework. It can use gevent for concurrency and ZeroMQ to communicate with
Mongrel2, it's webserver.

<http://brubeck.io>

~~~
espeed
So Mongrel2 sits in front of Brubeck and proxies requests back?

I've been playing with ZeroMQ quite a bit the last few weeks because I needed
a fast way to pump data from a Python app into the Neo4j Batch Loader, which
runs on the JVM. I wrote a lightweight ZeroMQ server in Jython called
Lightsocket and a Python client that (with multiple clients) was pushing about
10,000 inserts per second into Neo4j on my dev system.

Jython was the bottleneck, and I could have rewritten it in Java or Scala, but
the data was already loaded. However, I went looking around to see if there
was a ZeroMQ-based container or if anyone had modified Tomcat or Jetty to run
as a ZeroMQ server, but I couldn't find much on it. Does anyone know of a
ZeroMQ server/container for the JVM?

~~~
j2labs
Correct. Mongrel2 sends request messages to Brubeck which processes the
request and responds with a complete HTTP response.

------
jacques_chester
There's literature for this kind of architecture under the heading "Staged
Event-Driven Architectures" or SEDA.

[http://en.wikipedia.org/wiki/Staged_event-
driven_architectur...](http://en.wikipedia.org/wiki/Staged_event-
driven_architecture)

------
StavrosK
Can someone explain how this can be beneficial to more or less typical web
apps? I don't see where the extra concurrency could be used...

Alternatively, what sort of use cases is this good for?

~~~
freyrs3
Background workers, continuous caching, anything dealing with realtime data
acquisition.

~~~
StavrosK
Thanks, that last one sounds to be an especially good fit.

------
sneak
Haven't read the preso yet but I just wanted to chime in and say that I love
ZeroMQ - it's one of those simple libs that are so cool you want to run out
and build something with it the moment you see a four line example.

I've used it to build some cool stuff, and can't wait to use it more often.

------
rabidsnail
<http://svn.red-bean.com/bob/greenlet/trunk/greenlet.c>

There are real continuations for python!? Where has this been all my life?

------
getsat
Really great presentation and really great code examples! Thanks for uploading
this.

------
anotherjesse
I was just thinking of inviting jeff to give this talk at a openstack hacking
event and I see slide 10 he references us :)

------
iam
Is there by any chance any use cases of why would someone use one concurrency
model (e.g. ZeroMQ) over another (gevent)?

------
schlomie
> Ø Faster than TCP, for clustered products and supercomputing.

Not quite following this statement, what makes it faster?

~~~
xtacy
I think it means, it's faster than managing your own TCP connections for
cluster applications.

ZeroMQ's website itself makes no claim that they're faster than TCP:
<http://www.zeromq.org/area:faq>, under:

    
    
        The graph in the test results shows that ØMQ is slower than TCP/IP. What's the point then?

~~~
schlomie
Ah thanks, it must be something like that, though later on in the FAQ there
they also say:

> ØMQ gets you more throughput than TCP has using intelligent batching
> algorithms.

yes, intelligent, sounds like a good idea. ;-)

EDIT: ok here's a little more detail as to what they mean by that in the next
FAQ entry:

> ØMQ batches messages in opportunistic manner. Rather than waiting for a
> predefined number of messages and/or predefined time interval, it sends all
> the messages available at the moment in one go. Imagine the network
> interface card is busy sending data. Once it is ready to send more data it
> asks ØMQ for new messages. ØMQ sends all the messages available at the
> moment. Does it harm the latency of the first message in the batch? No. The
> message won't be sent earlier anyway because the networking card was busy.
> On the contrary, latency of subsequent messages will be improved because
> sending single batch to the card is faster then sending lot of small
> messages. On the other hand, if network card isn't busy, the message is sent
> straight away without waiting for following messages. Thus it'll have the
> best possible latency.

EDIT: This is interesting too for super low latency, Myricom actually makes
such a library (<http://www.myricom.com/kb/index.php?title=DBL>).

> We are working on delivering ØMQ over alternative networking stacks, thus
> having advantage of features like kernel bypass, avoiding TCP/IP overhead,
> using high-performance networking hardware etc. That way we can get the
> latency as low as 10 microseconds.

------
redouane
nice,i was just looking for something like this. thank you.

