
Zed Shaw's Advanced Network Architectures With ZeroMQ at Pycon 2011 [video] - atarashi
http://vodpod.com/watch/5757197-pycon-2011-advanced-network-architectures-with-zeromq
======
haberman
Can't watch the video right away, but I'll just say that every time I read the
ZeroMQ guide (<http://zguide.zeromq.org/page:all>) I am frustrated by the lack
of any clear technical explanation of ZeroMQ's basic architecture.

Today I gave it another shot, and after half an hour of trying to glean
technical detail and skipping any parts that read like a like a comic book
("zap-pow-kaboom satori paradigm-shift moment"), I think I've learned the
following:

    
    
        * ZMQ works by spawning a background thread that runs a
          poll() (or equivalent)-based async I/O loop.  This thread
          is created when you create a ZMQ "context."
    
        * On top of a context you can create a ZMQ "socket", which
          may map to N underlying UNIX sockets (or a comparable
          transport), and which will transparently queue data to slow
          or unavailable receivers.
    
        * Over a socket you can send or receive "messages", which
          are length-delimited strings which are always delivered
          in full with the original length. Send/receive can be
          either blocking or non-blocking.
    
        * Contexts can be shared across threads, but sockets are
          not thread-safe.  Communication between application threads
          and the ZMQ I/O thread is via a lock-free queue.
    

The main features seem to be topology-agnostic programming (since you don't
have to know what a socket is connected to to send/receive over it) and
transparent queuing. In my opinion transparent queues can be problematic
because they transparently use up memory that can be hard to account for.
Topology-agnostic programming certainly seems interesting, but in my
experience of distributed systems programming I never seem to need or want
complex messaging topologies. I guess I don't really get what all the hype is
about. Maybe I should watch the video.

~~~
dochtman
I started using ZeroMQ at work a few weeks ago after reading the guide.

The part that I like about ZeroMQ is that it's a simplifying abstraction, e.g.
it's a fairly simple wrapper about sockets that makes life easier for me. I
guess it's mostly like a library of patterns for socket usage, so it makes a
request-response loop and pub-sub scenarios really easy to implement. The
other thing that's great is that it's pretty much programming language
agnostic, meaning that C++ code and Python code look fairly similar and can
trivially be used together.

As for the transparent queueing, there are simple options to control the size
of the queue; it's even possible to spool the queue to disk as soon as the
memory limit is reached (see setsockopt docs and look for HWM, for high water
mark).

~~~
haberman
All of your reasons make sense to me, it's just that all the advocacy docs set
ZMQ up to be this game-changing system. I can totally buy "a fairly simple
wrapper about sockets that makes life easier for me." "world-saving superheros
of the networking world" not so much.

~~~
dochtman
I don't know, the simple building blocks it provides can be combined in a
bunch of ways that are really pretty cool. I think apenwarr would might call
it a "simplifying assumption". It's hardly ever the complex things that are
game-changing, right?

------
xtacy
I guess it would've been better if the title were "Advanced Network
Communication Architectures (or Patterns)" with ZeroMQ. Network Architectures
sounds more hardware-ish and (at least to me) sounds like it's talking about
the actual network topology... :-)

------
JoachimSchipper
Is there some text available somewhere (slides, perhaps)?

------
runjake
Previously (with comments): <http://news.ycombinator.com/item?id=2319875>

------
antidaily
Is it Zed Shaw week on HN or something?

~~~
pstuart
A submission like this is _exactly_ what HN is about.

