
A Badass Way to Connect Programs Together - norswap
http://joearms.github.io/2016/01/28/A-Badass-Way-To-Connect-Programs-Together.html
======
ggreer
I think OSC is interesting, but Armstrong goes too far in his disparagement of
XML and JSON:

> I think it is totally crazy to send JSON or XML “over the air” since this
> will degrade the performance of the applications giving a bad user
> experience and higher bills - since ultimately we pay for every bit of data.

If every JSON and XML API switched to OSC, the difference in data usage would
scarcely be measurable. First: The vast majority of mobile data is video,
audio, images, and application updates. Second: When sent over the wire, JSON
and XML are typically compressed with gzip or deflate.

And let's not forget: JSON and XML have some significant advantages over most
other serialization formats. They're ubiquitous. Practically every programming
language has libraries to parse and generate them. Practically every
programmer knows how to work with them. They're human-readable. No special
programs are needed to view their contents.

In other words, JSON and XML are only "wasteful" if you don't count developer
time. Once you do, it suddenly makes a lot of sense to "waste" hardware
resources. After all, programmers are expensive. Hardware is cheap.

~~~
rdtsc
> And let's not forget: JSON and XML have some significant advantages over
> most other serialization formats. They're ubiquitous. Practically every
> programming language has libraries to parse and generate them.

One of the interesting things about Erlang is it has binary pattern matching.
Matching any binary in it is just as trivial or even more trivial than parsing
json in other langauges.

Here is what an IP packet parsing might look like:

    
    
        <<4:4, HeaderLength:4, _DiffServ:8, _Length:16, _Identification:16,
          _Flags:3, _FragOffset:13, _TTL:8, Protocol:8, _HeaderChecksum:16,
          SrcAddr:32/bits, DstAddr:32/bits, OptionsAndData/bytes>>
    

It looks almost like casting a binary blob to a C struct, but you can do even
crazier things like take Length and match it later in the pattern, if it
specifies a length prefixed binary, for example.

> the difference in data usage would scarcely be measurable

It depends. See, Joe's background is from Ericsson. Their stuff controls
access of 50% of world's smartphone <-> internet traffic. When people say "but
who uses functional languages in the industry?" the answer is if functional
languages won't work, chances are high, you won't see picture of cats on your
smartphone.

But the idea is that mobile bandwith is still a precious resourse. Even if our
CPU and hard drives are getting bigger. That is one shared resource that is
expensive. Maybe if a 1M clients use JSON vs a binary protocol doesn't make a
difference much for some startup, when we talk about billions of connected
devices talking over radio waves, those things add up. Well, also, game
servers use UDP and binary often because of latency. Facebook knows a thing or
two about chatting with lots of users, so they use Flatbuffers
[https://code.facebook.com/posts/872547912839369/improving-
fa...](https://code.facebook.com/posts/872547912839369/improving-facebook-s-
performance-on-android-with-flatbuffers/) that's binary as well.

~~~
viperscape
I think it's also important to note that json isn't exactly something that can
be streamed while parsed.

~~~
ferrari8608
From personal experience, I can tell you that is can. The i3 window manager in
Linux comes with a status bar program called i3bar. It also includes a program
called i3status which constantly feeds i3bar information to display via JSON
through a standard pipeline.

I wrote an i3status replacement that does exactly this. It sends JSON to
STDOUT at a set time interval. i3bar takes the JSON as it comes through STDIN
and changes the look of the bar as it's read.

------
paperkettle
Major supporter of this thinking, and did several sketches in this direction a
few years ago: [http://www.illucia.com/](http://www.illucia.com/)

Made of: 1) a suite of interconnectable videogames & apps and 2) physical
patchbays to connect them

Came from thinking about software like modules in a modular synth. Playgrounds
of interconnectable control structures. Was patching games into samplers,
slaving text editors to drum machines, etc.

related writeups: [http://www.illucia.com/faq/](http://www.illucia.com/faq/)
another from 2010 or so when I was just starting coding (so a lot of it feels
silly now.. but still some related nuggets):
[http://www.paperkettle.com/codebending/](http://www.paperkettle.com/codebending/)

~~~
lbotos
Side Q: can I ask what typeface you used for illucia?

~~~
hjek
It's 'Square Serif Medium' (Look at the css, or do Right Click => Inspect
Element in your browser)

------
ChuckMcM
Ah OSC, my first encounter with it was with the Make Controller[1] (basically
an ARM chip with an OSC service endpoint). OSC always struck me as one of
those protocols that did something really useful in the wrong way. It just
"feels" awkward to me for some reason and I never quite put my finger on it.

That said it was great fun building a robot with the controller but hard to
get asynchronous feedback. If I were doing it again I'd try to figure out a
full duplex OSC channel for something like that.

[1] [http://makezine.com/2006/05/11/make-controller-
kit/](http://makezine.com/2006/05/11/make-controller-kit/)

------
dmix
> This is wonderful news for systems like Node.js whose concurrency model is
> non-existent and whose idea of having a fun time is throwing promises into
> the future and hoping that nobody breaks them.

Oh so true. I wish I learned Erlang (and to a lesser extent Go) much earlier
in my career after wasting so much time with Node. I now laugh when I see
people on Twitter/Reddit talk about Node's concurrency as a reason to learn it
over other languages.

~~~
mikekchar
The concurrency model in Node is essentially the reactor pattern. While there
are lots of concurrency patterns that you should learn, reactor pattern is
also one of them. The big thing to keep in mind is when you should use it and
when you should not use it. After you spend time learning about concurrency in
Erlang and Go (and practicing it), I recommend doing a mental kata and
thinking, "When would the reactor pattern be more appropriate than this." If
you answer is "never", then you should go back to Node (or simply implement it
in whatever language you like) and practice until you know what it's good for.
(Hint: it is quite useful when implementing UIs or anything else that has a
series of messages/events that have to processed in linear order. If you
wanted to do something else at the same time while having full control over
how much resources you are using...).

~~~
rdtsc
A few places I could see using the reactor pattern over the "concurrency unit
(i.e. green process, thread, co-routine) per connection context" is in very
high performance scenarios and where callback chains are very short.

Think proxies (haproxy), routers, forwarders, web servers (nginx) etc. Where
memory context per connection should be minimal, and everything should be as
close as possible to the select/poll/epoll loop.

Funny enough, this also includes demos, quick example scripts, and benchmarks.
I wonder if that what hooked most people to the reactor pattern -- small
examples in Twisted, Node, etc, look pretty easy and simple. But when you
start adding business logic and callback chains evolve into callback/errback
trees 10 levels deeps when things get very scary.

~~~
klodolph
Funny thing about "high-performance scenarios" is that the reactor pattern is
often slower than other patterns. If you want to use the reactor pattern, you
have to use asynchronous IO. But, asynchronous IO involves making more
syscalls. Blocking IO is actually rather fast. The performance is differences
are going to depend on a lot of particulars.

~~~
rdtsc
> Blocking IO is actually rather fast.

Right. Yeah there was a presentation about Java concurrency patterns about
that. Basically dispelling the myth that is everything non-blocking and
asynchronous will be faster than the old school blocking thread / socket.

Interestingly, I believe, haproxy and nginx use a hybrid model. They have a
worker thread per CPU, all listening to the same socket using epoll from
multiple threads! When data arrives they all get woken up and then use a
shared mutex to decide which one will handle the request. (Now later kernels
fixed that one can have exclusive wake-up across the same socket).

~~~
unscaled
Nginx implementation may have changed since the last time I delved into it
(and I delved pretty deep), but at least 2 years ago that was not the case.
Nginx did have some vestiges of an abandoned multi-threaded implementation
attempt from the 0.x days behind compiler flags, but beyond that it was
thoroughly single-threaded, but there is a master process which forks itself
into multiple processes.

There is a so-called accept_mutex, but I'm pretty sure you can't avoid that if
you want to have multiple cores handle connections from the same port. Even
Erlang would have to do that somewhere behind the scenes. Newer Linux kernel
versions support the SO_REUSEPORT which is meant to address this situation - I
guess this is what you're referring as exclusive wake-up accross the same
socket?

~~~
kuschku
Interestingly, Java 7’s "native non-blocking IO" actually uses exactly that –
you have a ServerSocket, and, if a user connects to the server, it spawns a
Thread and gives it a normal SocketChannel.

------
dayjah
The basis of the protocol is similar to that of protocol buffers [1]. Though
the tag field containing the map/list of segments in data is pretty novel IME.

[1] [https://developers.google.com/protocol-
buffers/docs/encoding](https://developers.google.com/protocol-
buffers/docs/encoding)

~~~
codemac
Message Pack[0] is another protocol that follows this very successful
simplicity.

A leading thought in a lot of this is the "unset and a zero value are
equivalent". It simplifies a lot of what the serializer and deserializers have
to understand, and places the burden on code generation or libraries rather
than complex serialization. It was unnatural for me to think this way, as I
take the opposite for memory (record whether something is set or not, don't
just assume it equals zero), but on the wire this makes for very compact and
backwards compatible messaging.

Cap'n Proto[1] I have not played with, but would love to learn more about to
see the advantages and disadvantages between protobuf, msgpack, and capn.

[0]: [http://msgpack.org/](http://msgpack.org/)

[1]: [https://capnproto.org/](https://capnproto.org/)

~~~
cypher543
For a standardized version of Message Pack, see CBOR (RFC 7049):
[http://cbor.io/](http://cbor.io/)

~~~
codemac
What makes it a standardized version of Message Pack?

Message Pack has a specification that is well documented[0] and even the rfc
you refer to specifically states it has different goals (though, eliding
specifics with pretty sad generalizations)[1]

[0]:
[https://github.com/msgpack/msgpack/blob/master/spec.md](https://github.com/msgpack/msgpack/blob/master/spec.md)

[1]:
[https://tools.ietf.org/html/rfc7049#section-9](https://tools.ietf.org/html/rfc7049#section-9)

~~~
simoncion
> ...and even the rfc you refer to specifically states it has different goals
> (though, eliding specifics with pretty sad generalizations) [link to
> RFC7049, section 9]

You find those generalizations sad because that is the _acknowledgements_
section!

If you start from the beginning, you only need to read three paragraphs to see
that "Appendix E lists some existing binary formats and discusses how well
they do or do not fit the design objectives of the Concise Binary Object
Representation (CBOR)." [0]

If we look at Appendix E, [1] we find that section E.2 contains an explanation
that you're likely to be more satisfied with. [2] However, before you go off
and read section E.2, I _strongly_ urge you to read the couple of paragraphs
in Appendix E, first. The authors of RFCs generally tend to try hard to remove
redundancy in their prose, and later sections _often_ elide information
covered in earlier sections.

[0]
[https://tools.ietf.org/html/rfc7049#section-1](https://tools.ietf.org/html/rfc7049#section-1)

[1]
[https://tools.ietf.org/html/rfc7049#appendix-E](https://tools.ietf.org/html/rfc7049#appendix-E)

[2]
[https://tools.ietf.org/html/rfc7049#appendix-E.2](https://tools.ietf.org/html/rfc7049#appendix-E.2)

~~~
codemac
The sad generalizations are in E.2

Things like stating that "evolution has stalled" while also recognizing that
the format is stable is hand wavy when you consider we're discussing things
that go over the wire and even end up on disk. Yes, stability should be a
goal.

The real difference between CBOR and MessagePack is that CBOR wants to be
"schemaless" in the _applications_ themselves instead of just on the wire.
They hold up json as the example format for something that doesn't require
schemas, and yet I see "json schemas" being published[0], and even people
trying to standardize the schema format[1]! Looking at any modern JSON API
would tell you that "schemas" in the xml sense are not required, but
applications all must be very knowledgeable of the format.

Having a data type for "PCRE" is just insanity on the wire, and I can't
imagine the type of API you'd be publishing where you would accept URLs or
Regular Expressions or Text or Binary, _AND_ want to be able to decode them
into proper types in memory all without applications on both ends knowing that
ahead of time.

Which brings me back to my initial point: CBOR is not just a "standardized
Message Pack", it's a very different approach to what they think the
applications on either end of a protocol should be doing.

[0]: [http://json-schema.org/](http://json-schema.org/)

[1]: tools.ietf.org/html/draft-zyp-json-schema-03

------
cyphar
OP hasn't addressed the fact that UDP will willfully drop packets and send
them out of order. What if I need to stream some data? What if I care about
the order my IPC messages arrive in?

~~~
tokenrove
Consider SCTP as an alternative if you want messages but want them in order.

~~~
im_dario
Also QUIC[0] should be a good candidate to be used instead of UDP.

[0]
[https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3...](https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3Pno2Xj_l_YShP40GLQE/edit)

------
nosuchthing
OSC is in use for a lot of DIY hardware interfaces fed into custom modular
(software) synths, DSP, or weird art programs using Pure Data, Max/MSP, Super
Collider, Ableton Live and MIDI bridges to other pieces of sound plugins or
external music hardware.

Nice to see a good write up on this project!!

[http://en.flossmanuals.net/pure-
data/ch065_osc/](http://en.flossmanuals.net/pure-data/ch065_osc/)

[https://puredata.info/community/pdwiki/OpenSoundControl](https://puredata.info/community/pdwiki/OpenSoundControl)

[http://livecontrol.q3f.org/ableton-
liveapi/liveosc/](http://livecontrol.q3f.org/ableton-liveapi/liveosc/)

------
jph
> nobody does this properly for purely local applications.

You may be interested in qmail and its sub-applications, which does it
properly IMHO. It's a great example of the article's key point, which is the
value of simple messaging.

------
watmough
Wow, odd that this comes up as I was casting around in another domain, but my
working assumption is that I would implement my GPU processing library using a
'tags data' style binary protocol.

However, I was thinking of it more from a Amiga COPPER instruction list
perspective, but it amounts to the same thing.

The really cool thing is that approaches like this tend to be language-
agnostic, easy to program to, and extremely fast. All points raised in the
article.

~~~
vidarh
Tag lists more generally appears all over AmigaOS as well - especially as they
started running into limitations with the existing APIs that depended heavily
on fixed structs with AmigaOS 2.0, they more and more opted for adding taglist
parameters to avoid running into the same situation again.

And while it's request-response based over message ports, personally I see
AREXX ports in most larger Amiga applications as a good example of the overall
approach of cooperating processes to build larger systems.

Dbus and similar systems today has lots of great functionality, but what they
miss is what ensured AREXX was "everywhere": It was trivially simple to
support.

Pointing to the simplicity of OSC in the article really resonated with me,
because a lot of these systems are far less useful than they could be because
it's extra work. Simple systems on the other hand end up shaping application
structure:

AmigaOS was massively message based from the outset: Message ports and
messages were "built in" and used all over the place. E.g. when communicating
with filesystems or drivers or the GUI you do it via messages.

Then came AREXX and there was suddenly a standard message format for RPC -
both application-to-application and user-to-application.

If Amiga-applications were message oriented before, a lot of "post-AREXX"
Amiga applications took it to the next level by being shaped around a dispatch
loop that included not only e.g. GUI updates, but defining internal APIs that
exposed pretty much every higher level action in the application, and used
that as an explicit implementation strategy.

------
gourneau
The best thing about OSC IMO are the many wonderful configurable controllers.
When I was messing with OSC a few years ago TouchOSC
[http://hexler.net/software/touchosc](http://hexler.net/software/touchosc) was
my favorite

It is really fun to use these beautiful Star Trek looking interfaces to make
your robots move around for example ;)

~~~
thirdsun
TouchOSC has been some kind of model app for me - was available on the iPad
very early and is still going strong to this day while countless other
controllers came and went.

------
jacobolus
So what happens if you want to transmit some data in a nested hierarchical
format? Just pack it into some (ad-hoc?) higher level protocol inside one of
the flat pieces?

Seems like a recipe for inevitable incompatibilities when two languages no
longer have codecs for the new underspecified layered format.

~~~
wallacoloo
Although the article may not have made it clear, OSC is designed primarily to
pass around _control signals_ , rather than arbitrary, _non-contextualized_
data. Control signals have inherent meaning, like "fast-forward <n> seconds",
"set color to <x>", etc. The important distinction is that "set color to <x>"
(a command) is distinct from "{ 'color': 'x'}" (non-contextualized data).

The layout of _controls_ is hierarchical in the protocol; each control has an
associated endpoint, which looks very much like a unix file path. For example,
to unpause an audio player, you would send the message "/player/is_playing/set
t" (or f, to pause). If you wanted to control the bands of a 10-band
equalizer, you could implement that as "/player/eq/band/4/amp/set 0.5" (there
are certainly other ways to approach it too).

If you want to send control signals to multiple parts of the audio player at
the same time, you could use a "bundle" composed of a separate message for
each parameter you're altering. All the messages will then be handled
simultaneously.

Because of this, data does not need to be hierarchical. If you _really_ want
to, you can consider the endpoints to be "keys" and the control data to be
"values" in a hierachical key -> value map, just to prove that everything you
could do with hierarchical data is still possible using OSC. But I would
encourage you to not think of it in that abstraction because the fact that OSC
is built to pass messages, and not non-contextualized data is a large part of
what differentiates it from other protocols like json streaming.

------
AndrewUnmuted
For an especially bad-ass implementation of OSC, check out the SuperCollider
project at
[https://supercollider.github.io/](https://supercollider.github.io/).
SuperCollider is a programming language for audio synthesis/analysis, and
algorithmic music composition. It works using a client-server paradigm, and
they communicate with each other via OSC.

Shameless plug time! I've been using SuperCollider to design a machine-
learning solo electroacoustic music performance platform, called Sonic
Multiplicities. You can hear it at
[http://www.sonicmultiplicities.net/](http://www.sonicmultiplicities.net/).

------
steeve
Reminds me a lot of bencoding, which is not as popular as should (outside of
bittorrent of course)

~~~
klodolph
It's unfortunate that bencoding doesn't support text strings.

------
qwertyuiop924
Binary protocols are only cheap if your time is free. If you can't connect to
your service via telnet, you're going to spend a lot more time debugging. The
only way to fix this is to develop a textual representation of your format,
and write your own telnet, which is feasible, but nobody seems to have done
that to OSC.

Also, I find Armstrong's fanaticism grating, and am inherently distrustful of
anybody who claims that their community and language has all the good ideas.
Because odds are, they're wrong.

~~~
59nadir
> Also, I find Armstrong's fanaticism grating, and am inherently distrustful
> of anybody who claims that their community and language has all the good
> ideas. Because odds are, they're wrong.

You might find this[0] blog post by Armstrong interesting, wherein he comments
on some things that were/are done wrong in Erlang. I would also argue that
this 'fanaticism' is made to look more severe as you can find at least 4
different talks where Armstrong argues for the same things over and over.

In any case, Armstrong is actually very humble, which I think seems fairly
obvious if you actually watch a few talks and read more than a couple of blog
posts by him.

Could you point to something specific where you feel Armstrong misrepresents a
strong side of Erlang, or is it just a general thing?

(Edit: Just to add, I think it should be said that the Erlang team created
their favorite language and that it probably represents the most useful
language in use today to them, so I don't think it's surprising that they'd
see more good ideas there than in any other language. On top of that Armstrong
is a very straight shooter, it seems, so I think this might be something to
take into account.)

0 - [http://joearms.github.io/2013/05/31/a-week-with-
elixir.html](http://joearms.github.io/2013/05/31/a-week-with-elixir.html)

~~~
qwertyuiop924
It isn't so much that Joe doesn't think that erlang did anything wrong, it
just feels like he believes that erlang's way of doing things, and erlang's
ideas are the only good way of doing things, and are the only right way of
doing things. This makes me suspicious, because that is almost never true of
any language or community.

------
ww520
I don't understand his emphasis on session on TCP and UDP. A session requires
nothing more than a "unique" number to identify it. You can certainly have
session in UDP; just attach a session number on all the messages belonging to
the session. And you can certainly be sessionless in TCP; just close the
connection after each message. HTTP request has no session by definition and
it sits on top of TCP.

~~~
rdtsc
> just close the connection after each message.

That is if you don't expect to handle more than one connection at a time in a
reasonable way. You can certainly do this loop:

    
    
       server_socket = create_and_listen_on_socket(...)
       while True:
          csock = server_sock.accept()
          csock.send(process(csock.recv()))
          csock.close()
    

TCP setup and teardown is not too cheap (especially compared to sending UDP
packets).

~~~
ww520
UDP is certainly faster than TCP, no question about it, because of different
levels of quality of service. However, coding for both is similarly simple,
given the same requirement for short receive-only message. TCP session has
nothing to do with the complexity. The complexity comes in how to do process()
in parallel.

A receive-only message can be handled in TCP:

    
    
       server_socket = create_and_listen_on_socket(..., MAXCONN)
       while True:
          csock = server_sock.accept()
          msg = csock.recv()
          csock.close()
          process(msg)
    

A UDP loop has similar steps, minus the accept and close. The complexity to
use threads or async to run process() is the same for UDP and TCP.

For a normal data volume remote command protocol, the simple TCP loop is more
than adequate. Just set a reasonable MAXCONN to queue up client connection
requests, which can drop connections if there are too many requests, just like
UDP dropping packets.

Edit: I don't believe lock can be avoided in Sonic Pi server when handling
concurrent incoming commands, whether it's written in Erlang or not. Sonic Pi
I believe has a single audio device, which makes it a shared resource.
Concurrent access to a shared resource has to be managed with lock somewhere
along the call path. In that case a single thread TCP server is perfectly
fine, serving as a lock as well.

~~~
im2w1l
Shouldn't you handle fragmentation? I don't think TCP guarantees that
csock.recv() will give you all of the message.

~~~
ww520
Fragmented IP datagram is re-assembled at the IP layer before it is handed up
to UDP or TCP layer. If it can't be re-assembled, the datagram is considered
lost.

UDP is unreliable and has small packet size. The TCP code is emulating that
simple requirement. Why expand the requirement? Looping to read fully would
block on one connection. One rogue client would hold up the whole server.

~~~
im2w1l
I looked it up, and it seems that while IP will reassemble packets, there is
no 1-1 mapping between _send_ calls and IP packets, when using TCP.

You are only guaranteed to get the bytes in the correct order but you have to
find the boundaries between messages yourself.

~~~
ww520
That's correct. A send call could call with a 1GB buffer and the IP layer
would have to break it up into multiple datagram packets. That's the nature of
TCP. Again sending large message is expanding the requirement beyond what is
capable in UDP, while we were striking to emulate UDP in sending short
unreliable message.

------
bjacks
Can anyone explain what a sequential programming language is?

Context here:

"I guess I’d underestimated the difficulty of implementing Anything-over-TCP
in a sequential language. Just because it’s really really easy in Erlang
doesn’t mean to say it’s easy in sequential languages."

~~~
Gracana
Imperative programming. Lists of statements, vs Erlang's declarative
programming, where you describe the program without specifying control flow.

~~~
white-flame
Erlang is absolutely a fixed in-order control flow programming language, on
message receipt. It is (generally) referentially transparent, but not
declarative.

~~~
Gracana
Oh, I don't know much about Erlang so I thought for sure that's what he meant.
What was that supposed to mean, then?

------
dschiptsov
My 5¢ is that 9P2000 protocol would be much better and canonical example.

Also I hope some day one would show how to implement most of functionality of
Hadoop orders of magnitude more effeciently in bunch of Plan9 shell scripts.
Or in Erlang with 9P protocol driver.

------
echochar
Is there any similarity between "tag-length-value" and netstrings (1997)?

------
vshan
Woah, funny I see this. I've been working on a distributed shell, basically
trying to combine sockets and pipes to create remote pipes, aka inter-host
piping.

Something like:

    
    
      $ echo foo | host2:grep bar | host3:grep baz

~~~
RickHull
> _basically trying to combine sockets and pipes to create remote pipes_

Sockets, or ssh? To get a remote shell, you're going to want ssh.

------
yomritoyj
From the title I thought that the ultimate monad tutorial had finally arrived.

------
chrisdew
The problem with flat messages is that optional composite types can be
partially present.

------
gtirloni
gRPC seems like a good alternative.

------
jessaustin
Title probably should be edited to remove "Joe Armstrong: ". He might be
badass, but he isn't a way to connect programs together.

~~~
dmix
I read the article because it said Joe Armstrong. That way I knew it was
someone who knows what he's talking about. Especially given the topic.

~~~
auvi
Same here. If I have not seen "Joe Armstrong" in the title I wouldn't have
clicked on it.

------
yarrel
| ?

