
A distributed, tag-based pub-sub service for modern web applications - sdomino
https://github.com/nanopack/mist
======
nodesocket
A lot of open-source projects by Nanobox look awesome, but most don't have any
documentation or demos :-(.

I'm most interested in:

    
    
      [Hooky D](https://github.com/nanopack/hookyd) - Remote hook execution layer  
      [Butter](https://github.com/nanopack/butter) - Git based deployment system  
      [Shuttle](https://github.com/nanopack/shuttle) - Transparent TCP proxy for web

~~~
tylerflint
Yeah we hear ya. The nanopack cloud-initiative was just announced last friday
([https://blog.nanobox.io/nanopack-a-new-vision-for-
automated-...](https://blog.nanobox.io/nanopack-a-new-vision-for-automated-
infrastructure/)). We're getting there slowly but surely!

------
sdomino
As a point of clarification (I work at Pagoda Box/Nanobox). Pagoda Box uses an
older suite of similar tools ([https://github.com/stormpack-
core](https://github.com/stormpack-core)) mostly written in Erlang. This newer
project (written in Golang) is part of a group of tools
([http://nanopack.io/](http://nanopack.io/)) being written for
[https://nanobox.io/](https://nanobox.io/)

~~~
polskibus
I'm curious why did you drop Erlang in favor of Go?

~~~
tylerflint
The actual reason is much less exciting than you might have hoped and actually
has nothing to do with erlang vs golang...

The erlang version has been running successfully for about 2 years and we
haven't had any issues whatsoever (excepting a wrestling match with mnesia
early on). We are erlang/elixir advocates and have used the erlang vm
successfully on highly critical multi-million-concurrency services for over 6
years.

When we set out to build nanobox desktop
([https://desktop.nanobox.io](https://desktop.nanobox.io)) our vision was to
provide a single pre-compiled executable that could be run without any
configuration. The design required a push layer and needed the same
functionality that the erlang project was already providing. It wasn't
feasible to package the erlang application into the nanobox binary, so we
emulated the original project into a consumable golang package. As time went
on we were porting more and more of the features into the golang port until
all that was lacking was distribution and authentication. At that point we
made the decision to consolidate our efforts into a single project, that could
be a standalone service or composed within a golang binary.

I regret to inform you that there isn't a mass exodus within our company to
ditch erlang for golang. While that certainly would make for a fun thread, in
this case it was simply a matter of fit and effort consolidation.

------
ShaneWilton
This looks like a great project! I'm curious about the following caveat from
the readme, however:

"Data flowing through mist is NOT touched in anyway. It is not verified in any
way, but it MUST NOT contain a newline character as this will break the mist
protocol."

I haven't read the code yet, but my gut reaction is that this suggests it
would be possible to inject commands using malicious user input. Given the
caveat, this may be allowed for by your threat model, but it may be worth some
effort to mitigate.

EDIT: To clarify, this may not be the case at all, in which case I'd be
curious to hear why this restriction is in place.

~~~
tylerflint
This is referring to messaging being sent through mist. Mist doesn't care what
the data looks like, and won't transform it.

Currently the public-facing websocket client is not allowed to publish
messages until this is mitigated, as you mentioned. Any feedback here would be
appreciated.

~~~
zrail
Netstrings are for this exact use case:

[http://cr.yp.to/proto/netstrings.txt](http://cr.yp.to/proto/netstrings.txt)

~~~
viraptor
Actually this proposal is quite bad compared to simple "<4 bytes size><size
bytes contents>". If you use netstrings recursively like djb recommends, you
will end up copying / reallocing the strings many times. Since your top-level
length depends on the lower level length, you cannot send it without actually
calculating the whole contents. Asn1 DER has the same issue because of the
weird size encoding.

RIFF is a bit better
[https://en.wikipedia.org/wiki/Resource_Interchange_File_Form...](https://en.wikipedia.org/wiki/Resource_Interchange_File_Format)
with strict length,tag,contents format.

~~~
zrail
Wasn't suggesting they use the recursive feature. Just a flat netstring for
the data segments, sort of how the Redis and Memcached protocols work.

------
rar_ram
> Messages are not guaranteed to be delivered, if the client is running behind
> on processing messages, newer messages could be dropped.

> Messages are not stored until they are delivered, if no client is available
> to receive the message, then it is dropped without being sent anywhere.

Not sure in which use cases can I use this ? Doesn't this defeat the purpose
of having a message queue.

~~~
tylerflint
So to the first question, there's not a transactional layer to guarantee the
client handled the message. Once a message is sent to the client, it is
assumed the client has a mechanism for queuing messages if it can't keep up.

As to the second, the caveat is basically saying that if a message is
broadcast and nobody cares about the message (ie: there aren't any
subscriptions) then mist isn't going to store the message anywhere, it just
gets dropped.

~~~
rar_ram
Err, I get that! but I am still wondering on where would I use this ? If I
dont need transactional layer or buffering, I could use event listener and
dispatcher pattern. Anything more than that, a message queue like ZMQ works.
Where does this fit in ?

~~~
tylerflint
Oh I see. Sorry for the confusion. The initial need was for a rails app to
send data model updates to dashboard clients. Later, as our infrastructure
became more micro-service-oriented, mist has been used as a place for the
dashboard javascript client to subscribe to a centralized event queue. Each
micro-service will publish messages to the mist cluster, and the dashboard can
respond accordingly.

------
foo42
Struggling to see when I'd use this over, say, rabbitmq. Feels a bit like "not
invented here" syndrome, though glad to be proven wrong. Also curious that
this replaces previous tooling written in erlang, be interested to here what
prompted the move to golang, I would think a message broker was right in
erlang's wheelhouse

~~~
lobster_johnson
Not the OP, but this looks like it's designed to be lossy, and prioritize
availability over deliverability. If clients are unable to keep up with
message volume, messages will be dropped.

That makes is okay for things like realtime dashboards (which this is
apparently designed for) and other rapid events where transactionality is not
required.

But it's obviously not suited for job-like scheduling like data processing or
email delivery. Nor as an event bus for inter-service coordination ("on event
X, do Y").

~~~
tylerflint
That's a pretty accurate assessment. Mist could be conceptualized as a self-
hosted replacement for pusher ([https://pusher.com/](https://pusher.com/)).
It's designed specifically to address building realtime web apps.

------
tylertreat
Curious how this compares to gnatsd: [https://github.com/nats-
io/gnatsd](https://github.com/nats-io/gnatsd)

~~~
lobster_johnson
Two things come to mind: gnats uses wildcard topics, very useful to create
filtered subscriptions. And gnats is pretty good at dealing with backpressure;
the stated aim of Mist is to silently drop messages if clients can't keep up.

I would also say that gnats is mature and has been proven in production for
several years, which might not be the case with this project.

------
mappu
I need one of these at $DAYJOB as of yesterday, so thanks!

Any chance of basing it on `igm/sockjs-go` or similar instead of raw
websockets?

~~~
jkarneges
An alternative is to use a proxy. For example Pushpin
([http://pushpin.org](http://pushpin.org)) can convert SockJS to native
WebSockets. There might be others too.

------
tylerflint
Also to clarify (and yes I also work at Pagoda Box on the Nanobox project),
this service was initially intended to provide push capabilities for rails and
other web frameworks that can't natively provide a push socket layer. It has
successfully filled that niche and has expanded to fill other needs as well.

------
cbsmith
It's like PubSubHubbub, except with JSON instead of XML, and bypassing that
whole HTTP protocol that was never going to catch on anyway...

------
sanderson1
Is this what what Pagoda Box uses for their in-dashboard live updates or is it
an off-shoot of that project?

------
zwily
How is this distributed?

~~~
tylerflint
When a node is brought online, it uses multicast discovery to learn about
other nodes. You can see the gist here:
[https://github.com/nanopack/mist/blob/master/main.go#L68-L76](https://github.com/nanopack/mist/blob/master/main.go#L68-L76)

Propagating events is really simple: When a client subscribes to a particular
node, the node will replicate the subscription to all nodes in the cluster,
and relay any messages from the 'proxied' subscriptions back to the client.

~~~
tylerflint
Also, if your network can't natively support multicast, you might want to
check out red
([https://github.com/nanopack/red](https://github.com/nanopack/red)) "A fast,
in-kernel, ad-hoc point-to-point vxlan network."

