
Show HN: Nchan – a pub/sub server as an Nginx module - slact
https://nchan.slact.net
======
slact
This is a huge refactoring of an old project of mine -- the Nginx HTTP Push
Module. I'm wondering if anyone here has used it.

Most importantly, I want feedback on the documentation. Did I overcomplicate
things? Does it need more examples? Does it need more live code? Is it too
long? Too short? Etc.

~~~
jkarneges
I love the clean client API, e.g. the use of ETags with long-polling.

In some ways it reminds me of our LiveResource protocol
([http://liveresource.org](http://liveresource.org)). It's still a work in
progress, but maybe we can consolidate ideas. It'd be cool if an LR client
could be pointed at an Nchan resource.

~~~
slact
Interesting, I'll read through this. Nchan code is quite modular, so it would
be pretty easy to add some kind of negotiation like this. Let's talk more.

For reference, here's the old document I wrote for the long-polling protocol
back in '09:
[https://pushmodule.slact.net/protocol.html](https://pushmodule.slact.net/protocol.html)

------
sams99
This is super interesting, I have a very similar project that we use at
Discourse
[https://github.com/SamSaffron/message_bus](https://github.com/SamSaffron/message_bus)
clearly being an NGINX module nchan is going to be a LOT faster

That said there are few little pointers that would make integrating MessageBus
with nchan a bit challenging

\- We have a concept of "reliable pub sub"
[https://github.com/SamSaffron/message_bus/blob/master/lib/me...](https://github.com/SamSaffron/message_bus/blob/master/lib/message_bus/redis/reliable_pub_sub.rb)
this means that if you shut your laptop and then open it stuff just magically
catches up with no loss of messages or ordering. Does the redis store you have
do something similar (I can not see backing lists in the redis store
implementation in store.c)

\- We implement per-message security, meaning you can publish to a
channel/group or channel/user and only those users get the message, I see you
can do auth upfront but can per-message security seems not doable.

\- We don't bother with per-channel URLS and instead just have a single
endpoint you multi-subscribe using POST params, we often subscribe to 10
channels on a request, the multiplexing seems a bit limited

\- When we subscribe to a channel due to the "reliable" nature of the store we
are able to tell the server what the last id is we subscribed to and catch up
from there

I definitely see us moving the moving our message bus server piece closer to
the web server like nchan has, its by far the most scalable way, but its tough
keeping feature parity.

~~~
slact
Nice project, and a curious featureset overlap, too...

> We have a concept of "reliable pub sub" [...] Does the redis store you have
> do something similar (I can not see backing lists in the redis store
> implementation in store.c)

Dig a little deeper...
[https://github.com/slact/nchan/blob/master/src/store/redis/s...](https://github.com/slact/nchan/blob/master/src/store/redis/scripts/get_message.lua)
I used lua scripts for all the fancy redis logic, that's where you'll find the
backing list accesses. Messages are stored as hashes, referenced by id in
lists, along with some other channel metadata. So a longpoll or EventSource
client knows its last message id, and can request the next avaliable message
as long as said message has not yet expired. Websocket clients don't have this
information, and I'm not yet sure how to relay it with each message while
remaining content-agnostic. Basically, regardless of the protocol, you can
send a If-Modified-Since + If-None-Match or Last-Event-ID headers and it will
resume from that position in the message queue for the given channel.

> I see you can do auth upfront but can per-message security seems not doable

Per-message access will definitely not be implemented. You could, however, do
this client-side by, say, encrypting the messages and sharing keys with
authorized subscribers. That's kind of roundabout though.

> We don't bother with per-channel URLS [...], we often subscribe to 10
> channels on a request, the multiplexing seems a bit limited

The main use-case I had in mind for multiplexing is that of a single channel
per user, and some shared broadcast channel. It's currently limited to 4 max
because I wanted to get this code out the door. Unlimited multiplexing will be
supported in the future, and you could trivially rebuild the module to support
up to 16 right now (At the cost of some memory per message per subscriber per
channel overhead).

> When we subscribe to a channel due to the "reliable" nature of the store we
> are able to tell the server what the last id is we subscribed to and catch
> up from there

Yep, Nchan does that too. (except for Websocket, and hopefully I'll find a
workaround)

How does that work for feature parity?

~~~
sams99
nice! I initially stayed away from lua due so I could support earlier redises,
but these days I would totally take that dependency on to simplify the code.

regarding that implementation, one diff I am spotting is that I also have a
concept of global message id / global channel, this allows us to subscribe to
a single spot and distribute from there (which keeps thread counts down and is
particularly useful for server -> server comms.)

A lot of parity with our projects :)

Regarding web sockets, I decided against supporting them (initial
implementations did) the issue is that web sockets are 100% flaky on HTTP and
just add tons of unneeded complexity, in advent of HTTP/2 in NGINX they would
be a net loss imo cause you would be wasting a connection.

~~~
slact
> global message id / global channel Yeah, I don't have global ids. every
> message is bound to the channel it was published to. But I do understand
> that for your use case, (user + arbitrary list of groups), you'd need a good
> deal more than 4 channel multiplexing. It's a pretty strong use case and
> I'll see what I can do in the next month.

Of course, I can be incentivized to work faster with a generous donation :)

> Regarding web sockets [...]

All the cool kids were talking about it, so I thought I might as well support
them, too.

~~~
solyaris
>All the cool kids were talking about it, so I thought I might as well support
them, too.

I agree with Sam Saffron (thanks Sam for al your githubcode) about some
perplexity using websockets as a panacea nowadays.

Nevertheless, I really appreciated Nchan structured approach, open to
different available protocols (poll, SSE, websockets) :)

------
thesmart
Does Nchan broadcast messages that may have been missed by a client that was
temporarily offline? When web client's navigate from page-to-page or enter
sleep mode, client's will miss messages during the offline delta. Ideally,
there would be a way to "catch-up" a client or to tell if a client is
permanently out-of-sync because it is beyond a message history window.

~~~
slact
Yes. All messages are buffered for a configurable length of time. Longpoll and
EventSource clients receive the last message id with each message, and if
interrupted can be resumed from there provided the message has not timed out.
I don't yet have a way of transferring the last-received message id for
websocket clients, but if it is known it can be set during the ws handshake.

------
geekuillaume
I'm using Nginx Push Stream with NodeJS for a high-scale chat system (Soon to
be released, completely Open-Source). The dependency on Nginx always bothers
me. How hard do you think it would be to do a system like yours but completely
standalone ? Then we would be able to integrate it to other languages via
plugins (NodeJS, Python, etc).

~~~
slact
Nchan is about 12K lines of C, I'd say 2-5K of that is dealing with Nginx
guts. To get rid of Nginx entirely, you'd need to add an event loop, forking
and multiprocess management, config parsing and reloading, and shared memory
allocation code. That's not a simple task, but it's certainly possible. The
reason I built this on top of Nginx is precisely because I didn't want to
handle those other things. Besides, nginx these days is a hulking scalable
monster. What's the bother?

If you really don't want an nginx dependency, I'd say you're better off
rolling your own pubsub server in Node.

~~~
geekuillaume
Okay, thanks for the information. The problem with doing the pubsub in NodeJS
entirely is the way NodeJS handle connections. Each is separated and is
accompanied by a big overhead from NodeJS. What would be great is a way to
interact with the pubsub server not by config but with an API. This would
allows, for example, the execution of middlewares when a user publishes a
message (to filter them or something else). Right now, I'm using two
websockets, one to Nginx PushStream only to receive messages and another to
the NodeJS server to publish messages. The NodeJS server is used to parse the
messages, format them, authenticate the user and apply the middlewares before
publishing the message to Redis. Then, each NodeJS get the message from Redis
and POST it to Nginx. The problem here is that there is two sockets for each
user and the complexity involved with the communication between Nginx and
NodeJS. That's why a really performant standalone websocket engine with an
extensive API would be awesome.

~~~
slact
> the execution of middlewares when a user publishes a message (to filter them
> or something else)

You can do that with nchan: [https://nchan.slact.net/details#authenticate-
with-nchan_auth...](https://nchan.slact.net/details#authenticate-with-
nchan_authorize_request)

> Right now, I'm using two websockets, one to Nginx PushStream only to receive
> messages and another to the NodeJS server to publish messages.

You can also multiplex several websockets into one for the client.

I can't offer you a standalone server, but I can offer some pretty fancy
features : )

~~~
geekuillaume
I saw this feature, but I think there is limitation preventing me to use it,
tell me if I'm wrong ;) \- I cannot use this to make a call each time a
message is published on a pubsub websocket \- I cannot modify the message sent
by the user (to add user information for example)

Also, what do you mean by multiplexing websockets ?

~~~
slact
That's correct, that feature is for authentication only. I may add a feature
to replace the message with the back end response.

By multiplexing I mean that a single websocket (or any other) subscriber can
subscribe to multiple channels.

------
uyoakaoma
Is this different from the other module nginx push stream which offers
websockets, long polling etc

~~~
slact
Yes, this is a different project, although they are related. Push Stream is a
fork of my Nginx HTTP Push Module, so they both descend from the same original
codebase. Push Stream uses a blocking concurrency model, whereas Nhan is
completely non-blocking. In theory, this means Nchan should scale better. In
practice, I haven't benchmarked it heavily enough yet to see a divergence.

Besides that, nchan and Push Stream offer different feature sets. For example,
Nchan has horizontal scaling and persistence through redis, whereas push
stream has customizable message transforms. There are other differences as
well, but that would take a whole article to elaborate. I should probably
write it soon.

------
jondubois
The "client-less" approach would be particularly nice for use with IoT devices
which don't have enough CPU/memory to run a beefy pub/sub client (which is
typical of other solutions). So that's nice.

Is there a way to perform access control on the backend? E.g. Is there a way
to prevent a specific user from subscribing to a specific channel (if they are
not allowed)? Or is this specifically designed to deal only with public
channels (that anyone can publish/subscribe to)?

~~~
slact
Yes, backend-authenticated access control is possible. See
[https://nchan.slact.net/details#authenticate-with-
nchan_auth...](https://nchan.slact.net/details#authenticate-with-
nchan_authorize_request)

------
ktt
Wow, excellent! I've been looking for lightweight websockets pubsub and had to
create something simple in Node to achieve it but this would've sufficed.

I can't wait to give it a try.

------
pablomolnar
Great project! Is there any concept of ack/nack a message? Let's say the
subscriber get a message and failed before processing it. Does the message go
back to the queue?

~~~
slact
No, ack/nack would need to be implemented in the client or the application.

However, messages do not disappear from the queue when received. All
subscriber requests are idempotent, and can be repeated so long as the message
queue is storing the message (which is a configurable parameter). So if a
subscriber failed before processing it, it's free to request the same message
again.

------
Beltiras
Interesting project. I've up till now used Twisted or Tornado (actually
swampdragon.net). My choice would be dependent on scalability and performance.

~~~
slact
I consider those two and socket.io my main competition. Nchan is built to
scale, but of course you'd need some numbers to back that up. I plan on doing
benchmarks once I iron out the documentation, although I realize it would be a
great selling point to have the numbers ready right away.

------
Raed667
How does this compare to MQTT ?

~~~
slact
I'm not familiar with MQTT, I've just glanced over the spec right now.

Nchan is basically a message broker with channels, optimized for message
broadcast.

MQTT is a TCP-level protocol, whereas all the currently implemented subscriber
clients for Nchan are HTTP-level (Longpoll, EventSource and Websocket, which
begins with an HTTP request). MQTT subscribers and publishers _could_ be
implemented in nchan, but I haven't yet written any raw-TCP connection
negotiation code, so I don't know how hard it would be. Aside from that, the
subscriber code is very modular and adding another protocol like MQTT would be
straightforward.

~~~
Raed667
I see. Thank you for this, I'll give it a shot and see how a simple client
compares to an MQTT client.

If I get a good performance on a C/C++ client then this could be a viable
option for push notification in the IoT domain.

~~~
solyaris
>this could be a viable option for push notification in the IoT domain.

Yes! :)

------
gauravphoenix
This is wonderful. I hope this module gets added to openresty.

------
userbinator
Is the description just a rather roundabout way of saying it can be used as an
anonymous messageboard? The name certainly evokes that.

~~~
slact
Well, it would be very easy to make a realtime 4chan-like thing with Nchan,
but it's far from the only use case. It's a general-purpose server and pubsub
proxy.

The name is a play on nginx and channels. I got tired of saying "nginx http
push module" all the time and renamed it.

