
DIY Meteor-like Realtime Functionality Using Socket.io and RethinkDB Changefeeds - ginkgotree
http://www.scotthasbrouck.com/blog/2016/3/13/using-socketio-with-rethinkdb-changefeeds-to-build-a-reactive-backend
======
jondubois
RethinkDB changefeeds are great for dealing with data streams which are common
to all users, but changefeeds can get tricky when you need to deal with
hundreds of thousands of data streams which are more or less unique to each
user (or unique to many small groups of users) - This is particularly true
when you need to scale your app beyond a single process/instance/machine.

The approach described in this article may not be suitable for a lot of use
cases. It's fine if all users shared the same central collection of Fruits
(E.g. a giant Fruit basket)... But what if users were divided into many small
groups and each group had its own independent Fruit basket? It becomes
extremely difficult to manage all these unique changefeeds on the backend
(especially if you have to scale beyond a single machine).

The team behind RethinkDB is currently working on a project called Horizon
(previously called Fusion) which should solve this problem.

~~~
kjstevo
What approach would you take to solve the problem you describe of individual
fruit baskets that can scale?

~~~
jondubois
The simplest way to implement this is using a client-side pub/sub engine and
have a channel subscription for each resource on the frontend.

So for example, if you have a Fruit basket with id 123 and you want to get
realtime updates when that basket changes (E.g. new fruits are added or
whatever), you could subscribe to a channel 'fruitbasket/123'.

You would have a separate channel for every basket but because the backend
treats all channels in a generic way, the complexity on the backend remains
minimal (and is handled by your pub/sub engine anyway).

You need to make sure that your pub/sub engine is good at handling lots of
distinct channels (and is designed to be user-facing - A lot of pubsub engines
like Redis are only designed for backend use). There are user-facing pub/sub
SaaS services which you can hook into your app but they are a bit expensive.

If you want a self-hosted user-facing pub/sub solution, there are only two
popular options:

\- SocketCluster
[https://github.com/socketcluster/socketcluster](https://github.com/socketcluster/socketcluster)

\- Faye [https://github.com/faye/faye](https://github.com/faye/faye)

Both Faye and SocketCluster are good at handling lots of unique channels -
Based on benchmarks I ran, both can handle the same load of about 10k new
subscriptions per second (per CPU core). SC makes scaling across multiple CPU
cores trivial so if you have an 8-core machine, you can handle 80K new
subscriptions per second with SC out of the box. You can also scale with Faye
but it will require a lot more work to get working.

Disclosure: I'm the main author of SocketCluster.

------
heron88
Great write up! I wrote about this idea last month, although in a less
detailed way. I've also included a repository in my article with a simple
React and Redux front end that demonstrates some of these concepts.

[http://www.andrewcoelho.com/essays/rethinkdb-realtime-
apps/](http://www.andrewcoelho.com/essays/rethinkdb-realtime-apps/)

~~~
arrmn
Hi a small note, it's impossible to see links on your blog, make them easier
to recognize.

~~~
ginkgotree
Noted, underlined them. Thanks!

------
phaedryx
One thing that I've never understood is why websockets instead of server sent
events?

~~~
jamescun
Possibly Internet Explorer support. Even Edge doesn't support them yet[1].

With deployments of HTTP/2 increasing, I think we'll see greater adoption of
server sent events. At the time of writing, HTTP/2 and WebSockets are two
incompatible standards. For non-latency sensitive applications (i.e. not
games) it is very easy to implement a WebSocket like interface over HTTP/2
using SSE (server -> client) and POST requests (client -> server) with very
little overhead (thanks to HTTP/2's multiplexing).

[1] [http://caniuse.com/eventsource](http://caniuse.com/eventsource)

------
KaoruAoiShiho
meatier is this but less DIY.

~~~
bryanlarsen
Perhaps downvoters thought you were making a joke?
[https://github.com/mattkrick/meatier](https://github.com/mattkrick/meatier)
is very appropriate to this discussion.

