
Rxdb: A reactive database where you can subscribe to the result of a query - dsun180
https://github.com/pubkey/rxdb
======
kiwicopple
I have been developing something that provides this functionality for
PostgreSQL:
[https://github.com/supabase/realtime](https://github.com/supabase/realtime)

It's still in _very early stages_ (although I am using it in production for my
company)

It's very similar to Debezium (mentioned in another comment), but it's built
with Phoenix (elixir), so great for listening via websockets.

Basically the Phoenix server listens to PostgreSQL's replication functionality
and converts the byte stream into JSON which it then broadcasts over
websockets. This is great since you can scale the Phoenix servers without any
additional load on your DB. Also it doesn't require wal2json (a Postgres
extension). The beauty of listening to the replication functionality is that
you can make changes to your database from anywhere - your api, directly in
the DB, via a console etc - and you will still receive the changes via
Phoenix.

I still have to document a lot of how it works and how to use it, but if
anyone is interested then I will make it a priority over the weekend

~~~
dsun180
There is a big difference between the changestream of many SQL and noSQL
databases, and what RxDB does. Having a stream of changes is useful but not
the whole solution. RxDB is capable of using single document changes of a
stream and recalculate the new results of an existing query. This saves you
not only much IO performance but makes developing much easier. See
[https://rxdb.info/query-change-detection.html](https://rxdb.info/query-
change-detection.html)

~~~
tjchear
I would love to use this just for the offline and query change detection
capabilities alone - but the latter is currently beta and disabled by default.
Is there a reason for that?

~~~
cryptonector
I do this sort of thing with triggers and table-based materializations of
views.

------
throwaway_bad
Having client state just be a replica of server state solves so many problems
I don't understand why the concept never caught on. Pouchdb/couchdb are still
the only ones doing it afaik.

Instead we have a bajillion layers of CRUD all in slightly different protocols
just to do the same read or write to the database.

~~~
koolba
When your data is public and immutable, this approach is very pleasant. The
client becomes just another caching layer and worst case it's presenting a
historical version of the truth. You can even extend this across tabs with
things like local storage.

This breaks down quickly once you have data that could become private or
mutate rather than append.

~~~
tehbeard
This what stopped me exploring couchdb further.

The "one db per user" model for private data made using other features like
views etc more difficult when you have to upgrade,edit,remove them.

Mutability wasn't really a problem, either present the conflicts to user and
pick one or write code to merge if possible.

~~~
fermuch
Couchbase (with its sync gateway) uses "channels" to sync data. It even lets
you change the channel of a doc, and to the sync client it shows as if the doc
was deleted (if the user is not part of the new channel)

------
tga
If you want to subscribe to changes in a Postgresql database, whether produced
by your app or not, you can also use Hasura or PostGraphile:

[https://docs.hasura.io/1.0/graphql/manual/subscriptions/inde...](https://docs.hasura.io/1.0/graphql/manual/subscriptions/index.html)

[https://www.graphile.org/postgraphile/realtime/](https://www.graphile.org/postgraphile/realtime/)

~~~
kiwicopple
I tried out Postgraphile Realtime and it's pretty cool.

For anyone who wants something similar that's not GraphQL, then I'm in the
early stages of developing a Phoenix (Elixir) implementation which broadcasts
changes over websockets:
[https://github.com/supabase/realtime](https://github.com/supabase/realtime)

See comments here:
[https://news.ycombinator.com/item?id=21354039](https://news.ycombinator.com/item?id=21354039)

~~~
BenjieGillam
Neat; happy to have inspired you!

~~~
kiwicopple
You did Benjie! Postgraphile is really an awesome piece of tech. I use
PostgREST a lot, and I see graphile as an awesome graphql equivalent.

------
jtmarmon
You can do this in datomic! I made an end-to-end proof of concept of this
using datomic and websockets. The client can open a websocket and subscribe to
a query and then the server will react to changes in the database and
automatically update the model and therefore the UI (no eventing code
required).

Demo vid
[https://v.usetapes.com/85knuAPruB](https://v.usetapes.com/85knuAPruB) Longer
description in github readme
[https://github.com/jtmarmon/hackerthreads](https://github.com/jtmarmon/hackerthreads)

~~~
dsun180
I think you have not understood what RxDB is. It is a client side database. It
works also when the client is offline. It does not need a stable internet
connection over a websocket.

Your example is similar to RethinkDB and others. A websocket streaming json.

~~~
kickopotomus
A more common solution includes using Datascript[1] as the client side DB and
then using something like Posh[2] or Datsync[3] to handle query/pull
subscription changes.

[1]:
[https://github.com/tonsky/datascript](https://github.com/tonsky/datascript)

[2]: [https://github.com/mpdairy/posh](https://github.com/mpdairy/posh)

[3]:
[https://github.com/metasoarous/datsync](https://github.com/metasoarous/datsync)

------
stigi
Another DB that plays in that field is Watermelon DB
([https://github.com/Nozbe/WatermelonDB](https://github.com/Nozbe/WatermelonDB))
which supports react-native based on SQLite and the web based on the LokiJS in
memory DB.

------
maxmcd
General question about reacting to database events: It seems like when
responding to DB events it would be easy to accidentally create an infinite
loop. Is that an issue with this pattern, or is it easy to avoid? Do any of
these data subscription tools have safeguards in place to prevent this?

~~~
rockostrich
Isn't this the case with any interaction between 2 systems? Between a client
and server, you can have logic in the client that reacts to a response from
the server that triggers another request to the server.

~~~
maxmcd
Very true, maybe I'm overthinking it? In my head: If I update full_name from
first_name and last_name whenever a user table row changes, if I do this
naively I will update the user on every update and trigger a continuous loop
of updates.

I have certainly created infinite loops while using React's
componentDidUpdate, maybe it's just important to define triggers on single
attributes rather than entire database rows.

------
dizaime
Meteor do this since inception with MongoDB on server and MiniMongo in client.

~~~
dsun180
Yes, mongodb and meteor and minimongo do something similar but with much more
restrictions on what you can do.

RxDB does exactly one thing which is being a client side database. You are not
tied to a specific ecosystem or backend database.

~~~
dizaime
Yeah, with RxDB, we are tied with RxJS, PouchDB and CouchDB.

Anyway, database is hard, opensource is great. We love them both, RxDB or
Meteor.

------
henriks
Isn't "subscribe to all state-changes like the result of a query" something
you'd implement efficiently using a forward chaining inference engine, like
something based on Rete?

~~~
digitalneoplasm
I don't know too much about database systems, but if we were in an inference
(e.g., production) system, then that would make a lot of sense! I build
logical inference systems and have implemented this kind of functionality in
forward, backward, and bi-directional logical inference.

------
MehdiHK
If you want a change-stream out of your regular MySQL or PostgreSQL database
systems, check out Debezium: [https://debezium.io/](https://debezium.io/)

It basically registers itself as a fake replica server so that it can get
updates from the master (like binlog in case of MySQL) and then it forwards
those updates to Kafka. The possibilities are endless what you can do with
those updates as Kafka consumers.

~~~
dsun180
There is a big difference between the changestream of many SQL and noSQL
databases, and what RxDB does. Having a stream of changes is useful but not
the whole solution. RxDB is capable of using single document changes of a
stream and recalculate the new results of an existing query. This saves you
not only much IO performance but makes developing much easier. See
[https://rxdb.info/query-change-detection.html](https://rxdb.info/query-
change-detection.html)

~~~
tga
Sure, that's handy as an extra feature to get better performance, but I see
that even in RxDB it's currently in beta and turned off by default.

I suppose this could be the killer feature to warrant using a new database --
because otherwise, from a user point of view, the result is the same.

------
bot1
So like firestore?

------
psoren
Maybe this is an uninformed comment, but how is this different from the
Firestore database in Firebase where you can listen for when a document
changes?

------
jinjin2
This is something Realm does really well, both locally on the device and also
with live subscriptions to data in the cloud.

We used this for our mobile apps and the experience was pretty awesome. The
ability to live observe both individual objects in the dB and results of
queries makes building reactive UI’s a very pleasant experience.

------
egeozcan
I was complaining about lack of such a database just today:
[https://news.ycombinator.com/item?id=21352235](https://news.ycombinator.com/item?id=21352235)

I'm not doing field projects anymore but knowing this exists would have helped
me tremendously in the last few years.

Thanks for sharing!

------
1337shadow
Nice ! reminds me of the Ryzom project in Python/Django/Postgres which
implements a meteorjs-like protocol and proposes the same example with a todo
list

[https://github.com/yourlabs/ryzom](https://github.com/yourlabs/ryzom)

------
lessname
What I personally like about RxDB is that it also works completely offline.
However, I don't really understand encryption: As far as I understood, it
encrypts on the client side with the db passwort which is sent to the server.

Another issue is Authorization and Authentication, I could not find a good
solution for me for CouchDB. Couchbase seems to have better solutions for this
but the premium plan seems really expensive and as far as I understood you
need a "server" and a "sync server" which don't have low system requirements,
at least for me.

~~~
dsun180
No you normally do not send the password to the server. You can ask the user
to enter the password when the application is openend.

Authentication is much easier when you use the GraphQL replication. There you
are much more flexible on which data you return depending on which user is
asking for it.

------
bilekas
I personally don't like the idea of that coupling, I prefer to use my own
events, and go through websockets for ex..

This is a nice feature for some, but not all situations..

------
jaime-ez
deepstream.io has similar functionality and supports postgres, rethinkdb and
other databases. It is a great tool and promotes a very nice paradigm.

------
venantius
I'm not sure why you would use this over something more battle-tested like
RethinkDB which was also built with this use-case in mind.

~~~
silasdavis
Because rethinkdb development has ceased? Last release was in 2017.

Is there still maintenance?

~~~
jjuel
The community has taken it over as far as I know. Looking at the github there
appears to be some active development still.
[https://github.com/rethinkdb/rethinkdb](https://github.com/rethinkdb/rethinkdb)

------
beaconstudios
hey, this is great! I've been building my own version of exactly this concept
based on an event-sourced in-memory graph for my visual programming
environment and it works really well for creating a collaborative editing
system. I'll be investigating this project now to see if I can't use it
instead of my hand-rolled solution.

------
mitar
I have made something similar trying to match more the concept of a
materialized view, but just on the client instead of inside the database:
[https://github.com/tozd/node-reactive-postgres](https://github.com/tozd/node-
reactive-postgres)

------
GrayShade
See also Noria: [https://notamonadtutorial.com/interview-with-norias-
creator-...](https://notamonadtutorial.com/interview-with-norias-creator-a-
promising-dataflow-database-implemented-in-rust-352e2c3d9d95).

EDIT: never mind, I should have read the article.

~~~
dsun180
I'm sorry but I think you comment is spam. The product in your article has
only one similarity with RxDB which is being a database.

~~~
GrayShade
You're right, sorry. I didn't click on the link and thought it was something
else. I can't delete my comment, unfortunately.

------
TheUndead96
This project has all of the goals that excited me about RethinkDB, I hope this
project is more successful.

------
winrid
Cool. But how is this better than Meteor for building webapps? Meteor would
scale better too with Apollo.

~~~
dsun180
I do not think that meteor will scale better then your GraphQL server with
whatever database you want to have. With meteor you are bound to a specific
ecosystem which is often a pain. RxDB does only one thing, it is a client side
database. Everything else in your stack is free to choose.

~~~
winrid
Oh, I didn't know this was just for the client. Missed that.

------
mizzao
Anyone know how this is different from the reactive query layer that Meteor
built on top of Mongo?

------
siscia
What is the time complexity of receiving a notification?

Are there triggers that run on every update on the dataset?

------
hashkb
Hasura does this for Postgres. What are some areas where I'd use this instead?

~~~
dsun180
RxDB is offline first. You can still query your data even when the user has no
internet. Hasura will not make your app workable without a stable connection
to the server.

------
amelius
How useful is real-time data really? For example, HN isn't updating in real
time and for me that's fine. If real-time updates come at a ridiculous
complexity and performance cost, then is it really worth it?

See also Google Wave, which had "real time" as its main novelty, but
ultimately failed.

~~~
beaconstudios
for my project it's very useful for collaborative features. Multiple people
need to be able to edit the same graph structures at the same time, which
would be difficult to scale out via a notify-and-poll architecture.

~~~
amelius
I've found that collaborative editing can be very confusing, and prefer a
commit/merge step over simultaneous editing. I think all collaborative editing
software should at least offer this kind of interface.

~~~
beaconstudios
agreed - there's a branching model in the roadmap :)

------
marknadal
There is also (mine)
[https://github.com/amark/gun](https://github.com/amark/gun)

It is run by Internet Archive, HackerNoon, DTube, Notabug, etc.

Handles about 8M monthly active users!

\+ end-to-end encryption, graphQL & graph data, upcoming Svelte support,
decentralized, etc.

MIT/Zlib/Apache2 licensed.

~~~
yc-kraln
I read through your whole readme and I still am not able to answer the
question: "what kind of project would I use GUN for?"

or, even, "what is gun replacing?"

I mean, it looks __interesting __and I would like to play around with it, but
I don 't understand enough to know what kind of stuff I can build with it.

~~~
dsun180
I tried several times to understand what GUNdb does and how its different or
which features it has. I have given up. As far as I can tell it is something
between Blockchain, graph-database and sync.

I also tried to understand the codebase which is impossible with obfuscated
code like this one:
[https://github.com/amark/gun/blob/master/lib/store.js#L16](https://github.com/amark/gun/blob/master/lib/store.js#L16)

~~~
marknadal
Thunderbong's answer is spot on (it replaces Firebase).

Here is a 5min interactive coding tutorial that shows that using GUN is more
powerful than reading about GUN. [https://gun.eco/docs/Todo-
Dapp](https://gun.eco/docs/Todo-Dapp)

For instance, Archive integrated in 1 week.

Notabug first version was built in 1 week with it. (p2p Reddit)

Organization can go a far ways in a short time.

------
ivanjaros
its all just a for loop anyway...you can do the same if you put tiny wrapper
on db-local server to avoid network and you'll get the same result.

------
Kuinox
IMO JavaScript is not compatible with "realtime".

~~~
ceejayoz
“Real-time” in JS tends to refer to approaches like websockets.

~~~
Kuinox
So it's not really realtime... No ?

I noticed an instant downvote. It's not the first time i see this behavior on
rxdb comments.

~~~
dsun180
The definition of realtime is vague so saying that something is "not really
realtime" just because it is not "realtime computing" is wrong.

~~~
discreteevent
In fairness it was a bit pretentious of whoever decided to call it real-time
which was a word with a specific meaning and a very hard earned reputation.
'Live' might have been better.

~~~
ceejayoz
"You see the updates in real time without having to refresh the page" seems
like a perfectly reasonable statement.

I submit that being pedantic about such things is frequently its own form of
pretension.

