With CRDTs (Conflict-free Replicated Data Types) becoming more mature and getting more attention lately, I decided to dive in and use it as the basis for a new project. I love the technology, and how it enables me to create Google Docs style collaborative apps, that seamlessly sync data across users / devices but also work offline.
However, I also found that it's not yet trivial to build! This is why I started building a library that makes developing multiplayer apps super-simple, with a very minimum API surface. (basically, you can just modify plain javascript objects and changes are shared across users).
I'm now excited to share SyncedStore - and looking forward to your feedback :)
It's designed specifically for React, Vue, Svelte but also works with plain JS. By using Javascript Proxies, the data store looks like plain javascript objects, except that they'll now sync automatically across devices / users!
By the way despite that particular repo (@localfirst/state) last being touched 6 months ago, Herb Caudill definitely seems still active in this space (I believe he's been working on other parts of this more recently -- e.g. ideas about authentication), and I think automerge development itself (Martin Kleppmann) is quite active right now leading up to a 1.0 release which seems fairly imminent, for which a lot of fundamental work has been done, also coordinating with automerge-rs.
First of all, SyncedStore does not implement any CRDT algorithms. Credits for this go to Yjs [1] (and its author Kevin), which it uses as underlying CRDT.
Yjs and Automerge are (afaik) the two most commonly used CRDT implementations. Both have their pros and cons, but Yjs has focused a lot on performance [2].
Automerge has a bit friendlier "Immer style" [3] API. I'm not too familiar with @localfirst/state, but it seems to add a Redux style API on top of Automerge.
My approach with SyncedStore was really to provide an API on top of Yjs that's as simple as possible to use in React / Vue / Svelte or plain JS app. I.e.: only use a single React Hook to observe changes, and use regular Javascript assigments to update values. The API is inspired mostly by Reactive Programming libraries such as MobX [4] (from the same author as Immer).
How/where does the data get saved for later? I'm guessing from the example, you don't need a database? E.g. I can just save the data to a document database, and next time someone works on it, just sync with the database? Or do I not need it at all because it's stored on the browser? And how would the information be associated with accounts, if someone wanted that feature?
Also curious how/where the demo site is deployed (guessing Vercel doesn't work since they don't seem to support websockets? I see that Hocuspocus is mentioned — is that required as a backend to make it all work?
Sorry I'm new to all of this and it sounds like magic, but I'd LOVE to learn how to implement this; especially on something like Vercel, if possible
Creator of Hocuspocus [1] (which I’ll open source next year) here. You can use y-websocket [2] aswell. Hocuspocus is a (very nice) wrapper/API around y-websocket written in TypeScript.
You can indeed store updates in a centralized database (wrapped with an API to handle auth / accounts) - this is probably the most common scenario.
But you can also go fully decentralized and use y-indexeddb to store updates in in the browser, and use a different transport to sync updates between users (e.g. webrtc).
—-
The cool thing is that the data model behind CRDTs is very flexible in this regard. All changes are captured in small “updates”, and as long as these updates at some point arrive at your device, your document can be updated with the changes (it’s eventually consistent).
The site runs on github pages without a backend (examples use y-webrtc).
Pretty interesting project. I'm curious what the data structure looks like that is sent over the network for diffs. I found the part about state-vector, but not about the actual data.
Does it diff the JSON string, or the objects/array themselves? Does it serialize the data or leave as plain text?
I ask, because I have a similar setup, that needs to sync dynamic data structures, but also keep the data deltas as small as possible when sharing them as JSON strings to save bandwidth.
It doesn't do any diffing, but keeps track of operations underneath, and syncs those. The gist is something like this:
- if you push a string "hello" to an array, it will sync an update [push "hello" to Array x] along with some metadata
- if you modify a property "prop" and set it to "value" it will sync an update [set "prop" to "value"] along with some metadata
It will not automatically diff these with the previous value (in a distributed system it's difficult to say what the "previous value" was, but it will sync only those properties that have been changed). For text heavy operations, you can use the Text structure which will sync fine-grained operations ([delete 4 characters at position 2] or [insert "world" at position 5]). This makes it suitable for collaborating on rich text documents (similar to Google Docs), see: https://syncedstore.org/docs/advanced/richtext.
Yjs author here. On network failure, the client will automatically reconnect and synchronize with the backend. And yes, you can get an event that notifies you about the current connection status. When syncing with the server, you will only exchange the differences that were created while offline.
Love this! Thanks for sharing, I have some amazing use cases for this type of work and I can't wait to dig into the code to learn more. The future for CRDT is indeed hot right now and everyone wants it into their app, Google paved the way and now other big names, like Figma, Invision etc are getting on this hard!
Thanks! As it builds on Yjs (many kudos to Kevin Jahns for creating that awesome project + community), any backend that's compatible with Yjs works.
The demos work based on WebRTC / BroadcastChannel (for the same browser / device), but I've listed different backends you can use here: https://syncedstore.org/docs/sync-providers
Hi! I just saw this reply — could you please put together a quick write up of an example deployment and the steps you took to do that (e.g. for the documentation site), including all the services you use?
Yjs author here. Yjs ships a flexible selective undo-redo manager. You can define one (or several) undo managers that listen to changes that you want to be able to undo&redo.
With CRDTs (Conflict-free Replicated Data Types) becoming more mature and getting more attention lately, I decided to dive in and use it as the basis for a new project. I love the technology, and how it enables me to create Google Docs style collaborative apps, that seamlessly sync data across users / devices but also work offline.
However, I also found that it's not yet trivial to build! This is why I started building a library that makes developing multiplayer apps super-simple, with a very minimum API surface. (basically, you can just modify plain javascript objects and changes are shared across users).
I'm now excited to share SyncedStore - and looking forward to your feedback :)
It's designed specifically for React, Vue, Svelte but also works with plain JS. By using Javascript Proxies, the data store looks like plain javascript objects, except that they'll now sync automatically across devices / users!