
Automerge: JSON-like data structure for building collaborative apps - goblin89
https://github.com/automerge/automerge
======
skrebbel
Wow! I really, really like this because the goal appears to be to expose CRDTs
in a developer-friendly way. I truly believe that well-designed CRDTs are the
future for a _lot_ of distributed systems problems, but they've been a bit
hard to get into for people who just want to solve problems and not read 10
papers first.

The only thing I miss here is a clear spec of the CRDT itself so that people
could implement compatible versions in other languages.

~~~
biztos
Conflict-free Replicated Data Type[0], for anyone else who didn't know what
CRDT means.

[0]: [https://en.wikipedia.org/wiki/Conflict-
free_replicated_data_...](https://en.wikipedia.org/wiki/Conflict-
free_replicated_data_type)

~~~
marknadal
I've spent almost a decade working on CRDTs, largely from an industry
perspective but also some academic side.

They're great, and not too hard to understand if you take time to think about
them.

Here are some animated/interactive cartoon explainers we've made to help teach
the concepts:

\- How a CRDT version of Operational Transformation (Google Docs) works:
[http://gun.js.org/explainers/school/class.html](http://gun.js.org/explainers/school/class.html)

\- Why CRDTs are better than centralized alternatives like PAXOS/RAFT:
[http://gun.js.org/distributed/matters.html](http://gun.js.org/distributed/matters.html)

Martin Kleppmann's work (author of Automerge) is outstanding, especially check
out:
[https://youtu.be/yCcWpzY8dIA?t=29m36s](https://youtu.be/yCcWpzY8dIA?t=29m36s)

~~~
justinpombrio
I have a rather specific question, in case you know. What would be the best
CRDT algorithm for editing a document consisting of (i) nested lists, (ii)
strings, and (iii) some primitive values?

I'm looking at Kleppermann's JSON work. But it doesn't look like it handles
collaborative editing of strings very well: they have to either be treated as
an immutable value in a register, which doesn't allow for collaborative
editing of the string, or treated as a list of characters, which would have to
represent large edits (e.g. deleting a word) as a sequence of character edits,
which sounds inefficient. Kleppermann's work also handles maps, which I don't
need, which is fine.

For context, I plan to make a tree/structure editor, and am considering
representing the document as a CRDT.

~~~
rollcat
(not an expert)

Treating the long string like a list of substrings should work, I guess the
optimum substring length could be determined experimentally based on real-
world interaction patterns.

------
danbruc
CRDTs can provide an elegant solution to some problems but they are not
powerful enough to be used as a general tool. If you change the title of your
task card on two different devices, there is not much you can do to
automatically merge those changes besides using some simplistic policy like
last write wins which will not yield the desired result in general.

CRDTs require the operations on them to commute or equivalently that state
changes are monotonic. If your application fits those constraints you can
probably build a quite elegant solution but unfortunately that is the
exception rather than the norm, most operations are not commutative.

EDIT: This was meant to be a reply to [1] instead of a top-level comment.

[1]
[https://news.ycombinator.com/item?id=16310656](https://news.ycombinator.com/item?id=16310656)

~~~
pvh
In fact, one of the purposes of this project was to test exactly what it was
like to use a Trello-like tool implemented this way.

What happens is that humans have intuitive coordination systems and tend to
fix the same problems.

In other words, if the card title is "rename auotmerge" then both of us are
likely to change it the same way, or at least either resolution is fine. In
the event of a conflict we flag the multi-value in the UI so that a human user
can decide if they care.

Where this approach shines is in domains where it helps avoid conflicts rather
than resolving them. Naively adding replies to a comment thread might result
in collisions or misordered comments across nodes of a distributed team.
Automerge captures enough metadata from the operations created by users that
changes "tend" to merge well.

I realize this is an argument not from first principles but from experience.
We weren't able to find examples of other people building web applications
with quite this approach (though we would love to hear from them if they are
out there) and so the question was "will this feel alright?"

Apparently it does.

If you find this result provocative, I would encourage you to challenge it by
attempting to build something with automerge and letting us know how it goes.
I believe it will work well in applications where all user operations can
always commute to produce a subjectively reasonable result. To make that
concrete, a bank account is a poor domain to model with a tool like this, but
a shared drawing is much better.

~~~
danbruc
As you say, I am certainly working in the wrong domain for this and such an
solution would never be acceptable. You certainly don't want edits to things
like insurance claims, medical records, or financial transaction made in a way
that edits somewhat randomly disappear and hope that people will notice and
fix it.

This might work to some extend for a toy example like a simple task management
application but I would bet even then users would quickly get annoyed about
losing their edits. And you also can not really claim that your are not losing
edits because you keep them in a list of conflicts because for all practical
purposes they are gone unless you build a solution to explicitly deal with
them which renders trying to use CRDTs mood.

I think unless your operations map cleanly to CRDTs and you get the expected
and correct behavior all the time, trying to use CRDTs will just cause
additional pains and not provide any benefits. Trying to force general data
structures with general, non-commuting operations and CRDTs together just
seems ill-fated to me.

------
wenc
Worth noting is the fact that this does not require a central transformation
server unlike Google Docs' OT approach.

This means you can sync peer-to-peer without an Internet connection if say you
have a Bluetooth connection between your phone and your laptop on an airplane.

Martin Kleppmann explains CRDT in detail in an episode of Software Engineering
Daily.

[https://softwareengineeringdaily.com/2017/12/08/decentralize...](https://softwareengineeringdaily.com/2017/12/08/decentralized-
objects-with-martin-kleppman/)

------
k__
When I learned one thing about collaborative apps, it's that there is no
automerge that works for all use-cases.

Couchdb does it the right way, it simply keeps all versions and lets the
application logic decide which is the "one true state".

~~~
jimpick
Automerge is very similar.

If there are conflicts, it returns a _conflicts property on the object so you
can retrieve the other conflicting alternatives so the application logic can
write another change if it wants to use a different alternative than the one
that was automatically chosen.

~~~
aeorgnoieang
I'm confused. What's the point of `_conflicts` property if conflicts are
automatically merged?

The linked project README, and even the CRDT Wikipedia page, seem to claim
that conflicts are _avoided_ , whereas, at least in my mind, these are more
like _strategies_ to automatically privilege one version in the event of a
conflict.

~~~
zachallaun
From my reading of the docs, Automerge will arbitrarily pick a "winning"
document in the case of a conflict, but guarantees that the same winner will
be chosen if the merge were to occur, for instance, in a different order:

    
    
        // Pretend doc1 and doc2 are already Automerge objects
        let doc1 = { x: 1 }
        let doc2 = { x: 2 }
    
        // x will be either 1 or 2 (arbitrarily chosen), but
        // res1 will always == res2 regardless of choice
        let res1 = Automerge.merge(doc1, doc2)
        let res2 = Automerge.merge(doc2, doc1)
        
        res1 == res2 // true
    

However, there may be cases where you want to manually resolve conflicts, and
if that is the case, the `_conflicts` property is there so that you can undo
whatever merge occurred automatically and set the "winner" yourself.

~~~
heynk
If it's "arbitrarily chosen", but the result is always the same regardless of
order, what is the algorithm that decides?

~~~
zachallaun
There's different ways you could do it, like calculating some hash of the
value and picking the one with the greater value. If the hash function is
good, then you get an "arbitrary" result that would still be consistent
regardless of order.

------
avian
Several years ago I made jsonmerge, a Python library for merging standard JSON
structures. The rules for resolving conflicts are encoded in a standard JSON
schema document that is extended with some (non-standard) keywords.

[https://github.com/avian2/jsonmerge](https://github.com/avian2/jsonmerge)

[https://www.tablix.org/~avian/blog/articles/talks/tomaz_solc...](https://www.tablix.org/~avian/blog/articles/talks/tomaz_solc_jsonmerge.pdf)

------
whoisjuan
This is how you're supposed to write documentation. Excellent descriptions,
explanations and examples.

------
namuol

        To change the state, you can use the regular JavaScript array mutation
        methods such as push(). Internally, Automerge translates this mutable API
        call into an update of the immutable state object. Note that we must pass in
        doc1, and get back an updated object which we assign to the same variable
        doc1. The original document object is not modified.
    

Hmm. I like having the _option_ to use mutations to describe changes to the
document, but in many cases I would actually prefer to use pure functions. Can
I just return a new document with the necessary changes? I don't see any such
examples in the docs here.

------
bringtheaction
> If the state was concurrently changed on different devices, Automerge
> automatically merges the changes together cleanly, so that everybody ends up
> in the same state, and no changes are lost.

> Different from git: no merge conflicts to resolve!

This is impossible unless there are significant restrictions on what kind of
operations are possible.

If I have a bag of 15 apples, and I take 10 of those apples at the same time
as somebody else takes more than five apples then we have a merge conflict
right there because we can't end up with a negative amount of apples in the
bag.

~~~
amelius
Another problem is this. Let's say there is a counter that is at value 100. I
increment the counter. Simultaneously another user increments the counter, so
the stable value should be 102. However, a operation-agnostic "merging"
approach like described here can never catch that, and sets the final value to
101.

~~~
kba
My experience is with OT, so I can't speak for CRDT, but I can imagine it's
similar.

With OT, you send a na[1] (number add) operation, so this would work fine.
Indeed, if you treat the number as a string, then you have a problem.

[1]: [https://github.com/ottypes/json0#summary-of-
operations](https://github.com/ottypes/json0#summary-of-operations)

~~~
taeric
This doesn't compose with all operations. One users adds and another user
multiplies, for example. That is to say, not all operations are associative.

I bet you can run really far with this general idea. But there is no panacea.

~~~
kba
You're absolutely right, but I'd also never suggest a collaborative system for
anything critical. As I've said earlier in this thread: If you're in a Google
Docs document with a buddy and you write "A" and he writes "B" at the same
time, you also don't know whether that'll show up as "AB" or "BA". In
practice, this isn't an issue.

~~~
taeric
To underscore our point, concatenation on strings is not associative, either.

To that end, the basics of math in associative, cumulative, distributive,
etc., go together to create an algebra of what you can automate rather easily.
I think most of us stopped thinking in terms of those laws years ago. To the
point that it is probably odd to folks that stayed close to them.

~~~
gmfawcett
> To underscore our point, concatenation on strings is not associative,
> either.

I think you mean commutative. It certainly is associative.

~~~
taeric
Ha! Yes. I meant to draw attention to all of the fundamental laws people are
used to. Messed up in some edit.

Thanks!

------
avodonosov
The description contradicts itself:

> Different from git: no merge conflicts to resolve!

But later:

> The only case Automerge cannot handle automatically, because there is no
> well-defined resolution, is when users concurrently update the same property
> in the same object (or, similarly, the same index in the same list). In this
> case, Automerge arbitrarily picks one of the concurrently written values as
> the "winner":

...

> Although only one of the concurrently written values shows up in the object,
> the other values are not lost. They are merely relegated to a _conflicts
> object

~~~
kadenshep
Sooooooo

>git merge --strategy-option=theirs

------
trjordan
This is cool! I've been thinking about this a lot in a similar domain (JSON /
YAML for configuration), and I've come to the conclusion that anything
approaching manual merge is pretty much the worst thing ever.

The domain I'm specifically thinking of is managing routing rules, like nginx
configs or control-plane configs for Envoy. It's really tempting to simplify
it down to a config file, then you version-control it, then you get merge
conflicts, and all of a sudden a bad merge makes 10% of your services
unroutable. Which is hilariously bad.

Having an auto-merge for those sort of configs seems like the only reasonable
way to do it. Escalating conflicts up, from easy merge, to trying an automatic
strategy, to surfacing conflicts to the user in a domain-specific way, seems
like the only way to do it!

------
benjaminjackman
Does anyone have any experience and can offer a suggestion on javascript
libraries that will handle text editing (like google docs or etherpad) and
which can be dropped into a site and which aren't locked into a particular
service or backend?

I.e. basically something like this but for text?

~~~
houshuang
[ShareDB]([https://github.com/share/sharedb](https://github.com/share/sharedb))
is the most commonly used, we use it extensively - also has bindings to rich
text or code editors. [Y-js]([http://y-js.org/](http://y-js.org/)) is a more
academic project, also based on CRDTs I think.

------
wehriam
I've been working on Observed-Remove Map and Set implementations that
replicate the native Javascript API, as well as versions that use IPFS PubSub
for synchronization.

* [https://github.com/wehriam/observed-remove](https://github.com/wehriam/observed-remove)

* [https://github.com/wehriam/ipfs-observed-remove](https://github.com/wehriam/ipfs-observed-remove)

The inherent tradeoffs can be challenging to explain. For example in these
implementations, there are caching constraints on certain delete operations.

Automerge looks fantastic, and I look forward to more progress in the space.

------
billconan
How does this compared to differential synchronization used by google drive?
[https://neil.fraser.name/writing/sync/](https://neil.fraser.name/writing/sync/)

~~~
toomim
I love Differential Synchronization. Howevr, Google Drive doesn't use it.

More specifically, by Google Drive, I assume you mean Google Docs. And Google
Docs doesn't use diffsync, it uses OT.

~~~
billconan
What's OT?

why doesn't google docs use DS? Who uses DS?

What's the best choice of making something similar to google docs' co-edit
feature?

~~~
billconan
I found this page informative and interesting

[https://operational-
transformation.github.io/visualization.h...](https://operational-
transformation.github.io/visualization.html)

it has a live visualization of the OT algorithm

~~~
eindiran
The live visualization was a lot of fun to play around with.

------
maelito
So this is to paths (keys and indexes) in a JSON object what git is to lines
in a text file ?

Could git be configured with a different diff algorithm to have this
functionality, or is it too tied to lines ?

------
superasn
This would work great with Firebase too. I remember writing a collaborative
document editor similar to Google docs and the "document" was basically JSON
with nodes for headlines, paragraphs, etc that multiple users could edit in
real time.

That time I just used to write the whole JSON to firebase (with flock like
locking) and it was really bad but since it was just a weekend project I
didn't care much at that time. Bet a library like this could have saved me a
lot of headache.

------
nmca
This seems to be Martin Kleppman's work - worth noting that his book,
"Designing data intensive applications" is also very good :)

------
paulddraper
This looks great, depending on the nature of the "collaborative app".

E.g. perhaps the most popular collaborative app is Google Docs. This would be
a terrible choice for Google Docs, as it doesn't have specialized transforms
for text editing/formatting (which turn out to be rather challenging).

------
kuon
My experience with collaborative sync is that it is really application
specific. Most of the time, I used a dumb store (just storing all versions)
and had the app decide of the logic to produce the current state.

But, this library looks nice, and I'm sure it can fit some use cases.

------
jpsim
Very similar to Realm (also syncing CRDTs), though this seems more lightweight
and works in client-side web apps. [https://realm.io](https://realm.io)

------
zeandcode
Recently I created jkt library for my NodeJs project.
[https://github.com/slaveofcode/jkt](https://github.com/slaveofcode/jkt)

------
davidw
This is pretty interesting. I've seen similar problems at various places I've
worked, and having a library that uses a CRDT to solve it is a winning
proposition.

------
softwarelimits
Thank you very much for the publication, may I ask a few questions:

* How to integrate this with a decent pagination library?

* How to integrate this with vue.js?

* How does this compare to [http://y-js.org](http://y-js.org) ?

* How does this compare to [http://gun.js.org](http://gun.js.org) ?

Thanks for your attention!

------
molikto
I am planning to doing the same thing, but using OT.

~~~
molikto
My idea about conflicts is, since OT is a centralized protocol, the server can
refuse some conflicts and let user to resolve it.

Another thing I don't like about CRDT is for text editing, it requires a uuid
for each char you type. this is a huge waste of memory and disk imho. on the
other hand OT requires no metadata

------
1zael
Can't wait to try this.

