
CRDT: Conflict-free replicated data type - tosh
https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type
======
canadaduane
Several interesting open source projects are working with CRDTs to make state
synchronization in distributed systems an easier problem to deal with:

    
    
      - Braid HTTP (https://braid.news/)
      - Automerge (https://github.com/automerge/automerge)
      - Gun (https://gun.eco/)
      - Yjs (http://y-js.org/)
      - Noms (https://github.com/attic-labs/noms)
      - DAT (https://dat.foundation/)
    

Personally, I'm most excited for Braid's effort to bring state sync to HTTP
through the IETF process, as well as Automerge's progress in P2P via
Hypermerge (and it's star app, Pushpin). I'd also like to see Gun succeed, but
have had a hard time getting started due to visually distracting quirks in its
documentation.

~~~
lucidstack
Horde [0] is a really interesting Elixir library for distributed, self-healing
state using CRDT

[0]:
[https://github.com/derekkraan/horde](https://github.com/derekkraan/horde)

~~~
qohen
Horde is indeed a really interesting Elixir library. It's based on the CRDT
library, DeltaCrdt [0], also by Derek Kraan. He discusses DeltaCrdt here [1]
and Horde here [2].

BTW, Horde is a distributed supervisor, i.e. analogous to what regular Elixir
OTP supervisors do, except that it can restart processes on different cluster
nodes. It also provides a distributed process registry.

Daniel Azuma of Google gave a great 39 minute talk at ElixirConf 2018, "Docker
and OTP: Friends or Foes?" [3], where he shows Horde (and DeltaCrdt) in action
to keep a multi-player tank video game going while the Docker container it's
running in gets killed and and the Elixir processes get respawned on another
container. The state is persisted across Elixir nodes in other containers in
an instance of DeltaCrdt.

(Again, the talk is really good -- if the above sounds interesting, you'll
want to check it out [3]).

[0]
[https://github.com/derekkraan/delta_crdt_ex](https://github.com/derekkraan/delta_crdt_ex)

[1]
[https://moosecode.nl/blog/how_deltacrdt_can_help_write_distr...](https://moosecode.nl/blog/how_deltacrdt_can_help_write_distributed_elixir_applications)

[2]
[https://moosecode.nl/blog/introducing_horde](https://moosecode.nl/blog/introducing_horde)

[3]
[https://www.youtube.com/watch?v=nLApFANtkHs](https://www.youtube.com/watch?v=nLApFANtkHs)

------
agazso
For anyone more deeply interested in this topic I recommend to read this blog
post from Archagon. It describes the different alternatives (OT, CmRDT, CvRDT,
diff sync) for writing a collaborative editor. And unlike academic papers it
is written in a format how a programmer does research and thinks about a
problem in real life, so it's very natural to follow, even if it's long and
complex.

[http://archagon.net/blog/2018/03/24/data-laced-with-
history/](http://archagon.net/blog/2018/03/24/data-laced-with-history/)

(I am not affiliated in any way, just enjoyed it very much)

~~~
dboreham
Alexei's work on this subject is very good: the best practical introduction to
the subject.

However, I feel that it is worthwhile (necessary?) to completely understand
the academic basis for CRDT. A good place to start is Shapiro et al's second
paper :
[https://hal.inria.fr/inria-00555588/document](https://hal.inria.fr/inria-00555588/document)
.

In order (sic) to understand the paper you need to have a grasp of Order
Theory. This is not terribly hard to get your head around. This is a good
place to start: [http://jtfmumm.com/blog/2015/11/17/crdt-
primer-1-defanging-o...](http://jtfmumm.com/blog/2015/11/17/crdt-
primer-1-defanging-order-theory/) also
[https://www.wikiwand.com/en/Order_theory](https://www.wikiwand.com/en/Order_theory)
and basically stop when you understand this :
[https://www.wikiwand.com/en/Lattice_(order)](https://www.wikiwand.com/en/Lattice_\(order\))

The reason why the formal basis is important is: that's the whole point of
CRDT -- previously (been there, got the t-shirt..) folks just made up
replication mechanisms they thought would work. Then they build them and
embarked on a process of fixing the bugs. Sometimes that took decades. CRDT is
nothing new in terms of : software to perform eventually consistent
replication. The new thing is that there's a way to formally prove that your
bright idea for replication will in fact work (as in : it will converge and it
will have the consistency properties you expect). So if you're not seeing that
aspect, then you're really not with the program.

btw originally the C stood for Commutative or Convergent, not Conflict-Free.
There are plenty of CRDTs that cope with conflicts consistently, rather than
being conflict free (e.g. LWW Register).

~~~
aboodman
FWIW, I disagree that it is necessary (or even desirable) to understand the
academic basis first.

The core idea of CRDTs are intuitive and easy to understand from an
engineering perspective. For me, the academic literature unnecessarily
complicates and obscures what's going on, and I would also point newcomers to
something like [http://archagon.net/blog/2018/03/24/data-laced-with-
history/](http://archagon.net/blog/2018/03/24/data-laced-with-history/).

It probably has to do with where you are coming from. If you're not a
mathematician or theoretical computer scientist, I don't think that reading
these documents is going to be a very helpful start.

~~~
dboreham
I hear you because I thought the same way until recently. I have worked with
eventually consistent replication since long before the CRDT papers and
thought initially it was just some academic mumbo jumbo layered over
intuition. However, based on my journey through the field I'd encourage you to
try again to grokk the academic side of the work because it's really the key
insight and (for me) it's very much not intuitive. In fact that's the tripping
hazard in all this: the simple examples are intuitive which leads to a sense
that everything is similarly intuitive. Not so, for me at least.

I'm neither Mathematician nor Computer Scientist (I'm an Electrical Engineer),
but I have learned not to fear formal notation and advanced abstract concepts.
They're really not that hard to understand once the terminology is decoded.

------
sbazerque
An interesting problem is how much interface mismatch you get when you try to
solve real-life problems using CRDTs.

Awhile back I implemented the "become a connection" logic for a project that
has purely CRDTs datatypes, and I kind of went back and forth between
composing well known CRDTs to model all the possible state, or just
implementing the solution as a finite state machine whose CRDT-ability had to
be reasoned for this particular purpose. Ending doing the second.

Got the intuition that this is a very nice way to think about distributed
software but the right abstractions may not be in place just yet.

~~~
zzzcpan
Could you elaborate on "become a connection" logic? What kind of problem is
that? You may need to change user interface a bit to fit into CRDTs.

~~~
dboreham
Like a LinkedIn connection, I assume.

~~~
sbazerque
Yes, it was for a chat proof-of-concept app. But the end result ended up being
so convoluted I thought no sane person would program like that, so I went back
to the drawing board.

Look at _init() here to get a feel of it:

[https://github.com/hyperhyperspace/hyperhyperspace-
web/blob/...](https://github.com/hyperhyperspace/hyperhyperspace-
web/blob/master/src/services/people/contacts.js)

~~~
dboreham
Your code didn't look so bad to me :)

I think you may have met some of my colleagues working on similar problems at
Dweb camp recently.

------
mbrock
I'm under the impression that CRDTs are somewhat limited by the lack of a
clear way to combine several structures along with application-specific logic.

In the 2012 paper "Logic and Lattices for Distributed Programming" (by Conway,
Marczak, Alvaro, Hellerstein, and Maier) they write:

> CvRDTs present two main problems: (a) the programmer bears responsibility
> for ensuring lattice properties for their methods (commutativity,
> associativity, idempotence), and (b) CvRDTs only provide guarantees for
> individual values, not for application logic in general.

They give this example of the second point:

> A replicated, fault-tolerant courseware application assigns students into
> study teams. It uses two set CvRDTs: one for Students, another for Teams.
> The application reads a version of Students and inserts the derived element
> <Alice, Bob> into Teams. Concurrently, Bob is removed from Students by
> another application replica. The use of CvRDTs ensures that all replicas
> will eventually agree that Bob is absent from Students, but this is not
> enough: application-level state is inconsistent unless the derived values in
> Teams are updated consistently to reflect Bob's removal. This is outside the
> scope of CvRDT guarantees.

They continue:

> Taken together, the problems with [CvRDTs] present a _scope dilemma_ : a
> small module (e.g. a set) makes lattice properties easy to inspect and test,
> but provides only simple semantic guarantees. Large CvRDTs (e.g., an
> eventually consistent shopping cart) provide higher-level application
> guarantees but require the programmer to ensure lattice properties hold for
> a complex module, resulting in software that is difficult to test, maintain,
> and trust.

This leads to a research project about monotonic logic as a distributed
programming paradigm.

[http://bloom-lang.net](http://bloom-lang.net)

[http://boom.cs.berkeley.edu](http://boom.cs.berkeley.edu)

------
arberavdullahu
I found it useful the IPFS research repository in Github [1], there are a lot
of resources there including background concepts.

[1] [https://github.com/ipfs/research-CRDT](https://github.com/ipfs/research-
CRDT)

------
phamilton
I find that we use CRDT's quite often in the form of append-only sets
(formally a G-Set). Rather than choose the CRDT specifically, we've seen that
an append-only set was the natural choice for a particular need and then we
identified the commutative and idempotent properties of the operation. Once
identified, we leverage them for scale.

One good example is processing items in an SQS queue, where (without enabling
FIFO) there is no guarantee of ordering. With a G-Set you can read and merge
batches of items (and retry naively) without affecting the outcome.

------
ericand
I've been researching these recently. Some things I've found: \- Macrometa is
a new startup and stateful edge (distributed database) service. I just tried
them this morning. User experience is very raw but it works great. You can
shim it in front of existing DBs (Dynamo, Firebase, etc) for better
performance. \- Gun: where macrometa runs on edge servers, Gun runs on the
client. Mark is a fantastic community and they are doing interesting things. I
had trouble implementing it but that's because I'm lousy at javascript. \-
Redis uses CRDTs or something similar: [https://redislabs.com/redis-
enterprise/technology/active-act...](https://redislabs.com/redis-
enterprise/technology/active-active-geo-distribution/) \- This presentation is
fantastic: [https://www.infoq.com/presentations/crdt-distributed-
consist...](https://www.infoq.com/presentations/crdt-distributed-consistency/)

------
bratsche
I wasn't familiar with CRDTs until Chris McCord started talking about them wrt
Phoenix Channels. Really cool and interesting stuff.

------
robto
I've been working my way through 'A comprehensive study of Convergent and
Commutative Replicated Data Types'[0] and I've been finding them very
approachable. I'm trying to implement them in Clojure as I go and that has
been interesting. I've also starting looking at 'Mergeable persistent data
structures'[1], and that one already has a repo with the code in Ocaml, but I
have to admit that I don't understand that queue structure yet.

It does seem like a good way to dodge some of the tough problems that come up
in distributed systems.

[0][https://hal.inria.fr/file/index/docid/555588/filename/techre...](https://hal.inria.fr/file/index/docid/555588/filename/techreport.pdf)
[1][http://gazagnaire.org/pub/FGM15.pdf](http://gazagnaire.org/pub/FGM15.pdf)

------
Karrot_Kream
CRDTs are very interesting, powerful, and simple datatypes. I've experimented
with a Go CRDT DAG, and it was simple to implement. The much more difficult
part with a CRDT is message routing. Even with routing, managing network
membership is a tough problem. There are multiple approaches, like Gun's
superpeers, or Gossip protocols, but this remains a very challenging aspect of
using CRDTs.

------
clarkmoody
I recently went down this rabbit hole and sincerely hope there will be some
sort of return to local-first software development, with an eye to
collaboration. CRDTs post a unique opportunity to develop software that
empowers users with their own data, yet retains the ability to collaborate and
store it on a server.

------
shepardrtc
I've been working with a forked version of ShareJS for a while and its worked
well over the years, so OT is certainly a viable option for anyone interested
in collaborative work. I would recommend anyone use ShareDB rather than
ShareJS, though. The former is a cleaner, more modern version of the latter.

------
dustingetz
Has anyone made a good sequential log CRDT yet? (It's a hard problem)

~~~
dboreham
What's your definition of sequential in this context?

------
jively
These are fascinating, I built a toy replicated K/V in go using CRDTs:

[https://github.com/lonelycode/yzma](https://github.com/lonelycode/yzma)

------
bob1029
I am trying to understand how CRDT compares to Lamport timestamps or vector
clocks. Couldn't you achieve most of the same objectives using these
abstractions as well?

~~~
trobertson
Directly discussed here:

[http://archagon.net/blog/2018/03/24/data-laced-with-
history/...](http://archagon.net/blog/2018/03/24/data-laced-with-
history/#convergence-techniques-a-high-level-overview)

------
mettamage
As a coding challenge I had to implement one based on the Wikipedia article.

I challenge you to do the same!

~~~
kgraves
Bonus points if you can complete the challenge under an hour with tests!

------
alexandernst
This was already posted here some months ago.

[https://www.google.com/search?q=hacker+news+crdt](https://www.google.com/search?q=hacker+news+crdt)

~~~
bascule
Call me crazy but I think CRDTs are interesting enough to deserve a repost

~~~
deepsun
I dunno, I heard and read so many times about CRDT that I wouldn't say it's
nerdy interesting to just posting wikipedia page here.

Something more, some failure stories like [1] would be interesting.

[1]
[https://news.ycombinator.com/item?id=19886883](https://news.ycombinator.com/item?id=19886883)

