
Show HN: Working code for Operational Transformation/CRDT hybrid - raphlinus
https://medium.com/@raphlinus/working-code-for-operational-transformation-crdt-hybrid-9d04a57309da
======
raphlinus
Author of the posted article here. For the last year or so, I've been doing a
deep dive into operational transformation, trying to figure out why it's so
hard, and whether CRDT is a better approach to collaborative text editing.

The result of this work is a hybrid technique which I think combines the best
features of Operational Transformation (especially the RAM-efficiency) and
CRDT (ability to work in arbitrary network topologies, not requiring a central
server). It also turned out to be appealingly simple, about 400 lines of
JavaScript. That's with a core data structure implemented as a balanced binary
tree, to make the merge algorithm O(n log n), as opposed to O(n^2) for the
best operational transform algorithm I know about (n is a measure of the
number of edits in flight concurrently).

For anyone spinning up a new collaborative editor, I believe these techniques
are a significant advance. Of course, if people disagree, I'd be more than
happy to discuss.

~~~
pjc50
(You should point out that you're the author of the article! :)

~~~
raphlinus
Good point, done. Otherwise, it would look like a pretty ham-handed attempt at
sock-puppetry :)

------
tarr11
I wish someone should offer a "firebase for OT" SAAS that offers an API, SDKs
for JS/iOS/Android and a few working samples.

Every time I want a collaborative OT like environment for apps that I build, I
end up:

1- implementing some sort of janky locking scheme, or

2- having to learn how to install and host a tightly coupled server and client
environment, or

3 - Abusing some other service (like firebase or couchdb) that has pseudo-OT
semantics and getting close enough

I know there a bunch of people working on this, but I've never seen it offered
as a service. If one of you do, please add my to your list of potential beta
testers.

~~~
jinjin2
Realm is pretty close (it is not offered as-a-service, but they have a free
version you can run yourself).

------
mochidusk
I would love a good in-depth tutorial on operational transformation but so far
haven't found one that is satisfactory (for a complete beginner). Maybe I've
been spoiled with all the great javascript/react tutorials out there that I'm
hoping someone could do the same for OT. A tutorial or series about OT with
detailed instructions from concept to implementation is something I would pay
for to learn.

~~~
raphlinus
I understand where you're coming from. Unfortunately, the academic literature
on OT is really difficult. A lot of the papers are flat-out wrong, and even
the ones that aren't have tons of confusing notation and terminology. It's a
specialized area, so I don't have hopes there will be an easy-to-follow
tutorial any time soon.

~~~
sounds
The linked article here is surprisingly accessible!

I would add just one small tidbit of advice that might help interested people
get up to speed: skim the paper, then look at the implementation.

Being able to work with actual code can make a lot of things easier.

(This article comes with working code! Try it out!)

------
marknadal
@raphlinus, fantastic work! I'd love to connect because I am also one of those
people who has been working on CRDT/OT algorithms for a couple years now. I
wound up building a custom database because of it, haha.

My approach (not finished) is to use linked lists and tombstones. I found
index transformations to not be provably correct in an arbitrary P2P system
because it always required a document root.

Here is a (terrible terrible) demo of the linked list approach:
[https://www.youtube.com/watch?v=rci89p0o2wQ](https://www.youtube.com/watch?v=rci89p0o2wQ)
.

What are your thoughts on your approach handling rich text? (document nesting)

Would love to chat more, mark [AT] gundb [DOT] io

~~~
raphlinus
Yes, let's connect offline.

I have thought some about rich text, but haven't implemented anything yet. My
general thinking is that attribute spans lend themselves well to the OT model,
but trees do not. The classic example is if you have three words, user A bolds
the first two, and user B italicizes the last two. With spans, they just
partially overlap, which is fine, but with trees you have an ambiguity.

Representing certain kinds of document structure (like nested bullet lists) is
not obvious with spans. I want to explore the combination of a marker
representing "beginning of tree" and a span, and define transformations in and
out of tree form that would be robust to concurrent edits. However, this is
all just ideas at this point.

~~~
amk_
Have you looked at the Quill delta format?

[https://quilljs.com/guides/designing-the-delta-
format/](https://quilljs.com/guides/designing-the-delta-format/)

~~~
raphlinus
I've heard of Quill but hadn't seen that document. Looks like interesting
reading, thanks!

