
Collaborative Editing in ProseMirror - espadrine
http://marijnhaverbeke.nl/blog/collaborative-editing.html
======
josephg
Hi! Joseph Gentle here, author of ShareJS. You quoted me in your article.

You're right about OT - it gets crazy complicated if you implement it in a
distributed fashion. But implementing it in a centralized fashion is actually
not so bad. Its the perfect choice for google docs.

Here is my implementation of OT for plain text:
[https://github.com/ottypes/text](https://github.com/ottypes/text) Note that
its only 400 lines of javascript, with liberal comments. To actually use OT
code like that, you need to do a little bookkeeping. Its nowhere near as bad
as you suggest.

In this tiny source-only demo I do all the bookkeeping, and implement a
working network protocol on websockets. The result? I can sync an object
between the server & client all in about 150 lines of code total:
[https://github.com/josephg/appstate](https://github.com/josephg/appstate)

This code has no external dependancies except for the ot type itself
(ottypes/json0 in this case - although it could be changed to text or rich
text or something with barely any code change).

Nice work making a cool demo - but I think you're using the wrong algorithm ;)

~~~
marijn
Yes, but that implementation deals only with plain text. The complexity seems
to ramp up pretty quickly as you support more types of operations, and since
extendability is an important concern for my project, I decided to avoid OT.

~~~
jahewson
If you have insert, update, and delete then you can build any other operation
from those primitives as long as you have the ability to batch a composite
operation. So there's no need to extend the OT algorithm to support other
kinds of primitive operation. For nested structures, addressing via ids or
linear addresses has to be taken into account but that doesn't affect the OT
transforms, it's one layer above. So it's possible (though not easy) to have
an extensible OT system without exponential complexity.

Still, avoiding OT has led to some interesting new ideas and I look forward to
seeing how things progress.

------
elisee
Looks great. CodeMirror ([http://codemirror.net/](http://codemirror.net/)) is
an amazing code editor OSS project by the same author. It powers Firefox and
Chrome's devtools and is used in hundreds of other sites and apps (including
the one I'm working on :)).

Direct link to ProseMirror's home page with live collaborative demo at the
bottom: [http://prosemirror.net/](http://prosemirror.net/) (click "Change" to
select a room)

And here's a link to the just-launched campaign to open source it (MIT
license):
[https://www.indiegogo.com/projects/prosemirror/](https://www.indiegogo.com/projects/prosemirror/)
. It's an all-or-nothing ("fixed") type indiegogo campaign, not the flexible
kind where the funds are kept even if the goal isn't reached.

------
braythwayt
If you like ProseMirror, there's an Indiegogo campaign to support the project:

[https://www.indiegogo.com/projects/prosemirror/#/story](https://www.indiegogo.com/projects/prosemirror/#/story)

------
williamstein
SageMathCloud ([https://cloud.sagemath.com](https://cloud.sagemath.com)) is a
completely open source
([https://github.com/sagemathinc/smc/tree/rethinkdb](https://github.com/sagemathinc/smc/tree/rethinkdb))
collaborative code editing environment built on CodeMirror. I love CodeMirror
and massively appreciate marijn's work on the project, support, leadership,
etc. I implemented Neil Fraser's differential sync
([https://neil.fraser.name/writing/sync/](https://neil.fraser.name/writing/sync/))
algorithm in it (back in 2013), and have refined it little by little since.
The core sync code is
([https://github.com/sagemathinc/smc/blob/rethinkdb/salvus/dif...](https://github.com/sagemathinc/smc/blob/rethinkdb/salvus/diffsync.coffee)),
and is BSD licensed. All development of SageMathCloud is done in the open
([https://github.com/sagemathinc/smc/commits/rethinkdb](https://github.com/sagemathinc/smc/commits/rethinkdb)),
and it's very active right now. Our next big release, which should be within
the week, involves rewriting the backend to use RethinkDB (instead of
Cassandra), and the frontend to use React.js + Flux heavily (instead of
jQuery).

------
marijn
Just commenting to say I (author) am here this is a good place for feedback or
pointing out relevant references.

~~~
erisds
First of all, this looks truly awesome! I've spent a lot of time recently
looking at the available open source editors, and with the exception of
specifically 'code editors' there isn't much out there that supports markdown.

I have been reading about the recently announced content-kit:
[http://madhatted.com/2015/7/31/announcing-content-kit-and-
mo...](http://madhatted.com/2015/7/31/announcing-content-kit-and-mobiledoc)
and am wondering if you have considered the 'card' style concept for
ProseMirror? I see ProseMirror has an interface for adding images, and says
that it will support different document models in future and I'm wondering how
extensible that will be.

What sort of APIs are going to be available, will it be possible to create
custom 'blocks' or 'cards' of data - e.g. defining a block for adding a table
/ spreadsheet - similar to what you see in things like Quip or readme.io?

~~~
marijn
Basically, yes, the document object is designed to be extended by user code,
so you could add your own kinds of nodes (even allowing them to contain
existing node types, as in a table), but for some nodes (like tables) the
impact will probably be big enough to require some specific UI-related code in
the core.

------
marknadal
I spent 7 months of my life dedicated to solving this problem and let me say,
it is not easy because I failed. But I learned something really important, we
are all doing it wrong.

An index/offset based approach isn't the best solution, even with OT. Instead,
we should have been looking at this problem as a linked list, where every
character has a unique ID that can be pointed to. Doing operations this way is
safe and becomes idempotent.

I'm busy building a database to back this type of data, but will be
implementing a solution using this approach soon. Great work to everybody else
hitting this problem, keep it up!

------
sntran
So, curious about ProseMirror, I read this:
[http://marijnhaverbeke.nl/blog/prosemirror.html](http://marijnhaverbeke.nl/blog/prosemirror.html).

Sounds very interesting to me. This is an editor I have been looking for so
long. Google's Kix generates a lot of DOM elements. Medium's sounds good in
theory, but I haven't been able to try it due to the requirement of Twitter,
and its clones just mimic the interface, not the functionality (still using
contentEditable).

The closest one I saw was Aloha Editor v2. A little bit buggy and lack of
document.

But I realize I don't need collaborative editing. Hopefully this will be
separated into a component instead of built-in. Not all rich text editors need
collaboration.

~~~
marijn
Collaboration is a separate module, yes (though some of the design of the core
was of course guided by its requirements)

------
houshuang
Very interesting to see the approach, especially his critique of OT, which
virtually every other system seems to be based on. His alternative seems to
make sense, but I'm not sure if the end result for the user will be better, or
if it's mainly that the implementation is easier/less bug prone?

I've actually been playing around with "translating" the server backend for
share.js to Elixir, to be able to host it as part of a Phoenix app. Playing
with ways of integrating with erlnode to run the actual test cases (including
a fuzzer) from JS directly on my Elixir code.

If this gets opensourced, I wonder how complex the backend will be (I'm
assuming it will be written in Node?), given that it seems like the front-end
is doing most of the work.

~~~
marijn
A minimal backend can be extremely simple, just relaying changes, but if you
want it to keep a running snapshot of the current document, you'll need the
capacity to apply those changes to document, so you'd need to use the module
used by the client, or a port of that.

------
falcolas
Sounds like the same problems that have been encountered (and mostly solved)
in many different online videogames. Might be worth investigating how they
solved these problems as well, since their solutions will be much more
practical than theoretical.

[http://gafferongames.com/networking-for-game-
programmers/](http://gafferongames.com/networking-for-game-programmers/)

------
leeoniya
hi marijn,

did you end up using [https://github.com/markdown-it/markdown-
it](https://github.com/markdown-it/markdown-it) for the AST pass?

