The "instantly syncs updates with all users" bit caught my eye. I always wonder how people go about implementing that.
Asana works that way, and they have their own framework named Luna that does some functional reactive magic. Meteor, which I've been working with a lot lately and have really enjoyed using, was also created by some ex-Asana people, and enables the same type of real-time synced updates.
I was just wondering if you could share anything about your technology stack or how you accomplish the real-time updates. Are you using Meteor?
First off, we're huge fans of both Meteor and Asana. I spoke with Geoff @ Meteor a couple years ago when we were first starting to build out the Airtable product and was very impressed by their approach and vision.
We've closely followed the developer blogs of both those projects (and in Meteor's case, their source code). With those learnings, we built our own realtime database engine that supports relational data (which Meteor doesn't yet support) and also some other major features like the ability to undo any user action out of order (like git revert), which is necessary to support undo in a multi-user context (because the last thing that you did may not be the last change globally if other people are concurrently making changes). Undo is a particularly challenging feature to implement in a structured relational database context, because it can't be reduced to a set of simplistic character insertion operations as is the case for a google word doc, or a spreadsheet (which is a simple 2d array of values without type constraints, foreign key relations, etc).
Very impressive, one more congratulation from me. And a couple quick questions:
How do you handle conflicts? E.g. If two users change the same scalar value (of an integer cell, let's say) at the same time. Do you accept one and throw a warning back to the unlucky users?
How much to you cache locally? I had begun a framework such as this at some point, with the ultimate goal of being usable offline via the localStorage cached version (and sync upon connect), so I have a good idea of how much pain you must have gone through sorting this out :)
You wrote your own real-time database engine. You're nuts! (in a very good way). Have you written anything about this? It would be interesting to hear both about the database and the experience of "rolling your own" at such a low level.
Hi, yes--we definitely want to get a dev blog up and start talking about some of this stuff!
Re: conflicts, in the case of a scalar value, last in wins, and the system (if you consider the server and each client as nodes in a distributed system) is eventually consistent (i.e. everyone will eventually see the exact same state). In that particular case, we show the other user's profile picture and highlight the cell to flag that change so you realize somebody else is editing at the same time as you.
Nice. There is probably an argument to be made for a notification when you update something and someone else also updates it and your update loses (i.e. small time difference updates or conflicts + your client wasn't last)
Although to be fair, I suppose "undo" covers this. Can you undo an action taken by someone else? (User A commits "aa", User B commits "bb", User A commits "yy"; Can User A, undo to both "bb" and "aa"?)
>Undo is a particularly challenging feature to implement in a structured relational database context, because it can't be reduced to a set of simplistic character insertion operations as is the case for a google word doc, or a spreadsheet.
I personally hit this roadblock while trying to build a revision management system for a spreadsheet like Google Drive has. I had a pretty fragile implementation that works within a tangled mess of has-many relations. It was a tough time generalizing the whole mess. I'm looking forward to your insights on this particular problem on your future developer blog! Congratulations.
You couldn't be more wrong. Yes, websockets would make this easier and more seamless than (say) polling, but the "big deal" is that your entire model changes from:
- There's a source of truth (server) and a single client at a time, ever.
To:
- There can be up to hundreds of clients at the same time, all having write access to the same data.
Meaning:
- Your old fashioned models don't quite cut it - you need something to handle changes happening locally [having to be propagated to the server/network ("committed") and potentially be rejected or changed], and at the same time receive remote updates, apply them to the local objects.
- You're going to have conflicting writes. You don't want to have global locking in such a system, no matter how fine-grained (cell, in this case, I suppose); enter: Merge conflict resolution code (you'll have fun with that). Does it happen on the server? Client? How are conflicting writes resolved? Can the client handle just about any write having to be cancelled, and instead accept another change to the object?
- Saving history states for all (ok, most) of your data, so that things can be rolled back. Kind of necessary when you can have a spreadsheet shared by 100 people, and any of them could delete the entire thing.
Do these features ring a bell? Yes, we're talking about a distributed version control system here, but "you just saying" that all it is, is bidirectional communication? That's why you're being downvoted.
Asana works that way, and they have their own framework named Luna that does some functional reactive magic. Meteor, which I've been working with a lot lately and have really enjoyed using, was also created by some ex-Asana people, and enables the same type of real-time synced updates.
I was just wondering if you could share anything about your technology stack or how you accomplish the real-time updates. Are you using Meteor?