

The Basis story: an accidental sync service in Clojure for iOS devices - mphillips
http://sapient-pair.com/blog/2014/04/21/basis-story/

======
bsaul
Seems to me like the merge part of the algorithm is a bit overkill :

Generate a uuid for each item on your list upon creation ( on the client
side), then sync the actions on the item ( create, update, delete). There's
never the need to "merge" on the server. All you need is to create a serial
"history" of all the actions on the server side, and replay them in that order
on clients. You do that by locking on the DB when a client wants to send new
actions.

PS : i'm currently coding exactly that. So it's more a way for me to share my
ideas rather than criticize.

~~~
mphillips
I'd be really interested to hear more about your approach. Maybe you’d write
it up?

I did consider simply storing and replaying actions, but realised I'd still
want a way to merge them: consider a series of actions like: "add item",
"update item", "add item", ...<lots more changes>..., "delete items 1 through
20". If there’s no merge, a lot of redundant actions will end up building up
in the offline case.

There are a lot of ways to approach this (I’m reading Brent Simmon’s
adventures
([http://inessential.com/2014/05/24/vesper_sync_diary_follow-u...](http://inessential.com/2014/05/24/vesper_sync_diary_follow-
up)) with interest at the moment).

~~~
bsaul
Yes i realized that it may be a bit long if one needs to sync everything from
the start. You may have to build a "compacted" history where all the items
that will later be removed are excluded. The thing is it's more of a
compression of serial actions than a merge between forked branches ( which
makes things easier).

------
notjosh
> And in fact Couch DB would have been close to perfect … if I could have run
> a Couch DB server on the client, which would is an insane thing to
> contemplate.

Couchbase Lite
([http://developer.couchbase.com/mobile/develop/guides/couchba...](http://developer.couchbase.com/mobile/develop/guides/couchbase-
lite/index.html)) does exactly this - except it's not an actual Couch
instance, but rather an SQLite database, and a client that speaks the
necessary Couch protocols for sync/replication. Not so insane, and (as far as
I've seen so far..besides git!) the only tool that really handles sync well
across platforms.

edit: Ah, CBL (and Datomic) are mentioned in part 3 of the article. All is
well :)

------
habosa
This is a great blog series, I'm really enjoying the read. Did you ever
consider using an implementation of the Operational Transform (OT) algorithm?
Google famously used this for Google Docs, which asynchronously merges a lot
of small changes from many clients into a single, real-time document. Sounds a
lot like what you built.

~~~
mphillips
OT is not so much an algorithm as an approach: a way of thinking about
sensibly merging parallel sequences of change operations against a data model
of some kind.

It’s a really interesting field, going back a long way: the canonical paper
that the Google Wave (RIP) people referenced is nearly 20 years old
([http://dl.acm.org/citation.cfm?doid=215585.215706](http://dl.acm.org/citation.cfm?doid=215585.215706)).
This is another, more recent, good paper: [http://hal.archives-
ouvertes.fr/docs/00/13/97/04/PDF/OsterIC...](http://hal.archives-
ouvertes.fr/docs/00/13/97/04/PDF/OsterICEIS07.pdf).

In fact, Basis does use an OT approach in the delta merge algorithm: it breaks
the deltas down into a canonical series of “micro operations”, merges them
piecewise, then generates a new delta.

Thanks for the kudos!

------
jeffmax
Just curious, is the basis client/server available going to be made available?
Thanks for the post.

~~~
mphillips
In the short term, no. I’d like to think that the (somewhat unplanned)
overhead of developing Basis will be paid off in future as a commercial
advantage for my apps — I guess we’ll see whether that pans out ;)

In the longer term, yes absolutely I’d like to make it open source. In the
meantime, I’m happy to share some of the lessons I’ve learnt along the way.

------
orandolabs
EnduroSync ([https://orandolabs.com](https://orandolabs.com)) is a new service
for syncing object stores. It has clients for iOS and Android now, with
several others right around the corner.

Data is modeled with objects. The object store works offline and online. If
you sign up for the service, then you get syncing.

EnduroSync also has a very nice permission model, enabling sharing of object
stores in a variety of ways (per user, per app, ...).

