
How we built our startup (it's all about balance) - k7d
https://whimsical.co/tech/how-we-built-whimsical/
======
morgancmartin
Great write up, thanks!

I hadn't heard of Terraform before, so thanks for the introduction. I have
recently been attempting to familiarize myself better with systems
infrastructure, and it looks interesting. I'll have to do some reading on it.

I have a question. You mentioned that Whimsical's back-end, "mostly serves as
a lightweight layer between the client and the data store." Does this mean
that you do most of the data munging client side so that once it arrives
server side, most of the work is already done? And if so, what do the trade-
offs look like in such a scenario? If I'm understanding correctly, this means
the client is fatter, but there's less load on the server, meaning cheaper
infrastructure costs. Is this a correct way of looking at it?

I'm just a junior dev trying to gain a better understanding of large systems,
so forgive me for any ignorance or misunderstanding.

~~~
k7d
Indeed, most of the application logic is in client side. In fact it's built in
a way that it could function as a standalone application without backend.
Currently it doesn't persist the data locally but that could be easily added.

The backend serves as a fairly generic datastore with an API that allows data
to be easily synced back and forth. It implements a logic (again non-specific
to Whimsical) for how to handle concurrent updates (basically it implements
CRDTs). I believe it's pretty similar to Google's Cloud Firestore - we might
have chosen that if it was out of beta and we didn't mind the lock-in.

Two main reasons we did this: 1) to be able to easily add offline mode - we
don't have it yet but it's fairly easy to add now 2) to minimize effect of
network latency on user experience

As for less load on server & cheaper infrastructure costs - you are right,
that is another benefit.

------
jwr
I would be very interested to learn how you manage changefeeds, e.g.
distributing updates to client apps. I used RethinkDB for this, as every other
solution seemed like lots of extra work.

~~~
k7d
We use combination of Redis pub/sub and Websockets. Each document has a unique
key and whenever any of the users/clients make a change to it, it's published
via that key. Other clients connect via WebSockets to backend and subscribe to
the same key in Redis. The notifications are fairly simple.

The complex part is how we sync actual data from clients to server while
ensuring consistency. We basically store all data in a tree format and having
a diffing logic how concurrent changes can be applied conflict-free in most
cases. I'm considering writing a separate post on that.

------
iorekz
Great post, thanks ! What is "red" above "re-frame" ?

~~~
k7d
It's an extension to re-frame that we built. It addresses some of the
(subjective) shortcomings of re-frame, mainly the over-reliance on keywords.
We'll be open-sourcing it soon.

