Hacker News new | past | comments | ask | show | jobs | submit login
The data model behind Notion's flexibility (notion.so)
338 points by pspeter3 77 days ago | hide | past | favorite | 120 comments



Not sure how "unique" this model is; for example Claris Works was built out of an even more powerful block model (they called them frames) back in the late 1980s:

> We came up with a frame-based approach. Most of the functionality particular to the various application types was packaged up into "frames": word processing frames, graphics frames, etc. These frames were then used as building blocks to make documents of the appropriate types, in a unified programming framework. E.g., a word processing document was essentially a bunch of text frames, one per page, linked together. (Doing this neatly was a big challenge - many subsequent efforts at building a component-based architecture (e.g. OpenDoc) have failed to take into account the top-level user interface requirements.) The result was that not only was most of the code shared across the document types, but the application was also truly integrated - the frames could be embedded in each other. E.g., you could plop a spreadsheet frame right into your word processing document. Text objects in a graphics document had a full-featured word processing engine behind them. The database form editor used the built-in graphics environment. Etc.

> One related cool thing we had was a "shared graphical context" mechanism: sometimes, stuff would wind up being displayed in multiple frames at once. E.g., maybe you're dragging an object across a page break in a document with multiple pages (like this). We developed a general architecture for displaying actions live in multiple contexts. Of course, a lot of this kind of stuff is old hat today, but it was new and exciting in 1989. Some creative programming was required to do these things efficiently on the hardware of the time.

> There were some cool features that didn't make it into the shipping product. For example, originally spreadsheet formulas were much more powerful: you could relate, e.g., graphical object positions and document properties to spreadsheet cells. So you could have the result of a calculation move objects around graphically, or vice-versa. (Further work in this direction led to a novel constraint-based programming paradigm called MOOSE, which I may resurrect some day...)

(From http://groups.csail.mit.edu/mac/users/bob/clarisworks.php#on... )

I've been kicking around the idea of writing a CRDT-based editor using this model.


I think it makes us unique among the popular editors of today, but we're very aware that the block model isn't original. Execution matters much more than innovation.


I’m simply commenting on your use of the word “unique” for the technological approach you took.

FWIW I use Notion.


Pretty sure Athens & Workflowy (both YC companies) use blocks.


Yes, enjoyed the article, but in case others are unaware, there are several other popular block style editors today which use the same data model. E.g. https://slite.com/, https://editorjs.io/ (open source), and https://www.slatejs.org/ (open source).


> I've been kicking around the idea of writing a CRDT-based editor using this model.

I got around to creating a data layer (p2p, browser-based, CRDT-backed) for something like this:

https://github.com/hyperhyperspace/hyperhyperspace-core

I'd be interested in collaborating on your editor


Excited to check this out!! I’ve got experience with this stuff, count me in! Check out svelt-yjs[0]. It makes complex CRDT apps trivial. Websockets or P2P WebRTC in a few lines. I’ve got a live collab Svelte REPL using Monaco almost complete. It uses Postgres via Supabase for auth / row level security / saving and sharing Apps you make. Heck, plug yjs into this Svelte based notion style block editor[1] and you’re more than halfway done in an evening.

[0] https://github.com/relm-us/svelt-yjs

[1] https://github.com/djyde/plastic-editor


That's cool!


I've also been toying with this idea for a while (p2p, CRDT backed, wasm-based)! It's a great concept, and I'd be interested in collaborating as well.

One thing I've been working on is rich structural editors for the scripting language used to build these local-first applications. So the environment becomes something like squeak or a lisp machine, where all application data is automatically synchronized over the network.


This is a very interesting idea. In general, it's interesting to think about synchronizing computation or memory state of a WASM module. Could you build an operational transform model for WASM memory state, to enable OTP-style process migration for more general WASM programs?


This is quite the question, and I'll do my best to address it.

First off, let's discuss more general Wasm programs. Currently, there are three primary modes of execution for Wasm modules - browser, node, or native runtime. The issue here is the dependence on JS - If you want to do any high-level interop between the language you're compiling to Wasm and some external library, you're most likely going to have to go through JS, pulling in a tool like emscripten to enforce the 'ABI' (so to speak) at the boundary between Wasm and JS. There are proposals, such as interface types, which may alleviate this in the future, but for now it's either go through JS or roll your own.

I'm not a huge fan of having to pull in JS for something like this, which is why I've instead been working on an alternative 'ABI' for Wasm, which supports algebraic datatypes. The basic idea is pretty simple: we hand the Wasm module a large hunk of Wasm linear memory that is encoded in accordance with the ABI. Wasm modules must expose an `update` function that takes this memory and updates it in-place. The runtime then 'deserializes' the shared memory, extracts useful data, and calls out to external libraries, modifying the shared memory (once again in-place) with any results. We toss this `update` function in a loop, run it in a background thread, and watch it to ensure it doesn't crash, write out bad data, or hang. That's the rough idea, but the actual format itself is pretty straightforward (similar to messagepack); all we need to implement is a library to serialize/deserialize this shared memory representation for each language that compiles to Wasm. All programs operate on these algebraic datatypes, so to save program state, we can just serialize the shared memory and write it directly to disk. By making a module stateless, we can easily synchronize computation and memory.

I'm currently using a hybrid operational transform / CRDT model that operates on the algebraic datatype model expressed by the ABI. Tombstones for text are largely inefficient, so we use a diff-based patch model for text inside blocks, but CRDTs for individual block-level transformations. (Wasm programs also have the ability to choose how to resolve conflicts.) Blocks are synchronized with a protocol similar to DAT, but, as stated, the protocol has built-in support for multiple writers and conflict resolution.

So, with respect to Erlang/OTP-style process migration - it's certainly a possibility. If we exposed an actor-style library to Wasm programs through the aforementioned ABI, implementing message passing would be as simple as sharing an append-only log between two computers on a network - the data synchronization would happen automatically. Spawning a new process is as simple as creating a new block in the algebraic datatype tree, and giving a Wasm module control over it. It's all pretty interesting to think about, and I'm just skimming here, but that's the basic idea.

To render out the user interface, we do something pretty simple - in addition to providing an `update` function, modules may provide multiple `translate` functions that take an ADT and translate it into another. Each node in the ADT is like a notion-style block, so all we need to do is define a `translate` function that takes the program-specific shared linear memory, and translates it to an ADT that represents a DOM (not a browser DOM, just a DOM). There are a number of optimizations that can be easily made here.

TL;DR: It's easy to synchronize Wasm state if modules are stateless, program state can be encoded as a block-based ADT, we can synchronize this state through a mixture of Operational Transforms and CRDTs, GUIs are a translation of program data.

(Woah, that's a lot of text...)


Any relation to the hypercore project? Sounds similar in tech and in name, yet I can't find any reference to it.


No relation, IIRC HyperCore was called DAT when I chose the Hyper Hyper Space name

\o/


How do you model text in your CRDT?


It's not really designed to model collaboratively edited text (yet?) but as the backend for multi user apps (social, workspace, commerce, etc)


This looks quite interesting.


> I've been kicking around the idea of writing a CRDT-based editor using this model.

Have you investigated OT at all? I'm not sure CRDTs are a great fit for complicated datatypes like an editor, and their strong convergence properties don't get you much in something as free form as text.


Didn’t Marvin Minsky come up with frames by 1974? Or are Clarisworks frames different?


These are not that kind of structure. The term "frame" is used in lots of contexts, e.g. these days in pandas' data frame.

By coincidence (with your question) Marvin later became Bob's PhD advisor. And at the time Claris works was being written I was working on a frame system with Marvin.


Notion clicked for me when I realised that everything is a "page with some attributes". Blocks are interesting at the engineering level, but as a user I found I was normally thinking at the level of pages.

Each page is some content (optional) with a set of arbitrary attributes of different types, e.g. date, string, number etc. When you create a table and populate it, you are creating a set of pages all of whom have the same attributes and the table is just a view of those pages. You can then create a calendar that's a different way of presenting those same pages. You can also deal with those pages directly and put content in, and reference them from elsewhere.

It's a really simple data model, and it's pretty flexible. We built all our processes on Notion initially and then moved them off when they got to an appropriate scale. I'm glad the API exist now, because then that approach can be incremental rather than a wholesale move. All the ways of displaying things - tables, calendars etc. are just displaying collections of pages.


This!

The design decision to feel "default-text" is awesome - text is a great foundation for a "custom workspace" product. But, it's only when the bulk of content is in Tables that things get extremely powerful.

Once Notion has in-line text references, ala Coda, the sky is really the limit. I think Notion nailed the UX and speed / performance.

Even if it's slow compared to Apple Notes, it's lightning fast compared to Coda, where "Documents" are extremely heavy, and relatively isolated.

Notion has a clean route towards eating Coda's most powerful functionality ([an incredible charting and formula](https://coda.io/formulas)). That's the kind of thing that a company as well-financed as Notion can dedicate a team to, and make incredible progress on.

Similarly, They can eat Coda's in-line text references, so your text can live-link to information in tables. A matter of resource allocation.

Coda, on the other hand, started from an "easier application builder" approach, like Retool for Excel experts. That's brilliant and powerful, but the UX creates a bigger dividing line between consumers and creators.

Notion feels enough like a normal word processor that consumers start doing crazy things without realizing it.


The amount of care I see our users investing in their Notion spaces on Reddit is inspiring. As is seeing Notion show up on my TikTok feed!?


What about a timeline first interface?


I've not used notion before and this is a great write up...

Having built my own private-ish collaborative editor/note/todo/sharing/pm tool back in 2012 (that uses indent level permissions) I don't see how you could have a non "block" implementation?

I also don't see how they are having so much performance difficulty moving up the inheritance tree at the client level?

Does anyone else who has worked on these kinds of problems take a different approach? This being HN and a space I develop in I am actively curious if any other architecture are in use generally?

Are there approaches outside of blocks for tools with this level of granularity?

Really cool of Notion to share these kinds of "inside ball" deep dives.


I don’t think you can do this without a block-based approach. That part isn’t super novel, for example I wrote textbook editing software circa 2014 which worked largely the same way.

What notion has done that is impressive IMO is made the editing experience enough like the MS-Word “inline” editing that people seem to understand it and use it without minding that it’s a block editor. The project I worked on back in the day wasn’t quite as slick, and there were a lot of users who got frustrated when things like copying text across multiple paragraphs didn’t always do what they had in mind (imagine a text selection from the middle of one paragraph (block) to the middle of another - that’s complicated to deal with).

Nice to see Notion take this model mainstream with such a nice UX.


I built a no-code web-app creator that also has a concept of blocks and I can relate to this. Each paragraph is a different block, and I still haven't quite figured out the best way of handling cross-block interactions like your example of cross-block text selection.

I think I would need to redo the browser-based editor, because even if I solved e.g. cross-block selection across blocks of the same type, it's unclear how it should work across, say, a text block and a video-embed block.


Excellent. Connected with ya.


Notion on iOS has exactly that problem for me, it’s not possible to select text across blocks or even move the text cursor between them continuously.


> I don't see how you could have a non "block" implementation?

I mean, documentation is mostly text, image and code. You can easily be document-based instead of block-based, like Confluence, Google docs ... you don't always need to be able to insert everything into everything.

I'm actually building a Confluence/Notion competitor, and it is document based. Being block-based makes a lot of things much more complex


Interesting, how do you handle line level or "intra-document" permissions then?


The permission is at the document level. What are the use case of intra document level permission?


E.g. managers having different information about tasks diplayed in a table than programmers. You could do a different page for each group, but it's not as nice - different URLs, duplication of surrounding content, etc.


Well in my case each row is a document and the table is just a a specific king of structure, but is not embeddable in a document. But the thing is how generic you want your tool to be. In the notion case, everything is just an instance of something. So that's super flexible, but also super complex. You can simplify things by enforcing a structure while still doing 95% of the use cases

You will never need to give a permission to a line of text


This is an armchair analysis but I feel like an ECS architecture here would really take this to the next level (flexibility and performance wise). But that is just my two cents.

Some examples: context dependent rendering, having multiple "types" at once, etc...


At http://story.ai the editor makes use of an ECS (Shipyard Crate in WASM). The flexibility it affords us has been super powerful particularly in support of constructs that are close to a programming language (values in and out of scope, suggestions and autocompletions, determining whether a "skill" is fully formed for publishing etc)

I haven't given extensive thought to where we diverge from Notion as the comment below mentions them moving away from it, but just wanted to say that your armchair analysis fits with our view.


Ah yes, thanks! This is definitely what I was curious about, I've never done game development and I wasn't aware of this architecture but I definitely ended up with a hybrid/better than naive ECS architecture. It's quite fast, that's why I wondering about notions struggles with performance issues, seems like it may be in part because they use inheritance of blocks and nodes instead of composition.


In a previous iteration of Notion used a kind of model kinda like ECS, where each block was represented by multiple records called "reference" and "composite", so a block could have different type depending on context. From an internal doc on history:

> We later decided this wasn't useful or necessary and collapsed them into one thing. WTF was I thinking??


I have no idea how to explain my model other than my very cozy rats nest that I’ve blown up 5 times and rebuilt —

And after reading this article, I realize it’s a lot like blocks... I guess the way I’d explain it is that I add a template element to the blocks so that untemplated versions with live data are like a black box to other ‘pages’, only when a user says ‘I want this visible again elsewhere’ aka templates do I extract out the non-sensitive bits (user identifies sensitive info) to be used again in other pages.


Notion is clearly a feat of engineering. The first time that you modify a link name, and it is automatically modified in all others connected users whether it is in the menu, inside a document or wherever it is. You are just impressed. You can also drag and drop links from a document to a menu.

I'm just worried that so much engineering complexity is going to make things slow to move. We will see in a few years if that was a good idea or not


It's one of those things that they don't really need to move. The product is perfect as-is. Maybe a more full-fledged API/plugin system would be nice, but that's it.

The only people that really want them to move and change things are investors, not customers.

Same with Dropbox. A folder that syncs is all I need from them as a customer. Everything else they try to do is just getting in my way.


> The product is perfect as-is.

> The only people that really want them to move and change things are investors, not customers.

I'm glad the Notion works for you, but I think we have a long way to go to reach perfection. To pick an example from the post, even indentation needs work. There's also always more work to do to make the product faster and easier to use.


We use Notion at work, and I would agree that Notion has some ways to go.

Notifications in the system are still pretty hard to work with, and email notifications are no replacement. Comments on pages are unwieldy and mess with cursor focus in annoying ways. I've definitely submitted this feedback to support before and I can understand that not everything can be worked on at once. However, I still have faith that the team will work the kinks out eventually.

EDIT: To be less negative, I should also note that Notion still has probably _the_ best LaTeX / mathematics support out of any documenting system I've ever used. I vastly appreciate this, as it is often easiest to share complex mathematics via a shared Notion doc since I work remotely.


Images in wide-page view will get stretched and pasted to 100% width no matter what their actual size is. You need to manually resize them to look normal.

I raised a bug for this, they said it was intended behavior. Like yea, ofcourse.


The product also desperately needs simple tables


It has tables. Just hit the + and select table.


Those are lists of pages and attributes and of little use for a table of numbers


Here's an example:

https://www.notion.so/dheeranet/Foo-a4b0f734fdab4ec4949a9b69...

What are you looking for that is not this?



As much as I like Notion it has _a lot_ of work to do tbefore anyone can call it "perfect".

It's text editor is very weak but I have stronger than average preferences so I'll refrain from commenting but more importantly the desktop clients are just down-right awful with no rebindable keys (even the keyboard shorcuts "help" needs to go online to a webpage), no plugins or any sort configurations at all. It's fine if you use notion once a week but if you work with it every day you really feel constricted and I can't imagine what people with disabilities even do.


Salesforce and JIRA both did something similar: their underlying database schema is very generic, basically keys and values, allowing arbitrary logical schemas to be defined at runtime. Yet in both cases, they ended up not really taking advantage of this flexibility. The logical schema of both systems is a very ordinary relational schema that could have been implemented directly on the database, with much better performance. I wonder if the Notion developers made a serious attempt to build on top of a more conventional structured schema, and found it really was unworkable?


> I wonder if the Notion developers made a serious attempt to build on top of a more conventional structured schema, and found it really was unworkable?

This is an interesting question. I actually think Notion's data model is much more conventional than the data stores behind document editors like Google Docs or Figma. Blocks are "just" (these quotes are doing a lot of work) rows in Postgres.

We use JSON for properties for flexibility and for user-defined property schemas. We could use an entity-attribute-value (https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...) table as you describe for that, but such a table would complicate our caching techniques. It would also be enormous, but, it might be time for another think about that since we finished sharding.


I was thinking more along the lines of, specific tables for different classes of entity, with a schema that’s as fixed as possible, so you get the most compact and indexable representation.


Been working with multiple document API's recently for my start-up. It's been a real challenge to understand how different document stores compare -- Confluence uses undocumented XML markup that "mostly" mirrors HTML, Google Docs gives you sometimes semantically-incorrect HTML with lots of inline styles, Slack has its own funny Markdown, etc. Then there's Notion blocks. I had to write my own HTML-to-Notion block parser, but I think I prefer working with it over everything else because structured data is just easier. One interesting gotcha: if you want to completely overwrite a page you currently have to remove every content block one at a time, and then add each new line one at a time. This is using their undocumented API, but the new API doesn't even offer deleting content yet. I'm eager to switch over but it may be a while...


Hopefully we get content editing soon! Most of the integrations I want to build need to edit text.


This is a great explainer on how and why Notion is so powerful, and certainly helps explain why it doesn't feel like the fastest tool (because it's inherently more complicated and more powerful than most).

I've tried using it for personal notes but it's speed and lack of offline prevented me from migrating to it. But I think it's the best choice for any organization that needs a collaborative docuement/wiki/Confluence-esque tool.

Not to mention that I think it has the best default fonts, spacing, and general aesthetic of any tool I can think of.


May I ask what you _do_ use for your personal notes instead? I'm always on the lookout for a new tool or workflow for my notes.


I’m not the person you asked, but I evaluated and passed on Notion (and Coda, and Airtable) for similar reasons.

I need a system that syncs, works offline, helps capture structured-ish data, is quick+flexible, and has fabulous linking.

I discovered and am currently using Obsidian.Md. I use Ulysses to access and edit on iPhone+iPad.

I love that it is markdown-based, linking is super easy, and everything is fast.

Cons: doesn’t work super well with native macOS features (e.g. services), native app for iOS/Android is still in private beta, and I don’t like the separation between edit and view modes. Also it could probably use a stronger UI for database-like features (e.g. tags). I wish there were better options for inking on mobile, but that’s not really their thing.

Previously was a OneNote devotee, but as I get into more complicated community engagement work, I needed something that could cross-link and find patterns.


I appreciate this write-up! I've been considering a migration from Evernote to Obsidian/Notion/other-competitor for a while as Evernote's UI gets steadily worse, and this reconfirms my perspective that Obsidian is leading the pack. Thanks!


> I wish there were better options for inking on mobile

Mobile beta is pretty similar to the desktop app. Like even most of the plugins I use work fine on my phone. The only real complaint I have is that it's kinda slow to start on a phone. Not as slow as Notion, but still takes a few seconds.

> and I don’t like the separation between edit and view modes.

That's next on the roadmap after apps become polished enough to exit the beta. Public roadmap: https://trello.com/b/Psqfqp7I/obsidian-roadmap


I use Bear on the Mac. Super simple, works on iOS too, fast, syncs cleanly.


OP here. I finally ditched Evernote when they made the Mac app Electron. I've tried Obsidian but it didn't stick (and it's Electron), but hear very good things about it.

I decided on DevonThink since it closely resembled what I used Evernote for but ditched it after just a month due to to bad iOS apps, weird default formatting issues, hostile developers (license covers max 2 computers, and refusal to support a few basic things).

Stumbled upon Ulysses and I've been using it since then. Excellent plain text/Markdown support (you can turn off the markdown formats you don't want to use), sensible default fonts and such, fast and native Mac app, great iOS apps, etc.


For personal, non shared stuff I really like the graph style apps, Roam is the one I'm using right now but there are a few out there now. Not "demanding" a hierarchy like notion etc do and everything having "backlinks" automatically is really a game changer.


Google Keep has been pretty sufficient for my needs. Nice search feature, able to capture things quickly when on a call or out and about. Not the most fully featured, but good for remembering things a few days or a week from now


I used Trilium Notes for a long time before shifting to Notion. It checks all the freedom boxes and is very customizable, without being alien like emacs.


Emacs org mode is worth a shot.


Apple notes

Google docs

Trello


As author of a Notion browser extension, "everything is block" concept made it significantly easy to understand the notion doc structure and tweak the UI to add multiple customisations on top of it. For instance showing a fixed static ToC, etc.

https://gourav.io/notion-boost


I think the power of Notion became apparent to me when I created a Page, accidentally changed it to some other type, then ctrl+z-ed it back to a Page, with all its contents intact. Seriously impressive. I wish search wasn't so crap though.


> accidentally changed it to some other type

This is the problem with Notion, in my experience at least. The docs are so limited and the maneuverability so unusual that it's very easy to do something like completely change the type of document accidentally.


Author here. I'm happy to answer any questions you might have.


I'm curious if Notion has any plans to make the "type" property user-extensible. Given the current data-structure, which decouples the block data from the way its rendered through the type property, a user has to define only one template for rendering arrangements of UI components (boxes, bullets, etc), titles and children. Extension could operate even at the level of derivation, where users could extend current base types with custom styling (color, font, size, border, etc) and child layout. As a plus, derivation would allow for blocks to be shared, with a fallback default rendering if users don't share custom types. Given the multi-dimensional nature of the uses of Notion (for work, personal projects, life management, etc), having types that were specific to their domain (grocery list, monthly budget table, contact card) would be a useful tool to semantically separate blocks by their presentation.


This sounds a bit like customized structured data types with style inheritance, à la DITA.


This kind of idea is very interesting, but I won't comment on future plans.


What’s the reasoning why Notion doesn’t have partial text selection across blocks? I originally thought it was an issue with each block being it’s separate contenteditable, but if you wrap all of the contenteditable blocks in another contenteditable div, partial text selection works (at least on Chrome).


I'm wondering ... so when a user requests a page which will have a full hierarchy of blocks, how are the db queries done.

Do you first query the root to see it's content blocks, then make additional queries to load the root's children block, then make additional queries to get those blocks children blocks (ie. recursively) until there are no more children?

Does that result in too many database queries? Or do you have other ways to optimize it?


It appears that Notion uses Postgres, which supports the recursive/hierarchical queries that are part of standard SQL [1]. While I don't know for sure that Notion uses this, it seems likely.

[1] https://en.wikipedia.org/wiki/Hierarchical_and_recursive_que...


Very interested in your data store. How do you store, query, and search across documents?

Also, are you adding presentational tables any time soon? :)


Our source-of-truth data store is Postgres, with a Memcached cache on top.

Most of our queries are "pointer chasing" - we follow a reference from one record in memory to fetch another record from the data store. To optimize recursive pointer-chasing queries, we cache the set of visited pointers in Memcached.

We use Elasticsearch for search features like QuickFind.

> Also, are you adding presentational tables any time soon? :)

Sorry, can't talk about future plans like that :)


Why not extend this to create forms/surveys as well in your application? You can render a checkboxInput block type as a <input type="checkbox" /> in a form. Your backend would match up the form POST parameter names with the associated form input names and that would be your "Submission".


I'm going to be writing a blog post for my engineering team soon and I'm really impressed by the storytelling and presentation here. How did you plan out your blog? Were there any resources or examples you found useful during the writing process? Thank you!


I'm glad you enjoyed it! It took a long time to put together. I started out by writing down how the entire system works with enough explanation for a technically-minded non-engineer, and enough detail to satisfy a newly hired infra engineer. That rough draft was about 10,000 words.

From there we started looking for a narrative. We extracted out the sections you see in the final post, and removed a lot of the superfluous technical detail so we didn't end up with technology buzzword soup; for example we cut discussion of Postgres, Memcached, etc etc, how we host the web servers; the kind of details that don't actually matter to the narrative.

The illustrations were in the post from the beginning as Mermaid diagrams (https://mermaid-js.github.io/mermaid-live-editor/). As we got close to publication we polished them up in Figma.

This is really the first engineering blog post we've put out, there was a fair amount of figuring-out-how-to-do-it going on. Now that we've had the experience, we're starting to write up our playbook internally.


I'd read a follow up with the Postgres part. I did a cmd+f for Postgres guessing that's what you used and was disappointed not to find it.


Would love to read the Director's Cut for more behind-the-scenes on storage, caching, infrastructure!


Did you... dogfood it ?


Really interessing.

This seem to be a really good use case for a NoSQL database. Am I wrong ?



Ltree is interesting, but if I understand correctly, to move a parent block, I'd also need to update the path column in all the child blocks -- at our scale such write amplification is a non-starter.


Yes, you're wrong. You're wrong because you need to JOIN a massive tree of blocks, to form the graph the author is referring to.

You can break out the "block" model into several tables and represent it in a relational database that way.

NoSQL = NO JOIN?

Hope that helps.


I don’t actually see a graph represented anywhere in the article; the author references wanting a graph at the start, but the only thing I’m seeing described are trees of nested blocks. Even the properties list seems to be a grab-bag of KV pairs that gets permanently attached to a block once initialized, to support roundtripping

Which is pretty much the ideal scenario for a document store. The article describes Notion as being very strictly hierarchal


A block has many properties. A property has a name, and a value.

The underlying persisted data doesn't necessarily have to be a bag of KV pairs.

A block is related to its parent and descendant blocks.

These relations are suitably represented in a relational database, not a document store.

EDIT: In graph theory, a tree is an undirected, connected and acyclic graph.


A document store is basically optimized for specifically hierarchal data situations — a tree. The data structure you’re describing, and what the article describes, is precisely that: a tree.

When comparing a document store versus a RDBMS, in terms of suitability and appropriateness, the distinction is primarily along the lines of a tree, versus an arbitrary graph (by which I mean that an RDBMS is more powerful, and more general, but not inherently as optimal in either performance, “scalability”, or UX in the places where a document store makes sense.

More specifically, the way the article describes it, you’re not interested in “give me every block of type X” — you’re only interested in “given block Y, what type is it?”.

That is, the question is one-way, and fits cleanly in a hierarchal format of a document store.

The only question posed that operates in the reverse direction is permissions, though even that’s a little odd, since it seems to me it should only go “downwards” as well — a block’s permission scope is the sum of all of its parents, and you can store it there upon iteration.

> The underlying persisted data doesn't necessarily have to be a bag of KV pairs.

It doesn’t have to be... but it can be, and appears to be.

> A block is related to its parent and descendant blocks.

Right; the singular parent, and the multiple children. A tree.

> In graph theory, a tree is an undirected, connected and acyclic graph.

When discussing trees and graphs, I think it’s obvious a distinction is being made between a graph forming a tree, and graph forming a not-tree (more complex than a tree). When I say that a square is easier to encode than a rectangle, I do not mean that a square is not a rectangle, but that a rectangle is not a square — that a square’s more specific properties give us opportunity to simplify/optimize (I only need to store one length to represent it).

A database can encode a tree just fine, but that doesn’t mean it’s the best tool to do so.

There are other properties to a document store I don’t care for, and I don’t like them in general (like the implicit schema, and total lack of data consistency validation by the data store, and the fact that you often don’t truly have a tree), but representing a tree is what’s been described, and it’s exactly what they’re specialized for.

If you want to argue against it, you need to specify why you think this isn’t a tree, because I feel it’s quite obvious it is.


The behaviour demonstrated at

https://www.notion.so/Tree-breaking-3a90e2bcd2154f4fab06a3c7...

Breaks the tree model, as 'Complete Task' would need to have both 'Subtasks' and the page itself as its direct parent.

That said… it's mostly a tree, and there may be merit to optimising for that access pattern.


@setr explained it really well. A side note, NoSQL also includes graph databases, dedicated to this type of node/relationship traversal.


We don't use JOIN for the content tree; I don't think I've seen one in any of our queries.


What do your queries look like? Are you using an ORM?


We don't use an ORM. Notion's codebase on the back-end is much more functional than object-oriented, in the sense that we have many more code that looks like `transformTheData(theData, theChangeToMake): ResultingData` than we have classes or methods.

We do lean very heavily on the TypeScript type system and try to make invalid states unrepresentable.


have you tried "data last" FP like `transformTheData(theChangeToMake, theData): ResultingData` instead? I learned this from Ramda.JS, makes it way easier to leverage currying, ex `change = transformTheData(theChangeToMake); change(theData)`


Notion is great for about a month, then the glacial performance and forced-hierarchical structure really start to grate. But by that time you’ve already bought it


Notion is cool but I went back to trello because its so damn slow. The ios app would refresh all the time and fail to sync. The web app was generally buggy.


Am I the only one who thinks the data model is very complex? How does it compare to a document based data model? What are the pros and cons of each?


You're not the only one.

The major issue with the document model is storage. How big can a document grow? How do you persist it, and make small changes inside it? How do you handle "hot" documents that are very popular? Moving data between documents or having part of a document reference part of another document are complex. Complexity comes from building derived data that looks at slices of a document.

With a block-oriented model, your records are much more manageably sized. It's easier to reference or move data between documents, but in turn, you need to do these kinds of recursive shenanigans because your individual records are smaller in scope. Complexity comes from derived data that composes blocks together into a document.


Thanks!


It's always nice to learn how tools implement this. Especially interesting to me, because I am currently working on a similar tool, although more focused on the plain text aspect (think Markdown + widgets). I am facing slightly different challenges because widgets have to live in a plain text context. For instance, having a widget (say interactive diagram) as part of a Markdown blockquote. On the other hand I don't have to deal with nested blocks (yet), because regular Markdown elements are just text (with fancy styling though). Another challenge is something like $a+b=c$ where it is shown as rendered LaTeX when the user's caret is not touching that part of the document. Also, contenteditable is hard.


Is there a reason why you did not use rdf for representation and some rdf aware encoding like jsonld for serialization?

Would be significantly easier for others to work with, could easily query it with SPARQL.


An early version of Notion (before my time) used Neo4j, but it turned out to be very slow for the kinds of data access Notion does.


You don't have to use neo4j or any graph database to use RDF. It is just your current model seems very graph based and actually not that difficult to map to RDF, it would likely be possible to do with a jsonld context, and if you provided such a context then it would make your data a lot easier for others to consume.


I used a similar approach when working on an editor/notes taker in Flash. Worked quite nicely but even back (early 2000s) then it wasn't really a new idea.


Great write up.

I have a requirement to allow users to create adhoc tables within a web application. I was wohoe beat to represent this on database. Any thoughts?


Tables (ad-hoc ones) are a beast. Code that aspires to do tame them, looks simple at first then evolve into crazy monsters that die at the altar of excel.

Here's a rough progression of (not entirely) imaginary user requirements for an imaginary, general purpose table rendering software.

"I just have a bunch of values, can we make this into a table?"

"That looks good. You know what would make it great ? Header rows. Just make them stay there while the table scrolls."

"Hey! can you give me some icons on the header row for each column that I can click and sort the data ?"

"Please give me some pagination!"

"Can you fix the sorting logic? A row with id 10 shows up before the row with 2?"

"Can you support merging cells ? Please !"

"Can you support re-ordering of columns ?"

"I want to adjust the width of my columns but the first column should be fixed width"

... A year later...

"This is great but you know its kinda slow. My users prefer to do this in excel and paste it here. Hey, here's a great idea, can you just parse data pasted from excel and just make a table from it ?"


A graph is so natural for pages, web pages are graphs too. So they choose a well fitting model for the problem, congrats


The webpage crashed as soon as I opened it. No idea what happened exactly.


you start a blog post with Engelbart, Nelson & Kay and you have my attention. At last someone who understands where the whole field of digital HID was started.


I liked the part where they talked about the blocks with the blocks and the block blocks and how they all block together.

I thought the reference to field pioneers was a bit much, I wouldn’t have the blocks to do that with a straight face. I wish these app makers (see also Figma, AirTable) would refrain from making their self-aggrandizing “we’re bringing computing to the masses”; what a load of blocks. What you’re doing is creating a product (another silo in waiting) serving a market that presently is interested. This block model is also not unique (see WordPress and Drupal): Notion may be refining some of that in a more seamless experience, but this is a different kind of statement. Perhaps “we added collaboration to blocks” would be a possible claim for uniqueness.

Aside from this, I found the technical decisions interesting and worth a read.


I think it really is that unique - Notion blending read/write is pretty revolutionary for an app like this. Wiki's are obsessed with toggles between read/write mode.

Those apps you mention - Drupal, Wordpress, etc - have write modes that exist in backends with alien experiences to the read modes.

By unifying read/write into a single continuous UI, with built-in relational databases usable by non-technical users, Notion really does let people create damned powerful custom apps.

If you squint, you can see how Notion looks a lot like the next version of the Internet. By hyper-focusing on company intranets they miss out on the power of connected workspaces ("domains"), but the thinking is there.

I'll be seriously surprised if Notion isn't looked back on as a breakthrough paradigm shift.


> I'll be seriously surprised if Notion isn't looked back on as a breakthrough paradigm shift.

To that end, I'd be surprised if they're even remembered much at all. I cancelled my subscription after being seriously disappointed in the feature-set.


If you want to set the line, I’ll take the over on “Notion succeeds”


They will be remembered, if only because a surprising number of tools have copied Notion's UI


I hadn't heard of Notion before this recent HN discussion, in which many, many people make the argument that it's slow, often unusably so: https://news.ycombinator.com/item?id=27144566

So it's a bit of a surprise to see them bragging about their technology today.


I tried notion several times and gave up. It is not as good as excel or google sheet. I want the data table view to be as powerful and efficient as google sheet or excel plus other notion views. Right now, it is very difficult to even style the the cells with different colors. I think it will be much better for existing google sheet to add other notion functions. Don't know why google stops innovating.


I totally agree but I also have the opposite problem.

You can’t have simple tables in Notion (you know, like a comparison table without structured data).

You always have to create what they call a “database”




Applications are open for YC Winter 2022

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: