Put your global tables in Postgres/CockroachDB/Vitess/etc. You don't need to force everything to be a SQLite table. Use the right tool for the job. SQLite can be the right tool, and it can not be the right tool.
Not sure what the poster meant but DuckDB is an analytics DB, it doesn’t have a btree index - at least not last time I looked. You could consider it the OLAP embedded DB to SQLite’s OLTP embedded db.
DuckDB can read SQLite so you can even imagine using them side by side in the same system, serving point reads and writes from SQLite and using DuckDB for stuff like aggregates and searches that SQLite is slower at.
The wild thing about computers though is you can use the same materials for a 1 meter bridge as for a 10,000 meter bridge; the kind of scaling laws & economics pervasive in physical engineering apply very differently if you plan ahead and make very smart choices.
The Internet gives you global reach for free.
B-trees work for storing for 10 items and for 10 billion items.
Classic headline “whatsapp scaled to 1 billion users with 50 engineers”.
JavaScript has plenty of stand alone ORMs like Drizzle, TypeORM, sql builders like Knex, there’s tons with a bit of googling.
I think that ActiveRecord is actually the weakest part of Rails, and the scariest and most difficult to manage code in a large rails app is models and stuff that is doing lots of active record stuff.
Rails can’t be reduced the goodness is that it’s more than the sum of its parts to get everything together, as DHH says, omakase. It’s not needing to build a framework before you can build your idea. It’s having such a solid and complete base that add-on packages can extend knowing they have a solid foundation. It’s being dominant enough in the language that most things try hard to work well with you.
Outside of React, I don’t think there’s any package in JS that comes close to have the gravitational pull that Rails has in Ruby.
I'm not actually a fan of ActiveRecord, i kind of detest it actually. But it's the beating heart of rails. The ORM aspect is almost a distraction from its primary benefit, defining a globally referencable model with lifecycle hooks and monkey-patching. Basically everything else falls apart without the model to leverage to rapidly produce code. The add-on packages you reference work so well because there are so many ways to hook into activerecord (and to a lesser extent, routing etc).
Like, the typical path is: 1) add gem, 2) code gen, 3) run migration, 4) you now have third party code managing a subset of your model. Crazy powerful pattern. applies to everything from auth to sending emails to async task workers to writing audit logs.
> RedwoodJS seems like the best attempt at a Rails in JavaScript these days, worth a look.
It's telling to me that Redwood/Adonis haven't really seen huge uptake. I feel like the JS ecosystem moves pretty slowly on a conceptual level overall. Backend dev is still heavily associated with express.js when that's seen as a minimal framework in other ecosystems.
I just love reading DevOps diaries from different companies, especially ones that I've used and had experience with. Thank you for sharing.
You mention briefly displaying metrics on office dashboard TV screens, that's a really fun idea. Is it just a bunch of Grafana graphs, or is the system more in depth? I might have to think about creating something similar at my place.
I think this one is a Datadog dashboard, so similar to Graphana. A few graphs but also some big numbers and top-5 tables. I've also seen some big-screen Notion dashboards where some graphs are Notion's native charts feature and some are embeds from elsewhere like Datadog.
It’s stock RDS on EBS, which as you say is super slow compared to Genuine Local Storage. Any large scan or large result set query that doesn’t fall into the Postgres cache or the underlying EBS cache is frustratingly slow.
However almost all of our reads are “get row by ID” lookups that hit memcached. Our query pattern is mostly recursive graph traversal which Postgres is bad at, with our table size even a 20x faster disk wouldn’t make it feasible to do in SQL. We also copy paste “notion database” data in to a stateful caching service, but that also needs specialized indexing for user defined schema, another thing Postgres can’t handle (my last project was an improvement to that service).
The place we really suffer from the EBS slowness is with full table scan data migrations, stuff like “for all blocks in Notion, block = f(block)”. For this we use “Big Data” approach reading from DB dumps in S3 instead of Postgres directly.
Ultimately I’d prefer to serve every read from your device’s local storage, which is my current project.
All platforms; we already cache and serve locally using SQLite (see https://www.notion.com/blog/how-we-sped-up-notion-in-the-bro...) but the cache is built as an optimization -- loads look like `isFastDevice ? (await loadLocal(request) ?? await loadRemote(request)) : (race(loadLocal(request), loadRemote(request))`
We're working on redesigning the data architecture to be more truly "local first" where every read should come from local, and falling back to the server is a rare exception. We want this to look like `await loadLocal(request) ?? await sync({ priority: request }).then(() => loadLocal(request))`
(Note: for features like content that make sense offline. Some features, like user invitation and management, only make sense online)
Oh nice. I had dismissed offline/local web apps for anything decently complex for a long time because of the file system limitations. But it looks like this will finally make that practical with OPFS & WASM SQlite. Thanks for sharing.
