I hope fly is able to make it. I’m rooting for them - however - I’m starting to wonder if the SQLite push isn’t more “this is fun and interesting to build” and less “customers want this”.
Don’t get me wrong - this is neat - but I’d never suggest anyone to actually use this outside of a fun experiment. The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.
If it turns out they’re building this for customers throwing cash at them, awesome. I just somehow doubt it. I think Planetscale has the better approach: a drop in replacement for MySQL/RDS with a smarter query planner. As a production engineer that’s what I want to pay for!
I’m excited about SQLite for web apps if only because it is one less moving piece in my stack, which focuses on prototyping and finding product market fit.
If I start hitting hundreds of writes per sec, then, thats either awesome or I wrote some horrible code.
I recently implemented SQLite for a metadata application at work. My constraints in design were pretty concise; the application will only ever need to scale vertically, my data persistence story is a matter of speed and accessibility over longevity, I have plenty of options for scaling disk IO, and my read performance is much more paramount than write performance.
The outcome is that my persistence later is not adding $200/m immediate service overhead cost and my deployment was easy to manage, which is a strong promise made to the rest of the team. I think there's a place for SQLite, but just like any tool you need to know that your design constraints match the constraints of the tool.
I think you're skipping the replicated read only use case, which is our use case, and it's super handy there. but i understand this is a restricted scenario where little could really go wrong, and it could be done other ways.
Any self respecting e-commerce site would want fault tolerance and strong consistency even with potential network partitions, so definitely not SQLite as described in article
They aren't necessarily going to have all this. Lots of smaller ecom shops can run on a single server per region. If you're a North American company selling a few hundred t-shirts per day in NA and EU, it could probably be fine, no? I'll admit I'm not speaking from experience. Rather, I have experience in everything I just said only I was using pg, not sqlite. But I've been very interested in sqlite recently.
There is a cost to fault tolerance as well, if you lose $100 per month due to fault tolerance thats worth about 30 developer minutes. Do you really get good fault tolerance for 30m of monthly work?
That's something for me to look into. Like what would it would look like in each? I've never been the dba-type in past jobs. I'm relatively well-versed in SQL but not administration.
8 years ago we built our own ecommerce site + warehouse app (inventory tracking, fulfillment, receiving) for a few hundred orders per day. The goal was to be able to better see profit margins by product, track where the money was going/coming from in detail, along with cleaning up the inventory management part of the operation.
The warehouse people loved the change because it really streamlined the fulfillment process and it basically reduced their errors to zero. Owners loved it because of the all the cost/price tracking. It took about 6 months of work by 2 devs.
Before this they were on woocommerce. It was slow as shit. We looked at shopify but the integrations with 3rd party warehouse stuff was bad. The other solutions we looked at were overly complicated swiss army knives.
This is very close to my experience as well. Unfortunately, it was in reverse for me :( We had a custom solution and people were happy. A new tech lead came in and didn't like that the custom solution was PHP that still had some legacy spaghetti in it, so we switched to an off the shelf solution. It was very painful. People were unhappy.
Your wild guess is flat out wrong. When the previous tech lead came in the entire thing was spaghetti. His strategy was to slowly start bringing in some order to the chaos which we did, diligently, over two years making tons of progress. It was a homebrewed framework but it was really nice (he'd had years of experience contributing to Drupal). All the business critical functionality had been converted to this. The spaghetti parts were features that were essentially "complete". Since they never needed updating, they worked fine as they were.
As with anything, the typical business can use something off-the-shelf, but a certain percentage are doing things differently enough that custom development becomes practical. I had to custom build the billing system for Beaker Studio to properly meter customers. Even Stripe's metering API wasn't flexible enough to handle per-hour metering.
> the typical business can use something off-the-shelf
I think we generally have this problem. It's not the typical business that can use that stuff, it's the average business. Unfortunately no business is the average business.
I find it hard to imagine a reason a shop selling 100 tee shirts in a day would need any custom functionality since this is basically the exact use case all these OOB e-commerce tools are built for.
I've worked for such a place. Off-the-shelf solutions are trying to be everything to everyone and you can end up customizing the crap out of them to the point where it can become more onerous than just building your own. The place I worked was also print-on-demand service and had hundreds of thousands of SKUs as well as allowed customers to make custom products and we also hosted some peoples' shops. Shoehorning that into a custom solution was painful.
I work at a very similar place now that uses Shopify. Managing that many SKUs on Shopify is crazy painful.
The thing is is that custom ecom solutions really aren't that hard. The off-the-shelf ones are complex because, as stated above, they are trying to be everything to everybody.
In our B2B space there are tons of custom business rules. For example, you place orders per manufacturer and each manufacturer has its own minimum and reorder amounts and reqs on an order, promotions, business rules, etc. This is for business that have been around 25-30 years.
We did an evaluation on several out of the box ecommerce solutions and none of them were able to meet the requirements that absolutely had to be there. Shopify flat out said no, they can't help us, etc.
One of the first commercial projects I worked on nearly 20 years ago was a tshirt shop. And the precious company I worked for was a comparatively huge logistics startup.
I’m fairly confident I could spend a month or so writing a custom solution for a shop selling and shipping a few hundred tshirts a day that would save them enough money to break even on the software in a few years (compared to off the shelf solutions).
If I was starting my own tshirt company, I’d definitely do it.
I have this same discussion at work frequently. Everything thinks they need instantly up to date, yet our customers work for 10 minutes on a stale document and no one cares.
> The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.
In my experience SQL databases are pretty poor at executing the kinds of deeply nested joins needed to return all of the data needed to render more complex UIs. It almost always ends up being faster to simply make nested selects in batches. So latency ends up mattering in these cases.
You can work around this by denormalizing the data, but this often explodes your data size.
I can't imagine many suitable production use cases too. Alternatives like Planetscale, Supabase or Neon are even easier to use and at the same time much more powerful. If latencies around 50 ms are too much for the specific use case, SQLite with Litestream could be a great solution. Otherwise I'd go with managed Postgres/MySQL solutions.
The alternative is to write good queries! Alternatively, a better query planner that has a more liberal approach to "I know what you want, not what you're asking for". A great article on this re: Planetscale's approach is at https://vitess.io/blog/2021-11-02-why-write-new-planner/
Don’t get me wrong - this is neat - but I’d never suggest anyone to actually use this outside of a fun experiment. The problem with existing SQL dbs isn’t really the architecture - its the awful queries that do in memory sorting or make temporary tables for no reason or read-after-write, etc, not network latency. SQLite won’t fix your current production problems.
If it turns out they’re building this for customers throwing cash at them, awesome. I just somehow doubt it. I think Planetscale has the better approach: a drop in replacement for MySQL/RDS with a smarter query planner. As a production engineer that’s what I want to pay for!