This is funny and sad to me. We had SQLite in the browser[0]. I only did a little bit of work with it but it seemed actually pretty nice.
It was torpedoed because it was SQL-based (and not trendy "key value" and "web scale").
There was the whole excuse that the specification was "whatever SQLite does" and, therefore, not suitable for being a standard. There would be worse things than SQLite upon which to base a standard, all things considered. I still believe it was torpedoed because of lack of trendiness and "not invented here".
As a sidenote, I worked a lot with the WebSQL API and it was not a very good API in the slightest, immaturity may excuse some of its flaws, and it isnt like Safari did a much better job with IndexedDB, its just a buggy browser and thats where WebSQL was used most, but a large part of the problem is that it was bolting an API that assumed a single threaded client when that is not the reality with web pages where multiple tabs exist
It’s funny that the top comment points out the obvious-- people want SQL in the browser-- and a NoSQL author immediately has to jump in to claim that SQL in the browser was wrong all along.
I think the NoSQL and MongoDB folks are worried for their careers.
> The excuse was that a standard needs to have multiple implementations otherwise we are standardising implementation details and bugs.
And yet we're are now at a point where Chrome rams its own APIs through standards bodies, and there are no (and often won't be) any independent competing implementations.
> And yet we're are now at a point where Chrome rams its own APIs through standards bodies, and there are no (and often won't be) any independent competing implementations.
Very few people are using Chrome-only APIs which are not in the standards yet. So it's not really a concern.
But otherwise, Chrome really has pushed the web forward more than any other browser. If it hadn't, native (and walled garden style) app stores would have totally taken over.
The web is in business (and thriving) as an application platform because it's being pushed forward relentlessly. The Storage Foundation API discussed in TFA is a good example.
> Very few people are using Chrome-only APIs which are not in the standards yet. So it's not really a concern.
Ah yes. But SQlite not having competing independent implementations somehow is?
Also, "not many people using something" is not as great an argument as you think it is. See, for example, the latest problem with browsers deciding to remove alert/prompt/confirm: https://dev.to/richharris/stay-alert-d
> The web is in business (and thriving) as an application platform because it's being pushed forward relentlessly.
A quote from the same article: "An ad company shouldn't have this much influence over something that belongs to all of us".
So somehow non-standards that Chrome pushes (many of which will never get a different implementation because both Safari and Mozilla consider them harmful) are good, and push the web forward. But SQlite is bad because there are no independent competing implementations.
> The Safari team did quite a lot of work re-styling their alerts and dialogs to be within the context of the webpage, yet phishing and scams that utilize this still run rampant on iOS. Repeated alerts are used to lock up the browser and make it unusable, forcing non-technical users to call a scam telephone number because they think their device was hacked.
A few bad actors ruining it even for totally different usecases.
> Ah yes. But SQlite not having competing independent implementations somehow is?
Except that's not why it was culled. The reasons are discussed in great detail on this thread and elsewhere.
> non-standards that Chrome pushes (many of which will never get a different implementation because both Safari and Mozilla consider them harmful) are good
I'm saying nobody uses those Chrome only APIs, until they become standards (which requires acceptance by other vendors). Or are you against experimentation?
> I'm saying nobody uses those Chrome only APIs, until they become standards
Ah yes. Once Chrome releases something in stable, literally no one ever uses those APIs, no one. And even Google's propaganda machine doesn't tell you to use them (example: https://web.dev/usb/)
My assumption is that a very small percentage of developers participate in it. Feel free to prove me wrong if you have any numbers.
> I'm not. What Google does isn't experimentation.
So the way this is done now (with origin trials to collect feedback from developers) is not good enough? What exactly is your definition of experimentation?
> My assumption is that a very small percentage of developers participate in it.
It doesn't really matter how many. It means that people are already using this functionality, and depend on it. I didn't link to an article on removing alert just for the fun of it.
> So the way this is done now (with origin trials to collect feedback from developers) is not good enough? What exactly is your definition of experimentation?
Running this as origin trial is a very good way of doing it. What's not a good way is:
- having an unrealistic timeline.
Example: Mozilla was asked for position on WebHID three months before Chrome released in stable. The "standard" was so badly written that Mozilla engineers couldn't understand it. Chrome still released it, and updated the "standard" two months after it was already in the wild.
- ignoring any and all objections, and favoring own needs only
Example: Constructible Stylesheets. The spec has a trivially reproducible problem. Several developers (not only from Mozilla and Safari) pointed this out, and proposed several changes to the spec that would get rid of the problem. However, Google's own lit-html was interested in the spec. So they released it in stable, said that "0.8% of pageviews now use this" (reality: only lit-html used it at the time), and refused to hide it back under a flag. Safari simply said they are not going to implement the spec in this shape.
- completely ignoring the realities and needs of the web
- presenting these "standards" as fait accompli even if other browser vendors will never ever implement them
They have co-opted web.dev as a propaganda machine. The site presents itself as a neutral resource for web develoeprs. It's not, it's a Chrome-only vehicle. They present various specs as already available and never specify issues with those specs. For example, WebUSB: https://web.dev/usb/
See, e.g., Mozilla's position on various specs. Scroll down to "harmful": https://mozilla.github.io/standards-positions/ I'll let you guess how many of those have already been shipped in Chrome. Then you can find what web.dev says about it
- gaslighting other browser vendors
I won't link to specific tweets because I don't need to be angry. But almost every single of Google's high visibility "standards people" and "developer advocates" and "community managers" will always, and I mean absolutely always paint all other browsers in as negative light as possible (as lagging behind, as hurting the web, as damaging the web, as being a detriment to the web etc.)
My feeling is that the web apis try to do too much. Why do we need a webSQL api. Why do we not just let websites create a file and then they can provide whatever kind of library they want. They could package a WASM version of sqlite and just work like they would as a desktop app.
That way you never have to deal with browser incompatibility or unchangeable specifications.
I'd agree with some of the existing web APIs. But I think some web SQL API should be built-in.
Why? Because it's a very common pattern. Almost fundamental. Take any project more complex than a toy calculator, and you'll quickly find places where the authors are hand-rolling relational operations.
Does your app have an array of records in memory, which it then searches for values, filters by conditions, and/or sorts? I'd give it a 50/50 chance that if you replaced that array with an in-memory SQLite table, and all operations on it with SQL queries on it, the result would be less code, more readable code and better performance[0].
It's not just the web - I'd argue that programming languages in general should all embed first-class in-memory relational database engines. It should be a part of the language standard - if it's easy to write this:
struct Foo {
int bar;
date baz;
string quux;
};
//somewhere in something
Array<Foo> foos;
it should be easy to write this:
table<Foo, index=[bar, quux]> foos;
(or some other, better way of describing intended data access patterns)
and get a thing that can be efficiently queried, filtered and transformed along dimensions specified, with the compiler turning your request into efficient bytecode/nativecode. Doesn't mean you have to write queries in vanilla SQL - there are more composable syntaxes out there. But arbitrary loops and map-reduces aren't that good either.
When you start looking at your data processing code as database operations, you'll see it everywhere. That map/reduce/zip blob? That's a JOIN query. That struct you're keeping in your event loop, that looks like:
that priority queue is an index on the tasks array, and looking at its front is just SELECT id FROM tasks ORDER BY priority LIMIT 1;. That Entity-Component-System pattern you're using in your videogame? That's literally a relational database. It was conceived of as such (and for massively multiplayer games, is often implemented as such). Wouldn't it be nice if the ability to express this came built into the language? With a column-oriented option for improved performance, too?
--
[0] - It's trivial to add indexes to SQL table. The engine will maintain them for you. Nobody habitually adds indexes to their own regular variables. They, along with the code to maintain them, would manifest as extreme code bloat. Meanwhile, in-memory SQLite is freakishly fast. If you're measuring performance in Python/Ruby units, you won't even notice the FFI overhead.
I think its fine for a web app to be multiple MB. Websites like HN/wikipedia/blogs have no need for an SQL database but if you are loading something like a full IM client, it makes sense to download a few MB to make it stable over all browsers.
Disagreed, this is a dangerous line of thinking that leads to using technologies like gwt, Vaadin, Blazor, Flutter or others that try to turn the browser into something that it's not suited to.
Sure, there could be cases where you have absolutely no alternatives to do something really specific, but in those cases I'd first invite you to reconsider whether what you're attempting to do actually needs to be a web app. Of course, people's thoughts will probably be split there.
However, the bottom line is still this - large sites load slower and cost more to load, they consume more battery and CPU, which on lower end devices could lead to system instability. If users don't outright leave your bloated solution (i.e. if they're forced to use it because of network effect) then in the case of any non-optional conditions (slow or unstable networks, low end devices, expensive data) their experience will be miserable.
Maybe have your IM client be a downloadable app instead? Better yet, why not have it be a native app instead of a bundled browser app, for excellent file size, small attack surface and great performance, while conforming to the OS look and feel?
Alternatively, why not make your IM client have a lightweight browser version that doesn't try to do everything under the sun? Personally, i feel like the answer is not to make browsers do more, but to try to do less yourself.
To that end, utilize server side rendering and pre-rendering with hydration when needed. Split some bundles and shake some trees. Compress as much as possible and use boring, common fonts while preferring local versions if available. Avoid videos in most cases and use smaller images, or vector graphics. The web can be a simple, beautiful and fast place if we make it one.
> why not have it be a native app instead of a bundled browser app, for excellent file size
There are other valid arguments, but "use a native app for smaller file size" specifically doesn't make sense at all - the opposite is true.
While webapps may be larger than simple web pages, they're still far far far smaller than native apps, and using service workers etc, you do only pay the download size for a web app once, just like a native app.
For example: the Twitter Android app is a 21MB download. I just tested loading the Twitter PWA (https://mobile.twitter.com) logged in but with an cleared cache: it's a <3MB, including all the content visible in my feed on the first load.
The native app is _7+ times larger_.
I suspect the same or worse is true for the vast majority of other apps, e.g. Uber on Android is a 50MB download, while m.uber.com (reloaded after login with a cleared cache) gives you the same features with <1MB.
That might be an apples to oranges comparison in some regards, to be honest.
Using native UI frameworks will almost always yield smaller file sizes than the equivalent implementation in any of the web technologies, regardless of whether served as a webpage, or bundled in a WebView or something similar (with few exceptions, one of which is listed below). Just consider a WinForms/GTK/Qt or a similar application in contrast to using React/Angular/Vue and including those within Electron, or even loading them in a web browser.
Many of the mobile applications out there actually include hybrid technologies (like Ionic, React Native, Xamarin or anything else of the sort), since nowadays telling a company that you'll have separate codebases with OS native widgets for iOS, Android and others would not be met positively. Therefore, concessions are made to speed up development, at the expense of having more abstraction layers and larger bundles.
One aspect in which you're probably correct, however, is that technically the web indeed affords you the possibility of having pretty good download space savings, if you choose which forms to make user download, as opposed to including all of them in the application. For example, if you have a billing form that actually needs 50 different variations based on countries and regions, and include them bundled into a native app, then on the web you'll instead be able to just download the one that you need dynamically. That said, none of this is alway easy, hence Twitter and Uber both employ a large number of engineers that work on screens like that, even if many people won't even see them.
My point here is that there are many different approaches and technologies that can be used, "native" in the context of my message meaning an application that uses the OS UI and widgets directly, without reinventing the wheel or attempting to encapsulate cross platform or web technologies within it. Of course, it's perfectly understandable and valid why companies that are highly focused on their own brands and shipping quickly don't necessarily adopt that approach.
I subscribe to an alternative premise - "The browser is effectively an OS and thus a standardized distribution target".
There are cases where the people I want to distribute to don't suffer from the constraints you mentioned. e.g. will tolerate a 20 second page load in exchange for the experience my product will provide.
Being able to reach all devices and have a super fast experience without consuming bandwidth is always good, but not always a high priority .
That is a valid premise indeed, but one that i cannot agree with for a plethora of reasons. There are articles that go into more detail about what this approach leads to: https://idlewords.com/talks/website_obesity.htm
There are also aspects which are probably entirely overlooked because of that approach, for example, consider:
Sometimes, there are also accessibility concerns (especially in the case of Flutter), all of which will work against letting your software be used by as many people as possible, in as many hardware and software configurations as possible.
If the target audience of your products and sites are wealthy and able bodied individuals in 1st world countries, then by all means go ahead. However, in certain other pursuits inclusivity is definitely an important goal as well and one that i personally value a bit higher than fancy looking sites with rich functionality.
That said, isn't 20 seconds a bit much? Sure, i get that it's just an example, but i think even heavier pages like GMail load in approximately half of that. I think that bundle splitting and some optimization are probably a good idea even in heavier pieces of software!
Funny, how this is the static/shared library problem again :). And where the hivemind seems to prefer static libraries for regular software, it's reverse on the web.
I suppose the problem is, on the web, you're redownloading the whole bundle each time you run. There are layers of caching in the browser that are supposed to help, but they're almost always defeated by a combination of app misconfiguration and developers' reluctance to use the cache given the ridiculous rate of redeployment of web software.
I'd still argue that SQL API is a prime candidate for a shared system-level library - AKA browser built-in. The concepts behind it have been worked out in the 1970s and honed through decades since. It's not experimental tech. The API won't be changing every month.
I view the "standards" argument as a red herring for building a NoSQL db in the browser. Which, to this day, is slow, buggy and requires third party libraries to be usable [1]
For those who are able to stomach an uncomfortable political history instead of an easy, technical answer, you can take a look at [2]. It's interesting that 7 years later, many the folks who pushed hard to get rid of SQL in favor of NoSQL seem to no longer occupy positions of prominence in the industry.
I am familiar with Nolans article, I created PouchDB (the project he is discussing), you seem to have misred the post as it discusses the technical nuance and tradeoffs involved in the decision at many points entirely agreeing with the position against WebSQL. While Nolan came to the a different conclusion than I did (a point he made in the post) he laid out challenges very well and made it very clear there was no obvious technical answer.
Regardless of how you view it, the benefit of hindsight shows the exact thing that people warned would happen did in fact happen (a widespread venerability in SQLite exposed across various browsers). Its also a fairly strange point to be personally insulting people involved in the process whose careers are doing perfectly well.
Edit: Nolan replied before and corrected me. I've removed my misinterpretation and kept my main point below.
Back in the day, there were people who strongly suggested MongoDB and IndexedDB were the future, and that PostgreSQL, MySQL, SQLite were trash. I've noticed the folks who rode that hype-train moved into other kinds of occupations that aren't exactly engineering-focused anymore.
I wrote that article 7 years ago, and FWIW I would side more with Dale these days. It's probably a good thing we didn't just slap a half-baked API on top of SQLite and call it a web standard.
The biggest problem is that yeah, WebSQL tends to be faster than IndexedDB. Or at least it was back when I was working on PouchDB. Biggest issue IIRC was that joins were faster in SQLite than implementing the same thing in userland on top of IndexedDB. Browsers eventually shipped getAll/getAllKeys which also helped with cursor slowness.
I haven't looked much at the Storage Foundation API [1], but it seems like a more reasonable approach moving forward. Just give developers the low-level tools and let them build SQLite on top of it. Also the Chromium devs have been working on relaxed durability, which apparently improves IDB perf in some scenarios [2] (although still not as fast as Firefox it seems [3]).
Six houses, all alike in dignity,
In fair IRC, where we lay our scene,
From ancient grudge to new mutiny,
Where civil blood makes civil hands unclean.
From Oracle, that SQL seer of IndexedDB,
To Google, the stronghold of search,
We add Mozilla, the Web SQL killa,
And Apple, peering from its mobile perch.
Here, a storage war would set keys to clack,
Tongues to wag, and specs to shatter,
There was also Microsoft and Opera,
Who don't really seem to matter.
THE PLAYERS
NIKUNJ MEHTA, of House ORACLE, an instigator
JONAS SICKING, of House MOZILLA, an assassin
MACIEJ STACHOWIAK, of House APPLE, a pugilist
IAN FETTE, of House GOOGLE, a pleader
CHARLES MCCATHIENEVILE, of House OPERA, a peacemaker
ACT 1
SCENE: A dark and gloomy day in Mountain View, or
perhaps a bright and cheery one, depending on your
IRC client's color scheme.
SQLite as a library is amazing, SQLite as a spec for a standard web API is not.
The issue is not just that there wasn't a competing implementation at the time, it is that you could not feasibly create a competing implementation. The set of features supported by SQLite is _massive_. And even you stick with SQLite, which version? And which extensions? Which features do you remove because they're not safe or not applicable for web?
Alternatively you could start from the ground up and create a new database spec that is similar to SQLite, but more limited in scope. But then it will not be directly compatible with SQLite, and need a translation layer. The browser would likely have to re-implement large parts of SQLite for this to work. And you'd miss out on so much of the functionality that makes SQLite attractive to use.
Arguably that would still have been better than IndexedDB. All that IndexedDB has going for it, is that it is simple to implement. But even then Safari manages to get the implementation horribly broken so often.
What we need instead is to have a low-level storage api that can be used to implement other databases as libraries, including SQLite. The Storage Foundation API, mentioned in the post, might just give us that.
>SQLite as a library is amazing, SQLite as a spec for a standard web API is not.
>The issue is not just that there wasn't a competing implementation at the time, it is that you could not feasibly create a competing implementation. The set of features supported by SQLite is _massive_. And even you stick with SQLite, which version? And which extensions? Which features do you remove because they're not safe or not applicable for web?
This seems like a case of perfect being the enemy of good.
I think the current state is fine. You ship your WASM-blob of SQLite, which has the exact bug-compatible version of SQLite that you've tested your app against. The browsers are not burdened with maintaining a huge API surface that can "break the web".
Otherwise you'd have to deal with different versions of SQLite in different browsers, most likely outdated, with many options turned off. SQLite is full of quirks and gotchas, so it's safest to ship your version in your app.
Would this not be solvable by just making it all arbitary WASM code? As long as the simple WASM sandbox is secure, it doesn't matter how buggy your internal SQLite implementation is.
This article only contains mostly information on adding more limits to avoid DoS—nothing there would harden WebSQL against SQLite vulnerabilities except for SQLITE_DBCONFIG_DEFENSIVE, which was only added in response to the zero-day bugs that were found in Chrome after implementing SQLite.
The justification was more that the standardization process requires at least 2 independent implementations, and nobody was working on rewriting SQLite from scratch.
I would have loved WebSQL, but it is reasonable to require multiple implementations for full standardization.
I don't know the reasoning being not doing it, but I'd guess it's related to those DB engines not being geared or interested in embedded use-cases and that no SQL engine seems to entirely agree with any other how to handle certain parts of SQL.
Indeed, it would have been a lot of work trimming down any of the major FOSS RDBMS towards embedding them in a browser.
Regarding the SQL dialect - that could have been handled by specifying a standard SQL dialect that's then dynamically translated to the target engine's dialetc.
I don't buy this reasoning because they could have standardized only the interface, letting it open for developers to choose SQLite or some other future implementation. There is no need to standardize SQLite behavior, including bugs.
What actually happens in that scenario is that one group now writes an adapter on top of SQLite to make it "standardized" to whatever interface you design, increasing the complexity and scope for bugs. To be clear: you're literally reimplementing something that SQLite already does, except probably much worse, in the name of the "standard."
Then that adapter is carbon copied by everyone, because again, nobody is going to reimplement an 80kLOC SQL database as well as whatever 10k LOC parser/lexer/typechecking adapter someone wrote, if they can avoid it. Then everyone just uses that forever, and you're back to square 1, using one implementation everywhere, which is the exact situation standards are supposed to avoid anyway.
The working group was correct to reject a "compromise" like that because that's never how it works out in practice; it's a submarine suggestion from the start. And a big part of this is all because, as evidenced by numerous responses in this thread, modern computer programmers seem to value their own immediate satisfaction and time over literally every other potential concern, no matter how significant.
That's not how the process works though. You could argue that they should change the process, but the reasoning behind it seems solid to me:
> Implementation experience is required to show that a specification is sufficiently clear, complete, and relevant to market needs, to ensure that independent interoperable implementations of each feature of the specification will be realized.
Although I do empathize with browers vendors. I worked at Mozilla at the time and was aware that this is a lot of things to think about when integrating something onto the web. I get why it happened, but practically speaking maybe it should have won. It's not like Chrome seems to care much about cross-browser standards these days.
It might actually be a better outcome if we get a storage layer with close to native performance, and then you can compile and db/lib/etc and it gets to use it.
I think offset based file access could be really powerful just based on what people are achieving in the browser with things like Flat buffers, proto buffers and even http range queries.
Building off what others have pointed out about sqlite not being a good choice for a browser standard, I just want to note that the approach laid out in the article is exactly the right approach to integrate sqlite into a web app: a web app links a version from a well-maintained “distro”. The issues inherent in trying to use an implementation as a specification and standard go away, while a “distro” maintained by experts will lower the bar to entry.
I think devs should be happy, not sad. It looks like we’re finally getting it right, and will have a feasible way to add the incredible sqlite to the set tools we have available to make web apps.
Hopefully absurd-sql keeps going and browsers adopt a good storage standard.
> It was torpedoed because it was SQL-based (and not trendy "key value" and "web scale").
This is almost certainly not even close to correct. There are substantial reasons why it wouldn't be a good idea, but this might be the biggest one: it's very hard to adequately sandbox an external C library.
(... and, also, Apple probably would prefer that the web didn't exist at all, but that's a different pandora's box...)
My "web scale" bit is snark. The "because it was SQL based" comes from the feeling I got at the time that Mozilla absolutely wouldn't accept anything was SQL-based. I strongly believe anti-SQL attitudes at Mozilla, and little else, are what killed WebSQL.
This is also how I justify my opinion that the "no independent implementations" was more of an afterthought excuse, and less of a primary motivation on Mozilla's part.
(I'm a little pressed on time here, but I could probably find more examples if I had more time to search... Sorry!)
Quoting Jonas Sicking from a W3C IRC log[0]:
> we've talked to a lot of developers
...
> the feedback we got is that we really don't want SQL
Quoting Maciej Stachowiak from the "public-webapps@w3.org" list[1]:
> Hixie has said before he's willing to fully spec the SQL dialect used by Web Database. But since Mozilla categorically refuses to implement the spec (apparently regardless of whether the SQL dialect is specified),
...
> At the face-to-face, Mozilla representatives said that most if not all of the developers they spoke to said they wanted "anything but SQL" in a storage solution.
The comments relating to JOIN in the comments on Mozilla's blog post comparing IndexedDB and WebSQL[2] betray the anti-SQL (and, arguably, anti-relational database) stance at Mozilla.
Mozilla's people didn't like SQL, so any excuse for dismissing WebSQL (or even a simplified SQL dialect that didn't have the "quirks" of SQLite) was a foregone conclusion.
And now here we are shipping >1MB transpiled WASM payloads around when we could have agreed on a query language feature set implemented in native code using a consistent back-end standard storage format.
I have no doubt that Jonas told the truth that that's the feedback he got. I was never aware of Jonas or anyone else involved at Mozilla having some kind of "anti-SQL" prejudice.
WebDatabase, with or without a spec, boiled down to shipping SQLite. We didn't want the Web to depend on that. It might even have required shipping a specific version of SQLite to make sure that query planning matches other browsers.
It seems that IndexedDB has turned out poorly for various reasons but one of those reasons is that Chrome has botched their implementation and that's not Mozilla's fault.
> WebDatabase, with or without a spec, boiled down to shipping SQLite. We didn't want the Web to depend on that.
Ironically, all the popular browsers already ship SQLite - they just not expose it directly. So, in a way, the Web already depends on SQLite. So does a lot of other technology - we're talking about one of the, if not the, most widely distributed pieces of software on the planet.
That's true, but "in a way" does a lot of work there. "In a way" the Web already depends on the C++ standard library, but we aren't exposing that as a JS API.
I don't think Jonas was dishonest in any way. I think Mozilla talked to front-end Javascript devs who came from a "NoSQL" background (after all, Javascript devs are what begat "web scale" MongoDB) and who had disdain for SQL. I think that disdain sealed the fate of any SQL-based API in HTML5, irrespective of the involvement of SQLlite.
The "no independent implementations" was certainly a convenient way to dismiss away the already-implemented SQLlite-based solution. It's a valid argument for rejecting SQLlite as the spec, for sure. Implementations notwithstanding it seems like Mozilla was unwilling to even consider anything SQL-based.
The same tone comes out in the "Beyond HTML5: Database APIs and the Road to IndexedDB" Mozilla blog post[0]:
> In order to really get Web SQL Database right, we’d have to first start with defining a meaningful subset of SQL for web applications. Why define a whole other language, when more elegant solutions exist within JavaScript itself?
...
> We were resolved that using strings representing SQL commands lacked the elegance of a "web native" JavaScript API ...
That sounds like the same rhetoric used in the early MongoDB / NoSQL craze days to dismiss RDMS, ACID, JOINs, etc. Never mind that SQL is a battle-hardened DSL that has proven itself to be suitable to the task-- it wasn't "elegant" enough for the front-end devs Mozilla surveyed.
As an aside, I also find the quote from that post very amusing:
> In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter.
The example for a JOIN using IndexedDB in the referenced article[1] looks strained. It really doesn't look like it has "syntactic simplicity" compared to the WebSQL version above it.
The sad irony is that if HN is anything to go by, SQLite is super trendy now.
I feel like "just sqlite" was really a very practical idea and now, because it has already failed once, we can‘t really try it again, or can we?
Having the spec basically be "bundle one particular piece of software into the browser" might not be in the spirit of web standards, but on the other hand SQLite is so widely bundled into everything, does it really matter?
> The sad irony is that if HN is anything to go by, SQLite is super trendy now.
If you look at the recent StackOverflow survey, the majority of the developers only have around 5 years of professional experience in the industry.
SQL has been around since the 1970s and is still around in force for reason. There's a good chance a lot of developers, especially the enormous number on the front end, do not have experience with SQL or are just getting into back-end work where they are exposed to it.
There have been a lot of SQLite articles recently, as well as node.js libraries to query SQL. I have a hunch this may be why.
Yeah I think so too. It‘s a cliche by now to say these things go in cycles but they do. If we like this part of the cycle, let‘s ride it and let SQL get hyped, even though its been around forever and never went away.
What gives you the confidence that it's going on circles? I cannot see our industry repeating the mantra that NoSQL solves all our persistence problems ever again. There are use cases for them, but not everything is a good fit and I'm pretty sure we found that out as an industry.
I'm sure there are always going to be individuals that will claim that they're better at everything, but that doesn't really mean anything. There are people that believe in a flat earth as well after all
The cycles don't happen exactly. But I'm pretty sure they will come up with something new that in the end is the same as NoSQL and give it a trendy name, so young developers will believe they found gold and run with it... It is just the way these things keep happening.
The cycles are longer than you're imagining. Relational database folk have been railing against the incursions of one generation of non-relational database type or another since at least the 1980's.
Actually, relational databases are the relative newcomers. The limitations of things like CODASYL were well known to 1960s programmers and Codd's relational model sought to address them.
Why the non-relational databases keep getting reinvented has been a bit of a mystery when there is already a rich history of development to look to. I get the feeling a lot of the industry isn't much into history, especially of pre-micro computer systems.
Network databases, key-value stores, graph databases, commercial offerings like Pick... you'd think the NoSQL people would have looked into it all before proclaiming their new found solutions, but apparently not.
Interesting, you wrote circles but OP wrote cycles, which can be circular, sinuslike, shaped like a sawtooth/ramp-wave (LISPs AI-winter anyone?) or even deformed variations with a bit of randomness or reinforcing patterns in them. Hypecycles and speculation bubbles come to mind.
This reminded me of the writings of a german Philsopher, Heidegger maybe? The proposition was, that development happens in cycles similar to a pendulum swinging while moving upwards. Thesis and Antithesis leading to Synthesis.
There's no such thing as "the SQLite" that you could even theoretically bundle.
Which version do you want to bundle? With which compile flags and which extensions? Do you have a checklist for bug-for-bug compatibility? Because you'll definitely need one when a future SQLite releases a security patch and breaks a million webpages.
same here. Firefox pulling out from websql is the reason I stopped from using it and recommending it. If the standard wasn't good enough, it should have been improved and refined, not thrown out.
first thought, was there a "standard" for key/value stores? mongodb the reference implementation?
overall, if you looked at HN like five years ago, every DB headline was key/value, mongodb, maybe some cassandra / couchdb, links to the "web scale" cartoon.
these days, it's SQL SQL SQL, with a heavy dose of SQLite and PostgreSQL. SQL survived the key/value fad despite the nebulousness of a workable "standard" (yes there's a SQL standard but no vendor DB implements all of it or doesn't add many many features, syntaxes, and behaviors on top of it). In particular SQLite recently seems to look to Postgresql for guidance on new syntaxes such as how it implemented upsert, it's RETURNING syntax is explicitly from PostgreSQL, and it interestingly uses the same "VACUUM" term for db utility cleanup.
I think sql is a bad fit for front end code in js. We may have ended up with another fight over ORMs. The Vietnam war of computer science as I think it was called.
I couldn't disagree any more, at least in my specific field, where my data is generally static and extremely relational. Writing JS to do what SQL does is... a pain. But having SQL on both the frontend and backend would dramatically simplify the translation abstractions between the two - for a similar reason that node on the backend simplifies translation abstractions. It ultimately means less code and less thinking.
How else would you suggest full applications run in the browser. Desktop apps make use of SQLite all the time even when they are mostly server backed.
An example is that Telegram maintains a local db which stores the most recently accessed content so you can scroll through past messages and media without having to request them every time you view them.
Desktop apps use SQLite because it's the embedded structured datastore that already exists, not because they want to use SQL. If it was easy to embed, IDK, some kind of file-backed redis, I'm pretty sure a lot of apps would use that instead.
> It was torpedoed because it was SQL-based (and not trendy "key value" and "web scale").
Whenever I need simple (but indexed) key-value (unless that's a hi-load server-side) I always just use SQLite anyway. I really don't understand why do we need any data storage other than SQLite (and HDF5 perhaps) to exist on the client side.
Another, probably not insurmountable, issue with SQLite in the browser (without having a formal specification that could produce cross-language alternatives):
Using SQLite results in a new, built-in reliance on a C library. Even though most languages can use C library bindings, it does present some issues in some build scenarios, such as static builds, and, given the enormous scope of SQLite, would prevent anyone from ever achieving a 100% rust browser.
James is one of the world's great techno-adventurers, & getting to para-socially share in wild adventures like this makes living on Spaceship Earth more lovely & lively! James has also done cool projects like sweet.js macros, helped kick off Firefox devtool's transition to react (iirc), oh and lead the basically industry standard JS formatter Priettier project. I'm forgetting a dozen other things over the years but it's always been fun.
Just a heads-up, the File System Access API[1] is underway in Chrome, which potentially removes nearly all of the absurdity here. It has other benefits too. A web page using this could write a .sql file on to your drive, that other programs could then access. One of the other bright stars in my world is Karli Koss, who has an extensive personal data-extraction setup for a ridiculously colossal variety of services & devices[2]. A vast amount of this massive massive data-gathering framework is just reading sqlite databases of the various devices and apps. If the web can help participate more actively, can let apps write sql files to store state: so much the better I say. Help externalize your state beyond the browser, please!
Several months ago I've made a proof-of-concept of exactly what you're talking about, feel free to check it out: https://shekhirin.com/sqlite-fs/.
I recommend downloading sample DB, writing some dummy query like "SELECT BILLINGCOUNTRY, COUNT(INVOICEID) FROM INVOICE GROUP BY 1 ORDER BY 2 DESC" and then pressing Execute.
I've been planning to write an extensive article about it and open sourcing the solution cleaning up the code a little bit, but still haven't got much time to do so.
now let's see what it takes to make absurd-fs, where we use https://github.com/guardianproject/libsqlfs to make a filesystem on top of sqlite on top of the File System Access API.
gotta keep ourselves fully looped! ⥀
(is there perchance a repo available with your work? that'd be lovely to see.)
> SQLite, even though it’s implemented on top of IndexedDB, easily beats out IndexedDB in every single performance metric. The absurdity!
This really is quite incredible. Same idea extends to your filesystem too. Tracking millions of 1KB objects on disk? You could load the whole set into memory substantially faster from SQLite using the same disk. If WAL is enabled with reasonable sync flags, the same applies going back out to disk as well.
SQLite is the most powerful dependency that our product uses today. We have been using it in production as the sole persistence mechanism for 100% of our data for the last 5-6 years now. Recently, we have started leveraging the actual SQL engine to process all of our business logic as well.
There are two pieces of tech that punch way above their weight: Lua and SQLite
So many apps could have just saved to a .db file instead of coming up with their own binary format. So many others could have just embedded a tiny, easily sandboxed, Lua library and added scripting languages, rather than embedding larger ones (or worse yet, implementing their own scripts).
sql.js is pretty hard to use as is otherwise you run out of memory really quickly. I was trying to use it as the in-memory SQL flavor for an open source data ide [0] but my naive approach of `SELECT * FROM VALUES (...), ...` would run out of memory after only a few hundred rows.
I ended up switching to https://github.com/agershun/alasql which could handle up to 80MB of data or so. (I haven't yet tested on larger datasets so I don't know the actual limits.)
I don't think this is a fundamental limitation of sql.js as the linked article proves that you can implement custom paging for sql.js. But unless you do that (which I haven't spent the time to figure out how to do) then sql.js will run out of memory very quickly.
Just something to be aware of if you're investigating it.
If there's a high-level library that makes more effective use of memory with sql.js under the hood let me know.
Unlike absurd-sql I don't need the results to be permanent. I just wanted an in-memory SQL for joining, filtering, grouping data.
Fascinating. I'm really curious what the use case is that so many people seem to have. Why do you need so much data in the browser, and to be doing queries and data manipulation there? Where does the data come from? Don't you need to sync it back to a server somewhere?
It can be used as a better form of cache to make actions happen faster. So an instant messaging app for example can cache messages and stuff in a db so it does not have to refetch everything every time you switch chats.
This lets you write any Serious App with "single-player data" as offline-first (though yes you still need to handle syncing to the cloud somehow – jlongster has done some very cool stuff for that too, looking forward to him sharing more about that).
This might be useful for a desktop-app like experience on the web. Imagine something like excel but you want to open a 100mb file and work with it right away. It can sync to the server as you are working but you just want to get working now.
Another use is privacy centric apps that send nothing to the server, using the web as a kind of “install” platform but nothing else.
Yeah, Jake made idb not too long after I made small-indexeddb. I think it's one of the most reasonable options (and it has TypeScript types!), but it's still about 5x as much code as small-indexeddb.
I would point out that the article you linked mentions that webapps "added to the home screen" on iOS Mobile Safari are exempt from this garbage-collection process. In such a case, the "add to the home screen" step is being taken as the user expressing the explicit desire to have that data around indefinitely. Which seems sensible to me; webapps should be treated as ephemeral-by-default, and only be allowed persistent storage if the user goes to lengths (i.e. beyond just clicking "Accept" on a modal) to express their desire to keep the data.
(And also, I've observed that webapps that notice they're on an iOS device can insist on being "added to the home screen" before they'll do anything, and so ensure their data stays around.)
The problematic aspect of this, is that there's no equivalent of this "exemption by user explicitly expressing the desire to keep the app around" for non-mobile Safari.
Maybe for desktop Safari, the data should be kept around if the web app's rel="canonical" URL is bookmarked in the browser?
Or desktop Safari could just support regular Progressive Web App standards, and so show an "Install App" indicator (like e.g. desktop Chrome does for this webapp: https://www.soundslice.com/). But I have a feeling Apple will never support this on desktop...
Google may have coined the term (not sure about this), but it's far from their own thing [1]. PWA should've been a blanket term for a set of standards and guidelines for developing web apps. Those include progressive enhancement, which I don't think most people would expect.
Unfortunately, the term has been co-opted to mean "website I can install/pin as an app". Again, Google is probably to blame for this, but AFAICT it was never meant to be the meaning of the term. What it does do is create misunderstandings, like a sibling thread claiming that (desktop) Firefox doesn't support PWAs because you can't install anything.
In general, the "Progressive" in "Progressive Web App" just means "progressively-enhanced." In conventional usage around most of the internet, any web app that supports any kind of progressive enhancement, is a PWA.
And "progressive enhancement" just means "being written in such a way that you'll work with a core set of APIs — all the way down to 'no javascript support' — but will take advantage of further APIs if they're available." Where the app forms a sort of Russian nesting doll of functionality, where it "launches" with just the innermost layer (e.g. just server-rendered HTML), and then in turn each new layer is enabled if possible (i.e. if Javascript is enabled, the browser will load some JS from a <script> tag that will then take over the DOM and turns the page into a SPA; etc.)
It's just a further refinement of the older concept of "graceful degradation", where the gracefully-degraded experience is delivered at the start, and then bootstrapped out into the non-degraded version.
Under this definition, the "further layers of functionality" provided to "installed" webapps in some browsers are just a specific kind of progressive enhancement. The app detects that it's being run "installed", and takes advantage of that. (And before that, the browser detects that the app has such a layer, and therefore treats the webapp as "installable.")
Thus, all "installable" apps are PWAs. (But not all PWAs are "installable"!)
I would note that webapps like Slack — i.e. webapps that have both web and "native" versions, where the "native" version is just the same remote webapp from the web version, loaded in a special browser that exposes extra capabilities that the webapp then takes advantage of — are also PWAs by this definition.
I wish these bookmarks extended to other browsers on iOS. It's already crazy they can't ship custom rendering engines, but basically this in and of itself could be considered anticompetitive (only Safari bookmarks can be exempted from this GC process) lol!
Apple did plan to make web apps first class on iPhones. But changed their minds when developers complained about not getting hard metal access. Maybe they thought that supporting web apps would get them more apps, but developers gave them apps for free and it allowed Apple to have a monopoly. Microsoft tried the same with their phones but somehow failed... Google went the web app route on ChromeOS with decent success. Mozilla tried with FirefoxOS but was too early - FirefoxOS is now very popular but under another name on low end hardware phones.
PWA are explicitly exempt from the data storage wipe behavior[1]. So there's that, at least. For now.
Although I agree that it's a fairly minor win in the grand scheme of how handicapped PWA's are on iOS. And like you said, the lack of install prompts and tucking away the PWA "installation" option in the share menu makes it less than intuitive and requires manual awareness efforts by devs[2].
That’s only half true. PWA are exempt from data storage wipe behaviour only if they have been added to the Home Screen. And as your second article points out:
> It’s important to ensure that our iOS visitor is using Safari because iOS doesn’t permit other browsers to install our awesome PWA’s!
So if the user is using Firefox or Chrome etc browser, they can’t do it. And since all browsers have to use WKWebView, the limitation applies to all browsers. Apple has basically crippled other browsers with this anti competitive behaviour.
But it's totally a valid option to have an app that is a web app and does not have push notifications. For the vast majority of apps regardless of platform, I absolutely do not want push notifications. It seems bizarre to consider that feature as a litmus test for whether the web is a usable platform on iPhones. Would you also require web apps to appear in the App Store in order to consider a web as a usable platform on iPhones?
That’s a fair point, but I’d argue that for cases where the app really needs notifications (for example an alerting system) or when the users want them, building a web app is not actually an alternative even if Apple considers it one.
I don't think Apple considers the web as an alternative platform for apps that are designed to alert you at any time of day for any reason, including when you are not actively using your devices. I think this is a very, very good thing, and doesn't mean that web apps are not a viable alternative to the App Store.
Clearing site data makes total sense. Otherwise after a year of web usage you have multiple GB of stored website data that the user does not know what it is being used for or how to clear it.
And the majority of uses for local data is to cache fetched data or store temporary preferences not worthy of being in the server side database.
I mean, he's correct in the same sense that a Buick can't be made to haul a fifth wheel trailer. It's because a modern Buick's design doesn't include the bed required to install the fifth wheel on.
Nevermind that there's nothing strictly stopping GM from changing the designs for Buicks to include a vehicle capable of a fifth wheel mount.
Clearly the solution to this is to keep a query log in an extra table and periodically stream that to the server as a form of logical replication (plus perhaps being able to load the initial database state from the server side as well, maybe even on-demand using the GH pages trickery until a write forces materialisation into IndexedDB).
As a bonus point this effectively adds yet another level of "Yo, Dawg" which I can't not love just as a matter of principle.
I wish every browser does that! Web site has no business storing data on my computer more than 4093 bytes (that is already too much) per domain in my computer just because I visited a web page. 10 MB - 10GB of data is too much.
I think that's a valid concern. Stuff like this should only be allowed after a prompt where the user explicitly allows it, just like any other permission
I would say it’s even more important in the case of a web app that stores data locally, because it’s imperative that the user understands that data is tied to that particular web browser and device, and that clearing site data (which I’ve known non-technical users to do periodically to reduce tracking footprint), changing devices, etc can/will result in data loss.
Oddly enough the handful of web apps I’ve known to use local storage don’t communicate this at all. Developers seemingly treat data in browser local storage as existing indefinitely and unlikely to be deleted, when in fact it’s probably the most easily accidentally lost form of storage an app can have.
If you're concerned about a web site silently having access to personal data from your phone, surely it doesn't matter whether that web site is storing it silently in your device's browser storage, or storing it on the web site's servers.
If, on the other hand, you're explicitly uploading or entering data on a web site, well then yes, the default assumption would certainly be that the data is getting saved on the web site's servers (if it's being saved at all). That's how the web has worked for a very long time, with the exception of cookies and a few other more recent but not widely known storage mechanisms which in most cases aren't considered sufficiently robust for persistent on-device storage.
So that is why a website I work on has complaints of Apple users who keep getting nagged by cookie banners - they store their settings in LocalStorage since cookies tend to have issues when the content is too long. Thank you so very much.
This is why I haven't used LocalStorage or any other browser storage solution for a long time. They seem really cool, but if they will just get wiped by some browsers, then what is the point? As of a couple of years ago, localStorage didn't work at all in a private windows in safari. Unless you have some very specific needs, I don't know how you would every use IndexedDB
Hmm... I was in the middle of rewriting an application of mine from JSON stringify into localStorage to IndexedDB, but was having issues with the API being so clunky. This is a tempting alternative. It does increase size from ~200kb by a whole mb, but the app's usage patterns are such that people open it and then use it for extended periods of time in the background.
Honestly it's a hobby project at the moment. Mac users can just use Chrome or Firefox. It's not worth my money to get the electron build running for mac, or my time to work around safari bugs.
The design is very much a 00s throwback with a lot of density and power compared to more modern apps in the same category, so Mac users are unlikely to like it anyway. I was just tired of the older apps lacking modern features I like while the apps with the modern features were clearly mobile/touch first and so much slower for bulk operations than their older competitors as a consequence.
the only thing I don't like is that SQL is a second class "stringified" citizen in this world (and often in any environment where you want to use SQL). It's missing all the advantages of syntax checking and dynamically building queries. In my C# projects, I tend to always work with SQL files which then get embedded into C# so I can always query against a DB and build queries more REPL like.
Having said that, I do like the idea of Sqlite in the front end for localstorage.
So, why is IndexedDB so slow on Chrome? Obviously LevelDB doesn't need 10ms for a point read. If it did, nobody would use it for anything. 10ms is a hell of a long time. Is it spawning a process to perform the read or ??
I was profiling on an older computer. On my newer one, summing 100 items takes ~8ms (use the raw idb mode). When I said "simple operations" I meant simple queries that you'd expect apps to write, not just 1 single read/write. It is a little faster for each read/write, but there seems to be a bottom floor. Even if reading an item itself is fast, opening a transaction is slow. So any query, even if it only reads one item, is going to suffer the perf hit of opening a transaction.
It's only twice as fast as Firefox, so overall IDB is still super slow when compared to running the same queries with native SQLite. We're talking summing 100 items taking ~.01ms or less. I have no idea why it's so slow.
This is great to see, and a project I considered attempting myself for a bit. I’m excited to test it out.
@jlongster I have a question about this:
> The backend calls it [Atomics.wait] to wait on the result from the worker and blocks until it’s done.
Does this mean the main (UI) thread is blocked during queries? Or are there more threads, like UI <- async messages -> SQLite main <- Atomics blocking -> SQLite FS backend?
————
At Notion, we’ve used IndexedDB for two purposes: (1) to durably persist a queue of changes to send to our backend, and (2) in the desktop app, to LRU cache the page data we read from the server to accelerate reads. Both of these used localStorage years ago, but we ported to IndexedDB because of data loss on localStorage. Porting was fine for the write queue, but we really noticed the slow when we tried porting the data cache. To get close to the original performance we coalesce reads, and we delay writes to the cache significantly so they can batch more effectively into a single readwrite transaction that we send after the reads for the current page load are complete.
That worked okay, but it was annoying to maintain the IDB cache code because our Android and iOS apps used SQLite for their caches, and it’s so much easier to add new queries using SQL compared to writing IDB iterations - and it’s faster. So we switched to using native SQLite via a bridge to a Node process. Now with absurd-sql, maybe we could bring the same caching logic to browsers.
The thing stopping me is how unreliable we’ve found IndexedDB to be - aside from the optimization work. We notice a lot of bugs in IDB implementations on different browsers. In Safari (especially on iOS) there’s a bunch of spooky issues that have caused stalls or spurious errors, sometimes requiring an app restart before the IDB database can be re-opened. Forget it on Android - weird vendor webview patches mean your storage might get cleared out from under you. On Firefox, we notice that sometimes the IndexedDB database doesn’t create all the object stores we request for some reason. Even on Chrome, IndexedDB can suddenly start refusing writes in the middle of a session with no clear explanation, and on Windows restarting the computer is sometimes the only fix.
If we can share SQLite queries with our native apps then maybe it’s worth wading deeper into these issues… but it really does feel like building on quicksand.
> Does this mean the main (UI) thread is blocked during queries? Or are there more threads, like UI <- async messages -> SQLite main <- Atomics blocking -> SQLite FS backend?
The latter! Your app running queries must be on a worker, and then the IDB backend will spawn another worker. `Atomics.wait` is not even available on the main thread.
Ideally in the future, there will be a better storage API that we don't even need all the Atomics silly-ness (hopefully it provides Sync methods)
That's really cool re: Notion! That's exactly the kind of thing I want too: a way to just build apps the same way everywhere, on mobile/desktop/web.
You are right about various issues, and I personally don't have to worry much about it on my app because I have native mobile apps and I don't support the web version on mobile. I intentionally do that -- the mobile web is just too broken in too many ways. My impression is the IDB is more stable on desktop, but because mobile is more memory sensitive there are more issues there.
However, you should try it out! I definitely discovered a lot of weird things; I definitely was able to get Safari into a weird state the required a complete app restart. Here's the thing though: I found ways around them. If you do a lot of read requests in a certain way, Safari will lock up permanently. However, if you make sure to wait until the `readonly` transaction is finished before starting a new one, the problem goes away. I was able to reliably reproduce that problem and it went away with that fix.
I think absurd-sql is so promising because it normalizes the patterns of how IDB is accessed, and it already includes fixes for a bunch of edge cases. There are probably more, but try it out! If you run into an edge case, we can tweak the IDB backend until it works. We can paper over these issues in the underlying backend and you don't have to worry about it because you aren't directly managing IDB read/writes.
While in-memory databases have their uses, it kneecaps
SQLite into something far less useful. To build any
kind of app with it, we need the ability to write and
persist.
Another approach than writing the data to a server could be to allow the user to store it on their own hard disk.
This could be done via the File System Access API:
Did you read the post? This project does exactly that. (but focuses in IndexedDB for now because it's the only cross-browser thing that works. I actually tried a webkitFileSystem backend and it was slower)
Well, I woudln't call using IndexedDB "exactly that". As IndexedDB is rather fleeting. You don't use a server, that is correct. But IndexedDB goes away under many circumstances. Saving a file via the File System Access API would give the user peace of mind that it is safe. I did not see any mention of the File System Access API in your post.
Hey, can you please not do this ("Did you read the post?", "Read harder", etc.), even when someone else hasn't read an article? I understand how frustrating it can be when people don't read what you write very closely (believe me, I understand), but it's one of the tropes that degrade discussion and we're trying to avoid sinking to that level here.
"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that.""
I'm going to build a business that offers SQLite as a web service. It will be backed by a P2P network of browser instances storing data in IndexedDB. Taking investment now.
Surely what the world really needs is a new, faster implementation of IndexedDB? I propose writing it on top of this sqlite implementation, so we get the full indexedDB on sqlite on indexedDB on sqlite experience.
> Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
> GRT that is staked in the protocol is subject to a thawing period and can be slashed if Indexers are malicious and serve incorrect data to applications or if they index incorrectly. Indexers can also be delegated stake from Delegators, to contribute to the network.
> Indexers select subgraphs to index based on the subgraph’s curation signal, where Curators stake GRT in order to indicate which subgraphs are high-quality and should be prioritized. Consumers (eg. applications) can also set parameters for which Indexers process queries for their subgraphs and set preferences for query fee pricing.
It's Ethereum though, so it's LevelDB, not SQLite on IndexedDB on SQLite.
To be clear: I was being sarcastic. But if I were to inject our comment chain into <insert random crypto appreciation thread here> it'd fit right in, proving your point that the level of silliness is getting to Monty Python levels. We need The Colonel to barge onto the stage and shut it down I feel.
You can’t trust localStorage anyways - it will silently drop some writes in Chrome and Firefox. Some years ago, we used localStorage for queuing writes to the backend and found that concurrent access to localStorage caused data loss - some transactions never came back out of localStorage. One of our engineers wrote a stress test and confirmed the issue. We switched to IndexedDB after that.
If I understand this correctly the db would be on the client/browser. Thus any persistence would happen via local storage or such. But:
1. People often reset the cache on their browser, after all it is just a cache. SO a big benefit of databases which is persistent data is kind of not there. Juts rest your cache.
2. The second great benefit of databases is that they are multi-user. There can be in fact millions of users. But this benefit would not be there if the database lives and executes in the browser.
I'd like to add one drawback with this solution: Complexity.
From my experience, complexity can easily lead to more problems than what it solves. With all the "blackboxes", e.g., WASM, JS, SQLite, IndexDB, ..., it might be hard to find bugs. Most of the tools used are somewhat stable and mature tough. SQLite, for example, has a whooping 100% test coverage (line coverage at least).
"In all browsers except Chrome, IndexedDB is implemented using SQLite". That's a strange way to phrase the status quo. That is Firefox and ... Opera? While Chrome includes Edge.
i wonder if it's possible to plug any kind of streaming replication onto this. i don't have much sqlite experience, but maybe someone here has an idea if it would be possible to run litestream or something of the sort, as both master and slave - in the browser.
that would solve the safari indexeddb 7 day ttl issue to start with.
and if replication could be made to work on top of something like webrtc we're looking at a great foundation to start building distributed, decentralized browser apps.
i must say that the experience is quite horrible - that torture of having to write map/reduce functions, added with some erratic behavior in regards to data integrity (inserted entries silently discarded, sync to the remote couchdb instance working somewhat whimsically). as soon as your dataset is sizable in any regard (tens of thousands of records in a collection, if i recall the terminology) it begins to just break apart.
was writing a browser extension, and used pouch with the hope of keeping its persistence local and avoid needing a server. seeing that it leaks tried to trade it for a couchdb server. seeing how bad sync is, and that couch is not very comfortable to work with either ended up throwing the thing in favor of a postgresql+postgrest backend.
No it's not, it has a terrible API. Not sure why people here are still defending that mess imposed by Mozilla when developers could have had something better, websql.
Another spec that failed from a practical perspective because of Mozilla's reluctance to implement key aspects of it is web components.
I just include a list of migration files in the app, iterate through them in startup and make sure they are all applied. It's pretty simple, but yeah you have to think about this if doing local apps
Yes, and it literally says "SQL on the web" and "SQLite on the web". Are we really confusing "the web" with "the browser" now, especially since we use "the browser" to implement apps that don't have a need to access "the web"?
"If you are writing a web app today, you’ll probably choose IndexedDB to store data. It’s the only option for something database-like that works across all browsers."
It was torpedoed because it was SQL-based (and not trendy "key value" and "web scale").
There was the whole excuse that the specification was "whatever SQLite does" and, therefore, not suitable for being a standard. There would be worse things than SQLite upon which to base a standard, all things considered. I still believe it was torpedoed because of lack of trendiness and "not invented here".
[0] https://www.w3.org/TR/webdatabase/