> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF file
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
Yeah... Original and clever, and a great read! But it seems mostly useful to the handful of people dealing with ELF internals, than to the vast majority of people executing ELF blissfully ignorant of its internals... Maybe if the latency and size trade-offs were the other way around it would be more appealing to the masses.
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
I didn't have this idea exactly but I did play around with sqlite as an embedded database for packing ruby apps into a single file a while ago. It was less direct than this: you basically compiled all the dependencies (including .so's) into a fake filesystem that an overridden `require` loaded from, which was unpacked to a `:memory:` database at runtime from a `.data` section that got bolted onto the interpreter. It died when 1.8->1
9 changed how the build system works and I never got round to updating.
A version based on this which carries around an overlay filesystem would be comparatively straightforward, the hard work is already done.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
Yes. In this interview Michael Stonebreaker, one of the original developers of PostgreSQL, discusses (among other topics) replacing the file system with a database :
(Even with ELF, a kernel can have a relatively simple parser and loader and leave more challenging work to user space... like relocations, dynamic libraries, etc.)
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
I’m wondering if the anti-clickbait filter could benefit from some lightweight LLM integration. E.g. when the submission hits the front page, if the title was “de-clickbaited” and wasn’t changed manually yet, run it through a cheap model and see if that was the right call.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
reply