Back then I was struggling very hard to get the low level details of the machine under control (Pi back then and some experiments with x86 BIOS stuff). It was super fun to have this ham-fisted brutalist JIT actually do something real and to connect to IRC on this mockery of a TCP stack and to make some VC4 GL triangles appear out of a bunch of commandstream arrays.
In the meantime I've built an FPGA graphics card for Amiga (VA2000) and an open laptop, the MNT Reform, whose beta units will ship very soon now (TM). A while ago I also worked with some really smart software engineers as a startup CTO for 1.5 years and all of these things together have given me a lot more perspective and depth compared to back then.
It will be fun to return to this and clean it up as a demo for Reform.
Need to start saving up now for when it goes live. Hopefully, my puny student wages can handle the cost.
Anyway, just wanted to express my enthusiasm for your project!!
That sounds pretty interesting! Every so often I think about the things I don’t enjoy when using computers and how I’d solve it (I’ve played around with hobbyist kernel/is development on occasion too, including contributing to Daeken’s RenrakuOS ten years ago..) but my motivation always drains when I think about how it would never be realistic to get hardware accelerated graphics working. Just the other day I was thinking that maybe my best bet is a minimal Linux system that boots into a full screen OpenGL application that basically pretends to be my OS. I even thought about doing it on a RPi3 since I have a few of them sitting around and it seems like a nice self-contained, expandable and cheap hardware platform to build on. Your projects sound like some of the things I’ve dreamed about but never actually did. :)
Precisely! When Lisp Machine programmer look at a screen dump, they see a lot more going on behind the scenes than meets the eye.
I'll attempt to explain the deep implications of what the article said about "Everything on the screen is an object, mouse-sensitive and reusable":
There's a legendary story about Gyro hacking away on a Lisp Machine, when he accidentally trashed the function cell of an important primitive like AREF (or something like that -- I can't remember the details -- do you, Scott? Or does Devon just make this stuff up? ;), and that totally crashed the operating system.
It dumped him into a "cold load stream" where he could poke around at the memory image, so he clamored around the display list, a graph of live objects (currently in suspended animation) behind the windows on the screen, and found an instance where the original value of the function pointer had been printed out in hex (which of course was a numeric object that let you click up a menu to change its presentation, etc).
He grabbed the value of the function pointer out of that numeric object, poked it back into the function cell where it belonged, pressed the "Please proceed, Governor" button, and was immediately back up and running where he left off before the crash, like nothing had ever happened!
Here's another example of someone pulling themselves back up by their bootstraps without actually cold rebooting, thanks to the real time help of the networked Lisp Machine user community:
Also eudox posted this link:
Related: A huge collections of images showing Symbolics UI and the software written for it:
The idea of flat text files sounds odd, when you can have structured S-exp's.
Now consider that anything that consumes a seekable byte-buffer as its backing store could just-as-well be modified to consume a block device as its backing store.
Voila: your OS now has “durable memory” in place of a filesystem.
(And, in fact, in modern systems you can skip the whole block-device layer and just sit your tuple store directly on top of NVMe.)
A filesystem is effectively a hierarchical model database with a very limited feature set (usually no transactions, minimal or no schema enforcement, very limited query language.)
Why not add some of those features?
Filesystem transactions: Windows supports them, but Microsoft has deprecated them; part of the reason, I think, is transactions required use of a different API to access files, which meant almost nobody used them; if they had been accessible through the same filesystem API as non-transactional operations, they might have seen more adoption.
Schema enforcement: I could create a directory called "images", and then specify that all files in images have to be of MIME type image/*, and the FS will refuse to let me put a text file or executable in there. I could have a directory called "logs", and require all file names in that directory to have names containing a valid date/timestamp. I could enforce the rule that a .json file has to contain well-formed JSON, or a .xml file must contain well-formed XML.
Querying: filesystem could support SQL queries over file extended attributes. (It doesn't have to be SQL, SQL syntax is pretty ugly.) "Find all files of MIME type image/png which are greater than 100KB in size?" "How many executable files are there? How much space do they consume?"
yeah, that's what a lot of people wondered in the 70s too and it failed spectacularly, and failed again spectacularly in the 2000s with microsoft's WinFS.
As far as I'm aware, the main difference is support for record oriented files, but the naming and lookup isn't so different.
But you are right, there is nothing especially database-oriented about file lookup and naming on IBM mainframes. Record-oriented files and key-sequenced VSAM files (and once upon a time ISAM too) are database-oriented features, but they relate to file contents not file naming/lookup/etc.
I think the idea of catalogs is interesting, in that they permit a separation between the naming of datasets and the volumes they are stored upon. A dataset can be moved to another volume without changing the name used to access it. That is arguably more complex on Unix-like systems, since you need to muck around with symlinks or bind mounts to get the same effect.
And documents themselves can be databases (e.g. SQLite3 databases) in a way that's understood by this same database-like system, such that there can be a vertical integration between changes to "documents" that happen through Desktop-Environment-level APIs, and events elsewhere in the system.
(For example: the interaction in macOS/iOS between Core Data "documents" and iCloud. If you have a KV-store-typed Core Data "document", then iCloud backs it with a KV-store on its backend, and syncs changes between your KV-store and the cloud KV-store keywise, rather than just re-uploading the entire document whenever it changes.)
From a userland perspective, it's a bit hard to understand why these abstractions haven't been "pushed down the stack" into the base system, such that even POSIX utilities can see and deal with "documents" as a whole.
But from a systems-development perspective, I think I understand why it hasn't happened—modern OSes are developed from the inside out, in a way where development on the kernel or the base-system usually involves stripping away all the complexities of the higher level. A good example of this layering is Android: there's the Linux base-system, and then the Android runtime on top.
People who need to develop new kernel extensions for Android devices, need to work on them "at a Linux level", rather than trying to figure out what's going on by peering into the Linux part of the stack from the Android-runtime part of the stack. When they're doing that, none of the Android niceties are available. If the whole filesystem was vertically integrated such that you didn't have one before the Android runtime successfully booted, that'd be really annoying in this case. A filesystem is very useful for doing this kind of development. (Picture if Linux didn't have the initramfs abstraction—if, before the real root filesystem was mounted, there was just no filesystem at all, and so no POSIX userland utilities to rely on. That'd make developing a boot process much harder.)
Personally, I think user documents shouldn't be a kernel concept -- it should be a protocol implemented by user-space system services. This would let any app expose its storage as "documents" without having to change its internal representation.
In fact, the only successful implementation of the concept (even if only partially) I've seen is the object-oriented storage on IBM i (formerly i5/OS and OS/400).
Now, you can call this a "database", although it is basically a flat-file database (ISAM/VSAM). But then, many mainframe/minicomputer relational database products use this as their underlying storage layer, and put SQL on top of it. But the SQL access is an application layer, not part of the actual OS-level filesystem. In some cases, you can have legacy apps directly accessing the files via ISAM/VSAM, bypassing the relational layer, while having newer apps going through the SQL interface instead to read/write the very same files.
Where OS/400 / IBM i makes this picture a bit more complex, is the relational layer is shipped as part of the OS, not as a separate product. What I can't work out, is how deep the integration actually is. Is it just like a relational database bundled with the OS (like how many Linux distros bundle MySQL and Postgres?) Or does it reach deep down into the OS kernel? There is not much technical info available on OS/400 internals, and I wonder if IBM's marketing/evangelism makes the integration sound deeper than it really is?
But, the idea I had was you can use SQL queries to locate ordinary files. Imagine if I had a view VW_FILES which contains metadata (name/directory/size/etc) of every file in my filesystem, and I could query it with SQL. I don't believe OS/400 / IBM i offers that kind of feature. It just allows SQL queries of the contents of database files.
I ended up reading OS/400 Redbooks years later.
However I remember that all file related activities were done via the catalog management tools, files weren't visible as such.
That being said, both share some similar design choices in data storage, among other things.
Even aliases aren't well thought out, at least on Windows. No automatic updating if the name changes, no "find folders that contain an alias to this one", etc.
Don't get me started on versioning. Apple's Time Machine is a decent interface, but it's solving a different sort of problem.
Each will have their own pros and cons but it’s fair to say there are a wealth of options available.
The other thing is that the model has been the basis for software's approach, in a way that has too much inertia. For example, Word documents that are drafts are saved as individual files. Why? If the filesystem enabled it, programs could save deltas and have an interface to freeze a draft.
You'd have to train people how to use versioning even if versioning was baked into the file system itself.
> The other thing is that the model has been the basis for software's approach, in a way that has too much inertia. For example, Word documents that are drafts are saved as individual files. Why? If the filesystem enabled it, programs could save deltas and have an interface to freeze a draft.
Those temporary files are there as a volatile backup of the running state of the program. It's not really the same thing as a "draft" so shouldn't be treated the same. That is unless you're literally talking about unsaved documents in which case I'm not really sure how you expect the file system to predict what you want to do with a file that even Word - you're exampled running application - doesn't yet know what the user intends.
So what you're asking for there doesn't really make a whole lot of sense. That is unless you're trying to describe an auto-save feature - in which case Word does have literally that feature already bundled as part of it's default set up. Of course you do still need to save the file once to at least nominate where to auto-save to. But saving your drafts is good practice anyway (particularly on Windows).
Vista/Longhorn was also supposed to feature WinFS, which was integrating NTFS with an embedded version of SQL Server, so you could do SQL queries to find files on your filesystem. This was pulled from Vista before release, because it wasn't ready. Some of the enhancements to SQL Server developed as part of the project were released as part of SQL Server 2008 (e.g. FILESTREAM data type), but the core idea was abandoned.
That wasn't the first time Microsoft had tried something like that. In the 1990s, their Cairo project had an object-based filesystem (OFS). That got abandoned as well, although just like WinFS, some of the technologies developed for it actually got released and used. (e.g. COM Structured Storage, which was used as the basis of the file format in older version of Office.)
Journaling file systems all have transactions, do you want to run multiple actions in the same transaction? that'd be neat.
> Schema enforcement
How would the database know that a blob of bytes is an image or an xml file? Will you need to implement heuristics for every possible file type, or do users establish the type of the file? if the later, what's the point?
unix `find` can do that. It's also pretty ugly
What I want is a transaction in the sense of "these actions should all succeed or all fail, and any observer should only see the initial or final state, never any intermediate states produced by the actions". The lack of such a mechanism feature leads to a lot of bugs and security critical race conditions.
For example let's say I want to do rename /var/logs and create a new /var/logs that should be used from then on. Normally I would do something like 'mv /var/logs /var/logs.old && mkdir /var/logs' (or the equivalent in my language's API). But what if the first command succeeds and the second fails? What if somebody executes 'ln /etc /var/logs' after the execution of the first and before the execution of the second command?
> unix `find` can do that. It's also pretty ugly
for windows, Everything  is great. It builds and maintains an index of all filenames (and some file attributes) and searches as fast as you can type. Great for finding everything if you know any part of its filename. Such a capability is the norm for databases, but rare for file systems.
It's part of a the standard install of various mainstream distros.
To find all cpp files, just:
"locate was first created in 1982. The BSD and GNU Findutils versions derive from the original implementation."
This said you couldn't have multiple operations in flight at once for the same dataset.
Sadly time commitments got in the way so I’d only gotten as far as building the read interfaces (in effect it only got as far as being a read only file system).
Sounds pretty much like a filesystem to me. Isn't this basically what Linux already does?
If an OS is an exclusive environment for a single language, then that language's abstractions for handling data could be pushed much deeper than we're used to.
I would expect a bit more Lisp in an OS that is supposed to be inspired by Lisp Machines.
It's like Dr. Bronner's soap: Dilute! Dilute!