
Programmer's critique of missing structure of operating systems - Bystroushaak
http://blog.rfox.eu/en/Programmer_s_critique_of_missing_structure_of_oper.html
======
GuB-42
This is mostly a critique of UNIX, but several of these concepts are already
implemented in other OSes, either in production or experimentally.

The database for a filesystem is an classic. WinFS, the filesystem that should
have been a key feature of Longhorn/Windows Vista is based on a relational
database.

The "death of text configuration files" is the idea behind the Windows
registry.

Powershell (Windows, again) is based on structured data rather then text
streams.

For the "programs as a collection of addressable code blocks", when we think
about it, we are almost there. An ELF executable for instance is not just a
blob that is loaded in memory. It is a collection of blocks with instructions
on how to load them, and it usually involves addressing other blocks in other
ELF files (i.e. dynamic libraries), with names as keys. We could imagine
splitting the executable apart to be saved in a database-like filesystem, that
would work, but if wouldn't change the fundamentals.

The problem I have with structure is that it implies a schema. And without
that schema there is nothing you can do. And of course, because we all have
different needs, there are going to be a lot of schemas. So now you turn one
problem into two problems: manage the schemas and manage the data. With a
UNIX-style system, even if you need some kind of structure to actually process
the data, the system is designed in such a way that for common operations (ex:
copy), you don't need an application-specific schema.

~~~
anonsivalley652
Yes, text configuration files are dumb because they require N parser/script
editors times M configurable programs. Furthermore, what people really want is
universal progmatic and CLI access to configuration.

Microsoft Microsoft'ed configuration management with the way they implemented
the registry. The Apple PList way sort-of goes there but doesn't quite master
it.

A better way would've been is configuration via code and command-line that's
easy to interface for all purposes. It's important to:

\- be able to transactionally backup and restore all settings

\- have multiple instances of the same program with different settings

\- wipe out settings to default

\- enumerate all settings within a program or within the whole system

\- subscribe and be notified of setting changes

\- ACLs or privileges to separate users and processes from each other (similar
to the Windows Registry)

\- audit and undo setting changes

\- hide secrets that are provided from dynamic providers

\- allow dynamic providers for values of settings

~~~
dragonwriter
> Yes, text configuration files are dumb because they require N parser/script
> editors times M configurable programs.

No, they don't. It's N editors + M parsers (where M=programs) I suppose if you
want really smart editors that are also developed completely independently,
such that each require it's own parser, its N+M parsers plus N editors.

> Furthermore, what people really want is universal progmatic and CLI access
> to configuration

I'm fairly certain most people don't want anything CLI anything.

~~~
bregma
> I'm fairly certain most people don't want anything CLI anything.

I'm fairly certain they all want CLI everything, it's just they don't know it
yet.

~~~
bvrmn
It's more convenient to call and export needed function directly via C API
than force everybody to make and use another "universal" layer.

------
teddyh
Old operating systems like VMS, MULTICS, OS/400, etc. aimed for this; they
were large and contained support for a _lot_ of structure. Problem is, they
always evolved to be _too_ complex, and/or the available complexity was not
what turned out to be needed, but different complexities had to be built on
top of the old, unneeded, complexities.

Along comes Unix and dispenses with all of that. Files? Bytes only. Devices?
Accessible as files in the file system. Etc. This became popular _for a
reason_.

~~~
hhas01
Worth bearing in mind here that files and file systems are themselves a
kludge, a 1950s workaround for not having enough primary storage to keep all
programs and data live at all times.† (Also, primary storage traditionally
tends to be volatile; not good when the power goes out.) I point this out
because institutionalizing kludges as The Way Things Should Be Done is an
awfully easy mistake to make, and in a long-lived systems like OSes have
serious architectural sequelae.

..

What’s interesting about the Unix File System is that it’s a _general_
abstraction: a hierarchical _namespace_ that can be mapped to a wide range of
resources; not just “files” in secondary storage but also IO devices, IPC
communication points, etc. And that’s all it did: mount resources at locations
and define a common API for reading/writing/inspecting all resources, without
getting bogged down on the internal details of each resource. Nice high
cohesion, low coupling design.

Plan 9 made much fuller use of the namespace idea than Unix did, but the core
concept was there for the start and it is excellent… except for one MASSIVE
mistake: individual resources are neither typed nor tagged.

Without formal type information there is no way for a client to determine the
type of data that a given resource represents. Either there is a common
informal agreement that a resource mounted at location X is of type Y (e.g.
/dev/*), OR there is an informal resource naming convention (typically DOS-
style name extensions), OR the client has to guess (e.g. by sniffing the first
few bytes for “tells” or by assuming ASCII).

Formally tagging each resource with, say, a MIME type (as in BeFS, or properly
implemented HTTP) would’ve made all the difference. THAT is K&R’s million-
dollar mistake mistake, because without that key metadata it is impossible to
ensure formal correctness(!) or implement intelligent helpers for data
interchange (e.g. automatic coercions).

Arguments over the pros and cons of alternative namespacing/association
arrangements (e.g. relational vs hierarchical) are all secondary to that one
fundamental cockup.

..

Unix became popular not because it was Good, but because it was Just Good
Enough to entertain the cowboys who built and used it, who enjoy that sort of
pants-down edge-of-the-seat living. And because a lot of them were in
education, they spread it; and so made cowboy coders the status quo for much
of this profession and culture. And while I admire and largely approve of
their Just Do It attitude, I abhor their frightening disregard for safety and
accountability.

And while I’m heartened to see some signs of finally growing up (e.g. Rust),
there is a LOT of legacy damage already out there still to be undone. And
retrofitting fixes and patches to endemic, flawed systems like Unix and C is
and will be infinitely more pain and work than if they’d just been built with
a little more thought and care in the first place.

\--

† If/When memristors finally get off the ground, the primary vs secondary
storage split can go away again and we finally get back to single flat storage
space, eliminating lots of of complex bureaucracy and ritual. And when it
does, there’ll still the need for some sort of abstract namespace for locating
and exchanging data.

~~~
pjc50
> Formally tagging each resource with, say, a MIME type (as in BeFS, or
> properly implemented HTTP) would’ve made all the difference.

That gives you a "global naming of things" problem, which is surprisingly
hard. Who controls the namespace? Who gets to define new identifiers? Do they
end up as intellectual property barriers where company A can't write software
to work with files of company B?

> without that key metadata it is impossible to ensure formal correctness(!)

That seems irrelevant - even with the metadata you have to allow for the
possibility of a malformed payload or simple metadata mismatch. I don't
believe this alone would prevent people from sneaking attack payloads through
images or PDFs, for example.

> THAT is K&R’s million-dollar mistake mistake

K&R wrote UNIX before MIME. Not only that, but before the internet, JPEG, PDF,
and indeed almost all the file types defined in MIME except plain text.

Refusing to choose also prevented UNIX from being locked into choices that
later turned out to be inconvenient, like Windows deciding to standardise on
UCS-2 too early rather than wait for everyone to converge on UTF-8.

Even the divergent choice of path separators and line endings has turned out
to be a mess.

> cowboys

The "cowboy" system is the one that beat the others, many of which never
launched (WinFS, competing hypertext protocols) or were commercially invisible
(BeOS, Plan9 etc).

Both Windows and MacOS have alternate file streams which can be used for
metadata, but very rarely are.

Memristors aren't going to save you either. Physical space ultimately
determines response time. You can only fit so much storage inside a small
light cone. We're going to end up with five or six different layers at
different distances from the CPU, plus two or three more out over the network,
getting cheaper and slower like Glacier.

We probably _are_ going to move to something more content-addressable, in the
manner of git blobs or IPFS, and probably a lot closer to immutable or write-
once semantics because consistency is such a pain otherwise. It would be
interesting to see a device offering S3-style blob interface plus content-
addressable search ... over the PCIe interface.

Oh, and there's a whole other paper to be written on how access control has
evolved from "how can we protect users from each other, but all the software
on the system is trusted" to "everything is single-user now, but we need to
protect applications from each other".

~~~
pdimitar
> _That gives you a "global naming of things" problem, which is surprisingly
> hard. Who controls the namespace? Who gets to define new identifiers? Do
> they end up as intellectual property barriers where company A can't write
> software to work with files of company B?_

Well, you could go the sqlite route: if you want the DB to be globally
accessible for everyone else, pass an absolute path; if you want it to be
anonymous and for your temporary usage only, pass ":memory:" as the DB path.

Additionally, UUIDs / ULIDs / what-have-you can be extended but IMO just make
512-bit global identifiers the norm and say "screw it" for the next 1_000 -
100_000 years. (While making a standard library to work with those IDs baked
in every popular OS's kernel).

Sure, it might get complicated. Still, let it be one complicated thing on one
specific topic. Still a better future status quo compared to what we have
today.

\---

I mean, nobody here seems to claim that changing things will be instantly
easier and stuff will automagically fall in place in a year. No reasonable
person claims that.

But freeing us from the "everything is strings on shell" and the "zero
conventions" conundrums themselves will probably solve like 50% of all
technical problems in programming and system/network administration.

(While we're at it, we should redefine what the hell a "shell" even is.)

------
erling
I understand the natural frustrations articulated here, especially given OP’s
experience working with files, but it seems to dismiss what is actually a core
strength of current operating systems: they _work_. Given a program supporting
16 bit address spaces from the 1970s, you can load it into a modern x86 OS
today and it works. This is an incredible feat and one that deserves a lot
more recognition than offered here! Throughout an exponential explosion of
complexity in computing systems since the 70s, every rational effort has been
made to preserve compatibility.

The system outlined here seems to purposefully avoid it! Some sort of ACID
compliant database analogy to a filesystem sounds nice until 20 years down the
line when ACIDXYZABC2.4 is released and you have to bend over backwards to
remain compatible. Or until Windows has a bug in their OS-native YAML parser
(as suggested here) so now your program doesn’t work on Windows until they
patch it. But when they do, oh no you can’t just tell your users to download a
new binary. Now they have to upgrade their whole OS! Absolute chaos. And if
you’re betting on the longevity of YAML/JSON over binary data, well just look
at XML.

~~~
jmiskovic
Want to admire your fancy After Dark win 3.1 screensaver? Just emulate the
whole environment! We don't want to keep suporting the broken architectures
and leaky abstractions of past, they drag us down. Microsoft's dedication to
backwards compatibility is admirable but IMO misguided and unsustainable in
the long run. The IT industry has huge problem with complexity. We need to
simplify the whole computing stack in interest of reliability, security and
future innovations.

The proposed improvement as I understood it would be future proof. It seems
trivial to build a rock-solid YAML/XML/JSON/EDN parser on OS level, and since
it would be so crucial part of OS the mistakes would be caught and fixed
quickly. It shouldn't even matter if structured data syntax is replaced or
expanded in future, as long as it is versioned and never redacted. Rich
Hickey's talk "Spec-ulation" has much wisdom about future-proofing the data
structure.

~~~
gibbonsrcool
> The IT industry has huge problem with complexity. We need to simplify the
> whole computing stack in interest of reliability, security and future
> innovations.

Yes! I really hope I keep hearing more of this sentiment and that eventually
we collectively take action. What would be the first practical step? There's a
lot of effort duplicating the same functionality across different languages
and frameworks. Is reducing this duplication a good first goal? Should we
start at the bottom and convince ARM/x86/AMD64 to use the same instruction
set? After that, should we reduce the number of programming languages? It
seems there's still a lot of innovation going on, would it be worth stifling
that?

~~~
jmiskovic
The actual non-snarky first step would be to admit that we are over our depth
and we can no longer deliver software that is reliable, secure and
maintainable. We can only guarantee that our software works for at least some
users, on current versions of OS/browser, and is hopefully secure against some
of poorer attackers.

Countless variants of programming languages and of instruction sets are not an
issue. The problem is lack of well-defined non-leaky interface on boundaries
of abstraction layers.

------
bvrmn
Article resume: biggest cs problem now is a diversity of serialization
formats, because most of current code consist of parsing various formats, so
OS must to do something about it.

No, it is not a so big problem. And no it will not do our life easier. Also
author did not mention about the _real_ problem of semantic. How client should
interpret a structure to compose a valid request.

OS should not know about userspace structures because OS don't do anything
with it. It stores and transfers chunks of bytes and its semantic is defined
by userspace. And forcing current popular serializing format on OS level is
the most dumb idea ever.

~~~
thetanil
most dumb idea ever is a bit strong, but yeah. If you could get OSs to adopt
this, then as an app writer, you're going to have to worry about how
Microsoft's jacked up version of the standard broke your content when it was
moved between computers or even OS versions. You'd have the UNIX/MS line
ending problem not just in text files, but with every Type recognized by the
database

~~~
de_watcher
Additionally we've got enough fun already with the case insensitive
filesystems.

------
jayd16
Problem: We make a lot of CRUD apps.

Solution: The OS should do it.

Ehhh... Why is that the obvious solution? We can't decide on the right way to
do it in user space, why does moving the problem to the OS help? This seems to
be based on the whimsical idea that having the OS do it would somehow fix the
varied problems of structured communication. Are we enforcing WSLDs in the OS?
One size fits all structures defined by the OS? I don't think the rambling
thoughts really made it back to the thesis.

That said, I suggest to anyone interested in this stuff to try Powershell...no
really! I don't use it often but it is a window into another world where
everything has a structured definition behind the text output.

~~~
speedplane
> Problem: We make a lot of CRUD apps. > Solution: The OS should do it. > >
> Ehhh... Why is that the obvious solution? We can't decide on the right way
> to do it in user space, why does moving the problem to the OS help?

The article is indeed ridiculous. An OS _should not_ do everything. Hardware
storage resources are generally the memory, disk, and network connection (and
if you're getting really deep, the cache and registers). A good OS _should
only_ provide access to those resources as efficiently as possible across a
wide variety of hardware.

There is a vast myriad of ways of utilizing those resources, and it would be a
fools errand to implement a one-size-fits-all approach. The better approach is
to provide access to the resources, and let higher level software developers
build on top of them.

A disk only database is far different than a disk database with a memory
cache, is far different than a memory only database, is far different than any
collection of the above coordinated via a network connection. Further, storing
text is different than storing images, which is different from storing video,
which is different from storing JSON or XML.

Pushing everything to the OS will often give you worse performance, locks you
into a single OS vendor, and slows down innovation from third parties. Bad
idea.

~~~
clarry
> An OS should not do everything. Hardware storage resources are generally the
> memory, disk, and network connection (and if you're getting really deep, the
> cache and registers). A good OS should only provide access to those
> resources as efficiently as possible across a wide variety of hardware.

It sounds like you're talking about kernel only. So I guess your OS of choice
is something like LFS?

My view of an operating system is very different; it's supposed to be a
complete system ready for productive work as well as a programming environment
and platform for additional third party software.

> Pushing everything to the OS will often give you worse performance, locks
> you into a single OS vendor, and slows down innovation from third parties.
> Bad idea.

Pushing everything to third parties will often give you massive duplication of
effort and dependencies, excessively deep stacks that eat performance and make
debugging harder, locks you into a clusterfuck of dependency hell, and slows
innovation from first party because now they must be very sure not to break
the huge stack of third party stuff that everyone critically depends on.
There'll be no cohesion because third parties invent their own ways of doing
things as the stripped-to-the-bones OS has no unified vision, documentation is
going to be all over the place, there's nothing holding back churn...
development of third party applications is slow and frustrating because the
lowest common denominator (underlying OS) is basically magnetic needle. Bad
idea.

This is largely why I prefer BSD over Linux, but I share the author's
frustrations with Unix in general.

------
houseinthewoods
Preach! I've felt this way too, that adding structured data as a universal
feature to operating systems would be a pretty agreeable next step.

I wonder if we're past the point of return, though, in terms of technical
divergence. It sounds like, in the Ancient Times, there was a handful of great
programmers whose work created the world we program in now. But now, there's
way more programmers being paid to make slightly different versions of this
"next step", and it would require widespread agreement/coordination to
implement it on a scale where it's a seamless feature that's taken for granted
the way the shell/network/fs are.

------
leosarev
Logging: Structured logging with automatic rotation etc was implemented in
Windows Event log

Structured data passing between programs instead of just text is part of
Powershell concept.

Calling of other programs to request specific actions with smooth UI called
Android intents.

If you want to store structured data, you should use well, a database.

So, part of critique of author is Linux specific.

But generally I agree with author: OS are poor abstractions and really need to
be improved.

~~~
p2t2p
> Structured data passing between programs instead of just text is part of
> Powershell concept.

dbus, CORBA and COM would like to have couple words with you

~~~
qwerty456127
I don't know about DBus and CORBA but COM is unreasonably hard compared to
REST or Protobuf.

------
hamilyon2
Imagine interoperability nightmare when we cannot rely on everything being
just bytes being streamed.

I mean, everything would be stored and transmitted one time or another. Word
document, sqlite database, an email and it's attachment. Imagine you could not
send something as simple as word document because ip protocol assumes stream
of bytes, and your operating system talks custom storage format. Imagine you
cannot store, use sqlite database efficiently because operating system does
not present you with efficient, fast, compatible byte storage.

~~~
hhas01
“ Imagine interoperability nightmare when we cannot rely on everything being
just bytes being streamed.”

As I’ve noted above, the problem isn’t transmittability; the problem is never
knowing what the bytes being transmitted _represent_.

I mean, C is hardly renowned for the robustness or expressivity of its “type”
system, but untagged untyped byte streams are tantamount to declaring all data
as void*. That is a ridiculously shaky foundation to build on, yet could have
been entirely avoided by simple addition of one more piece of metadata and an
ftype() API.

K&R were brilliant, but also kinda dumb. I certainly wouldn’t want to eat
chicken cooked by either one.

------
andrewflnr
I suspect that one of the keys to user-friendly structured data will be
storing sparse data, in the sense of incompletely defined data (note that's
real persistence, not just existing in the editor). It's also important to
allow data that doesn't (yet) conform to whatever schema is relevant, but I
think sparse data will generalize to that case.

One of the undersold features of plain text is being able to exist in
"invalid" states while it's being edited. Structured data UI needs to have a
least that level of malleability to have a chance over the current paradigm.

~~~
sedatk
Windows supports storage of sparse data but probably it’s not what you mean.

~~~
nitrogen
If you are referring to sparse files, those are also present elsewhere (see
the _seek_ parameter to _dd_ ).

------
iamwil
Wait, Isn't the appeal of a stream of bytes that the OS is agnostic to
whatever the application layers above are doing with it? That way, you can
separate concerns and evolve each layer independently. It's probably the
reason the OS has had such a long staying power of decades. It seems to be the
same strategy of IP (internet protocol)--that it's a narrow waist architecture
--so the layers above and below it can evolve independently.

How would versioning of the data structures work? Would it be append only? I'm
pretty sure part of the reason abstractions are hard is because we don't know
exactly how to model the domain the first couple of times around. So if these
data structures in the OS aren't right the first time, we end up lugging it
around and it'd be more expensive than the parsing that we currently do. When
I look at the EDI spec, it just has a lot of cruft I don't need.

And lastly, I'm afraid when we invoke meaning in a stream of bytes, different
groups with different interests will have different ontologies they want to
enforce. We see these in format wars and internet working groups. I'm not sure
we can easily agree what these data structures are, and how to evolve them.
Maybe there's a good way and I just haven't seen it.

~~~
pdimitar
> _Wait, Isn 't the appeal of a stream of bytes that the OS is agnostic to
> whatever the application layers above are doing with it? That way, you can
> separate concerns and evolve each layer independently._

Dude, we all know the supposed theoretical ideal state of things. And yet it
almost never happens.

> _It 's probably the reason the OS has had such a long staying power of
> decades._

Or the fact that without an OS your hardware is a simple paperweight? Not sure
what is your point here, can you clarify?

> _It seems to be the same strategy of IP (internet protocol)--that it 's a
> narrow waist architecture--so the layers above and below it can evolve
> independently._

We don't have that much choice there either. Networks have to be streams of
bytes due to the physics and realities of our network hardware -- which aren't
likely to change anytime soon. So I don't think yours is a transferable
analogy.

> _How would versioning of the data structures work? Would it be append only?_

Yes. That's how FlatBuffers work and every single team I talked with are
grateful that they moved to it and no longer use JSON. Has to mean something.

(FlatBuffers also allows you deprecate and thus ignore parts of the structure
you are transmiting. But indeed and as you alluded to, anything new can only
be appended at the end of the byte stream.)

> _I 'm pretty sure part of the reason abstractions are hard is because we
> don't know exactly how to model the domain the first couple of times
> around._

Sure, agreed. But as an area IT already has a lot of experience and can tap
into some past lessons already.

Configuration: zero programming language constructs inside it, please! You
need config with programming? Here's that small subset of LISP or Lua, you can
only use that.

I see nothing wrong with an approach like that. People will grumble a lot and
will adapt to it as they always do. It's not like Ansible's YAML programming
language is better, no?

> _And lastly, I 'm afraid when we invoke meaning in a stream of bytes,
> different groups with different interests will have different ontologies
> they want to enforce. We see these in format wars and internet working
> groups. I'm not sure we can easily agree what these data structures are, and
> how to evolve them. Maybe there's a good way and I just haven't seen it._

I also haven't seen it and I am sure most people don't as well but it doesn't
mean we shouldn't try. Stuff like RDF were pretty good for describing many
types of data, for example. And if somebody needs something drastically
different, well, they shouldn't use that format.

\---

IMO the author's point isn't that tomorrow we can have the one ultimate
programming language and data format to rule them all -- of course not. Their
point is that nobody is even _trying_. And we actually have a lot of low-
hanging fruit, the "everything is a string on the shell" being one of the best
examples.

~~~
iamwil
I'm not against trying. I'm just saying, ok, if we're going to try it, these
are the things that immediately come to mind as hurdles to get over. I wish
they addressed those hurdles in the post (or maybe in a later post).

------
akubera
I think the browser is the described OS here: every-api passes objects around,
IndexDB is your filesystem, remote REST & GraphQL servers are returning
structured JSON objects to your program, you can add a <script> tag to your
<html> to pull in your single-function scripts (i.e. leftpad).

Seriously though, the question of "why isn't it done this way" is answered a
couple times: "it's all just bytes in the end". Bytes & arrays-of-bytes are
really the only thing that you can trust that all systems and all languages
interpret the same (even 2-byte integers can be interpreted differently by
different systems). Presumably the author just wants to be able to memcpy
structures between programs or from disk, or maybe with some high efficiency
pre-allocated heap stuff, but you really need to have code to validate the
bytes, essentially requiring the parsing of your data-structure which the
author is trying to avoid.

Running programs with some common argument notation does sound really nice,
but ideally that's just calling a function from your language's binding to
their library (as their executable should do, right?).

~~~
bmn__
> remote REST & GraphQL servers are returning structured JSON objects to your
> program

They are not. They are sending a HTTP message, which consists of just octets
and has to be parsed. The parsers have to have a ridiculous complexity for
various reasons. The HTTP header contains sublanguages such as RFC 7231 §5.3.1
or RFC 8288 §3.

JSON meanwhile has amassed at least six incompatible specifications with the
same exact media type, so there is not even a theoretic hope of parsing all
messages correctly.
[https://news.ycombinator.com/item?id=20736665](https://news.ycombinator.com/item?id=20736665)

~~~
squiggleblaz
Wow that statement is completely untrue.

It links to several RFCs. RFC-7158 and RFC-7159 differ only in (a) that one is
RFC-7158 and the other is RFC-7159 (b) that one is dated in March 2013 and the
other is March 2014 (c) that one obsoletes RFC-7158 in addition to RFC-4627
which they both obsolete. It appears that 7159 was pushed only to handle an
error in the dating of 7158, given that the irrelevant RFC-7157 is dated March
2014.

So to say that there's more that five incompatible specs is just a cruel
disregard of the facts. No implementation can differ in its parsing of a text
based on the metadata of the spec. There needs to be a substantive difference
in the spec.

As to the current RFC 8259, it makes no substantial differences from the
previous version. There are a few typos that have been fixed ("though"
becoming "through", when "though" was not a possible interpretation). It
eliminates the obligation to parse public texts not coded in UTF-8.
Previously, documents SHALL be UTF-8 - that's barely a change, but it is a
change. It has specified more concretely its relationship with ECMA-404. It's
possible that someone reading earlier RFCs would have concluded "a valid RFC
7159 json parser can parse and only the documents produced by ECMA-404". If
there were any discrepancies between ECMA-404 and RFC-7159, therefore, an
interpretation along these lines would lead a person to conclude you cannot
parse a document according to RFC-7159, so the spec doesn't really count as an
incompatible spec. Under the current spec, it identifies that there are
possible discrepancies, and RFC-8259 is intended to accept fewer documents
that ECMA-404.

The first RFC did not permit the json documents "true", "false" and "null",
accepting only json documents that comprise an object or an array. It
permitted any Unicode encoding, defaulting to UTF-8, and non-Unicode encodings
as well (so a Latin-1 document is valid RFC-4627, but not valid RFC-7158+). It
also included some incorrect statements about its relationship to javascript.
These were reduced in later versions and eventually eliminated. They do not
affect the specification of the language, but merely how you may handle data
in the language.

No document that is accepted by the current RFC will be rejected by any RFC
other than the first. The only such documents to be rejected by the first are
those which consist entirely of "true", "false", or "null". No UTF-8 document
that is accepted by the first RFC will be rejected by subsequent RFC parsers,
unless a person reads a paragraph called "security considerations" that says
"you can kinda do this but it's insecure" as somehow trumping the clear
statements of the grammar in earlier sections.

I have not investigated the other specs and I probably won't. But the idea
that there are 6 incompatible specs is false. There is not more than 5, and it
is almost trivial to accept documents according to 3 of 5 or 4 of 6 specs at
once by ignoring the restrictions on character coding.

~~~
pdimitar
> _It links to several RFCs. RFC-7158 and RFC-7159 differ only in (a) that one
> is RFC-7158 and the other is RFC-7159 (b) that one is dated in March 2013
> and the other is March 2014 (c) that one obsoletes RFC-7158 in addition to
> RFC-4627 which they both obsolete. It appears that 7159 was pushed only to
> handle an error in the dating of 7158, given that the irrelevant RFC-7157 is
> dated March 2014._

"they only differ in", yeah, indeed.

Dude. We all have work to do and grappling with some obscure JSON parsing
corner is the last thing we need in our workday. This applies to 95% of
commercial programmers, I am willing to bet.

What you described does sound simple and small in isolation. Now multiply it
by 50 and see how "they only differ in" is a problem that _must not be allowed
in the first place_.

Where's the-one-and-only JSON spec? Why no organisation has the courage to
step forward and standardise it once and for all?

Given the current realities, just cut your very small and ignorable losses by
losing those obscure JSON parsing corners and let people describe JSON simply
as:

"RFC #ABCD".

That's it. Nothing else. Only that. Everything is contained in it.

Complexity compounds and makes everything non-deterministic. We should stop
allowing that.

------
fulafel
Let's reminisce on some OS-bundled databases of Unix: The libdb files, the
utmp/wtmp/btmp style files, the /etc/services and /etc/passwd style ascii
databases.

I think looking back we can be thankful that structured databasey stuff has
been decoupled from OS interfaces and has been allowed to evolve on its own.

~~~
Bystroushaak
> Let's reminisce on some OS-bundled databases of Unix: The libdb files, the
> utmp/wtmp/btmp style files, the /etc/services and /etc/passwd style ascii
> databases.

I would argue that you can't really call this a (ascii) database, because it
isn't (no transactions, no data types, no query language, no parallel write,
nothing). It's what you get with unix - shitty stream of bytes.

~~~
ratboy666
I would disagree.

Take /etc/passwd. It does have a query language. getpwent(), login, passwd

It has data types -- again getpwent() (man getpwent) needs pwd.h

It has "parallel writes". Multiple users can use passwd at the same time. It
has "transactions" \-- mostly because of inodes.

Similar comments apply to utmp and wtmp (note that utmp and wtmp are not ascii
-- they are binary records). /etc/services is ALSO structured but is ascii.
Less controlled, because it is almost always "read-only".

Of course, in a very strange sense, you are correct. Any structure imposed is
NOT by the kernel -- this is "post init", or user level.

FredW

------
PaulDavisThe1st
One reason that we don't "just stored structured data as-is" is because there
are many kinds of structured data where the in-memory representation is
architecture and/or compiler specific.

Many a novice programmer has no doubt made the mistake of thinking that you
could, for instance, do this to deliver a C/C++ struct across a socket:

    
    
            writer: write (socketfd, &struct, sizeof(struct));
            reader: read (socketfd, &struct, sizeof(struct));
    

that works until the reader and writer are on machines with different
architectures.

Same observation could be made about filesystems used to store "structured
data". We serialize and deserialize it because the in-memory representation is
not inherently portable.

~~~
clarry
The author's point is that if our APIs worked with structured data instead of
byte streams, then we wouldn't need to care about the in-memory format. The
standard API would do the hard work for us, allowing us to call send(fd,
my_struct);

Since there's no standard API (blessed by the OS) and the lowest common
denominator is byte streams, we're seeing all these ad-hoc solutions and a
hodgepodge of formats and libraries to deal with. That's lots and lots of time
and money spent on rather basic stuff.

~~~
PaulDavisThe1st
And that's why I think the author is wrong about this (he is probably right
about thinking (more often) about filesystems as databases, but that's
somewhat orthogonal).

The approach you're describing only works for POD-style "structured data".
Once you start using OOP of almost any type (though not __every __type), you
no longer have ... well, POD that you can move to /from a storage medium. You
have objects whose in-memory format __IS __important and compiler dependent.

There are other concerns too. His WAV example (I write pro-audio software for
a living) doesn't even begin to touch on the actual complexities of dealing
with substantial quantities of audio (or any other data that cannot reasonably
be expected to fit into memory). Nor on the cost of changing the data ... does
the entire data set need to be rewritten, or just part of it? How would you
know which sections matter? Oh wait, the data fundamentally __IS __a byte
stream, so now you have to treat it like that. If you don 't care about
performance (or storage overhead, but that's less and less of a concern these
days), there are all kinds of ways of hiding these sorts of details. But the
moment that performance starts to matter, you need to get back the bytestream
level.

And so yes, there's no standard API and yes the lowest common denominator is
byte streams ... because the __only__ common denominator is byte streams.
Thinking about this any other way is a repeat of a somewhat typical geek dream
that the world (or a representation of the world or part of it) can be
completely ordered and part of a universal system.

~~~
clarry
Structured data can be streamed too, and indeed there is software that does it
at scale. Data with much more complex structure than audio frames.

~~~
PaulDavisThe1st
Of course it can be streamed! Nobody ever suggested it could not be. The point
is that to stream it portably (i.e. without knowing the hardware
characteristics - and possibly software characteristics too - of the reciever)
you have to first serialize it and then deserialize it, because the in-memory
representation within the sender is NOT portable.

~~~
clarry
You're too hung up on in-memory representation. Yes, if it's not right, then
it needs to be converted. That can be done for you, or you can do it manually
with byte streams like cave man. If you can do it manually fast, then it can
be done just as fast automatically based on the declared structure.

~~~
cycloptic
It isn't possible to be in a world where the in-memory representation doesn't
matter. Someone has to write and maintain that code, and if it's you, then you
get the fun task of telling some users their workload won't fit into it. And
then they go off and write their own in-memory representation.

------
taneq
Problem: There are N different ways of serializing data and it's a lot of work
to support them all.

Solution: Create a new, unified, universal way of serializing data.

Problem: There are N+1 different ways of serializing data and it's a lot of
work to support them all.

~~~
earenndil
That's a bit of a knee-jerk reaction, IMO.

Problem: the computer systems we use are insufficiently flexible/too flexible,
encouraging the use of N different ways of serializing data.

Solution: make a computer system which encourages centralizing the mechanism
by which data is serialized.

~~~
Ragnarork
New problem: Data serialization is now centralized and there are N ways of
converting between the centralized way it's been serialized and everyone's
incompatible structural needs.

~~~
taneq
Hey, at least then we've improved the format conversion problem from O(N^2) to
O(N). :)

In practice, though, you end up with something like XML, which (to steal a
phrase, I wish I could remember where I read it) is less a common language
than it is an alphabet. I guess it's better than CSV.

------
qwerty456127
I always disliked the file systems being hierarchical databases with the
limitations they have.

Even from the simple, non programmer user (or a "power user") perspective I
don't actually want the files the way they are. A file usually needs a name
(which is not good for anything) and stores both the data and the metadata
inside, a metadata correction automatically changes the file hash and the file
system doesn't consider the fact most of my files (or at least the data part
of them) are never meant to change (taking this fact in account could provide
extra optimization and protection). Ideally a file should only store data
(e.g. a pure sound stream for an audio file) + a collection of metadata fields
(including tags!) I can query to find the files I need. AFAIK something like
that was supposed to be implemented in WinFS but it was cancelled.

In fact you can use extended attributes/streams or even use SQLite as a file
system (or a file format). But nobody does that. I feel interested in
implementing a file manager to store all my files in SQLite and provide a
convenient GUI to work with them but it seems too much work (I don't even have
an idea of the UI) and its usefulness is going to be questionable as no
application developed by others is going to be able to open files directly
from that database anyway. Nevertheless it can be a good idea to use SQLite as
a file system for particular projects like those you have described.

~~~
squiggleblaz
> its usefulness is going to be questionable as no application developed by
> others is going to be able to open files directly from that database anyway

Don't most operating systems provide for some kind of vfs nowadays? So you can
indeed expose it as a legacy hierarchical filesystem if you want.

I guess the reason every extended database type filesystem has failed is the
same one: basically, you need legacy support. Otherwise the user agents don't
exist to access your data. Several older operating systems had support for
attributes of various type and purpose, but they roughly died as the web
became popular, because HTTP has no way to handle that problem.

The nearest exception is on handheld devices that run non-legacy operating
systems, but then power users complain they don't have adequate control.

~~~
qwerty456127
> but then power users complain they don't have adequate control.

They wouldn't if they were actually given adequate control. The problem of iOS
is not the fact it doesn't expose a traditional file system, it's the fact it
doesn't expose any and won't let you control the apps beyond the level Apple
likes nor to sideload custom apps you or the free software community might
have developed (which certainly isn't a 100% evil policy - it actually
protects you from evil agents like the Chinese TSA which is known to install
hidden apps on Android phones it searches). I couldn't even find a way to play
OPUS audio files on a recent iPhone (despite the fact iPhone supports OPUS in
hardware). As a power user I can be pretty well-satisfied without access to
the file system if only the UIs and APIs above it provided the level of
control and functionality I want.

~~~
squiggleblaz
Well, you convinced me. It's an interesting project idea.

Concerning user interface, I think the ordinary user interface of such a
system would have to be application based. Rather than increasingly impure
desktop metaphor, you simply list the applications available to user. They
open the application and select from the available documents. You would have
some kind of list of filters, including automatically maintained filters (like
"time last opened") and manually maintained tags.

You could probably have a power users interface that is "the file manager".
But it's just the normal file picker, without a default filter of "files i
know how to open".

But you still have a problem when you export a file. How do you export that
compressed audio stream when you want to send it to your friend? You need to
identify all the metadata and reconstruct an MP3 file. Presumably you could
implement version 0 of this without changing the file storage, since file
importer/exporter is kinda crucial. So you just cache the important data
somewhere.

It would all be more important if files were still as relevant as they used to
be. I used to have ogg vorbis files and photos and everything. Nowadays, it's
really only code and occasionally a document. Otherwise I'm in my web browser
or on an app on my phone.

------
z3t4
One thing that annoys me is that when "directing" other programs, you need to
write tests that check if each program is alive and kicking. If you for
example is checking a http server, you need a http client to test it. And the
most frustrating part is that you don't know when the program is ready to do
whatever it's supposed to do, so you have to constantly poll until it answers,
but then when you disconnect it might have some warmup/warmdown mechanism, so
you can't put it to use right away, you have to do manually testing - then set
a timeout with a good margin. Then when you "director" worker crashes (and
restarted by another "init" service) those children are still alive, so when
you try to start another instance it will complain about the port being used,
so the "director" worker also have to check for existing orphans. .. It's
sometimes easier to just rewrite the program yourself so it becomes a port of
you "monolith". I like the unix philosophy and micro-services. But I feel it
can be better...

~~~
Bystroushaak
Hah, yeah.

------
stikypad
I get where the author is coming from, and I'm often dismayed by how much of
programming is just shuffling data from one
platform/format/structure/persistence layer to another. It's tedious AF.

Still, there appears to be a lot of conflation of concepts here, and I don't
see anything approaching crystallization of a coherent solution.

What do I gain by telling the OS that I want a JPG rather than a sequence of
bytes that I know is a JPG? I'm probably already using a library to
deserialize this for me, unless I'm a masochist, so the heavy lifting has
already been done. And that object/struct/whatever is going to be represented
somewhat differently in memory for Java than for C++ than for Python. There is
no way to reconcile those differences without creating even more problems. So
a programming language would need to either be written or adopted to
standardize on.

Most application-specific data is already stored in a database of some sort
anyway, which is itself (potentially) platform agnostic. To assert that
applications should instead rely on the OS and filesystem to provide this
persistence layer is to assert that there is a universally appropriate choice
of database -- a bold claim.

Moreover, if structure can (must?) be defined by the application which is
interfacing with the OS, then nothing is gained but overhead. Every object
would need a description of itself, so we either end up with redundant
structural information for every file, or else we have some centralized table
of "object types" that the OS has to look at every time we request something.

Maybe I'm missing something, but I don't see the appeal here. I understand the
desire to reduce overhead, but as far as I can see, this just creates more.

~~~
sliken
Ok, today we have shells like bash that can help you navigate around a file
system, find files, run files, search files, view files etc. Common tools are
grep, less, cat, "|", awk, sed, perl, tr, wc, find, etc. The <cr> is a poor
mans record seperator.

Experienced people can often do non-trivial things in shell by combining a
tools, regular expressions, and manually filtering out false positives. I.e.
using grep foo | grep bar for that one email you are looking for.

But as a result things that need more structure require significant coding and
create a sandbox that doesn't work well with other systems. Like say
thunderbird (an email client).

Now imagine something different that has some higher level of abstraction.
Maybe every file gets by default a list of functions to help the OS understand
it. How about dump (raw byte encoding), list, add, view, delete, and search.
Each file type supported by the OS would get those features, one of those
might be jpeg. So of you created an address book, you'd define a record type
called person, and a list of fields. One of those fields might even be a JPEG
for a image of the person. In a GUI (like thunderbird) you'd just wrap a
function called addressbook.insert much like just about every GUI platform has
a file picker.

Ideally every application on this OS would make use of these function calls so
every app that needed an addressbook could share the OS calls to interact with
the addressbook. But also instead of being frustrated at thunderbird you could
use your addressbook for new and different things. Like say a map viewer might
put icons up for every home address in your addressbook. Or you could query
you addressbook for the home address nearest you... from the command line.

Ideally this files are actually objects that include data and code. Said code
could inherit based on primitives from the OS like records and fields. That
would enable things like extending the addressbook to handle a new field like
your keybase ID, PGP key, 3D representation of your face or whatever.

Similarly any image viewer, or even a pipeline using standard image tools
could iterate over all your JPEGs and extract geo tags.

The object aware "find" replacement could access records/fields for all file
types so you could find photos within a distance of a long/lat.

By combining the above with a relational database instead of a filesystem you
could mix and match and create virtual folders for things like the top 1000
newest images on your "filesystem". Suddenly replacements awk, sed, wc, less,
etc would understand email message metadata.

Or make a directory that contains the newest email messages you haven't
replied to. Running ls --fields=From,To,Date,subject would give you a summary
of email in a folder.

The code replication for understanding a file format, serialization,
communication, etc would be greatly reduced and moved largely from the
application space to the OS space and result in significantly increased
compatibility between applications. Imagine instead of a zillion files under
~/.thunderbird that instead it was all in a database compatible with any email
client that's supporting the new record/field standard.

------
32gbsd
I get what the programmer is trying to say but I think the problem is when you
think everything is a nail because all you have is a OOP hammer. OOP lets you
wire things up in a very unnatural memory hungry way kinda way.

When you start dealing with 500,000 files in one directory and you try to do a
files->getname(10)->size->to_string(). You are bound to run out of physical
memory. This is probably why newer operating systems are so slow. Caching will
only save you for so long.

~~~
Bystroushaak
I don't actually really like the concept of the OOP as it is understood today.
I mean it more in the Self / Smalltalk sense, which is quite different and
thanks to late binding more similar to lisp.

Also your example is weird and you won't run out of memory if you use
generators.

~~~
32gbsd
Generators are OOP service packs. It feels like the intention is to make
things simple and easy to mesh together at the cost of overall flexibility.
This would make a case for a new operating system built around this idea.

~~~
Bystroushaak
Well, its just criticism. I am working on my own programming language
([http://blog.rfox.eu/en/tinySelf.html](http://blog.rfox.eu/en/tinySelf.html))
which could be probably used for experiments on this field of work, but mostly
I am focused on creating personal wiki systems these days.

~~~
32gbsd
interesting and confusing language. if we really want to get new break
throughs we will eventually have to tackle the root cause of issues and re-
write the OS or we will be patching languages forever. For example; image
every variable would be data + meta? It would take up twice the space but
allow for stranger types.

------
zupa-hu
I've came to the same conclusion and I decided to build it. It's called Boomla
[0]. Every OS needs a killer app or they will die. I decided to create a
website builder or CMS as the first product, hence I call it a Website OS.
Please cut through that noise when you look at the docs. It is 2 products, the
OS and the website builder built on top of it. They will be properly separated
in time.

[https://boomla.com/](https://boomla.com/)

~~~
pdimitar
This looks really impressive. Thanks for posting your work! I'll do a deep
dive at some point since it got me curious.

~~~
zupa-hu
Cool! I'll shoot you an email so that you can easily find me.

------
Bystroushaak
I would like to point out, that having structured information as "bytes" or
"text" is general and universal, but pretty much no one uses it as such. Maybe
for simple append, or maybe for tasks like "count number of bytes". But every
time you want to do something with the actual structure, you end up using ad-
hoc parsers.

Unix utilities sound like a great thing, because how universal they are. Want
to find something in text? Just use grep, or maybe with regexp. But what this
really say is "use a parser made from simple condition", or "create parser
using single character pattern matching language". And ok, this would be fine,
if it really worked, but it can't really handle the structure. It may be great
for admin, who just wants to quickly find something, but it is horrible and
broken by design for anyone who really want to make something more permanent.
So you end up with writing better parser / using library. And you are not
working with text anymore, but with typed graphs / trees. And this happens
every single time you actually do something even slightly complicated with
"just text".

~~~
ratboy666
Text is structured: character, word, line, possibly field. Even "bytes"
(octets) have some structure. On top of text, I usually layer, these days,
usually JSON. Bytes? usually sqlite3

Yes, creating a parser with grep isn't usually desired. But, "plain text" is
quite useful. And, with JSON and sqlite3 in easy reach, I don't see the
massive issue.

Please -- I would like examples where you had issues with doing something even
slightly complicated with "just text", and had difficulty. I really want to
examine this. Either the text toolkits are not adequate, or I will be
convinced that I am wrong, and will investigate structured CLI and OS
interfaces.

FredW

------
teddyh
Regarding structured logging, the systemd journal supports structured logs,
for those who want it.

------
PaulHoule
It's hard to really innovate in operating systems because people still want to
run existing code, and they want to run it really badly, even if means
compiling from source code.

If an operating system supports fundamentally different APIs, old code won't
run. You can build an abstraction layer, but if the OS is different enough
inside to be simpler, lighter, and higher performance running native apps, the
abstraction layer would run code with awful performance and terrible
reliability.

I think there's a market for an OS for systems that don't have a traditional
MMU, getting rid of the TLB could save a lot of die area and make memory
access more predictable. Smaller systems could live comfortably in a 24-bit
address space, but 32-bits and no MMU would also be good.

I have dreamed about stripping down a Linux kernel and distro to the point
where it has just a very simple root filesystem and everything is services
offered through unix domain sockets (e.g. when you want to access files you
use 9P in user space)

When I work it out I start realizing that you lose the benefits of the page
cache, some IPC applications really take advantage of memory mapping, and
pretty soon you're back to the "not-so-micro" microkernel.

So far as structured serialization, it's a tough nut.

Whenever it is tried, people tend to hate it. Back in the 1980s and 1990s
people were obsessed with performance and made inscrutable systems like Sun's
RPC and Microsoft's COM based on C. More recent systems are easier to
understand, except for the notorious Protocol Buffers from Google. I love the
idea of Apache Arrow but like all the others it has no real answer for text
(e.g. variable length fields, unicode, etc.)

"Database as part of the OS" was common in mainframe operating systems (which
have a filetype which looks like a single SQLlite table) and in minicomputer
operating systems (Pick, AS/400) that have something like a relational and/or
object db built in.

~~~
pjc50
> OS for systems that don't have a traditional MMU

Running a single program from a single vendor for a single user?

~~~
PaulHoule
Maybe.

You can do memory protection by having registers that limit memory access to a
certain range. Segmentation isn't fashionable these days, but it could
implement something just a bit better than that. System/360 had memory
protection without an MMU.

Other options would be something like the Java virtual machine or Rust that
let you statically check that code is safe.

The worst problem for MMU-less systems is memory fragmentation. This was a
problem throughout much of the history of Classic MacOS. An MMU lets you
"move" memory without the application knowing anything moved. Without an MMU
you either have to live with fragmentation or have some kind of moving garbage
collector (possibly quite coarse-grained)

Other advantages of MMU are paging (less popular these days, not viable for
real-time) and being able to map the same pages into multiple address spaces.

------
rini17
I agree. I see major practical obstacles:

General purpose editing of structured data. Org-mode is the closest thing I
know of but it is still infuriatingly ad-hoc and text-first.

Grepping (search in) and diff/patch of structured data. While many text
manipulations are easy to implement in search/replace manner, structured data
are more involved.

Errors and other kinds of unexpected input/output. Semantics of unix process
failure are clear. State of the system after one process crashes is readily
known. We can compare the filesystem before/after.

If we instead throw an exception in object-oriented runtime, the result is not
so easy to analyze or compare, there may be dangling references or other
complications. I don't know of any language runtime that can dump itself whole
in (semi-)readable format for later comparisons.

~~~
wizzwizz4
Smalltalk can.

------
bvrmn
There is a very entertaining discussion about fs-as-db-and-more from core
linux devs:
[https://yarchive.net/comp/linux/reiser4.html](https://yarchive.net/comp/linux/reiser4.html).

------
Tyr42
Wow. After living on the google tech island for a while, it's surprising how
much of this people have tried to "solve". Need a message? Pass a protobuf.
Want to log something? Got you covered. Files? What's a file? I only make rpc
calls to some database server.

So I think there's definitely some truth to what he is saying. However I'm not
sure if we can climb down from our local optimum long enough to all climb up
there together. I think even at google where it's pretty restricted you still
might have two ways to do some things (the deprecated way and the not yet
ready way).

~~~
forgottenpass
>After living on the google tech island for a while, it's surprising how much
of this people have tried to "solve"

You're missing the point of the article. Most people and organizations have
written libraries and services that make common development tasks much easier
when developing within their own software ecosystems. Google is just one of
many.

The question this article is getting towards is solving those problems in a
pattern that transcends individual implementations and the conceptual model
becomes as ubiquitous as the filesystem hierarchy.

Dumping a "this works for $DAYJOB" solution onto the public by publishing a
standard isn't the answer. If that worked, those problems would be solved and
this article wouldn't exist.

~~~
Tyr42
Before it can become as ubiquitous as the filesystem, people need to be using
it _somewhere_. And also talking about it.

It's reasonable to say that "google implements some of those ideas" counts as
evidence for "those ideas are right", which is mostly what I wanted to say.

------
kazinator
Concept of the memory. Think about what it really is: a limited flat key-value
database. It is limited because it not only limits the allowed subset and size
of the key (e.g. 32 bit or 64 bit address) but also the value itself: a region
of bytes or words starting at that address, of some unknown size, which spans
across adjacent addresses. That seems reasonable only until you realize that
it is a flat database which doesn't allow you to store structured data
directly. How can we improve this shitty, eighty-year-old metaphor?

------
leoc
See
[https://news.ycombinator.com/item?id=14542595](https://news.ycombinator.com/item?id=14542595)
if you're seriously interested in this.

~~~
Bystroushaak
I'll look into that, thanks.

------
axilmar
We certainly need some standardization over binary formats, that's for sure.
It could take away a lot of redundant work that's been done over and over
again (i.e. serialization/deserialization).

This need not be at O/S level though. It can be a bunch of standards, like
HTML, and each O/S could provide the appropriate implementations.

And the data format does not need a particular schema, it just needs a
structural description.

The same format could be used to describe schemas.

------
pmontra
The sentiment about application developers reinventing languages (the example
of Ansible) is understandable. Some standard OS service offering structure
could be convenient for some use cases.

However how would that help this common consumer scenario? I record a video
using my camera (some unknown OS). It's stored on a SD Card as a list of
bytes. I guess there is little to do about that. I either send the video to my
Android phone over WiFi (my backup strategy when travelling, the camera is an
AP) or I insert the SD Card into my computer. When the video is on the phone I
syncthing it to my computer, so it always ends up there. Then I could process
it or not, send it to friends using several different OSes or upload it to
YouTube. Friends could receive the video over Google Driver or via a https
link to a server of mine.

A video file definitely has some internal structure, mandated by standards and
it must be interoperable across OSes. Maybe a list of byte is all we need
here. Probably most YAML files must be cross OS (definitely docker, also
ansible). The list of bytes is the minimum common denominator for all OSes so
I guess it's here to stay.

~~~
ieRei6ae
The article makes a good point about the dumpster fires named ansible and
docker.

------
imtringued
I like the complaint against the command line interface.

Parsing commands should be the responsibility of the shell and all the program
should do is check for the existence of flags and values.

bash: keytool -list -keystore keystore.p12

customshell: keytool --list --keystore=keystore.p12

One could convert the above into this JSON Object (or any other format):

["keytool", {"flag": "list"}, {"flag":"keystore", "value":"keystore.p12"}]

~~~
kevinoid
> Parsing commands should be the responsibility of the shell and all the
> program should do is check for the existence of flags and values.

I would love to see more consistency in command line parsing, and I think
argument splitting has helped add consistency for Unix-likes compared to
Windows, but going further has some major challenges:

getopt(3) parsing currently depends on the accepted options. For example, the
parsing of `command --option value` depends on whether --option requires an
argument. Similarly for `command -cmyfile` which depends on whether -c
requires an argument (or -m if -c doesn't, etc.).

For argument, lets say the shell implements the One True CLI Option Syntax™
and passes the parse result to programs. What are the big advantages over the
status quo? A stronger encouragement for consistency than getopt(3)? Slightly
less work for new option parsing libraries? Unfortunately, most of the code
for option handling, which parses and/or validates options and option
arguments, prints errors, prints --help, etc. would likely remain largely
unchanged.

However, in the same vein, one area that I would love to see more
standardization (although probably not at the OS level) is a more expressive
way to declare supported options and arguments. There is all sorts of
information beyond `struct getopt_long *` which would be useful for
parsing/validation, documentation (--help and man pages), and shell
autocompletion that is currently ad-hoc.

------
ww520
The goal of the OS is to have the minimal support to do it jobs and gets out
of the way. Cramming more feature into the OS just increases the complexity
unnecessary.

If you need a database, why not just use a database? Why force the solution on
the file system?

Microsoft had WinFS 20 years ago that's exactly what you ask for but it never
caught on because the complexity in the kernel wasn't worth it.

------
alexisread
[https://github.com/apache/arrow/tree/master/format](https://github.com/apache/arrow/tree/master/format)

Are your in-memory formatted structures. Even better, there are libraries in
many languages for accessing them. As such you can consider the data almost as
an ffi between languages. Metadata can also be attached to the arrow data.

As far as the rest goes, A2/bluebottle defines that standard program unit
similar to COM etc, ot has an object shell similar to Oberon or powershell, so
looks to be pretty much what the author is looking for.

Having said that, I'd prefer an OS with managed memory but no GC as per
Composita.

[https://github.com/btreut/a2](https://github.com/btreut/a2)

[http://www.composita.net/Composita.html](http://www.composita.net/Composita.html)

~~~
Bystroushaak
Interesting, thanks for pointers.

------
kthielen
I agree especially with the proposition that we should exchange structured
data (via files, network connections, ...).

I started this project several years ago to support live code updates in
production systems without sacrificing performance, but I eventually hit the
same issues with exchanging structured data: [https://github.com/Morgan-
Stanley/hobbes](https://github.com/Morgan-Stanley/hobbes)

I split out structured logging into a single header-only lib, structured RPC
into a single header-only lib, etc. It's not quite the same as having native
support in the OS, but because the mechanism is so lightweight (don't even
need to link anything special), it's a pretty good substitute.

------
epr
The most practical way to give these ideas an opportunity to grow (IMO), is to
start with code. Rather than code being stored as text, we store it as
"objects" (as described in the article). To the extent which you are already
using high quality tooling we already get many of these benefits, with the
unfortunate addition of tons of parsing code duplicated between your
compiler/interpreter, ide, text editor (syntax highlighting), and other tools
(refactoring, etc.), as well even more duplicate code covering the mechanics
of the internal model of the language(s) you are using (linters, intellisense,
etc.).

------
_pmf_
In the case of the Linux kernel, the internal APIs are much nicer that the
user space POSIX API.

People love to praise "everything is a file", but the other side of the medal
is "every piece of configuration is a byte array".

------
julienfr112
Microsoft had a project to make the FS a database. It failed, didn't it ?

~~~
arethuza
WinFS - it had some interesting ideas although I can see why it was abandoned:

[https://en.wikipedia.org/wiki/WinFS](https://en.wikipedia.org/wiki/WinFS)

------
jeroenhd
A lot of these criticisms are about the OS not doing enough high level stuff
for the programmer. If you want to see what happens when the OS does all that,
have a look at Android, where the "OS" is what's giving you sockets, key value
stores, structured logging, etc. because the only real APIs it's giving you
are the Java ones. Now look at the horrible workarounds you need to apply to
make old code compatible with newer versions as demands changed over the years
tk focus more on user privacy and battery life.

For nearly every problem encountered here there's a standard people just
ignore. You can implement structured logging at OS level, like Windows does,
and see random binary data drops and text files appear all over your logging
interface and your file system. Linux config should follow the XDG standard so
that configuration for applications can be structured into directories except
half the programs don't do that. Windows provides an API for this stuff (the
registry) and it's near impossible to configure an application using the
registry. The registry is also combined with random config files all over the
system, of course, to make it extra difficult to modify a program's behaviour.

Most nice modern features are also missing because operating systems are still
being made in C, not C++. The OS doesn't have a concept of objects so yes your
socket function will have to fall back to calling select(). A lot of these
abstractions for basic communications have been solved on Linux using DBUS,
which provides a somewhat standardised interface for many OS daemon and GUI
features, all not being used because programmers forget that it exists or
because programmers want to use their own solution instead.

On Windows there's COM to try and help with that and well, see where that
ended up: a versioned mess of pointers and factories to try to make it easier
for everyone, where functionality sometimes completely breaks or needs to be
emulated because it turned out the high level concept had a design flaw and
now programs won't work if you fix it.

I've done some thought experiments about structuring an OS and a file system
to store data consistently and easily parsable, with modern bindings for most
concepts. In the end I've had to conclude that the only way to keep the system
working like intended would be to either convince everyone to do exactly as I
say or to only allow me to write software for such a system. Whatever
structure I can think off will inevitably be too constrained for someone else
and the middleware abstraction problem starts all over again.

My only conclusion is that I want the OS to be as simple as possible with
people following common standards when they write applications, such as using
YAML/ini/XML configuration with Syslogd logging and XDG directory structures
for user data, with the technical abstractions left at the library level. If
we can just get that, most of the inconsistency problem would be solved, but
even this is too difficult to do in practice as it turns out.

------
fao_
It's probably worth the owner checking out the Lisp Machine, Oberon, and
Inferon (A descendent of Plan9 where they improved over the original concept).
They say Plan9 is a step in the right direction, but rather than attacking a
specific aspect of Plan9, they critique how the concepts were imported into
Linux, which seems inherently silly to me.

It's like critiquing a master craftman's tools by observing how a master
craftman uses his tools, and then critiquing how the apprentice uses those
tools. Sure, some of that criticism of the concepts is valid, but they picked
the worst implementation of those concepts to critique.

One of the things mentioned in the critique is how the filesystem in question
communicates the changes back to the user, and how the user is expected to
know what parts of the filesystem to alter. While in the examples given, it
does seem relatively obscure, the decry the fact that they had to _check the
manual_ for the position to write to.

However, both of those would still be the case for the system that they talk
about. Let's talk about object orientated systems, which already exist, in
limited (compared to the operating system) forms. You still have to pick up a
manual to figure out what keys and functions are relevant to you. In Pharo
(which is more or less the system they're after!), you can inspect objects
that exist and the methods that exist for them -- however -- often,
similarily-named functions do different things, which also requires reading a
manual. In addition, using these tools as a first-time user, I was overwhelmed
by the number of functions available, most of which I could only guess at what
their purpose was. Pharo integrates the manual into the system, but the manual
is still there!

Standardized error codes are given in structures passed back (As in Erlang) or
in Exceptions, but there are problems with those too. You still have to figure
out _what those exceptions or errors mean_ , and there isn't necessarily a
_standardized format for those errors_. As a developer I've been recently been
working on writing a Python api. As someone who has little experience with the
intimate details of the Python language, I do not always know of the
exceptions that exist that I am able to throw, I do not always know of which
exceptions would be most appropriate. The same problem exists (obscure error
values), but in a different form. There are existing libraries that I rely on
that implement more exceptions, sometimes these aren't documented, and even
with the venerable Requests library, I have still had to crack open the
objects that exist to find easier methods of passing data that the Requests
library _has_ , and _uses_ , but does not _document_ for external use.

Let's look at the windows hive database. That's a database of keys for values
in the operating system. As a random developer, would you be able to open it
and figure out what it does? I wouldn't. As a windows power user I often
relyied on articles from howtogeek without really fully understanding what the
keys were doing (Although because of my experience an a systems developer, I
could guess, but only _after the fact_ ). Again, the same problem ("I know
what I want to do, but I do not know how to do it") is exposed in a different
form, and the methods and practices of the Microsoft organization make that
difficult hard to reach. Yet again, the same problems are there, but in a
different form.

I do agree that the shell should handle program arguments, a program could
expose keys and values and a shell could read that and tell the user about it.
I would be interested to see the problems that the Oil shell encounter in
their implementation of an object-orientated shell.

~~~
Bystroushaak
I've checked Genera (Lisp Machine).

Oberon is on my todolist, Inferon is new for me.

> They say Plan9 is a step in the right direction, but rather than attacking a
> specific aspect of Plan9, they critique how the concepts were imported into
> Linux, which seems inherently silly to me.

Well, I've played with Plan9 several times, but I don't really feel like
criticizing it, because I don't really know it that much. Only thing worth of
criticizing is that it almost feels like objects, but not implemented fully.

I agree, that criticism of Plan9 would make more sense from philosophical
point of view, but it wouldn't be authentic.

> However, both of those would still be the case for the system that they talk
> about. Let's talk about object orientated systems, which already exist, in
> limited (compared to the operating system) forms. You still have to pick up
> a manual to figure out what keys and functions are relevant to you. In Pharo
> (which is more or less the system they're after!), you can inspect objects
> that exist and the methods that exist for them -- however -- often,
> similarily-named functions do different things, which also requires reading
> a manual. In addition, using these tools as a first-time user, I was
> overwhelmed by the number of functions available, most of which I could only
> guess at what their purpose was. Pharo integrates the manual into the
> system, but the manual is still there!

Thats true, but you can also do a type-checking and infer a lot only from the
proper naming of the objects and methods in the "Clean code" (book) style.

> Let's look at the windows hive database. That's a database of keys for
> values in the operating system. As a random developer, would you be able to
> open it and figure out what it does? I wouldn't.

One of the things that is not strongly captured in this article and which I
since consider increasingly important is ability to use _reflection_ on the
system. Database that you can't open and figure out what it does is not worth
using.

~~~
fao_
oops! I typo'd, it's Inferno :D

------
imtringued
>But why do we still use unstructured format of binary data transmission even
today when all communication is structured? Even seemingly binary stuff like
streamed audio is structured.

This reminds me of Bluetooth with it's billions of different profiles. Want to
send FLAC or opus audio streams over Bluetooth? Tough luck, you better use the
blessed profiles or make your own proprietary implementation (aptX).

------
zaarn
I've imagined what a perfect OS would look like in my opinion;

the primary way of exchanging access is handles. Opening a file is obtaining a
handle, which represents both the file and the driver/type behind it.

The file handle has some functions associated with it; write, read, flush,
close. You can pass the handle to other processes, either cloned (RO or RW) or
as ownership (loosing access to it entirely).

A printer has a different handle and different functions.

Programs can be the source of handles, so you can write a program that wraps a
file behind a different set of primitives. Or that wraps the raw printer
network socket and allows passing a PDF file handle into a function to print
the PDF.

And if the printer goes away, the printer wrapper is notified that it's gone
away, so it can queue up any prints. Or exit and notify everyone owning a PDF
printer handle that the handle is now invalid.

Device files like on Linux wouldn't exist, you only need handles to operate on
devices and you abstract over those, if you need it, you can abstract the
filetree handle to provide a simulation of the /dev/lp0 file in it.

The filesystem would be a database, split into various tables ("default-
configuration", "system-configuration", "user-configuration", "user-homes",
"executables", etc.) with some tables merged into a view ("configuration"), so
you can completely forget about where a config file is. Files themselves can
either be formatted as a table themselves (key-value, json/nested data, etc.)
or used as text or binary files. Of course a program can choose to use the FS
transactionally, including SERIALIZABLE for your backup programs, that need a
very consistent view of the FS.

Every interface would be malleable, a program with higher privilege can
override it and change it for programs of lower privilege. Chroot becomes as
simple as prepending the chroot path onto access to the filesystem functions.
Going down the scheduler itself being runtime modifiable so a user can swap to
a different scheduler at runtime or define a scheduler for a specific subset
of programs.

If everything can be used, manipulated and introspected freely, then the
programmer will have to do minimal work. Why use JSON configs when there is a
perfectly good database system shipped with the OS? Why bother with logging
when there is an interface for the system logger that supports structured
logging and can ship logs to anywhere you want in any format at any detail
level?

Sadly, some of these ideas aren't very efficient or hard to implement. But
luckily not impossible. Just not wanted by the greybeards of the industry.

------
nottorp
DOS used to have record oriented file functions. More like some kind of legacy
from CP/M.

Guess what, they were abandoned. I don't know, maybe they were not flexible
enough?

Does the OP suffer from not invented here? Most of the file formats he
describes have free (at least to use) libraries available.

Even for sockets there are countless message passing solutions. He just need
to research a bit and pick one.

------
TheOtherHobbes
I think this misses the point in a lot of ways. There are showstoppingly huge
practical issues with schematising data, including standardisation, version
control, schema access, schema localisation, schema politics - who owns the
definitions? - and many more.

But there is a valid point about data ownership. A lot of current privacy
issues and political problems are created by the way that applications own
your data. If I want to access a Photoshop file ten years from now, chances
are good I'm going to be paying Adobe a fee to do it - likely in the form of a
subscription.

This is crippling, because it means Adobe (MS, Apple, Amazon, etc, etc) have a
choke point over my own personal access to my work.

Of course I can export documents to some other format, but _in principle_ what
I can and can't do with my work is controlled by big corporations.

Open source alternatives don't fix this, because outside of developer tool
space they're usually poor competitors and never have the leverage - nor the
quality - to become an industry standard.

At some point this data siloing became a "personal" computing principle, which
is unfortunate because it undermines the idea of a computer as a personal
tool.

OS schematising wouldn't really change this. But forcing open access to the
internal file structures used by large corporate applications would be a game
changer, because it would allow shell-like automation and composition of data
that is currently trapped behind corporate paywalls with either no low-level
access at all, or high-friction save/load only import/export features.

------
laughingbovine
ITT Software Developers: This is a great idea! System Admins: Why on earth
would you want to do this?

I'm not sure this situation (devs and admins disagreeing) will ever go away.
Is there some future where our opinions will converge?

~~~
SubjectToChange
I doubt system administrators are against operating systems having better data
structures for configuration or storage. In fact administrators might be more
in favor of structure since they are the ones who do the most ad-hoc scripting
on machines. Developers can largely insulate themselves from the operating
system with frameworks, libraries, and databases.

Anyway, system administrators, outside of BOFH types, and developers generally
aren't at odds with each other.

------
NotUsingLinux
Too bad the author doesn't yet seemsto have found
[https://www.unisonweb.org/](https://www.unisonweb.org/) it really covers some
of his points.

------
anonsivalley652
I'd go further:

= Everything as text - There's a fundamental problem with turning and treating
everything as text: N programs having to know M parsing formats. Structured
data that is readily usable without de/serializing would be an improvement.

\- Logging - logging is broken because of the previous point, because log
entries are streams of events poorly serialized to text files, log rotation
and loss of structured data.

\- Processes - Processes should be pause-able to disk, migrate-able to other
systems (as long as I/O and files can be migrated)

\- Indexing - there ought to be a tunable Spotlight-like system that doesn't
need to constantly reindex because it indexes everything that was changed
using file notifications in the background during idle (not on battery)

\- Caching - there ought to be a central web cache on a computer that all web
browsers and web operations can share

\- Hypercard - we need more of this

\- Introspection - the system, processes and every variable should be profile-
able, inspectable and queryable

\- Databases / files - there really shouldn't be any files or database, every
program should be able to offer data types and format transformation services
that the OS can use to add new functionality to all other programs and
services. The notion of a file should be containers of user data that programs
can use directly. IOW, a programming language should have a NoSQL/SQL-like
query language built-in so there's no need for ORMs... the OS should handle
how best to store and index data with the storage allocated.

\- Typed memory / VM - it would be much easier to accomplish all of the above
if the OS were a lightweight system similar to Erlang BEAM, Pony, LLVM runtime
features.

\- Kernel security, isolation and performance - something like seL4 but with
additional transactional (N > 3)-process support to be able to make a bunch of
changes, validate state and commit/rollback on failure.

\- Fragmentation - the problem with systems like Linux that are developed as
zillions of piece-meal, semi-interchangeable parts that duplicate
functionality and have many options are the mess and confusion of trying to
integrate them. The BSD's have it right in terms of base system development.

\- Complexity - Look at how much stuff is cram-packed into the Linux kernel or
OpenSSL.

\- Churn - Deprecating and changing APIs creates incompatibilities. An OS
should pick one set of APIs for all time and make it very difficult to change
them. Heck, every OS should have the same API and same VM opcode format so
that there's never again an obsolete or incompatible program. There's no point
to re-inventing the same thing over and over and over that doesn't work with
anything else that came before.

------
f2f
the article mentions plan9 but says that it left a lot to be desired due to
unstructured text input/output interface.

i want to note that Inferno, the other OS that came from Bell-Labs in the 90s
did have a typed shell called alphabet:

[http://www.vitanuova.com/inferno/man/1/sh-
alphabet.html](http://www.vitanuova.com/inferno/man/1/sh-alphabet.html)

mayhaps you'll find this interesting.

~~~
Bystroushaak
I've come across this several times in the past, but never paid too much
attention to it. I'll check it again. Thanks.

------
amelius
Filesystem as a database is complicated because let's assume that you have a
network drive, then suddenly a transaction spans more than one computer.

------
musicale
Did anyone else have trouble reading this because of the bizarre automatic
font sizing of the page?

I wish sites wouldn't try annoying tricks like this.

~~~
Bystroushaak
What device did you used to read the page? The CSS is pretty basic, but I did
some hacks to make text more readable on iphone.

~~~
dvh
Not op but practically unreadable on mobile with thin noodle of text covering
left half of screen.

~~~
pmontra
I read it on my Android phone with Firefox, no problems. Only some unused
white space to the right and no margin to the left. Same with Chrome.

------
jillesvangurp
There are a bunch of things that don't really make sense in modern software.
For example, we now commonly package things up as docker containers with
immutable/ephemeral file systems. For better or for worse, that's how a lot of
server-side stuff gets deployed.

The whole point of that is simply emulating what the program is expecting to
such an extent that it can run as if it was running on a normal file system.
Often that means working around many broken assumptions. For example it might
expect some configuration in a particular place in the filesystem. However,
since the filesystem is immutable, you can't really modify that after you
build the docker container. So you work around it by e.g. using dockerize to
template the config file and then inject the actual configuration from the
environment variables on the docker command line. Likewise, applications
produce logs but writing those to an ephemeral file system is kind of
pointless; so you instead write to stderr/stdout and leave it to the host OS,
Docker, Kubernetes, or whatever is running your container.

These workarounds are something to avoid if you are writing new software. When
you ship as a Docker container, you don't care about the host OS. It might be
Linux, BSD, Mac, Windows, or whatever else is capable of running docker
containers these days. Chances are that the host OS itself is also running in
some VM if you are shipping to a cloud environment. The nice thing with Docker
is that you (mostly) don't have to care about any of that.

With WASM this is extending to other places. People are running edge computing
functions in WASM form, running WASM programs in a browser, or even in
operating system kernels. Most of those places don't necessarily even have
filesystems, environment variables, or log files that you can access (or
should access).

IMHO modern development requires a bit of upfront thinking on how to configure
things, where to send logs, and where to store data. Rarely is the answer for
any of those things a file on an (ephemeral) disk. Logs need to be aggregated
so they can be analyzed. Local storage is not always available/reliable.
Configuration gets injected rather than loaded from a file. Files/state gets
written to some specialized service (a networked file bucket, some DB, a
queue, REST service, etc.). Most of those things are accessed via networking
rather than a file system.

This is also true of most frontend development these days. A browser
application does not have a filesystem; it has no access to environment
variables; and while it has a console, it's considered bad form to actually
log to it in production apps. Instead browser apps write to remote services
(including logging and analytics data) and get their configuration from things
like cookies, in browser databases, and remotely stored user preferences.

Come to think of it, most modern development is kind of detached from the
operating system these days.

------
tonymet
i think android & iOS provide the data structures the author is looking for
e.g. queues scheduled tasks, relational storage, key value and mostly typesafe

~~~
earenndil
Good luck using one as a regular workstation (without resorting to classic
unix, at which point we're back where we started).

------
betimsl
My brain hurts from this article. I really tried to make some sense out of it,
but man, seems to me that author has very little knowledge of how some things
work.

