A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now.
An anecdote I like to share is that while I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) was that the student projects could run on real - if old - hardware. There was a PC that could boot your stuff if you put it on a disk.
This PC had to have PS/2 interface to the keyboard, though. A newer PC would be all USB. Apparently, the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building (and, of course, considerably less educational). I mean, this wasn't a full-functioned OS, but it allowed multitasking and preemption of kernel calls, so seriously non-trivial.
Multiply this story by all the devices a reasonable OS would be expected to talk to and it's a scary prospect for the OS researcher... and if you don't have those devices up and running, good luck supporting most workloads anyone cares about.
PS/2 interfaces are actually still quite common for "enthusiast" motherboards, because of the low latency (the simplicity compared to USB certainly has a lot to do with it.) Laptops also tend to have PS/2 keyboards and a "mouse" (the trackpad, which can be switched to a proprietary protocol later for more features) because the EC they use provides that "for free" and that industry tends to have a "it works, why change it and break things" attitude.[1]
I think there was a Ben Eater video debunking the "PS/2 is faster than USB" myth.
That used to be true, but the PS/2 is so much slower (lower frequency) that by the time a package containing the mouse or keyboard info is finished transmitting the usb port started and finished a poll
USB is faster and it is better when there is nothing saturating the USB with traffic and CPU can keep up.
Most of the time there will be no noticeable difference using the USB mouse or PS/2 mouse.
But if CPU is busy and USB bus is saturated, USB mouse will start to be less responsive than PS/2 mouse.
In the extreme cases even USB keyboard will become unresponsive where the old PS/2 keyboard still works fine.
This can make a difference between being able to terminate a misbehaving process and resume working or having to power-cycle the computer.
I frequently have this problem and I'm always aghast that no OS that I'm aware of has a good method to restrict resource usage to ensure the user can still perform essential tasks like terminate a problematic process. Instead, the machine just grinds to a halt and all you can do is sit and wait or else train your finger on the power switch.
It seems like the done thing these days (for people who care about running on real hardware) is to bring up educational/experimental kernels on Raspberry Pi, using the "mini UART" for console I/O and the SD interface for storage.
Admittedly, I got a B in 410 and only managed to TA 213, but I do work in the space now.
Yes, it’s hard to do research if you start at “let’s design a kernel from scratch,” but you’d never need to do that. You can just hack Linux or a bsd, or even use something like bpf to extend it.
The thing that annoyed me about 410 is that when I switched to Linux I realized that pusha/popa didn’t matter at all. It was a good course for writing reentrant C and learning the very basics of hardware, but the really hard stuff is in the weird dynamics of memory management on NUMA systems and work conserving io, which you can’t get anywhere near if you are starting from scratch.
I feel you are focusing on the literal aspect of what I wrote and ignoring the broader implications. The tradeoff here between "student OS kernel principles" and "grind through a USB driver implementation" will recur as you try to add drivers and hardware to run meaningful-to-users workloads.
So you wind up with a research operating system that has finger-painting graphics, can't talk to modern SSDs or networks cards... or you somehow conjure up the resources to build an enormous number of drivers.
If you "hack Linux or a bsd" - or even more timidly - use ebpf - the constraints of what you build are going to result in you basically building "more UNIX stuff". This isn't bad research, but the sheer mass of constraints you're taking on if you accept all the design tradeoffs of Linux/bsd/eBPF/whatever means you're certainly not doing the sort of OS research that the original author talked about.
The linked talk specifically complains about everyone building off of Linux, because doing so bakes in a whole bunch of assumptions that are decades old and might not be the best options for today's hardware...
Then start with something else. There are dozens of operating systems out that there make an attempt to be better (for some definition of better), choose one and work with it. Starting from scratch means you spend years doing all the things they all get right before you can work on the interesting parts.
Complexity increases over time but it's not for nought. USB, while complicated is a great deal simpler than dealing with a dozen ill documented buses. Additionally the amount of things you can do with it are much greater than what you could do with ps/2. The table stakes in general for operating systems is much higher which means making a new one from scratch is impractical for most folks but this is true for most things. Implementing a CPU that's competitive with the ones Intel it ARM is very challenging for a college student but I'm not sure I see that as a problem. Projects like RISC-V exist to give you a baseline to start from for CPU design just as Linux does got os design and they give you a fighting chance to compete. That's probably for the best, and starting from scratch, while sometimes useful, isn't really necessary for most.
> the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building
For the complete USB stack, that's very likely. However, for minimal code needed to read keys from keyboard USB shouldn't be that bad.
These students only need USB 1.1. They don't need to support bulk transfers (USB protocol has 2 distinct transports, one low latency another one high throughput). They don't need to support USB hubs between computer and the keyboard. The only device type they need is HID.
USB consortium wouldn't allow to call that protocol USB due to the missing features, but it's good enough for education purposes.
The professor could implement these boring lower-level pieces in a library which fires couple callbacks, one on connect/disconnect to provide two descriptors for HID device and their reports, another one for handling the HID report packets.
BTW, USB specifies a small subset of HID interface called “boot interface”. It’s normally consumed by UEFI firmware who needs mouse and keyboard for the setup GUI which runs before OS launches. The subset only supports basic keyboards and mice.
Then it isn't write your own OS from scratch. The constraints of the library will force your implementation around it, even if you think you have a better idea.
I think for the USB in that OS, a C API with intentionally very narrow scope would be OK for the job.
The main reason why the real-life code is so complex is that scope being very wide. We have USB 2 and 3, mass storage, two-way audio, cameras and GPUs, hubs and composite devices, numerous wireless protocols on top, OTG, power delivery, power saving features, and more.
Sure, but even one assumption limits innovation. Even if the innovation is stupid, learning the hard way is useful, so I want at least one student every few years to try that stupid thing and then from personal experience tell the others why it didn't work out.
If we were talking about a serious project there are a lot of best practices that should be rarely (and then only carefully) questioned, but for something intended to be thrown away at the end of the semester challenging best practices is a great way to learn.
That said, PS/2 is a simplifying assumption as well.
> A perennial complaint; Rob Pike had a trollier but similar version years back. I'm not nearly as optimistic that we have the resources to cope with the monstrous complexity of hardware now.
"Hardware" (read: device drivers) is not notably complicated IMO.
That's not to say they're simple, but driving devices from a software point of view is pretty similar to interacting with other software. You read the spec (APIs), write code to set up data structures or registers a certain way, parse responses, etc.
Writing a modern full fledged USB stack is very complex, but would not be more complex than writing a modern full TCP/IP stack for example.
A lot of device programming models have gotten simpler too. Device drivers used to be notorious deep voodoo magic e.g., in cases of the IDE disk driver, but that was not really "complexity" of the software logic so much as hundreds of special cases and deviations from standards or odd behavior in all manner of IDE controllers and devices. To the point where the real wizards were the ones who had access to internal data sheets, errata, or reverse engineered firmware from these devices, or otherwise spent countless hours poking at things and reconstructing quirks for themselves, probably bricking a lot of silicon and spinning rust in the process.
But systems and devices are now getting to the point where things don't work that way anymore, silicon power is so cheap and firmware is on everything. The NVMe device for example sets up pretty simple packet-type command queues and sends requests and receives responses for operations - query device information, perform a read or write, etc). It's not quite that simple, there are some quirks and details, but it's quite a lot like writing a client for a server (HTTP, perhaps). I think other device interfaces will evolve toward cleaner simpler models like this too.
One exception to this is GPUs of course. The other thing is with moore's law continuing to slow down there will be increased incentive to move more processing out to devices. It's long been happening with high end network devices, but with technologies like CXL.cache coming soon, I would expect that trend to keep ramping up and come to other accelerators (crypto, AI, even disk).
So.. it's a mix. Things are definitely getting more complex, but AFAIKS the complexity of interacting with hardware is not increasing at a much greater rate than the complexity of interacting with other software. And, as always this complexity is made possible by abstractions and layers and interfaces. It's not clearly exceeding our ability to cope with it.
This is entirely wrong, and I'm sorry that those who do work at the hardware/software interface have done their work apparently so well as to allow you to harbor this delusion. Hardware is complex -- exceedingly so -- but (just as the talk relays) more of that complexity is in hidden cores running operating systems you can't see. Indeed, the talk addresses you (yes, you!) more or less directly at ~31:47: "This is called 'ignorance.'" While you are not entirely to blame for your ignorance (if the talk errs, it's in pretending that these hidden systems are well documented -- or even documented at all), you are complicit it in. Please educate yourself as to the massive complexity that lies beneath you, perhaps by just watching the talk that you are commenting on.
Get off your high horse and shove your accusations of complicity and ignorance, and you educate yourself before you start throwing stones: Re-read my post and don't bother replying unless you can understand and accept that I never once suggested hardware was not complex.
I said the software interfaces to them have not exploded in complexity, i.e., specifically refuting the suggestion that we don't "have the resources to cope with the monstrous complexity of hardware now", from the point of view of driving them with software. You utterly failed to refute anything that I actually wrote, but after your tantrum is over feel free to have an attempt.
And I work at the line and on both sides of it logic, architecture, and software, so don't bother with the vapid appeals to authority.
I haven't listened to the whole thing but that part of the talk you linked is really cringey actually. To paraphrase, "don't trust a person to write software if they haven't written a first level interrupt handler".
Utter rubbish. And he goes on to talk about programming for an embedded SoC, etc. Hah. Some of the most utterly wretched code, vhdl, and development practices I've ever had the displeasure of being acquainted with have been in embedded devices, devices, device driers, firmware, etc.
And first level interrupt handlers? He's saying that like it takes some genius to do it, or the process of doing it confers some deep understanding on the writer. It doesn't. It's not particularly complex, outside buggy or stupid architecture but even that's just grinding work.
And the there is some horrific buggy crappy interrupt handler code around and I can say that because I've written some. And it sounds like you may well have too, if your interactions with David Miller relating to the the performance of Sun OS system calls is anything to go by.
Arguably it's more valuable to understand how the C environment (stack, etc) is set up and how to wrangle the toolchain into emiting code at particular locations and such. But you can get all that many other ways. And even then, I don't actually agree that you need to know the minutiae of those details in this day and age which is a great thing. This is not where most of the interesting work is happening, like it or not.
This kind of elitist gatekeeping attitude is just sad. It reeks of the has-been (or maybe never-was) mindset. I would just as well trust code written by someone who deeply understands what they do writing a word processor or video game or distributed database, than someone who has hacked out an interrupt entry vector. And the sad fact is that being an OS developer does not require or confer a deep understanding of how to get the most out of hardware, understanding caches or multiprocessing or branch predictors or performant scalable data structures and synchronization techniques. No more than writing a web server prevents a person from understanding all those things deeply.
Unrelated to the article, but your description of the 15-410 course piqued my interest. Is there a way to access previous versions of the course materials? I took a look at https://www.cs.cmu.edu/~410/, but it seems like some parts of the site are still being worked on for fall 2021.
While I totally agree with your post (especially the point that a course where you implement your own OS for real hardware is great), I would like to add that USB is a massive foundation/abstraction layer. Once you got USB going for HID, you are much closer to USB for storage, USB for networking, etc. So it certainly is troublesome from an educational perspective but the complexity is where it belongs.
Implementing the minimal “bootstrap” HID interface can be done without most of the abstraction layers. It’s effectively hard-wired and barely parsing of any descriptors or anything.
It was a fantastic experience, although TA'ing it was daunting. Like the apocryphal Ledru-Rollin quote: "There go the people. I must follow them, for I am their leader", it was frankly hard to keep up with all the different approaches students conjured up.
I believe that the course number changed at some point - or some shallow aspect of the course was set to change - and a bunch of industry people called the university to express concern. It is/was (I can't speak for the current iteration of the course, although I have no indication that quality has slipper or anything) truly transformative.
I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option; our job was at least in part to pass engineering students who didn't care that much about computing.
> I offered to run a 'franchise' of the course at the University of Sydney a while back and was informed that anything quite that transformative wasn't really an option
Thats very disappointing. You might have more luck joining the University of New South Wales OS team. (For context, UNSW is a local rival of USYD).
When I was a student there, the OS course was legendary. I only did the "basic" OS course. Our assignments led us to implement a bunch of syscalls for handling file IO, and write a page fault handler for a toy operating system. I'm not sure what they do in the advanced OS course - but it has a reputation for a reason. And it looks like[1] its still run by Gernot Heiser, who's a legend. He's the brain behind SeL4 - which is the world's first (and only?) formally verified OS kernel.
I'm kicking myself for not doing his advanced OS course while I was a student.
I have a modern consumer motherboard that has intermittent problems connecting all it's USB ports, with the latest "stable" BIOS supposedly fixing this but also reportedly having worse performance.
I'd argue that implementing a USB driver would probably be far more educational, in a practical sense, than implementing a kernel. The number of students who will go on to work on core kernels is vanishingly small, but driver work is never going to end.
Plus, if we ever are to transition to the world of microkernels (something that OS researchers have been pushing for decades now), we're gonna need a massive amount of people writing drivers just to get us to the point where things are barely functional.
No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable.
Conversely, the work they did in kernels has a huge amount of transfer to thinking about concurrency, which is truly valuable - and deep. It also meant that they acquired a far less magical idea of where things like processes and threads come from, having actually built the mechanisms to make them happen (you build a thread library in a warm-up project).
As for microkernels - they often punt a lot of the hard concurrency stuff up to user space servers, which will wind up needing exactly the kind of concurrency that the students learned to build.
> No. Driver work is painful but it mostly teaches you about the h/w in question; you don't learn deep principles from a lot of it. You just grind away, spec in hand, until stuff works. A lot of this winds up being "one damn thing after another". There's a good deal of craft to it, and I don't mean to deprecate it, but it's not broadly applicable.
I agree you don't learn deep principles from it, but I disagree that it's not broadly applicable based on your description. Grinding away with spec in hand, "one damn thing after another" sounds exactly like most programming people are likely to encounter in their career.
Sure, but you're not learning anything per se - perhaps some discipline. To some approximation, any two decently competent programmers will take similar amounts of time writing a device driver for a complex novel device regardless of previous experience - the vast majority of the job is in simply going through the spec.
I think ingraining the right discipline might be the hardest part of becoming an effective software engineer.
Also, this course's OS kernel presumably also has a spec, and implementing such a kernel is also "simply going through the spec".
I think the point you're trying to make is that the contents of the spec have be relevant to an operating systems course, and most hardware specs are maybe only tangentially related to the kind of information you have to teach. I'm not sure that's fully true either though, because isolation and safety are core OS properties, eg. DMA has all kinds of security implications. Maybe not stuff for a beginner OS course, but hardware interfacing is critical.
> I think ingraining the right discipline might be the hardest part of becoming an effective software engineer.
Sure, but that's not the business of a university course to teach.
> Also, this course's OS kernel presumably also has a spec, and implementing such a kernel is also "simply going through the spec".
Per some of the other discussions, this doesn't seem to be the case (i.e. the spec that was offered to students seems to have been quite open ended), as people mention that a huge challenge as a TA was in adapting to the large diversity of approaches that students were trying.
There is also a huge difference between a spec that says "the task scheduler must accept tasks in this format and ensure they are scheduled fairly with an O(n) algorithm" and a spec that says "to enable PVM set bits 1 and 3 in register 7; to issue a new read cycle clear all bits in register 21". Device drivers deal mostly with the latter, unless you move to extremely complex devices like video card drivers, that are probably way more out of scope than a simplistic kernel.
I suppose it would be useful if it taught students how much this industry is hare rather than tortoise nonsense where the interfaces are sheer accidents, and the implementations are mad-dash to keep up.
But what you propose is preprofesional drival. It's bad enough that education is subsidized for employers already, we don't need to stoop to them further.
It's not just a USB driver though. You've got to support PCI (or something) to find the USB controller, then you've got to fiddle with that, then you have to discover the devices, then you have to interface with them.
Reading from a keyboard is much simpler. Probably makes more sense to run the computer headless with a serial terminal and make the students poke at a UART. PC standard UARTs are pretty easy, and I think it's easier to find a motherboard with a COM1 header than a PS/2 port (at least my latest hardware matches that description).
You've got to support PCI (or something) to find the USB controller
PCI is nothing in comparison to the complexity of USB, especially since the BIOS will usually have set up the I/O ranges for the devices already. It's literally a few dozen bytes of machine code to scan the PCI(e) bus for the device you want.
Oh sure, PCI isn't too bad, but it's way more work than inb(0x60) or inb(0x3f8). (although to be fair, you probably want to setup an interrupt, which is real work)
Interesting talk but I'm not convinced of the value proposition of "rediscovering hardware". The speaker points out that there are lots of small processors inside a modern SoC that are often running their own software hidden from Linux. I agree that there are security implications here a la Qualcomm Hexagon vulnerabilities or Intel Management Engine backdoors but I'm not sure what the speaker's proposed alternative is. Is it that a "true" OS would directly execute the duties of these hidden processors? This is extra work that would make systems slower. Having dedicated hardware handle these duties is an optimization (hardware acceleration). Alternatively you could replace these small processors with fixed function hardware. This would make any system flaws permanent although it would be much harder to exploit any vulnerabilities. Does the lack of hidden processors make the system "better" somehow? (aside from potentially more secure) Are bus masters and hardware accelerators part of the OS?
It would certainly be a more beautiful system that I would love to see, but I need help justifying the engineering cost. I hope to see some open hardware passion projects along these lines, but I doubt they will ever be mainstream.
I think the "speaker's proposed alternative" is to do more academic research in this area.
You have good questions in your comment, and I think he is calling out for searching answers to them and other similar questions. Apparently, the operating system researchers haven't been doing this enough, or at all. His speech is a lament for this state of affairs, and a call to get excited about finding out what benefits would be gained by doing things (very) differently.
But that's just my take, and I'm not working on operating systems research.
I guess I'm struggling to understand what the problem to solve here is (besides security bugs). Maybe that is wrong, and the idea is just to try new things. But seems to me that 'try new things' isn't a particularly strong call to action when you are competing for research funds/time.
Actually there are a few bright spots on the horizon. RISC-V is gaining mindshare, and I've seen separate events/papers which push for using modified RISC-V cores as specialised processors, similar to eg. The Nintendo64 which used MIPS cores. If that gains traction then there's a good chance that the software will improve.
On the radio side of things, Dash7 firmware will run on many Lora devices, meaning you don't have to use the proprietary Lora software.
More likely its that a "full" OS would at least be be aware of the heterogeneous elements. Then it could provide api's for negotiation between them, if not directly providing a method to run them.
One method could be, say, extending microkernel capability systems to incorporate these remote processors. Or perhaps even the ability to provide WASM or ePBF control blocks to customize the power management system. Though its true that'd require more engineering resources.
Actually USB is somewhat like this in that there's a hardware api and spec that the OS knows about. Linux has troubles with even that in my experience (often requiring a reboot to fix.
Most of these components already provide all their functionality over SPI, I2C, PCI or whatever.
Maybe, instead of opening up all that complexity, simply making better ways to fake a dumb component would be more effective. (That is with verified, memory safe parsers and API interfaces. Then it doesn't matter what other gunk it has behind that interface.)
And this could be implemented as yet another managed layer between the main CPU that runs user space and the hardware SoC. (Basically a low-level "WAF".) Yeah, it's a herculean task, but at least we have access to the Linux drivers.
I'm increasingly seeing the value of Apple style products where almost every part of the device from the hardware, to firmware, to OS, to UI is all from the same company.
The whole combined experience just makes things run so smoothly and allows security issues to be patched since Apple has the resources that maintain each part internally.
Yeah, of course that works, for a while at least. Also developed in the open, patches are welcome, you can update your own NIC/GPU/Wifi-dongle components would work too. Unfortunately the whole industry (Apple as one of the main culprits) is quite hostile to anything like that.
Though maybe we'll have a RedHat of hardware someday.
> Is it that a "true" OS would directly execute the duties of these hidden processors?
I think a “true” OS would run its code on all these processors, implementing both sides of the communication protocols connecting these pieces of hardware into a computer.
For one, such hypothetical OS may upgrade or replace these communication protocols with OS updates, or because user changed some system preferences.
This also allows to re-configure these things based on runtime information. For instance, when AC power is disconnected from a battery-powered device, it makes sense to do stuff slightly differently, like stop all processes on all CPUs who are polling to get events faster, and switch to slower but energy efficient interrupts.
This.. This is exactly what I was thinking. Well put. The complexity it would add the Linux kernel code base would probably, inadvertently, create more security issues. The more code, the larger the attack scope. In general, as a security researcher, I would tell people to let hardware do most of the lifting when possible as software being involved usually means introducing a higher probability of bugs/vulnerabilities.
Pffftttt "hardware acceleration" of features we literally never wanted. We don't even have the freedom to buy computers that aren't "enriched" with this junk.
I've worked on firmware for a few SoC's that my company has designed or purchased and we use embedded linux on some. I think his observation about how Linux doesn't abstract the hardware accurately is totally right. The system isn't that complicated, but implementing everything within the world of Linux has been painful imo, so for future implementations we're evaluating RTOS's. The main reason people use Linux on these custom SoCs because of driver availability of key components (ethernet phys and persistent storage), large library of tools, and probably $ (who wants to pay for QNX). Those are good reasons, but if you have to write a large volume of code, it doesn't really feel like it's providing too much. It definitely felt like we were just using a hammer and seeing everything as a nail. Those are my 2 cents.
This issue with needing more flexibility on SoC's makes me wonder if its part of why the Linux Foundation has put a lot of work and resources into Zephyr RTOS [1]. Its gaining a lot of popularity and incorporates some of nice elements of linux like device tree syntax (DTS) files [2]. I could readily imagine using Linux & Zephyr on a SoC, and using shared DTS to describe the SoC architecture.
Actually, come to think of it extending DTS syntax to describe the pointer swizzling the OP video mentions. Then both OO'es could derive correct physical address handling.
So, it's funny. It's not uncommon to run RTOSs and Linux on the same CPU, where the rtos coordinates tasks between the components that have real time requirements and the Linux side does stuff like stream data from a buffer that the rtos writes to. We had this architecture on some systems and the amount of cludging to get this all working was annoying to say the least. We specifically had issues are figuring out how to partition the memory regions for each OS and it wasn't trivial to validate.
I wasn't around when the decisions at my employer when the decisions made around OS options, but this would've been a viable one if it was mature enough at that time. Thanks for the info!
Sounds way too familiar! What’s amazing (to me) is that there’s even a VSCode plugin for DTS files that checks the memory offsets and gives you nice error messages like “flash partition 1 overlap partition 2 at 0x3e000, …”. It’s a bit like magic after doing it manually.
P.S. huh looks like Linux and Zephyr _are_ doing this via RPMsg subsystem as well. Wow, much nicer than the last time I looked into it!
Unfortunately it isn't that simple. One of the comments on the youtube video gets to the heart of the matter:
> I don't see how this might be changed. Most of those little red circles are not just "tiny computers" but also different implementers in different companies, all trying to protect their little fiefdom against their customers
There are of course initiatives like libreboot/freeboot/whatever, and efforts to reverse-engineer the firmware of microcontrollers in peripherals, and there exists fully documented hardware such as the Raspberry Pi 4 (I think?), but even with full documentation, it's incredibly hard to write custom firmware, and the benefit is very slim (better security perhaps?)
That actually goes to the heart of the problem, modern SOCs are basically proprietary components gobbled together on a chip with some internal networking/wiring that each come with their own software that does a lot of stuff before even getting around to booting Linux.
The problem is the proprietary nature of this stuff and the fact that a lot of it is outside of the scope of scrutiny that goes into Linux. It's held together with what basically amounts to glorified duct tape. It's complicated. It has weird failure modes and occasionally this stuff has expensive issues. Like for example security issues.
Very relevant if you want to build an open source phone platform or laptop. Not impossible; but it requires addressing a few things. Like connecting to a 5G network without relying on Qualcomm's proprietary software and hardware.
The real solution is not merely integrating these things as is and trying to reverse engineer them but engineer them from the ground up to work together and make sense together. That larger, open operating system does not really exist. And many proprietary equivalents on the market right now have an emergent/accidental design rather than something that was designed from the ground up.
When programmable shaders were introduced in GeForce 3 (2001), nobody really understood what to do with them. The consensus was that it’s incredibly hard to write custom shaders. OpenGL consortium refused to support shaders for almost a decade (GLSL 1.0 released in 2008), that’s one of the reasons of the success of Direct3D https://softwareengineering.stackexchange.com/a/88055
Yet look at them after 20 years of evolution. People no longer coding shaders in assembly, we have quite a few higher-level languages for them. People have built better APIs to interface with programmable GPUs, both low level close to metal (D3D12, Metal, Vulkan) and high-level (CUDA, TensorFlow, game engines). People built debuggers and profilers for these shaders. Overall, people embraced programmable GPUs, leaned to use them, and the results are amazing. Visual quality of games and other real-time 3D skyrocketed, despite the increase of pixel count and density in the displays.
I can see how the same path is possible for the rest of the programmable chips and pieces inside modern computers.
> and the benefit is very slim
Better products in very broad sense. For instance, when idle, a smartphone OS could power off not just the screen, but also CPU, RAM, and a few other components, and downscale itself to the slowest and the most power efficient core in the system.
> when idle, a smartphone OS could power off not just the screen, but also CPU, RAM, and a few other components, and downscale itself to the slowest and the most power efficient core in the system
Don't the proprietary versions already do that, though?
I don’t think they do. They switch to very low power states, on big.little systems they power off half of the CPU cores. But some CPU cores, and all of the RAM are still powered, the main OS doesn’t have other place to live.
With different OS design it might be possible to power off more components of the phone when idle, while being able to wake up fast enough.
One one hand there is great vertical disintegration, on the other hand, even Apple is too lazy to actually do this right, still treating their software stack as a black box where breaking changes are fine but elegance is lost on the suits.
It's very vindicating of our worldview, yes -- and also vindicates the de novo operating system (Rust kernel, Rust tasks) that we have developed for some of the use cases that Roscoe outlines. Looking forward to getting it all out there!
As a regular-old application developer, I had no idea that operating systems were being so aggressively sequestered from the perspective of computer architecture. It seems to be quite a joke, actually, to call Linux an "operating system." Combine this awareness with how much closed-source code is running these chips without any knowledge by the operating system and thus the user... it seems pretty scary.
I suspect much of the problem is "wall of confusion" differences: operating systems are primarily concerned with compatibility and reliability while computer architecture seems to be about rapid innovation and change.
So this has actually always been the case. Hardware firmware blobs that initialize the device to talk on the bus yielded to EEPROM chips that had SW firmware burned into it to do that initialization as the EEPROM chips became cheaper and let the bringup process be more flexible. This yielded to microcontrollers that had programmable flash (still some EEPROM for some parts of HW, but now you can glue multiple pieces together). Now you have full-blown ARM cores doing that orchestration (but HW still has EEPROM for bug fixes on top of the base NOR-flashed code). It’s all about flexibility and cost.
I don’t know though if I fully agree with the characterization that Linux has been shuttled off to the corner as chip vendors work around it. It’s more that the computer has actually always been a distributed system and OS/hardware developers have ignored that for a very long time. You wouldn’t say that Linux isn’t the OS for distributed clusters even though a lot of work goes into making those clusters run and that all typically runs in user space outside the OS boundary.
It’s possible that there’s a better model out there that manages to unify things but I’m skeptical. All this code runs on different chips with different clock speeds. Code also resides in drastically different memory spaces and that’s unavoidable - an M3 might be able to load the firmware over PCIE, but it’s not executing out of main memory. Additionally these chips typically have drastically different cache coherency properties.
Now maybe we need to revisit this all holistically, which I think is the actual pitch being made and one I can support. That needs to look like defining interfaces for how this stuff works (& yes, making it possible to integrate into Linux) and getting chip vendors to adopt. Sunk costs are real and trying to rework not just HW architecture but also the entire SW ecosystem can be a dead end. I’d certainly be keenly interested to hear the speaker’s ideas. He’s certainly far more knowledgeable about this space than I am.
It is a interesting debate to have but this is unlikely to happen for consumer hardware, since no OEM wants you to have that kind of access(talking of those phone SoCs), they will just give you a slice of ccNUMA to protect their "IP". That qualcomm bug that he showed is unlikely to go away for consumers.
There is a message to build your own computers and that's probably what people will do and it could be interesting research for custom servers and future heterogeneous architectures. I suspect people in this industry are already doing it.
I am confused. He is basically saying Linux isn't powering all the hardware simply because firmware is running on other more complex peripherals/SoCs on a device. That will always become the case as hardware gets more complicated and advanced. Linux simply need to provide the means to abstract the nitty gritty. Linux is still acting as the central "manager" of all the hardware. As far as security. Linux can act as the final security boundary for getting the machine to do anything useful.
Academics can sometimes learn a lot from industry. The ScyllaDB database and its underlying SeaStar library essentially implement a userspace OS that handles the hardware very differently than a typical modern OS with phenomenal improvements in performance.
Historically, it started as a unikernel called OSv designed to run directly on hypervisors. But they eventually realized that it can run just as well on linux with kernel bypass working and io_uring and is easier to deploy.
The real progress nowadays, since OSes have not stepped up, is in Kernel Bypass. (Show me a fancy new abstraction, and my reaction is, "Great—how do I bypass it?") We have para-kernels, demi-kernels, uni-kernels, all to get closer to the hardware.
Probably future machines should have lots of FPGAs, and easy access to claim and program them. We will need good languages for the latter, although probably C++ will end up being that.
Having seen the coalface, the problem is not the OS, it's the (stupendously retarded) application model.
There's very little (interesting) you can do when application developers are (for the most part) incapable and unwilling to embrace anything even remotely different from the DOS application model.
The OS is, at its heart, a mapping from that application model onto the hardware. If a hardware feature or characteristic can't be expressed / surfaced within the application model, then there's little to no value in attempting to exploit it.
Someone like at least has internal customers, and a vertically-integrated embedded system can sometimes do better, but general-purpose systems are shackled to some minor variation on the POSIX application model for the forseeable future.
No one wants apps to directly interface with hardware. Imagine loading up a notes app and it directly trys to interface with your hard drive using its own built in FS driver and a bug in that causes your whole drive to be wiped.
Or that you open two programs at once and they try to access the same device at once causing some kind of error and likely data loss or system instability.
I don't even want apps to directly have access to the file level interface of my storage but rather an extra abstraction which lets me pick which files it can access.
> No one wants apps to directly interface with hardware.
I build computer kiosks and I agree wholeheartedly. So many devices/peripherals require some sort of custom implementation. Most have some sort of SDK, but not all. Better hope that SDK comes in your tech stack's language(s) of choice.
Why isn't it just data in / data out?
Even printers and scanners are a pain to deal with. Sure, Windows (and TWAIN) abstracts some of the printer/scanner communication, but it all seems forgotten. Like, why is using new hardware in my software so difficult and how come I can't just implement new hardware the same way every time? To me, it's all just data in/data out from my language runtime to the device. I couldn't care less about the implementation details in between.
I certainly write plenty of code where I write to registers directly to configure hardware modules and don't think I'm using an OS in any way. I assume most of the security design and hardware permissions are handled at a higher level than I usually write code.
Oh, oh the ragnarok of systems programming, moores plateau stretches on, parallel is coming to a limit and now its goodbye abstractions & no-skill-lib-glue-code, hello 80s hardware hacks and efficient programming again? Will OSes be produced like games, were they start with a "reuseable" oo-engine and then that is broken down into data-orientated design.
An anecdote I like to share is that while I was TA'ing CMU's 15-410 operating systems course (a great course - build a preemptive multitasking operating system from scratch) was that the student projects could run on real - if old - hardware. There was a PC that could boot your stuff if you put it on a disk.
This PC had to have PS/2 interface to the keyboard, though. A newer PC would be all USB. Apparently, the complexity of the codebase to talk to the USB device was around the similar level of complexity to the entire preemptive multitasking operating system the students were building (and, of course, considerably less educational). I mean, this wasn't a full-functioned OS, but it allowed multitasking and preemption of kernel calls, so seriously non-trivial.
Multiply this story by all the devices a reasonable OS would be expected to talk to and it's a scary prospect for the OS researcher... and if you don't have those devices up and running, good luck supporting most workloads anyone cares about.