Windows Internals is also a good book for learning about operating systems, even if you're not specifically interested in Windows. Mac OS X is more detailed since the source code itself can be published, but it's not as easy to read (gets pretty dry at times). I also enjoyed reading Robert Love's "Linux Kernel Development".
One of the things that really helped me learn more was reading both of those books - understanding two very different kernels (NT vs. Linux) really helps you grok the problems an OS has to solve, and to separate "This is a NT thing" vs. "This is a hardware thing that every OS has". Kind of like learning a foreign language makes you think more about grammar in your native language (at least for me).
OSDI is "the MINIX book". It shows, about as much detail as possible, how a microkernel-based OS is written, down to the actual (complete) source code.
Modern Operating Systems, on the other hand, is a bit higher level (but still fairly nitty-gritty) and examines two monolithic kernels (Linux/Unix and Windows) as case studies.
The tutorial certainly lacks any emphasis on security. It ought to be mentioned right up front. Does the choice of development environment matter? Say so!!! Trying to add security after the fact is painful and tends to be only marginally successful.
OS security is something that really aught to be discussed separately from OS texts. I mean, not all OSes are multiprogramming or need security. Earlier version of his Operating Systems book taught MS DOS, along with others, for example.
Security exposition is something best left to a separate text, and Matt Bishop's fat book does a good job (not the thin watered down one.)
Xinu was originally designed as a vehicle for teaching Operating System design concepts and is used by many educational institutions for this purpose. Later versions supported TCP/IP, these versions are often used in Data Communications courses.
I also used Xinu in my O/S course. I'll mention that one of Doug Comer's (http://www.wikipedia.org/wiki/Douglas_Comer) intentions with Xinu was to purposefully make a different design decision than Unix when pedagogically relevant. For example, instead of treating everything like a file (Unix/Linux) everything is treated like a device.
Its various iterations over the year do not necessarily remain true to that principle, but it is something different about the O/S.
Some good tutorials can be found at http://www.osdever.net/tutorials/index - these are generally more hands-on than academic textbooks. Also, feel free to have a look at http://flick.cvs.sourceforge.net/flick/src/, my attempt at writing an OS (long time ago, now); the code is clear enough to give you a rough idea on how to get started.
That's why I love the software industry. We have such a low financial barrier to if you want to play around with even the deepest systems programming techniques, or the highest-level functional programming languages. All you need is a $400 PC.
These days, a $50 PC (or a free PC) will probably serve you better, because it's more likely to have a serial port.
You should probably try to find one that can either PXE boot or boot from a USB stick, since those are pretty much the easiest ways to quickly test your software.
Makes me wonder, can I make that money doing stuff like this? I know systems research is irrelevant and all that (Rob Pike)... AFAIK the last successful operating systems startup was VMware (is that true?). Does the world have place for more OS startups these days?
I tried to build one from scratch based on my own L4 microkernel. Given the bloated nature of monolithic kernels and their irrelevance to embedded space, I had the thought that if I wrote something small and neat it could gain traction for embedded systems. For that I took the BSD virtual memory design, and implemented a complete VM system from scratch. Then I wrote a VFS layer but that one was partially complete. I had demand paging implemented and supporting calls like fork() execve() etc. It was challenging and fun - but an incomplete project, also my biggest error from a startup/making money point of view. Basically having a new kernel is a hard sell nowadays, no matter how neat it is. It was more of an intellectual challenge for me rather than a startup - but still I was in the expectation that it could be a success.
About exactly a year ago, I changed my direction to virtualization. I ditched about %60 of the software that includes all that VM layer and VFS for which I put loads of effort (many man months...). I realized I was out of focus, so I changed my focus to enhancing only the microkernel itself. I decided upon three specific goals: 1) Adding multi-core support for latest ARM cores (e.g. the ones that will be on most high-end mobile phones and devices) 2) Adding fine-grain security and control based on capabilities. (in short, having control over system calls) 3) Adding virtualization support for the linux kernel. Right now I am done with the first two, and about to get done with the 3rd one.
My interest has always been to create something new in the OS/kernel space. After doing much research I decided that L4 microkernel design represents the most promising work in this field. This is because a) monolithic kernels are mature and good for what they are doing b) microkernels can solve certain new problems like virtualization.
Eventually I was convinced that embedded virtualization would be the most interesting problem I could solve. In my opinion there is still room for systems-level work particularly in Virtualization. The APIs are not set in stone. Not many people know how virtualization works, let alone design an interface for it from scratch.
I think new areas like this will come up but its quite important to spot an unsolved problem because if you dive in to write a kernel its an expensive journey.
There's definitely room for improvement over currently used server and desktop operating systems; the security models used by desktop Linux, OS X, Windows, and BSD don't meet the needs of users in a world of malicious software. iOS and Android were able to improve on the situation a good deal by adopting higher-level security models that protect applications (and their data) from one another, but the desktop lags behind.
As far as I know there's no technological reason the same couldn't be done with the server or desktop, it would just be an enormous shift in momentum to get there from where we are now. But if we're willing to stomach shifts in momentum, we could even go further and consider other, more radically different approaches to security than what's commonly used now, such as Microsoft's Singularity.
I don't know enough to say whether an operating system startup could be successful, but I think if it were, it would be less about basic operating systems research than about getting what research we already have into the hands of the masses on the desktop or the server. Presumably by targeting sectors where customers care enough about security to make the difficult switch to a fundamentally incompatible system.
Systems software research was irrelevant ten years ago when Rob wrote that. Thanks in part to his efforts, there's a great deal of relevant systems software research now: Golang, MapReduce, Hadoop, Tahoe-LAFS, Caja, Chrome (the browser), LuaJIT, seL4, Capsicum, the CLR, HotSpot, V8, Nitro, Tracemonkey, GFS, Sawzall, Pig, Redis, Puppet, Chef, Tor, HTML5, ...
Most of this, but not all, comes from the traditional academic and industrial-lab research context.
In http://doc.cat-v.org/bell_labs/utah2000/ (the point of reference for this discussion), "systems" is defined as "Operating systems, networking, languages; the things that connect programs together." Examples of what he's talking about from other parts of those slides include:
> By contrast, a new language or OS can make the machine feel different, give excitement, novelty. But today that's done by a cool Web site or a higher CPU clock rate or some cute little device that should be a computer but isn't.
> Work on how systems behave and work, not just how they compare. Concentrate on interfaces and architecture, not just engineering.
> Only one GUI has ever been seriously tried, and its best ideas date from the 1970s. (In some ways, it's been getting worse; today the screen is covered with confusing little pictures.) Surely there are other possibilities. (Linux's interface isn't even as good as Windows!)
> There has been much talk about component architectures but only one true success: Unix pipes. It should be possible to build interactive and distributed applications from piece parts.
Chrome is an environment for running AJAX web applications and enabling them to talk to each other; among other services, it provides a graphical user interface toolkit (DHTML), a SQL database (SQLite), security mechanism and policy (via tab-per-process, the same-origin policy, incognito mode, and restrictions in the JS engine), a JIT compiler for a language, and process management (both at the user level, with its process viewer, and at the language level with Web Workers). It has a component architecture built in; several of them, actually: iframes, plugins, JSONP. It is possible in Chrome to build an interactive application from "piece parts"; this is currently called a "mashup".
So Chrome is right in the center of the issues Pike's talk was talking about. Much of this, of course, is made of ideas that don't come originally from Chrome; but Chrome is on the cutting edge of making new stuff possible.
If this post were from, say 5 years ago I would have figured it was spot on. But today? Look at all the new OSes coming out now in the mobile/appliance space. Not to mention that Windows Vista/7 was enough of a shift that the learning cost of moving operating systems has gone down relatively (because you'll have to do it regardless).
I think there is. Every OS that exists is a mess. Make an OS whose inner workings I can undestand (and therefore are very well documented) while allowing to make commonly used software, and I will worship you.
There's no need for a new operating system if your criteria is just "inner workings I can understand". OpenBSD is very thoughtfully implemented, its documentation is top notch, it's open source, and yet it is a general purpose operating system that runs real world applications. It isn't a "mess"; in terms of its design, I think OpenBSD is about as clean as a real-life operating system can possibly be, so much so that my college operating systems professor even chose it over Minix for teaching the course.
I still think there's room for improving operating system security models, but I don't think it's possible to improve over OpenBSD in this particular dimension.
My friend was part of a group of students who decided to write XOmB. They just apparently decided it would be a neat project, now it's a pretty cool exokernel:
http://xomb.net/index.php?title=Main_Page
Not only are we working on XOmB, (pronounced 'zombie') but we extracted all of the stuff "from power on 'till kmain" and called it XOmB Bare Bones, so that other people who want to write kernels in D don't have to worry about that ugliness: http://wiki.xomb.org/index.php?title=XOmB_Bare_Bones
That is great to hear. I've heard a few stories on how painful that stuff was to track down and write. Bring up always sucks. This sounds like an excellent way for people to dig into writing kernels without forcing people to write all the bootup code before they can do anything.
How do you do device handoff? Are people restricted to an exokernel model or can they handoff to a macrokernel kexec style?
(VMware used to use the Linux kernel to take care of init before they wrote it themselves, so there's definitely a need for this type of thing, even beyond hobbyist.)
In classic exokernel fashion... you have a choice! Someone can write a libos to handle that stuff however they'd like, or you can just go do the work yourself.
I took a class like this once except instead of i386 we implemented on 16-bit x86 so we had to fiddle with the stack and registers manually to switch kernel and user mode, or to switch processes. It was great fun despite a few eccentricities.
I'd much recommend Tannenbeum's classic text that covers implementing Minix: http://www.amazon.com/Operating-Systems-Implementation-Prent...
But since this text is rather old, it doesn't have some of the later developments in OS. So for that additional material, I'd recommend his latest book to have at your side: http://www.amazon.com/Modern-Operating-Systems-Andrew-Tanenb...