As someone that has just spent the last 27 months building an OS from scratch in a University research environment (an OS that can survive catastrophic hardware and software failure [0]), I am very impressed with the quality of this document. I only wish I had found it previously!
OS dev can be great fun and gives you a level of understanding about how computers actually work that is hard to gain from other sources. I have frequently found myself using the knowledge I gained from building the OS when working on other projects. It has been particularly useful when it comes to understanding performance issues. I would highly recommend it to anyone that is even remotely interested.
Other superb resources to help you get started include the OSdev wiki[1], along with this document[2] from the University of Birmingham, UK.
Thanks! At the moment, the best source of information is the talk I gave at the Erlang User Conference 2016 [0]. Four papers and a thesis are in the pipeline for next year, though! I will update the site once PDFs of those are available.
Thanks for sharing, really enjoyed the video.
Isn't it amazing that Ericsson were achieving nine 9's (31.5 milliseconds downtime per year) [1], using Erlang in the AXD switch back in 1998 [2].
What is your opinion on some of the issues? I see the project seems dead, and some of the simple issues like fixing the frame buffer instructions, and replacing external with extern. I'd love to get into OS dev, and help make this 'book' better, but it seems dead and no one has taken the time to fix such issues
Do you mean with the 'little book of OS dev'? Or one of the other resources?
In general, my experience has been that OS dev is a topic that is best approached through the study of a variety of different documents, simultaneously. For example, you would do well to read both the originally posted 'little book...', as well as the University of Birmingham text. Then it might be a good idea to look at the OSdev wiki pages for the specific tasks that you want to work on.
> but it seems dead and no one has taken the time to fix such issues
Unfortunately, there seem to be few /completed/, and high-quality texts available. It is certainly a shame that there is not a modern, high-quality book about OS development (that I know of), but the motivated student can certainly find the needed information. Good luck and have fun!
If you want a real bang up tutorial that covers absolutely everything theory and implementation check out Broken Thorns. You'll need to Google for it because I'm on my phone.
Two small notes: the website is down (it's quite old and I'd be sad if the owner has a bit of bus-syndrome), you can find it on the waybackmachine
It's the best tutorial out there for an x86 based OS
Other than the educational value of doing so, what would be a reason to write a brand new operating system? Obviously there is a lot to learn about existing operating systems by coding you own, much in the way that writing a new languages teaches you about existing languages, but what are some creative applications of a hand made operating system?
> Other than the educational value of doing so, what would be a reason to write a brand new operating system?
Well an operating system talks to hardware and allocates resources, right? The motivation to write a non-toy operating system must be some strong ideas about how to do those things, and they'd have to be pretty fundamental to give up on the vast advantages of interoperability and pre-existing software that would come from just implementing your ideas on top of Linux or whatever.
Past motivations for writing OSes might be expressed as licensing (GNU/Linux), disagreements about fundamental architecture of operating systems (microkernels), need for hard real-time (vxWorks), or the belief that you have the sheer resources and market power to start again and build something better (Windows NT). Security seems like another likely motivation (There's probably a better example than OpenBSD because I don't think they started from scratch).
I quite like the idea that hardware intended for autonomy might benefit from being implemented on bare metal. Devices that AREN'T general-purpose computers, but communicate only with their own hardware and with the outside world by their actions and by communications protocols, well why shouldn't they run on a custom and highly stripped-down OS? You'd expect to get benefits in performance and a smaller codebase for potential security holes. But I don't think the industry is actually headed in that direction.
I quite like the idea that hardware intended for autonomy might benefit from being implemented on bare metal.
Some embedded OSes work by basically writing your application as an extension of the OS. So you're essentially running on bare metal there. Usually even for simple devices you want more than one thread/process....and then you start wanting OS stuff.
There's probably a better example than OpenBSD
Kaspersky's new OS is built for exactly that reason.
Also, for completeness, the osDev wiki has a list of OSes built for various reasons....some serious, some for research: http://wiki.osdev.org/Projects
Two, older examples are RTEMS and eCos. RTEMS is basically a single application with some OS code linked in. eCos was interesting in that it let you delete what you didnt need out of the OS. JX is an interesting, modern example where the OS is mostly a microkernel, parts of drivers, and JVM. Everything else, including driver logic, is in the JVM and depends on developer's needs.
Well, for one thing, you can explore computing ideas and paradigms that fall outside the realm of UNIX. What if instead of a filesystem, there was a persistant collection of live objects (with attributes and functions and everything), like Smalltalk? How about a system where there's a unified abstraction for both network and local I/O, ala plan9?
How about some new idea that hasn't been tried before? Maybe you'll find a better way of doing things.
This is where the education part is also helpful. It's been so long since I made a primitive OS that I wouldn't even know where to improve. But even some of the concepts in TempleOS are fascinating.
The funny thing is I was going ro reply to qwerty with that exact OS but thought it's shown up in comments too often. So, I didnt. Then it still surprises people when you do. Maybe I need to keep posting it when people ask these questions if it consistently has such an effect. Maybe see about getting the author over here too while we're at it.
I know Ive wanted to make changes to my own for a while. At very least default ui, menus and shortcuts after installation. These are modtly ux changes though
Additionally theres a few features I would prefer from a security user experience standpoint
- symlinks are indexed anD the move or deletion of the target would trigger a "would you like to change/delete the symlink as well?"
- all applications run in a container
- the application can never touch os filesystem or know about other apps but themself and what the user allows
- requests for hardware trigger permission dialog
- requests for files are handled outside the container
- shared folders are mounted to the container that the user knows are vulnerable
- some folders are an aggregate of the applications folders but the application only views its own
- deletion of an applications folder will permanently delete all if its data. No messy installations or artifacts
- system monitor shows cpu, memory and network usage per application
- applications require permission to use network and outward calls can be captured before being made
Additionally there are features from a dev standpoint
- there is a discoverability system for modified and alternative applications
- there is a focus on ensuring all parts of the os are easily found and modified
- changes to the os filesystem creates forces the user to create a virtual machine, make changes to that then "commit" those changes in bulk
- the user can make these patches available online along with keywords, problem and a tutorial inorder to recreate for discoverability
- heavy focus on user interfaces that can be reused in terminal and gui. Applications that only are available in one or have different experiences will be flagged as such.
> symlinks are indexed anD the move or deletion of the target would trigger a "would you like to change/delete the symlink as well?"
Patch your coreutils. Difficulty: easy.
Most of your other needs can be addressed by making a specialized Linux distro (difficulty: hard), rather than writing a kernel that does all of this (difficulty: max). Build your packages to be containerized by default. I suggest using chroot containers and writing some kind of FUSE filesystem to mount in the container and provide controlled access to user files. You'll also want to write or modify a Wayland compositor to suit your needs. You can have this provide the permission prompts and similar things. Difficulty: hard, but very much doable.
Skills to learn: Linux, thoroughly. You should be comfortable making packages, running things in containers, and writing C before you embark on this.
You should definitely build this. I'm working on something similar but my design appeals to fewer people. Feel free to reach out to me if you have questions with anything, email is in my profile.
Several of the features you've described are present in mobile operating systems, like Android and iOS. For example, "the application can never touch os filesystem or know about other apps but themself and what the user allows" and "requests for hardware trigger permission dialog". Android and iOS both have this model. Most of what you've described can be implemented in the application layer, and don't require writing a new kernel, and the few others could be implemented with small patches to existing well-established kernels like Linux or FreeBSD. We've already got the infrastructure for the container-related features you've described, so implementing what you want is a matter of creating an system-level policy that allows applications to only run in containers and enforces the other restrictions you've described.
Starting from scratch is one way to achieve this. But I suggest that you look into the GNU Hurd if starting from scratch seems daunting. Its potential is underused by the likes of Arch Hurd and Debian Hurd. One of the things that it can do is substitute user-supplied intermediaries, for just about anything, for selected processes.
This is probably a good thread to ask: one of my hobby projects is writing an OS, but for x86-64. My next task is user mode, and while I feel pretty solid conceptually, the details around the TSS and such I'm much shakier on. Does anyone have a good, detailed, with-code resource on such things?
I'm part-way through my implementation of an x86_64 experimental OS and had to address all the issues you're facing right now. I didn't really document my process, but the code to get everything booted into x64 mode is there, including setting up the TSS.
Neat thanks! I will check it out. It's really the context switching itself that's the trickiest part, I think. Or all of it. To me, it feels exactly like when I was learning about interrupts: while I was doing it, it was so tough... but as soon as I got something working, it made perfect sense and I wondered why I found it so hard in the first place.
One resource I found myself consulting over and over again was the Intel Architecture Software Development manuals (https://software.intel.com/en-us/articles/intel-sdm). They're huge and obviously a bit dry, but they had about 90% of what I needed to know about x86 memory architecture and interrupts.
I included some other resources I consulted in the README. The OSDev wiki (http://wiki.osdev.org/Main_Page) is also good, although I found some of it to be dated or incorrect.
Good luck on developing your OS. Even though I'm not finished, it's been an extraordinarily valuable and educational process.
Yeah, I probably need to just take a weekend day and actually read rather than skim and look for secondary sources. The OSDev wiki's pages on this particular topic are all kind of okay, but often aren't talking about long mode, only protected mode, or only have bits and pieces.
Thank you! Good luck with finishing yours, as though these things are ever finished :)
Does anyone have a similar guide for ARM, and specifically an Android device? I wrote a small x86 kernel for my class project in school. Being able to run an OS on a phone seems like a lot more fun.
The board I use is the mini2440 single board computer. You can bring up the same on a beagle bone with some minor modifications when you are dealing with the peripherals.
I also would add DMA (direct memory access) transfers to round the IO model. It's important for performance and affects a quantity of devices (disk, net, sound...).
OS dev can be great fun and gives you a level of understanding about how computers actually work that is hard to gain from other sources. I have frequently found myself using the knowledge I gained from building the OS when working on other projects. It has been particularly useful when it comes to understanding performance issues. I would highly recommend it to anyone that is even remotely interested.
Other superb resources to help you get started include the OSdev wiki[1], along with this document[2] from the University of Birmingham, UK.
[0] http://www.hydros-project.org
[1] http://wiki.osdev.org/
[2] https://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures...