
Ask HN: Recommended resources to learn the Linux kernel and OS theory? - non-entity
So recently I did a couple of minor patches on the FreeBSD and NetBSD kernels and played with some linux kernel. It was the first time in a a few years, I&#x27;ve been excited about programming.<p>Unfortunately I&#x27;m still completely lost. It seems that there&#x27;s so much to learn and every kernel update breaks whatever you just finished writing. I see terms in unfamiliar with and often find myself googling basic OS concepts.<p>Are there any recommendations reading materials to get a better grasp on OS theory and &#x2F; or the Linux &#x2F; UNIX kernels and programming for them?
======
waddlesplash
The OSDev wiki can probably teach you anything you need to know about "OS
theory" (and practice, mostly on x86):
[https://wiki.osdev.org/Expanded_Main_Page](https://wiki.osdev.org/Expanded_Main_Page)

Depending on your style of programming, I'd recommend maybe not working on
Linux if you are trying to learn kernel development and OS internals.

Have you looked at Haiku ([https://www.haiku-os.org/](https://www.haiku-
os.org/))? We have a _very_ well organized and commented modular-monolitic
kernel, and a pretty active development team with a wide range of experience
levels. I'm more than happy to help you (or anyone else) learn OS development!

Some example Haiku kernel code:

* "load_image_internal()", which is responsible for creating processes: [https://github.com/haiku/haiku/blob/master/src/system/kernel...](https://github.com/haiku/haiku/blob/master/src/system/kernel/team.cpp#L1679)

* a recent change by me to replace a global lock with two local ones: [https://github.com/haiku/haiku/commit/37eda488be1c9fee242e8e...](https://github.com/haiku/haiku/commit/37eda488be1c9fee242e8e4bf6ca644dd13441d8)

~~~
xz0r
> I'd recommend maybe not working on Linux if you are trying to learn kernel
> development and OS internals.

Could you also tell why ?

~~~
waddlesplash
The Linux kernel is really, really large, and changes extremely rapidly, to
the point of breaking such "experiments" (as the original poster already
noted.) It's also rather loosely commented and, aside from a handful of core
systems, not so well documented.

The BSDs do much better here; they have excellent manpages for almost all
kernel APIs, manuals for how the internals work, and very good source code
commenting and organization practices. Haiku is not as good as the BSDs on
manpages and manuals, but that's the only real way we differ from them (in
this respect.)

------
hatsubai
One important part about learning how the Linux kernel works is understanding
the details of the system's interface the kernel provides. In my opinion,
there is no better book out there than the "Linux Programming Interface" by
Michael Kerrisk: [http://man7.org/tlpi/](http://man7.org/tlpi/)

It provides extremely detailed information about everything going on in Linux,
as well as example programs and exercises to help you further your knowledge.
While it doesn't get deep into kernel theory like Tanenbaum's books tend to
do, it will provide you with a greater understanding of how things work, IMO.

~~~
aidos
£50 on kindle! Looks like a great book, but that’s punchy. (Maybe that’s
standard for a text book like this, not something I would normally buy)

~~~
geofft
It's a huge book (1552 pages), much larger than even your average OS textbook.
It's more like a Linux encyclopedia.

------
filereaper
Many folks here are recommending reading up on production grade operating
systems which might be a very steep learning curve.

I'd recommend starting with an academic operating system to nail the
fundamentals down, once you have a solid foundation you can then use all the
excellent links provided by everyone else and ramp up on production grade
kernels.

I have linked to multiple course syllabus'es below from excellent institutes,
pick whichever feels most comfortable.

Harvard with MIPS based OS/161 (I learned this personally):
[http://www.eecs.harvard.edu/~cs161/syllabus.html](http://www.eecs.harvard.edu/~cs161/syllabus.html)

Berkeley EECS 162 (John Kubiatowicz is amazing and is behind RISC V)
[https://inst.eecs.berkeley.edu/~cs162/sp19/](https://inst.eecs.berkeley.edu/~cs162/sp19/)

MIT OCW: [https://ocw.mit.edu/courses/electrical-engineering-and-
compu...](https://ocw.mit.edu/courses/electrical-engineering-and-computer-
science/6-828-operating-system-engineering-fall-2012/lecture-notes-and-
readings/)

------
geofft
Yes, there is just so much, and it is not thoroughly documented. "BPF maps" in
particular are a) a special feature of a special feature that's Linux-
specific, not anything that generalizes across OSes, and b) a feature for
userspace anyway, not for kernel internals. But even longstanding kernel-
internal features aren't well documented (e.g., the other day I was trying to
figure out what struct file's f_version does, and I think there's genuinely no
docs for it.) So honestly I think the answer there is to not feel bad about
not knowing everything.

The basic trick of dealing with the kernel is becoming comfortable working in
a large codebase most of which you don't understand, and figuring out how to
find what you need. Honestly, git grep is one of the best tools here. Get some
practice finding some specific thing and where it's implemented, e.g., find a
syscall (git grep SYSCALL.*foo) and trace what it calls. Find the definition
of a structure inside include/ and see who uses it. Get comfortable with the
kernel's OO-ish system of operations structs, and get some practice tracing
both "this function makes a generic call, here's a sample driver that
implements it" and "this is an implementation of a generic function, here's
the syscall that calls it."

Beyond that, reading [https://lwn.net](https://lwn.net) 's articles is
invaluable, partly for the clear prose coverage and partly for the breadth of
what they talk about. (You don't need to pay unless you care strongly about
this week's updates - you'll learn plenty from reading articles a week behind
- but support them if you can, they're an important resource.) Again, you're
not going to follow exactly why e.g. Google wants a new syscall for
"restartable sequences" on your first read, but you'll get a sense of what is
involved in adding a syscall, how various concurrency models work, what other
kernel features are relevant, etc.

What sort of OS concepts are you finding yourself Googling? I will say that
actually doing a college OS class is what made things like virtual memory
management click in my head. It's an intensive approach but writing code in a
much smaller kernel than Linux is a valuable way to understand concepts
without being drowned in real-world optimizations and edge cases and
portability.

~~~
non-entity
Perhaps I should've said more. I do often end up googling basic OS stuff,
especially related to memory (software interrupts still confuse me though, I
understand the concepts, but never the implementation), but even more so have
been my ignorance of hardware level concepts. For example trying to patch the
code for a network card the other week, I learned what watchdog timer is, I've
never heard of that prior.

~~~
geofft
Working through an academic OS class where you implement interrupt handling is
super helpful for that - hardware interrupts and software interrupts are
similar but not _quite_ the same and actually writing the interrupt handlers
makes some things clear.

I can endorse MIT's 6.828 (I've taken it and taught a version if it), which
(at least before this semester, they seem to be redoing things) has a lab
about setting up interrupt handling:
[https://pdos.csail.mit.edu/6.828/2018/labs/lab3/](https://pdos.csail.mit.edu/6.828/2018/labs/lab3/)
It's entirely possible to complete the labs on your own, and doing labs 1-3 is
worthwhile, but it's also a significant time commitment.

Watchdog timers fall in the category of "there is so much" \- they're not
crucial to how network drivers work, but drivers in a production OS have to
care about every feature that might interact with them. :(

------
pm215
My somewhat off-beat recommendation is "Lions' Commentary on UNIX 6th Edition
with Source Code". The sources to a 1976 version of Unix written in an archaic
dialect of C and targeting a long-dead CPU architecture are obviously not of
immediate relevance to the modern world, and may well not be to your taste
unless you already have some sympathy with the idea of retrocomputing. But if
you do: 6th Edition is small enough that you really can read through the whole
set of sources and understand pretty much how the whole thing is put together;
the basic bones of the design are not so far away from Linux, so it gives you
a workable conceptual model of what the shape of the Linux sources are and
what the various important parts are; the commentary is really good,
explaining the inscrutable but nudging you to figure things out yourself too;
and overall it is practice in reading a large volume of somebody else's code,
which is something you're going to be doing all the time if you work on Linux
or any modern kernel. You probably also want a more theoretical and general
book too, of course.

~~~
jiveturkey
And you can witness the glory of swtch() and the original "You are not
expected to understand this" comment.

Highly recommended.

------
whydoyoucare
"Operating Systems: Three Easy Pieces" is a more modern book to OS concepts,
explained with real source code as opposed to algorithms, and is regularly
updated.

[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/)

(Online version above, you can also order a print copy).

~~~
0x262d
I took this class (although not from him). it's a good textbook and was pretty
much sufficient.

------
lorenzfx
If you are still interested in FreeBSD, "The Design and Implementation of the
Freebsd Operating System" is a fantastic book. It's a great guide in how a
UNIX system and especially FreeBSD works and written by some of FreeBSD's most
respected developers.

~~~
non-entity
I actually did get a copy of that! Unfortunately, I only have the ebook at the
moment, which I find difficult to read

~~~
ashafer
If you want something more hands on you can try "FreeBSD Device Drivers: A
Guide for the Intrepid" by Joseph Kong. It is much easier to read, and my
guess is you ran into it while making those BSD patches.

He also has a short book on building rootkits for BSD, which is a very fun
read and demonstrates how things like system calls work.

If you want a guide to reading actual kernel code, here is a great resource
explaining where some of the foundational pieces are. With a tool like cscope,
reading the FreeBSD kernel is actually pretty easy.

[http://www.watson.org/~robert/freebsd/reading/](http://www.watson.org/~robert/freebsd/reading/)

Hope that helps!

------
qntty
If you want to understand Linux systems programming, I've never found anything
better than the book _The Linux Programming Interface_. For Linux internals,
_Linux Kernel Development_ by Robert Love is good. For something more removed
from current systems, _The Design of the UNIX Operating System_ and _Operating
Systems: Design and Implementation_ are good.

------
weinzierl
Not _very_ Linux specific but Tanenbaum's _" Modern Operating Systems"_ is an
excellent source for OS theory and very readable as well. It's an expensive
book, but I found it worth its money.

~~~
Noe2097
Seconded. I would add "Structured Computer Organization", as a preamble book.
I read both as if they were novels - they really are interesting and written
in such a way that can hook you just as a good story

------
molteanu
Operating Systems: Three Easy Pieces [1]

XinuOS [2]

MIT's xv6 OS [3]

[1]
[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/)

[2] [https://xinu.cs.purdue.edu/](https://xinu.cs.purdue.edu/)

[3] [https://github.com/mit-pdos/xv6-public](https://github.com/mit-
pdos/xv6-public)

~~~
apengwin
+1 for xv6. The commentary book is really well written, and is a
straightforward introduction to a lot of important ideas.

[https://pdos.csail.mit.edu/6.828/2014/xv6/book-
rev8.pdf](https://pdos.csail.mit.edu/6.828/2014/xv6/book-rev8.pdf)

------
bytematic
OSTEP is the best book on Operating systems easily.
[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/)

~~~
apacheCamel
Not sure if it is the best book overall, but I used it for one of my college
classes and I think it is a great book for learning much more about operating
systems. And it is available for free online so my college student wallet
loved it!

------
yodsanklai
Not Linux, but I'd recommend this lab from MIT.

[https://pdos.csail.mit.edu/6.828/2014/overview.html](https://pdos.csail.mit.edu/6.828/2014/overview.html)

I completed this one, but they may have more recent versions. They give you
some initial code that you need to complete, as well as tests to check your
work before going to the next step. It covers a lot of material. It takes some
patience and it's not for complete beginners but it's a great fun project. Not
sure, but I'd say it took me 1 or 2 weeks full time.

------
peterkelly
I recommend starting with something smaller.

Here's something I put together some years ago - it's about 5,000 lines of
code and supports a number of key Unix features. There's a PDF of lecture
notes accompanying it.

[http://adelaideos.sourceforge.net/](http://adelaideos.sourceforge.net/)

Once you've gone through this you'll be in a better position to tackle the
Linux kernel and books on OS architecture. My personal favorite is "The Design
of the UNIX Operating System" by Maurice J. Bach. It's an old book but
explains things well, and I believe it was one of the references Linux
Torvalds relied on when creating Linux.

------
gamescodedogs
Years ago in school, I decided to figure out how the Linux kernel works. So
I've downloaded the sources and open them with a notepad++. After an hour I've
figured out that I have no idea what am I reading in the code, close the
notepad++ and never open it back. Buy the sources are still awaiting in some
old directory at my home's PC :)

------
lignux
This looks like a good start: [https://0xax.gitbooks.io/linux-
insides/content/index.html](https://0xax.gitbooks.io/linux-
insides/content/index.html)

------
yourbandsucks
The O'Reilly book Understanding The Linux Kernel is fantastic. Starts you off
in the deep end with memory management/addressing and fans out from there.
Notably excludes networking, as the book is long enough without it.

~~~
DanAtC
Any suggestions for Linux networking? I have Understanding Linux Network
Internals but it’s a bit dated.

~~~
yourbandsucks
I think it comes down to just understanding IP and TCP in general. Which I
don't really (IP) or do sorta (TCP).

As far as kernel architecture, the rest of the system can work with the
assumption "writes go into a buffer, reads come from a buffer, and the
networking subsystem does the right thing in between", plus some stuff about
DMA optimizations.

------
kernyan
[https://github.com/s-matyukevich/raspberry-pi-
os](https://github.com/s-matyukevich/raspberry-pi-os)

Matyukevich page on GitHub has lessons with accompanying code to develop a
Linux OS for raspberry Pi. The author goes into kernel, processor
initialization, interrupt handling, scheduler, implementing syscall, and
virtual memory.

I like his approach for these reasons 1\. Minimal workable code, 2\. Points
you to entry function in linux repo 3\. Line commentary of those entry
function

Note that it's also a somewhat short read as he focuses on the practical
implementations instead of the theory.

I myself was hoping that the author continue developing the chapters on file
system, drivers, and networking but seemed to have been on hiatus.

------
throwawaypls
Linux Kernel Development By Robert Love and Operating Systems Concepts by
Silberschatz are great resources.

------
rubyn00bie
This book has been immensely helpful to me, it's "Writing an OS in Rust"
[https://os.phil-opp.com/](https://os.phil-opp.com/) and I can't recommend it
enough.

Also, Redox OS is an OS written in Rust and has a ton of wonderful
documentation on it that's been helping me get into systems development:
[https://www.redox-os.org/docs/](https://www.redox-os.org/docs/)

------
alfiedotwtf
For the Linux kernel, there is an excellent book called "Linux Core Kernel
Commentary" (which is based off the SVR4 Lion's Commentary book), which has
the source of the kernel (an old version) printed out and then a commentary on
all the bits (code and data structures). It was an eye opener for me.

As for OS theory, "Operating Systems Vade Macum" and "Operating Systems Design
and Implementation" are my favorites.

------
rpark
A classic: The Design of the UNIX Operating System by Maurice Bach.
[https://www.amazon.com/Design-UNIX-Operating-
System/dp/01320...](https://www.amazon.com/Design-UNIX-Operating-
System/dp/0132017997)

I learned more about OS from this book than any academic material used at the
university and gave me a good foundational handle on *NIX's.

------
ndesaulniers
Join us in /r/kernel. As a mod I'm always helping folks with questions like
this, and our community frequently beats me to the punch.

~~~
non-entity
Didn't know that existed. Will definitely check it out.

------
aeyes
The Linux kernel is unfortunately pretty far from UNIX nowadays but if you
want to understand the basic concepts behind UNIX I think Lions' Commentary on
UNIX is still the best resource out there:

[https://www.amazon.com/Lions-Commentary-Unix-
John/dp/1573980...](https://www.amazon.com/Lions-Commentary-Unix-
John/dp/1573980137)

------
Jach
Zhao Jiong released this excellent work in English not too long ago:
[http://www.oldlinux.org/download/ECLK-5.0-WithCover.pdf](http://www.oldlinux.org/download/ECLK-5.0-WithCover.pdf)
If you know Chinese it was originally done in that language, details are on
[http://www.oldlinux.org/](http://www.oldlinux.org/) It heavily comments /
explains everything in the linux kernel v0.12. Quite some time ago, but I
agree with the assessment on its relevance for an "inductive understanding".
The growth since then is largely in new subsystems, new drivers, but maybe not
so much in core theory...

If you're more into the core theory and less about Linux in particular, a fun
project-oriented approach to get more familiar with the basics would be to get
a little development board for an ARM CPU and try to make a multi-tasking
(real-time) OS for it from scratch.

~~~
gabrielblack
Thanks, I haven't this one, it seems interesting.

------
moron4hire
I've semi-seriously been considering starting my own OS project to circumvent
some of the IO latency issues with VR and AR applications.

The big metric is motion-to-photon latency, which is a measure (in
milliseconds) of how quickly a user's head motion is reflected on the display.
Anything greater than ~10ms is considered insufficient, as you've missed a
whole frame of rendering for VR headsets that run at 90hz refresh rate.

Current XR applications need to wait for the USB or Bluetooth stack to return
the head pose before modifying the view, then wait for the graphics stack to
reflect that change. There are many ring transitions along the way. This leads
to an absolute minimum motion-to-photon latency that can't be broken just
because of OS overhead. This eats into your frame budget, as every ms of
context-switching out of the IO drivers leaves you fewer ms to render a frame
in time to get it back into the kernel for context-switching back to the
graphics driver.

------
linguae
For learning the basics about how an operating system works, I recommend Remzi
and Andrea Arpaci-Dusseau's "Three Easy Pieces", which is freely available at
[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/).
Both Remzi and Andrea Arpaci-Dusseau are great writers and are prolific
researchers in the areas of operating systems and file systems, and their work
is top-notch.

For learning the basics about how the kernels of Unix-like systems are
designed, I recommend the following books:

\- The Design of the UNIX Operating System (1986) by Maurice Bach. You could
also compare this book to the source code of Version 7 Unix, which is freely
available under a BSD-style license. Many of the basic design principles of
Unix-like operating systems have not changed dramatically over the years, and
so studying the design and implementation of older, smaller implementations of
Unix-like operating systems can aid with figuring out the code base of Linux,
which is huge.

\- UNIX Internals: The New Frontier (1996) by Uresh Vahalia. This discusses
some more modern features that weren't part of AT&T Unix.

\- The Design and Implementation of the FreeBSD Operating System (2014) by
Marshall Kirk McKusick et al. I've met "Kirk" in person a few times at USENIX
FAST conferences; he's not only a great writer, but he is also very friendly
and approachable. This is a well-written text describing FreeBSD 11, including
modern features such as ZFS.

There are some good Linux kernel books (I'm partial to "Linux Kernel
Development" by Robert Love, which helped me out nine years ago when I was
writing a device-mapper device), but what I've found during my experience
working with the Linux kernel is that the Linux kernel changes much faster
than the books describing it get updated. Nevertheless, the best thing to do
is to get a solid understanding of how Unix works, then get a basic
understanding of the Linux kernel, then start diving into the source code,
which is ultimately the definitive documentation of the Linux kernel.

When I was working on that device-mapper device, I found that perusing the
Linux kernel mailing list was excellent for learning about how relevant
portions of the kernel changed over time.

Good luck and have fun! I haven't done much kernel-level work lately, but I
miss working at that level!

------
theonewolf
The Minix OS was designed as an academic, teaching OS:
[https://www.minix3.org/](https://www.minix3.org/)

There is a textbook that goes along with it:
[https://www.pearson.com/us/higher-
education/program/Tanenbau...](https://www.pearson.com/us/higher-
education/program/Tanenbaum-Operating-Systems-Design-and-Implementation-3rd-
Edition/PGM228096.html)

The author famously had interactions with Linus Torvalds when he was starting
Linux!

And, Minix is __inside __every modern Intel CPU
([https://www.zdnet.com/article/minix-intels-hidden-in-chip-
op...](https://www.zdnet.com/article/minix-intels-hidden-in-chip-operating-
system/)).

~~~
non-entity
Oh yes, I've been curious about breaking into the Minix in intel chips.

------
darkr
If you’re looking for something specific to Linux, that is less about the
kernel, and more about how the whole operating system is strung together, I
can highly recommend:
[http://www.linuxfromscratch.org/](http://www.linuxfromscratch.org/)

------
jiveturkey
The old standards.

Stevens to start. If you don't already know Stevens' thoroughly first, I
wouldn't go deeper. The kernel is not separable from the userland-facing API.

I'm not convinced you actually need to / want to understand actual internals.
But assuming you do, next is Tanenbaum (maybe start with the newer "Modern"
book).

"Operating Systems: Three Easy Pieces" also seems well regarded, although I
haven't read it myself.

The FreeBSD book is great, if you're specifically interested in BSD. There's
also a video lecture you can buy on McKusick's site. I had the pleasure of an
on-site presentation and it is absolutely fantastic.

------
Diederich
This is a somewhat different take, and quite possibly impractical.

The main thing I've done over the years to get reasonably 'good' at the Linux
kernel (and quite a few other things) has been to read every technical article
appearing in [https://www.lwn.net/](https://www.lwn.net/), every week, in
full, even if I didn't really understand the material. I've been doing that
since 1997, and it's paid off quite well.

Also: please buy a membership, even though you don't have to. It's well worth
supporting.

------
djhworld
I've been making my way through this course
[https://classroom.udacity.com/courses/ud923](https://classroom.udacity.com/courses/ud923)

It's about 9 years old I think so might be too out of date for you, but I've
been really enjoying it. I think they have an advanced version of the course
as well if you're already familiar with the basics

~~~
Mxtetris
That appears to be the Intro to Operating Systems course from the Georgia Tech
OMSCS program.

Udacity also hosts the GT Advanced Operating Systems course:
[https://www.udacity.com/course/advanced-operating-systems--
u...](https://www.udacity.com/course/advanced-operating-systems--ud189). AOS
is not a "how to" course, but rather a tour of the challenges operating
systems may face (distribution, fault tolerance, scale) and of academic papers
attempting to solve them (mostly from the 90s and 00s). As such, the material
might not be a good fit for the original post, but this commenter found it
fascinating.

The reading list is here:
[https://www.udacity.com/wiki/ud156-readings](https://www.udacity.com/wiki/ud156-readings)

------
gravypod
These are some good resources on the things that go into creating your own
kernel:

    
    
      - http://www.brokenthorn.com/Resources/OSDevIndex.html
      - http://www.osdever.net/bkerndev/Docs/intro.htm
    

For learning about Linux specifically:

    
    
      - https://0xax.gitbooks.io/linux-insides/content/index.html

------
no-dr-onboard
In school we used Tannenbaum's MINIX book (Operating Systems Design and
Implementation). It was generally a pretty good introduction to "OS Theory",
directly references the source code in the back. Useful for a desk book if
you're constantly elbows deep in writing syscalls.

------
rapphil
In the past I did this: [http://eudyptula-challenge.org/](http://eudyptula-
challenge.org/)

Not sure if they are accepting new applicants, but you can try. Also, you can
try to find the assignments online.

~~~
neuland
The assignments are online in various places. Here is a copy I made from
someone's Github (not linking GitHub b/c it had answers too):
[https://pastebin.com/cUXuHcDc](https://pastebin.com/cUXuHcDc)

------
n_t
From my previous comment -
[https://news.ycombinator.com/item?id=18451260](https://news.ycombinator.com/item?id=18451260)

------
apples_oranges
From not too long ago:
[https://news.ycombinator.com/item?id=19225268](https://news.ycombinator.com/item?id=19225268)

------
mesarvagya
In addition to OSTEP, Linux From Scratch [1] is also a good resource.

[1] [http://www.linuxfromscratch.org](http://www.linuxfromscratch.org)

------
master_yoda_1
[http://inst.eecs.berkeley.edu/~cs162/sp19/](http://inst.eecs.berkeley.edu/~cs162/sp19/)

------
hd4
[https://github.com/0xAX/linux-insides](https://github.com/0xAX/linux-insides)

------
duckqlz
Beyond the exceptional mentioned texts dedicated to the subject; my favorite
resources are:

lwn.net

github.com/torvalds/linux

------
ekr
Start with something smaller, such as lk.

