
Operating Systems: From 0 to 1 - tuhdo
https://tuhdo.github.io/os01/
======
brw12
tuhdo, can you say more about how the process works to convert your book from
lyx text source to pdf? Since the current pdf seems to have particular
formatting added to the source text (eg, terms not emphasized in remaining.txt
are italicized in the pdf), it's a little tricky to tell. Is the raw lyx file
where the formatting rules live? Does the lyx file store its text source in
text files, but its formatting rules in binary?

------
radisb
From the book:

"If a programmer writes software for a living, he should better be specialized
in one or two problem domains outside of software if he does not want his job
taken by domain experts who learn programming in their spare time."

Seems a bizarre sentiment, but after reading this sentence, I feel like I
really wanna donate some money to the guy. If he gives a way I will surely do.

~~~
shacharz
Can you explain more what inspired you in this sentence or are you still
thinking it's bizarre? And if you do still think it's bizarre why would that
convince you to donate?

~~~
karmajunkie
not OP, but one thing we as developers should recognize is that programming
itself is likely to become merely a skill like operating a spreadsheet or
typing. In much the same way, writing was once a skill available (or at least,
practiced) by a very small subset of people, and the spread of literacy made
the job of "scribe" largely irrelevant over time. Children today are learning
how to program—perhaps not yet in the numbers we might have imagined, but a
growing number. As they grow up, they are not going to pursue jobs as
"programmers" or "developers", any more than I would pursue a job in a typing
pool. They will merely incorporate this skill into whatever role they end up
in.

Those best suited to compete in that scenario are going to have deep
experience in some area where programming is an applied skill, not an end unto
itself.

~~~
mbrumlow
While a agree to some extent and have even explored the idea of how my
programming skills could be augmented with other non-programming jobs I still
feel that for the foreseeable future we will have programmers.

I see lots of jobs being done by non-programmers that would be done much much
better had that person been a programmer. The future will undoubtedly be full
of people in traditionally non-programming jobs with programming skills. But
we should not trivialize the systems that allow this to happen. While most
programmers work very high in the stack there are many very deep layers that
sit below that will require a army of programmers to keep going and keep
innovating so more productivity can be had at the higher layers.

As an example I feel people often trivialize one or two liners programs, or
short examples that seem to do so much for such a few terms. But even those
simple examples are backed by sometimes 10 of thousands of lines of codes. A
wave of a hand, and a bark of spoken command to a computer will -- at least
with today's software and hardware -- never end up producing something new and
innovated that will be widely used.

To get where you are suggesting many things have to change. And it's not all
software. If you even take a cursory glance at what it takes today to
bootstrap a operating system -- you would then see how complicated things get.
For the future I feel you are envisioning the hardware will have to be worked
along with software to make things much much much more modular than they are.
More standardised terms to describe things and more finite building blocks
will need to be made. In fact the closest thing I could get your future vision
with today's technology is FPGAs and it is no trivial task to do anything
useful with a hardware described language.

tl;dr We have not _solved_ computers or software, therefor; a army of
programmers will be required for the foreseeable future.

~~~
karmajunkie
I don't think that day is as far off as you seem to. Yes, there are some kinds
of programming which require deep expertise in programming itself, but these
jobs are far from the majority.

If you've ever read Vinge's _Deepness in the Sky_, recall that the
protagonists occupation for a period of time was that of a code archeologist,
attempting to collect, categorize, and understand code that was centuries (or
was it millenia?) old... We may be a ways off from that, but I don't think its
far from the truth of the future.

------
bogomipz
This looks great!

I would also like to recommend another free resource that might be a good
complement(theory vs implementation) to this:

"Operating Systems: Three Easy Pieces"

available online at:

[http://pages.cs.wisc.edu/~remzi/OSTEP/](http://pages.cs.wisc.edu/~remzi/OSTEP/)

~~~
redpanda_ua
I haven't yet finished this book, but this is very good book imo. Can
recommend.

------
js8
Looks very nice so far. One minor nitpick - there should definitely be a
chapter on inter-process communication, it's an important part of operating
systems.

------
itsmemattchung
Every time an OS book (about weekly) is posted on HN, I immediately want to
jump all over it but I gently remind myself to finish: CMU's Computer System's
from a programmer's perspective and Elements of computing (nand2tetris).

~~~
harrisi
It seems you've already decided that these are worth reading, but in case you
want some motivation, I had CS:APP as my textbook last term and I thought it
was quite nice. I also worked through TECS several years ago and it was also a
great resource. CS:APP is quite dry but by the end of the course I was able to
write a real memory manager in C, which was super fun! If you're ever looking
for an internet book reading buddy, I also need motivation. I worked through
TECS with a friend over a few weeks and found it very useful to have someone
else seeing and working through the same problems.

Good luck!

~~~
dbancajas
TECS? I would like to have online buddy to do this self-learning courses. Most
of my projects fizzled due to lack of accountability.

~~~
harrisi
TECS is The Elements of Computing System. Feel free to send me an email!

------
koolba
For those of you that have read, or at least skimmed, this, how does it
compare to the Minix book? ( _which was a joy to read!_ )

------
lorenzfx
How much time would it approximately take to work through this?

------
kriro
Seems to be for a x86 operating system. I'd have preferred some other
architecture because so much of OSdev for x86 that I remember was working
around quirks of the architecture (A20 gate etc.). I guess it's a valuable
lesson but I'd really enjoy a fork of the book for the hardware you find in a
Beagle Bone or Pi3 or something. Maybe this could be crowdfunded if the x86
version is popular?

~~~
tuhdo
Author here. Yes, it's a x86 operating system. However, rather than getting
around A20, it focused on protected mode instead.

The book not only teaches x86, but how to use the official resources from the
hardware manufacturer to write the OS. In sum, a reader when reaching part 3
for writing the OS, he will need to use the official document, in this case,
the "System Programming Guide" manual from Intel to write C code that complies
with the documents. Once he learned how to do so, learning other platforms
will be much easier given how complex x86 is.

~~~
wolfgke
> Author here. Yes, it's a x86 operating system. However, rather than getting
> around A20, it focused on protected mode instead.

You still have to open the A20 gate in the bootloader if you want to access a
memory adress that has bit 20 (counting from 0) be set to 1 (you probably
want) - even if you switch to protected mode. The only exception is if you
boot from UEFI instead of BIOS - in this case the A20 gate is already set. But
the book uses BIOS as far as I see it.

~~~
tuhdo
In the Volume 3, it is said that:

"In protected mode, the IA-32 architecture provides a normal physical address
space of 4 GBytes (2 32 bytes). This is the address space that the processor
can address on its address bus. This address space is flat (unsegmented), with
addresses ranging continuously from 0 to FFFFFFFFH. This physical address
space can be mapped to read- write memory, read-only memory, and memory mapped
I/O. The memory mapping facilities described in this chapter can be used to
divide this physical memory up into segments and/or pages."

It correlates to my experience of developing in protected mode in QEMU. Once
entered protected mode, I can access to any address above 0x10000 without
being wrapped around. When I was writing my first kernel
([https://github.com/tuhdo/os-study](https://github.com/tuhdo/os-study)) in
real-mode, indeed A20 must be enabled.

~~~
Vendan
The A20 gate would affect both real and protected mode, so to be truely
compliant, you should be opening it even for a protected mode OS. It was
originally a purely hardware hack forcing an address line(A20, or the 21st
address line) to stay low, thus "wrapping" addresses from 1mb-2mb to 0-1mb, so
it would have affected real or protected mode identically. On modern hardware,
there is a chance that A20 is (against the accepted spec) default open, or
possible not implemented.

~~~
wolfgke
> On modern hardware, there is a chance that A20 is (against the accepted
> spec) default open, or possible not implemented.

To nitpick on this a little bit (consider it as a polite supplement):

In Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3 (3A,
3B, 3C & 3D): System Programming Guide

in section

8.7.13.4 External Signal Compatibility

one can read (emphasis by me):

"A20M# pin — On an IA-32 processor, the A20M# pin is typically provided for
compatibility with the Intel 286 processor. Asserting this pin causes bit 20
of the physical address to be masked (forced to zero) for all external bus
memory accesses. Processors supporting Intel Hyper-Threading Technology
provide one A20M# pin, which affects the operation of both logical processors
within the physical processor.

The functionality of A20M# is used primarily by older operating systems and
not used by modern operating systems. _On newer Intel 64 processors, A20M# may
be absent._ ".

TLDR: The accepted spec is that A20M# might not exist.

~~~
Vendan
fair point, I hadn't bothered to look more, but it's still true that A20(when
implemented) gates both real and protected modes, and therefore "ignoring it
to focus on protected mode" is invalid

------
emiliobumachar
It's a from-first-principles guide to building an OS.

From the title, I had mistakenly assumed it was about the first OS ever.

~~~
devnonymous
Like almost everything in Computer science there would be no clear definition
of first OS ever. There used to be monitor programs and schedulers before they
iteratively evolved into OS programs. Just like, programming languages,
servers, 'cloud'... There seldom is a first ever something in our field.

~~~
brett40324
Im curious what "monitor programs" means here in this context?

The first OS was OS/360 at IBM in 1964.

~~~
skissane
OS/360 was not the first OS. Quite a number of operating systems came before
it.

Examples of earlier operating systems include SHARE Operating System (SOS),
first version in 1959, for the IBM 709 mainframe; which in turn was itself was
based on an earlier operating system, GM-NAA I/O, first version in 1956.

A very notable pre-OS/360 operating system was MIT's CTSS operating system,
the first timesharing OS, first version in 1961 for the IBM 7094. The first
versions of OS/360, by contrast, notably lacked timesharing – that was
initially provided by alternative IBM operating systems (TSS/360 was the
official answer and CP/CMS, which later became VM/CMS, the principal
unofficial one); OS/360 only gained timesharing support itself when TSO was
released in 1971.

------
laxentasken
I have on my list to read something along the lines like this, but this seems
incomplete. Does anyone have anything similar?

~~~
sagischwarz
This is what I'm planning to read as soon as I've learned assembly:
[http://www.brokenthorn.com/Resources/OSDevIndex.html](http://www.brokenthorn.com/Resources/OSDevIndex.html)

~~~
tuhdo
I graduated from that guide :) You can see the repo here, written entirely in
NASM: [https://github.com/tuhdo/os-study](https://github.com/tuhdo/os-study)

The problem is that the guide is out of date in terms of toolchain, and you
need to figure out many things by yourself, especially if you want to develop
on Linux. My book helps you to understand how to learn and write x86 with
Intel manuals (this is really important!), understand how to craft a custom
ELF binary that is debuggable on bare metal, which in turn requires you to
understand a bit of how debugger works.

Once you get gdb working, it is much easier to learn how to write an operating
system.

~~~
sagischwarz
Oh, good to know! I'll keep that in mind and keep a bookmark of your book and
your implementation. Actually, I wanted to start writing an OS by following
the BrokenThorn tutorial and was quite naive. After reading some pages it came
to me that I don't know much assembly and so I started learning from Jeff
Duntemanns Assembly book [1]. As far as I can see, your book also teaches the
basics of assembly, it seems more friendly to beginners. Maybe it will be a
better start for me. Thanks for putting in the hard work!

[1]
[http://www.duntemann.com/assembly.html](http://www.duntemann.com/assembly.html)

------
gigatexal
Well this has got to be one of the more ambitious things I've seen on HN. I
wonder if the guy behind SkyOS is still doing operating systems.
[http://www.skyos.org/](http://www.skyos.org/)

~~~
wolfgke
> I wonder if the guy behind SkyOS is still doing operating systems.
> [http://www.skyos.org/](http://www.skyos.org/)

No, he (Robert Szeleney,
[http://www.skyos.org/?q=node/411](http://www.skyos.org/?q=node/411)) doesn't.
He (together with Heiko Hufnagl) founded a studio for creating software and
games for mobile devices:

>
> [https://en.wikipedia.org/wiki/Djinnworks](https://en.wikipedia.org/wiki/Djinnworks)

> [http://djinnworks.at/about-new/](http://djinnworks.at/about-new/)

------
ejanus
I will read your book....and thanks for such a book

------
jerianasmith
The hallmark of a Good book is that it should leave you wanting more. That's
what the book is all about.

~~~
dbancajas
Have you read the book?

------
seewhat
On pg 38: _Field Gate Programmable Array (FPGA)_

Surely: _Field Programmable Gate Array (FPGA)_ ?

~~~
tuhdo
Thanks. I will fixed it.

------
lanbanger
The grammar makes that book fairly painful to read :-|

~~~
tuhdo
Hi, I would love to hear your feedback. It is the style or the correctness? I
am really appreciate if you can show me the few incorrect usages of the
grammar.

~~~
dbancajas
I can tell you're not native. Have the book proofread by someone.

~~~
tuhdo
Yes, I'm not native. For that reason I tried to use simple grammar and
sentences, and did go through paragraph by paragraph through a grammar checker
until the end of chapter 4. I will try to improve it gradually.

~~~
palerdot
It is better to proof read for correctness. I myself want to point out few
grammatical stuffs and looking for the source in Github. You can simple create
a repo or add the source text to the current repo, so that people can easily
fix these issues for you by giving pull requests.

You don't want people judging this excellent book based on few language quirks
here and there.

~~~
tuhdo
Thanks. I will upload the source soon. As it is written in Lyx, not sure
everyone will find it comfortable though. You can always open an issue in the
meantime though.

~~~
molteanu
Just curious why you chose Lyx and not Org, for example. I still have your
"emacs mini manual" open in my browser all the time, so I'm sure you feel
natural in org-mode too.

~~~
tuhdo
Well, Org is suitable for small notes. But for the scope of this book, it's
difficult to make Org work for such a complex layout (largely because I don't
want to mess with the CSS).

I used Lyx because it enabled me to focus on the content without all the
markup text. Writing Latex in Emacs can reduce the distraction, but not
enough. I just wanted to focus on the content at the time. Learning Latex is
difficult enough, learning how to use the major mode at the same time doubles
the difficulty.

Obviously, I still use Emacs daily for writing code and other things. Just not
for writing book.

------
anhldbk
Nice job man!

