
How modern Linux systems boot - adamnemecek
https://utcc.utoronto.ca/~cks/space/blog/linux/LinuxBootOverview?
======
voidmain
One day, through the primeval wood, A calf walked home, as good calves
should...

[1] [http://holyjoe.org/poetry/foss3.htm](http://holyjoe.org/poetry/foss3.htm)

~~~
bjackman
This is wonderful. By "S.W. Foss" as well...

------
pmalynin
For those interested: nowadays Linux has support for what is called the EFI
Stub boot, which allows the kernel to be booted directly by UEFI. So in a
sense steps 1-3 are rolled into the kernel, and the kernel acts as its own
boot loader.

~~~
secure
Do you know of any major distributions using this mechanism? I’ve only ever
seen GRUB on UEFI.

~~~
rl3
Arch supports it:

[https://wiki.archlinux.org/index.php/EFISTUB](https://wiki.archlinux.org/index.php/EFISTUB)

Being able to directly edit your motherboard's EFI boot order via command line
is admittedly pretty neat. That said, using a traditional bootloader seems to
be a popular choice still for simplicity and portability reasons. EFISTUB also
can make certain crypto setups more difficult as far as I'm aware.

~~~
checkyoursudo
On Gentoo, for a couple of years, I used an initramfs compiled into the kernel
to boot via EFI_STUB with full disk encryption. It's possible. It's been a
while. I don't remember it being very easy to wrangle. I don't use that laptop
anymore, but I still boot via EFI_STUB on my desktop. It's super simple to set
up when you're not using FDE.

I don't remember there being any specific problem that make some crypto setups
more difficult than others, with the caveat that implementing FDE at all makes
EFI_STUB somewhat more complicated no matter what.

Like I said, it's been a couple of years since I figured all that stuff out,
so the passage of time may have softened the terror of configuration and
clouded my memories.

~~~
rl3
> _... the terror of configuration ..._

That's an excellent way of phrasing it.

------
organsnyder
When I was first learning Linux (almost 20 years ago [yikes]), I installed it
by hand using the Linux From Scratch guide. It was invaluable for learning a
ton about how Linux works at a deep level. While a lot has changed since then
(mainly due to UEFI), that knowledge is still valuable for understanding how a
system functions.

[http://linuxfromscratch.org/](http://linuxfromscratch.org/)

~~~
BJanecke
A bit more lightweight version of that is
[https://wiki.archlinux.org/index.php/installation_guide](https://wiki.archlinux.org/index.php/installation_guide)
The first time I tried to install arch I had many aha! moments

~~~
zshrdlu
Similarly, the Gentoo Installation Handbook:
[https://wiki.gentoo.org/wiki/Handbook:AMD64](https://wiki.gentoo.org/wiki/Handbook:AMD64)

~~~
watersb
Gentoo taught me Linux. The Handbooks are really good.

I think that much of early Gentoo ideas - the OpenRC init system, the ports
system, the handbooks - were inspired by BSD traditions. OpenBSD and FreeBSD
each have good documentation. Working through the FreeBSD handbook, and then
study of McKusik's BSD Book [0], are a good way to get another perspective if
you get into this sort of thing.

[0] [http://www.worldcat.org/title/design-and-implementation-
of-t...](http://www.worldcat.org/title/design-and-implementation-of-the-
freebsd-operating-system/oclc/870847694)

------
AceJohnny2
> _How the initramfs /init pivots into running your real system's init daemon
> on your real system's root filesystem is beyond the scope of this entry. The
> commands may be simple (systemd just runs 'systemctl switch-root'), but how
> they work is complicated._

Darn, I was actually hoping this would be elaborated on. Does anyone know of
other (just as understandable) sources?

~~~
xfs
dracut does the work. The reason it is complicated is because it supports
booting from btrfs, DM RAID, MD RAID, LVM2, device mapper multipath I/O, dm-
crypt, cifs, FCoE, iSCSI, NBD and NFS.

[https://en.wikipedia.org/wiki/Dracut_(software)](https://en.wikipedia.org/wiki/Dracut_\(software\))

~~~
AceJohnny2
Hm, Dracut looks like a userspace framework for building initramfs. I was
wondering more about what's going on at the kernel level to switch the root.
What depends on / ? Do any processes survive whose CWD is in the initramfs?
Later in the article it's pointed out that systemd(from initramfs) passes the
torch to systemd(in final root) while preserving state, so are any
processes/ressources from the initramfs leftover or is everything cleared?

~~~
xfs
A process spawned from initramfs can certainly live on but you wouldn't want
to do that because that would prevent initramfs from being unmounted. By
switching to the new root you can run real binaries. You can also not switch
the root and also run real binaries from the new mount but that's not very
convenient is it?

Pivoting root is not the complicated. The complicated is the process of
detecting and setting up the real root filesystem which can involve loading
kernel modules to dhcp negotiation to http requests and more. Dracut handles
all these.

------
herpderperator
What this article fails to mention is that having an initramfs is not
required. It's only necessary if additional tools are needed to mount a
particular filesystem, otherwise the kernel can run init directly from the
root.

~~~
textmode
True or false: An initrd (cf. initramfs) is still required, where "rd" stands
for ramdisk.

As a BSD user, I prefer to use a mfs root (ramdisk) for personal reasons, but
AFAIK BSD has never _required_ a ramdisk in order to boot.

If I recall correctly Linux needed to use a ramdisk in its early days. Is this
still true today?

~~~
ajross
Linux has never needed a ramdisk. If it's built with drivers for the boot
device and it probes in an unambiguous manner, you can pass
root=/dev/whatever99. That's as true today as it was in 1992.

~~~
checkyoursudo
I pass root=PARTUUID=whatever to be safe, but yes this is how I boot an efi
stub kernel directly with no rd/initramfs.

Just need to, for example, build the block device driver for the root disk
into the kernel instead of a module.

------
scott_s
If anyone is curious what it looked like in 2004: [http://www.scott-
a-s.com/the-linux-boot-process-of-2004/](http://www.scott-a-s.com/the-linux-
boot-process-of-2004/)

~~~
amorousf00p
Was a much better time imho.

------
jbuzbee
Back in the day, I seem to recall having to binary patch LILO every time I
installed a new kernel to tell it what partition to boot. I suppose that dates
me...

------
Florin_Andrei
It's not really that different from how things were 20 years ago. There's a
few bells and whistles added now, but for the most part it's still the old
sequence.

(added this comment because of the "modern" qualifier in the title)

~~~
thatcks
How much things have changed depends on what level you look at the boot
process at. For example, before 2.3.41 introduced the pivot_root system call,
the starting process in the initial ramdisk wasn't run as PID 1 (as far as I
remember and can tell from the remaining kernel code for this). The kernel
also used to be far more willing to do things itself, such as assemble
software RAID devices; modern Linux pushes all of that into the initial
ramdisk user level code, or even later in boot.

At the broad level, though, yes absolutely. Unixes with System V init have
been drawing a distinction between 'single user' boot activities like fsck'ing
the filesystems and 'multi-user' ones like starting daemons for a long time,
so that's a two stage boot. Linux booting became three stage once it added
initial ramdisks so that the core kernel didn't have to have to build in all
of the pieces necessary to get the root filesystem.

(I'm the author of the linked-to entry.)

~~~
JdeBP
You did, however, proffer a discussion of the _Linux_ boot process, and Linux
operating systems do not and did not have AT&T System 5 init. They have/had
something resembling it, written by Miquel van Smoorenburg in the 1990s. But
it isn't AT&T System 5 init, and it actively diverged from it early on.

Ironically, 20 years ago was almost three years _after_ van Smoorenburg
init+rc had diverged from the old idea of one "single-user" mode and had
instead taken the route of two modes, emergency and rescue. In fact, we only
need to wait a year or so until it has been 20 years since those names can be
found in widespread use. It has already been more than 20 years since the name
"emergency" gained traction.

* [http://jdebp.info./FGA/emergency-and-rescue-mode-bootstrap.h...](http://jdebp.info./FGA/emergency-and-rescue-mode-bootstrap.html#history)

So whilst things _are_ like they were 20 years ago, how things were 20 years
ago isn't in fact the old AT&T System 5 world that people nowadays tell one
another it was. That was the 1980s, and it wasn't Linux-based. Indeed, by 25
years ago the AT&T world itself had already introduced ideas changing the
original model of multi-user login, such as the Service Access Facility.

~~~
siebenmann
Belatedly: that was a very interesting read on the history of this in both
Linux and Unix more generally. Thank you for the link.

------
forgot-my-pw
Brief overview on how ChromeOS boots (8 years ago):
[https://www.youtube.com/watch?v=mTFfl7AjNfI](https://www.youtube.com/watch?v=mTFfl7AjNfI)

Would love to watch an updated version today, if anything changes.

------
JdeBP
Contrast a different explanation discussed at
[https://news.ycombinator.com/item?id=17326691](https://news.ycombinator.com/item?id=17326691)

~~~
JdeBP
I notice that I and M. Siebenmann reacted to this in similar ways. (-:

------
rl3
Related:

[http://iam.tj/prototype/guides/boot/](http://iam.tj/prototype/guides/boot/)

There's something to be said for flow charts when trying to explain a complex
topic such as the boot process.

------
watersb
Google terms: "Early Userspace"

------
21
I have a full disk encryption setup for my Ubuntu. It uses LVM, LUKS, and
cryptosetup in GRUB.

The problem is that it takes forever to boot (5 min, not kidding). Apparently
the whole boot partition is decrypted to a RAM disk or something.

~~~
lgunsch
I have my whole disk encrypted, including swap, and use LVM with BTRFS on top.
My key is included in the initramfs. After I type my password in the Grub
prompt, it only takes about 60 to 90 seconds to do a full boot. Although, the
disk is a SSD. I'm sure if I didn't have a SSD disk, the boot process would
take a very long time.

~~~
CBLT
> 60 to 90 seconds

That seems pretty long, what's your setup?

I noticed mine taking much longer than it should, and the reason seemed to be
a lot of useless PKDF time. For every disk I had to decrypt, my system went
through the LUKS table and tried the keyfile in order. By putting the keyfile
entry first in the table and configuring it to minimum PKDF time (just for the
keyfile, still millions of sha512 for passwords) my disks all decrypted in a
couple seconds.

~~~
lgunsch
I probably over estimated a bit, and I only reboot about once a week or so.

I only have the one disk to decrypt, and one key. When I was setting it all up
I did run cryptsetup benchmark to choose the best hashing algorithm for my
architecture. It ended up to be aes-xts sha256. Has enough iterations for
about 2.5 seconds.

