
Linux Kernel Fastboot [pdf] - agumonkey
https://www.linuxplumbersconf.org/event/4/contributions/281/attachments/216/435/LPC_2019_kernel_fastboot_on_the_way.pdf
======
jmccorm
How nostalgic! Mind you, I say that in a good way!

This brings me back to the heydays of Sun Microsystems where they controlled
both the hardware (SPARC) and the operating system (Solaris). Sun didn't
hesitate to use this to their advantage. With Clear Linux, I see Intel doing
some very similar things. Yes, this was just a story about boot time, but it
led me to take a look at Clear Linux and it looks like it could be a star. I'm
interested in how much more performance their distribtuion is able to squeeze
out. I'm also wondering how they're dividing their focus here and what high
notes they're trying to achieve.

At a strategic level, I'm assuming that they're wanting to distinguish
themselves from ARM? Are they looking to create a well-tuned reference for
other Linux distributions on Intel to adopt? At a tactical level, I see
they're focusing on performance, availability (which boot time would be a
function of)... but are they tweaking for other things such as reliability?
Fault tolerance?

Until now, Clear Linux wasn't even on my radar. Today, it is. Anyone have some
experiences with this distro (for better or worse) that they'd like to share?

~~~
Filligree
> Anyone have some experiences with this distro (for better or worse) that
> they'd like to share?

Not much of one, but for a while I was playing with multiple distros trying to
find the best possible power management for my laptop. (XPS 13 9380.)

Clear did better than the Ubuntu installation that came with it -- but a dead
octopus would have done better than _that_. It was sitting at 6W for non-CPU-
intensive work, and was also noticeably snapppier. Windows on the same
hardware sits at 8W when idling; same as Ubuntu.

More recently I've tried Arch, though. Which somehow got that same workload
down to 3.3W, albeit with a lot of adjustments... that didn't seem to affect
performance, but did affect power use, clearly.

It's just too bad that all of this optimization goes out the window if I fire
up a browser, but I've had a fun time finding ways to avoid that.

Apart from that, obviously Clear's software repository is far more limited
than Arch's, and it also seems harder to contribute to. I don't think that
problem will be going away soon.

~~~
nobleach
I'd love to hear more about the tweaks you've done to Arch. I've gone little
beyond a few power management things myself (On a Thinkpad X1C Gen 6). I get
decent battery life, but I know it could be better. And giving up Arch would
be a tough sell to myself.

~~~
tbrock
The thing with arch is it’s mostly not more tweaks that give you better
battery, it’s not having all the crap that Ubuntu preloads running that makes
that happen.

~~~
nobleach
Folks think I'm crazy for using Arch as my main dev machine. I'm not an Ubuntu
hater at all (it runs my main server in the basement at home)... but it'd be
awfully hard to go back to using it as my main machine.

~~~
doubleunplussed
Really? Arch is... boring. Once you've set it up, nothing happens without your
permission and you can just keep developing away. Sure you have updates
imposed on you, but you can delay them a bit if you dont have time right now,
and its not like upgrades are indefinitely avoidable on any other OS.

Using Arch often gives me advance warning that my apps will break on users
machines due to changes in libraries. I see the changes first and can add
compatibility for the new library version such that it's fixed before
affecting my users.

I love it as a daily driver.

~~~
nobleach
Yeah, for me, it's "just right". Most detractors claim it's "bleeding edge".
What? just because it's got something newer than Postgres 9.1 available??
(this was my argument against Debian/Ubuntu for a long time)

Totally agree that it's a great daily driver. It has EVERYTHING!

------
NelsonMinar
I love the bit on slide 16 where initializing 8GB of RAM takes 100MS, so they
boot with only 2GB and then hotplug in more memory later.

~~~
cperciva
Just last weekend I was talking to one of FreeBSD's VM gurus about doing
exactly this in FreeBSD too. You have to be careful about it though -- some
data structures are autosized at boot, and if you're not careful you can end
up with systems "running out of memory" because they have _too much_ memory.

~~~
toast0
> some data structures are autosized at boot

It's probably a good idea to audit all the things that are autosized at boot
by ram size; some of them are probably much too large on a system with 128 GB
or so, given some of the sizing was written when 128 MB was big. I know I've
run into the size of IP fragment buffers being way too big, but not sure what
else.

------
cperciva
Those of you who are interested in such things might also like to watch my
BSDCan 2018 talk:
[https://www.youtube.com/watch?v=HMywejyXB9k](https://www.youtube.com/watch?v=HMywejyXB9k)

~~~
cocoa19
Sorry for off-topic: I checked your HN profile and tarsnap looks like what
I've been searching for a long time. Looks much more straight forward than
nextcloud and open source too. Wish you success with your company.

~~~
cperciva
Thanks!

------
zamadatix
> “mem=4096m” in cmdline to only init 2 GB

What is going on that makes this 2:1?

~~~
sedatk
2gb user + 2gb kernel?

~~~
trdtaylor1
2 gb user + 1 gb swap + 1 gb rootfs unpacked in ram perhaps. They discuss a
decompression operation and the only portion of the sequence that would occur
in normally is unpacking a rootfs in RAM.

~~~
wtallis
The kernel itself is compressed, and among the first things that happen after
the bootloader hands off to the kernel is that the stub at the beginning of
the kernel decompresses the rest of the kernel.

And in this case they don't appear to be using any initrd for the root
filesystem, just hurrying to bring up access to the full filesystem over eMMC.

------
jtchang
> SATA driver init takes 100 to 200 ms even without a real disk

That's pretty interesting. Wonder what it is doing?

~~~
ComputerGuru
SATA controller firmware has a hard-coded “drive predelay” that waits before
enumerating attached devices, a relic from the days long before SSDs. Even
when an option to set a custom amount is available in the BIOS, it is in
addition to and not instead of, as there’s no bus master to tell it all
attached disks have init and the controller may speak to them.

It wouldn’t necessarily have to be done synchronously up front at all, except
when the BIOS hands off control of the system to the SATA controller it
expects that afterward it has a definitive list of available boot devices and
their properties, for selection/display in the boot menu and BIOS
configuration.

Theoretically with UEFI and it’s NVRAM-resident settings, a boot device could
be expressed as a namespace and path, and the firmware need not enumerate
devices until it has determined that there’s a need to search for
alternative/fallback boot devices, but the stack hasn’t been updated with that
in mind.

~~~
wtallis
The delay in question is in initializing the Linux driver, not in the system
firmware scanning for bootable devices before running Linux. The Linux driver
may be taking a while for similar reasons as the firmware delay you describe.

The hard drive predelay is also not as obsolete as you may think; I've
encountered high-capacity SATA SSDs that take a surprisingly long time to come
up after being hotplugged, to the point that Linux doesn't always succeed in
bringing up the link on the first try.

~~~
ComputerGuru
Sorry, I missed that in the discussion. Thanks.

I’m willing to bet if the stack enforced tighter tolerances for modern non-
rotating disks, manufacturers wouldn’t be able to slip such shoddy firmwares
into production.

------
bmurphy1976
I'd really like to get my raspberry pi booting in a couple seconds. I have a
few ideas for things I'd like to try but the boot time is too long. Don't need
X11 but I would like to use raspbian.

~~~
fmntf
If you do not need the graphical stack, there is room for optimization both in
kernel and user space. On a similar board (4 ARM cores) with Ubuntu I managed
to start user space applications in <2 seconds (from power on) and the
complete boot (including Xorg/MATE) in <10 seconds. If you are interested, on
Medium I have some write-up (same username, last post).

~~~
akx
A link to said write-up: [https://medium.com/@fmntf/adapting-ubuntu-for-the-
automotive...](https://medium.com/@fmntf/adapting-ubuntu-for-the-automotive-
quick-boot-and-power-loss-resilience-74d25915250c)

------
doener
Discussion on Reddit:
[https://www.reddit.com/r/linux/comments/d34rvg/how_intels_cl...](https://www.reddit.com/r/linux/comments/d34rvg/how_intels_clear_linux_team_cut_the_kernel_boot/)

~~~
craftyguy
> Just remove the sleep(2700).

Such high quality discussion too!

~~~
why-oh-why
> SATA driver init takes 100 to 200 ms even without a real disk

While it’s a joke, they’re not particularly far from the truth, as noted in a
different comment here. They do indeed wait for some parts to respond.

~~~
msclrhd
Would it be possible on multithreaded architectures to have a thread listening
for the response, then setting a flag in /proc or somewhere else to indicate
that the part is now active? That way, init code that does not depend on that
device can continue to run.

------
mises
I've seriously got to look into running arch with a clear linux kernel. I knew
they did some pretty serious optimization work, but this is definitely
impressive.

~~~
ComputerGuru
Most of the bloat isn’t from the kernel. That’s the last thing you really
optimize after getting rid of all the background services, startup tasks, etc.

~~~
mises
Clear kernel performed better in every single test run by phoronix, except one
(because it has retpoline and others did not).
[https://www.phoronix.com/scan.php?page=article&item=arch-
ant...](https://www.phoronix.com/scan.php?page=article&item=arch-antergos-
clear&num=4)

~~~
ComputerGuru
I think you misunderstood. I was saying in a typical stack, time from POST to
desktop is significantly overshadowed by userland bloat, not that Clear kernel
is not worth the optimization.

~~~
big_chungus
Yeah, but a stock arch system is significantly less slowed than a typical
stack. Getting to bootloader is still significantly longer than actual OS
boot, though. I'm not usually a supporter of the "all software has to be open
source" thing, but between spying issues and horrible code, it seems like
coreboot is a serious benefit. Wish I could get it on my machine.

------
newnewpdro
> Systemd is ~1.5MB - the loading time for emmc is 100ms

Surely not _all_ of systemd is faulted in immediately, is this really an
issue?

~~~
amluto
If you want just the important parts faulted in, you need to get the important
parts into their own pages, preferably in order. I don’t know whether profile-
guided linking is a thing.

~~~
PeCaN
PGO detects hot/cold functions and places them together¹ ², but I imagine
whatever it needs on startup is probably not especially hot. …Though it might
all be cold, which would have the same effect, actually.

―

¹ At least on GCC but I would be very surprised if clang doesn't also do this.

² You can also do this manually with __attribute__((hot)) and
__attribute__((cold))

~~~
the8472
PGO may not be the right tool for this. Mozilla used custom instrumentation
and linker scripts to achieve symbol ordering optimized for startup.

[https://web.archive.org/web/20100223202038/http://blog.mozil...](https://web.archive.org/web/20100223202038/http://blog.mozilla.com/tglek/2010/02/19/teaching-
ld-to-optimize-binaries-for-startup/)

------
hungra
Awesome! I'm seriously thinking about running Gentoo with Linux Clear kernel!

~~~
jjuhl
"running Gentoo" \- Sure, waste your life compiling code for next to nothing
performance benefits that will never add up to the extra time it took to
compile the code..

------
maika72
Site seems Not Found, is there any backup?

------
baybal2
I have hard time understanding why do they have to have a supervisor there to
begin with

------
katzeilla
P10

> Rootfs Mouting -> Rootfs Mounting ?

------
KKKKkkkk1
Do I understand correctly that the motivation for this is that the rear-view
camera on the Chery Exeed LX runs Linux? Is it a good idea to ship a car with
safety components running Linux?

~~~
kchamplewski
As opposed to running what?

Linux (as in, the kernel) is extremely stable and well tested.

If the components in question utilise any significant features of the kernel,
I strongly doubt some homegrown kernel-esque project will achieve more
stability and reliability than Linux.

Sure, if it can be done with a (minimal) amount of embedded code then it quite
possibly should be, but if a whole kernel is actually needed then Linux is a
fine choice.

~~~
detaro
QNX is still fairly popular in cars. Sometimes also QNX for the more critical
things and Linux for the rest.

