
How is the Linux kernel tested? - sprado
https://embeddedbits.org/how-is-the-linux-kernel-tested/
======
ndesaulniers
We run our own ci for building Linux kernels with clang.
(ClangBuiltLinux.github.io). We take Debian's nightly package of ToT, make a
docker image with the minimum tools we need, and use buildroot ramdisks that
have a custom init script that powers down the machine once it reaches init.
Our CI fetches various kernel trees and branches, builds them, then boots them
in Qemu. The machine has 2 minutes to power up and down (usually takes less
than 10s), otherwise we consider the machine hung and fail the run. We use
travisci for the reporting, but are looking to offload the building.

Also, Linaro's Tool chain Working Group runs a ton of CI on the kernel as
well. There's an effort from RedHat called KCI to aggregate all of these
reports.

~~~
cat199
so you have a very well tested boot sequence -

is this not a slightly better kernel equivalent of 'it builds, ship it' ?

~~~
yjftsjthsd-h
> a slightly better kernel equivalent of 'it builds, ship it' ?

Surely, "it _runs_ , ship it"? That seems quite a bit better.

~~~
blattimwind
The grand majority of code in the Linux kernel will never be hit by booting it
in a QEMU with init=/bin/shutdown. The pure line coverage of these builds is
probably like 5 or 10 %.

~~~
viraptor
The majority of code will never be hit in QEMU or any testing in general,
because 56%+ lines are in drivers.
[https://unix.stackexchange.com/a/223763/29119](https://unix.stackexchange.com/a/223763/29119)

With 16%+ lines in all 24 architectures combined, you're not going to reach
more than one of them at a time.

10% is going to be an extremely happy case. If we can test 5-10% that's a
great achievement. To get past that you need to start booting real hardware
with specific configurations.

------
alkonaut
Why is there such little emphasis on “traditional” testing, that is, regular
unit tests? At least some portion of the code base is surely suitable for
normal unit tests. For example data structures, scheduling algorithms, file
systems, ...

~~~
EdSchouten
I think the main reason is that the Linux kernel (and similarly the *BSD
kernels) are written in a programming language (C) that doesn't make it easy
to do that. Code is often directly built on top of other kernel subsystems
without any dependency injection whatsoever. This means that it's still
possible to do unit testing of parts of the kernel, but it takes a crazy
amount of effort, such as overriding symbols, overriding include paths and
provide stub headers, etc..

I am well aware that it's also possible to have dependency injection in C by
using structs with function pointers, but I think we can all agree that it's a
lot less pleasant to use than C++ abstract base classes, Go interfaces or Rust
traits. This is why the Linux kernel only tends to use this sparingly (e.g.,
inode operations).

~~~
eschaton
You don’t need a dependency injection framework to write unit tests, you just
need cleanly separable units with well defined interfaces.

~~~
scubbo
And, in the absence of a dependency injection framework, it's likely that the
units are _not_ cleanly separable - because, without a DI framework, all
classes are (presumably?) instantiating their dependencies directly.

Unless I've missed something? I've only ever worked in Java so maybe things
are different in C-world,

~~~
asdfasgasdgasdg
> without a DI framework, all classes are (presumably?) instantiating their
> dependencies directly. ... I've only ever worked in Java so maybe things are
> different in C-world,

Well, for one thing, there are no classes in C. :) It is possible but unfun to
emulate them with function pointers. Iiuc, little of the Linux kernel is
written in that style.

Also, FYI, for many years we did DI without frameworks, using the factory
pattern and other techniques. It wasn't always fun but it can certainly be
done without Spring or whatever the new thing on the block is.

~~~
blattimwind
> Well, for one thing, there are no classes in C. :) It is possible but unfun
> to emulate them with function pointers. Iiuc, little of the Linux kernel is
> written in that style.

object-structs with function-pointers-for-methods are super-common in the
Linux kernel and basically used everywhere for everything where modules can
plug something into the kernel (e.g. virtually all drivers have at least one
of these).

~~~
asdfasgasdgasdg
Thanks for the correction. I was going off the little bit of Linux code I've
read, which seems to call most functions directly. And also another comment on
this story. I don't know what to think now.

~~~
blattimwind
Linux is monolithic, but also modular. While drivers are almost entirely
implemented with these kinds of objects, "core" modules have less pluggable
functionality, and so you don't see it as much. For example, contrast the page
cache code (that's basically mm/) with the VFS code (fs/). You'll notice how
almost anything I/O uses these kinds of objects heavily.

------
jimmyswimmy
> Another major challenge is to automate device drivers and hardware platform
> tests because they require the hardware to be tested.

Anyone have any thoughts on how to automate testing of coupled hardware-
software systems? This is really hard; we've attacked it in the past by
writing hardware simulators in accordance with the ICD. This falls flat once
you find that the hardware doesn't precisely match the ICD, and usually it's
much cheaper to change the software than the hardware. And at that point, the
simulator hasn't actually helped you at all.

I recently built a system which involved several tightly coupled hardware
components and we fought many bugs on a tight schedule. It would have been
nice to find a good way to think about this beyond the basic hardware-in-the-
loop manual testing.

~~~
rxhernandez
I can't tell you precisely how it was done but I worked at a company with many
different types of hardware that contained many complex configurations of
FPGAs, lasers, optics, and microcontrollers, coupled to a computer and they
managed to simulate it quite well for what seems like a decade. One of the
scientists there was one of the few geniuses I've ever met and they managed to
simulate all those devices and a sufficient amount of their variations. So I
can confirm it's possible, maybe it just requires an overworked genius?

~~~
yjftsjthsd-h
So I can confirm it's possible, maybe it just requires an overworked genius?

I believe that, but unless you can find a genius and/or mass-produce their
work, does it help the rest of us?

~~~
rxhernandez
Maybe. Knowing something is possible is sufficient for enough people to give
things a try.

I know I've been successful in doing so; I've never built a simulator of this
magnitude but I've successfully solved difficult problems with novel solutions
simply from hearing it was possible to solve them in a given manner.

------
aorth
There is automated static analysis from smatch for over ten years now:

[https://lwn.net/Articles/691882/](https://lwn.net/Articles/691882/)

This has found thousands of bugs in the kernel.

------
umvi
So what happens when Linus Torvalds dies? Is there someone else who will
become BDFL?

~~~
LukeShu
Greg Kroah-Hartman has filled in before when Linus has had to take time off.

------
thedance
Long way of saying "it's not".

~~~
DSingularity
No, it is.

[https://github.com/linux-test-project/ltp](https://github.com/linux-test-
project/ltp)

~~~
0xFFC
This has not been mainlined as far as I know.

~~~
recov
The commit history looks pretty healthy

~~~
yjftsjthsd-h
"Mainlined", not "maintained".

------
dk8086
Oh my! I though they were doing TDD :)

