On the bright side updating to 5.10 fixed a regression of a 5.4 to 5.8 kernel upgrade to me. The fix might have been in 5.9 but I only got the idea of upgrading after the 5.10 release.
Anyways, Linux needs some more CI so that such bugs can be found during the RC phase.
Where in the current CI that we have today is lacking that needs to be improved? We always want more testing and testers, what is preventing everyone from helping with this?
I'm a software engineer who's not involved in Linux Kernel Dev... but I've got a stack of old laptops that I'd be happy to set up to run automated CI if that'd be helpful.
Is there a webpage or doc somewhere I can look at?
(I'm not trying to snark - the fact that you're you and you're here asking for help is making me want to dip my toe in).
Simplest thing to do, just run Linus's latest releases (the -rc releases), or from his git tree, on your machine and report any problem.
Second-simplest thing to do is to run the linux-next branch/tree on your machines and report any build warnings and runtime issues you find. That's what will be the "next" kernel releases and is where all of the developer/maintainer trees are merged together before they are sent to Linus.
Both of those should be very easy to do, and any problems found there should be easy to fix and resolve before they get to a "real" release.
I haven't been following kernel dev for years; what does the CI setup look like? Did the Phoronix Test Suite ever find its way into widespread use?
Back when I was building kernels for embedded hardware (Sheevaplug) in the 2.6.33 timeframe, I found a USB audio regression between 2.6.33.7 and later versions. If there were a semi-turnkey way to set up a testbench that could automatically reboot hardware in every new kernel, run through some basic tests, and report any deviation, I probably would have been more likely to do so. At the time I was working solo trying to release a polished consumer product (sadly though the product was released the business didn't work out) and didn't have time to dig into and report bugs.
We have so many different CI systems running on the kernel on a hourly basis.
We have the 0-day bot from Intel that runs so many things on all developer trees. We have kernelci running on many many different hardware platforms, and we have Linaro test systems also running on many different branches and hardware platforms.
If you want to tie your own hardware into the system, kernelci is the best place to start, I recommend looking into that.
Anyways, Linux needs some more CI so that such bugs can be found during the RC phase.