I look at Ingo, Dave Miller, Richard Henderson and Andrew Tridgell who are ~ my generation, all both better coders than I will ever be and really nice people to work with.
Then I look at young coders like Olaoluwa Osuntokun and Bastien Teinturier who are also great to work with and who I can only keep up with because I have years of experience, and it completely keeps my massive ego in check!
Or are you asking how we create more people who will do this in their free time?
Also, as a Clojure dev, you cannot recompile the running kernel while using it at the same time using something akin to REPL driven development. That would be the other side of the live-patch productivity boost in so many areas.
If I am not mistaken, the CPU and other devices don't care, if everything is compiled ahead of time and a few functions tacked on later, as they are changed by e.g. patches, or if the kernel is completely recompiled.
Obviously, I am simplifying a lot but it is a very serious question. We really should strive to build maintainable systems that don't have to reboot just to apply updates.
Here, you would need to involve a compiler of course and do more due diligence and then run ftrace or something to exchange the code. Also, you would probably in some cases need to make sure, the ordering of code is right as to not hurt performance. (See Emery Berger's talk https://www.youtube.com/watch?v=7g1Acy5eGbE which probably applies here as well.)
Of course, with great power comes great responsibility. You could add some optional checking facility, that would be able to roll back the changes to code if some conditions were met.
The rest could probably be solved by a very short "stop the world and switch a few pointers" routine.
It may well be that same sense of ownership creates an "inner circle" effect that is a barrier to entry for creating more such developers.
I've attempted something similar myself - a refactoring of a very dirty Java project that was orders of magnitude smaller than the Linux kernel. I didn't have good refactoring tools, nor even a good SCCS; my month-long effort had to be backed-out. I didn't enjoy it - it was boring, but needed doing, because the accumulating cruft was impacting my ability to work on the codebase (and presumably everyone else's). The project stayed dirty, and my reputation suffered.
Credit to Ingo for his perseverance. That is massive commitment.
It makes a big difference for the thousands of kernel developers . There's also more than a million linux users globally, assuming only 1% might compile their own kernel, that still leaves tens of thousands of linux users benefiting.
Not to forget about the side benefits of a refactor. There could be runtime improvements coming out of this too down the road.
That said, in your case it felt a bit like an organizational failure also. As you could read in Ingo's mail he had to start over after plenty of work put in, for him there wasn't any reputation damage because he seems to have had free reign to start this as he wished.
Reading code is fun and it's a skill that took me time to develop. People need to told that's a skill you need and yes (shock) not emphasize creativity as much as we do. Creativity is great but it has enough encouragement in society already. We need people willing to read what already exists as well.
If you do a from-scratch rewrite, you could stay in igorance and might make those mistakes. But if you do a refactor, especially one broken up into many steps like this Linux one, it in fact demonstrates a better understanding the existing code and any nuance than merely leaving it in place!
Not touching scary-looking code is just bad, and not a wise form of humility.
But I think throwing away the changes should be an option if there's no objective improvement. Very often those changes end up on the main branch but they only cause changes, not improvements.
The real outcome is in someone's head.
That or they advocate doing a from-scratch rewrite.
I seem to get satisfaction from things that a lot of others might find tedious/boring.
People like us though just need to watch ourselves that what we’re doing is actually useful opposed to just scratching an itch. Which of course is not exclusive to us as developers often find many other ways of scratching itches without actually adding value.
On my best projects, I've done the first 90% so that a great many other can collective do the 2nd, 3rd, and 4th 90%s :). Without being "goal oriented" you can feel good about unlocking that part that would be too annoying for others, and they in turn can do the finishing work that wouldn't be fun for you.
Don't feel like you're not doing important work just because others are slotting in the keystones.
Good team builders will find a variety of different employee talents and interests and let people do what they like and develop those skills.
So we need to restructure our society to have economy the promotes leisure not output.
That will at least align the incentives right with a lot of more extreme engineering that just doesn't make economic sense today.
On a completely different level, yes, this is the sort of thing that does require passion. There is a lot of obscure stuff that is hard to plan or otherwise validate society. I'm a bit of a utopian I suppose in thinking if we had more leisure and guaranteed consumption, people would be able to do more passion projects, and we could give people hindsight recognition just instead like this.
Basic needs can be unconditional, renown can be the reward for "extra" work.
Finally, I hope as more monoliths are broken into libraries, we get more of this sort of stuff organically. Beyond the economics, conways law holds us back. (Even within the kernel!) Need to make sure people feel free and safe to really get down in other people's code, to develop the macro view, to see stuff like this.
This is the sort of "macro" refactor you can't get to with just local optimizations alone. We can and should break it into steps, but by not means does every commit have its own clear perf benefit.
In other words, in a perfect world, someone recognizes the value-add of your passion, and offers you money to do what you would do for free.
Evidence of this is all around us, and while most won't get this, some significant proportion of people pull it off.
So what do I win?
Health problems -> Money problems -> Fun problems.
You get the privilege to work on "fun problems" once you don't have "money" or "health" problems (otherwise, the "fun" problems are the least of your "problems").
Linus Torvalds worked on Linux for a long time before being directly compensated for it, but he probably would never have done so had he needed to worry about whether he could afford his next meal and rent that month.
I had free time, and an interest. Often the same characteristics of people who contribute to FOSS.
> The fact that they were still using mailing lists
In my personal experience, going back some 30 years or so online with my own personal paid email account – which is still live and still works – I find that most people I've worked or interacted with who dislike using email for workflow or other important comms do not know how to use email effectively. Strange as it may seem, this now applies to the developers of most email clients and online email services.
Email and mailing lists are remarkably powerful and capable tools, if used properly. Most people do not use it or them properly. I have yet to see anything else by anyone that is an improvement in every way on email.
> I'm not sure how it will progress when they retire or die off.
TBPH, as a Linux user and professional, I rather hope it does not.
Linux is an amazingly useful tool, but OTOH it's now, in and of itself, a vast hairball of technical debt. The traditional UNIX model itself is.
It is long past time that we should have moved past it on to better things. There have been many attempts but none have ever achieved critical mass.
At the rate things are going it looks quite likely that our technological civilization will collapse due to out-of-control global warming and the ongoing and accelerating mass extinction event. As the parent of a 2YO I very much hope I am wrong.
But if I am and it doesn't, the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
Linux, perhaps, but "the model"? I'm not so sure.
> It is long past time that we should have moved past it on to better things.
That's what they said about SQL RDBMSes too.
> There have been many attempts but none have ever achieved critical mass.
Maybe for good reason.
> the world needs better tech, and a ½ century old UNIX clone already does not really cut it today.
That's what they... Eh, I'm repeating myself.
> If Linux and all the other monolithic Unixes die when their dev teams die, we'll have to move on to newer, smaller, simpler systems that a new generation of programmers can actually read from top to bottom, understand, and do useful work on.
That's probably what they... No, not quite. But it may have been what Linus thought -- at least, judging from what he made.
> If monolithic Unices become the mid-21st-century COBOL, just kept around for a few old systems and only critical bugs patched, that will be a big win for us all.
I doubt that'll happen. I'll be happy if they become the mid-21st-century SQL -- which I suppose will also be still going strong.
But the alternatives are there.
The fast-headers thing put me in mind of the way that Plan 9 changed the C language, not only formalising indentation and things, but to forbid nested #includes, which apparently vastly reduced compilation times.
Inferno moved on to largely supplant C with Limbo, which is one of the ancestors of Go. I'd say Go, D and Rust all show that there's demand for a newer, better C and that C++ is not it.
Apple's xnu is widely held not to be a true microkernel because of its big in-kernel Unix server, but Minix 3 is... and for all that Minix 3 is not quite there yet, QNX shows this is possible and doable and performant.
It's 2022. We have boxes with terabytes of RAM now and hardware-assisted virtualisation even on £5 ARM boards. We don't need a total clean-sweep replacement; we can virtualise the old stuff and slot in underneath something smaller, simpler and cleaner that mere mortals can understand in less than a lifetime.
3D Xpoint memory is on the market now and makes non-volatile RAM doable, multiple orders of magnitude faster and longer-lasting than flash memory. Computers don't even need drives any more: we could just have a few terabytes of persistent memory, and no more shuffling stuff into memory. True single-level store is doable now. Who needs files when everything is in RAM forever?
ISTM that much of the computer industry has its eyes on the ground now, and is just shuffling round and round in the same old groove, rather than trying new stuff. In the middle of my working life there were dozens of exciting new OSes trying to do things differently. Now, we have 30 million lines of C instead, and need a billion-dollar industry to aim all those many eyes at all those many bugs.
As Molnár himself said:
“Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away."
Very few companies or other management structures would ever sanction this kind of work. They might recognize it as important, but it's too big and open ended to "allocate resources to". So for this to happen you need to pay people for more of a general "make stuff better for us" role. Even that's hard because you never know what you're going to get.
Without them, quality degrades and you get a lot of busy work from working in inefficient systems.
I did this at work too early on in my career too but it was quickly made clear to me that if there's no ticket that the customer has opened (or at least approved), they're not paying for it and I can't spend time on it. Tickets I opened myself were 99% ignored or only brought up when a problem I had anticipated actually manifested in product (told ya.. now this 18-month-old ticket is suddenly relevant?). I quickly learned not to give a crap about the code base (hard to give a shit if you're not really allowed to give a shit?). I guess that also slashed my motivation (and productivity with it). It's pretty frustrating to work this way.
Creative freedom and autonomy are key. I'm sure there was no micromanager assigning Ingo Molnar the "make kernel builds faster (est. 80 hours of work)" ticket.
Obviously this is an extreme case, but it scales. Something like 8 years ago I noticed something was horribly wrong with the ARM32 boot wrapper, submitted a fix, and my approach was mostly shot down by the maintainer. Last week I submitted a 34-patch RFC series to finally add WiFi support for 5 years' worth of Macs (which required quite a bit of new scaffolding in that driver, as well as fixes and new features) and it's gotten positive reviews so far, modulo nits. I wouldn't have been able to pull that off 8 years ago. And it's not like I spent those 8 years doing kernel dev, but I've sent in a few fixes and watched how other kernel developers work. I started on a big kernel project a year ago and all that sitting and watching has been very helpful in putting out stuff that people like.
1] remove their technical blockers. Often a person like this doesn't have a universal skillset — they're usually good, but they can't do everything. If there's a 5% of the job that's hard-blocking them, make sure it gets done so their tires don't get stuck in the mud.
2] remove their bureaucratic blockers. Make absolutely sure the project WILL proceed, even in a skunkworks capacity (this is the primary value of skunkworks — it's the ability to proceed with something you know will work, in spite of authorities expressly forbidding it because they think it won't work and don't want it to happen for what's usually a petty reason like a "it's waste of resources").
3] remove their emotional blockers. This is really the apex — THE primary value of a person like this is not technical skill, but rather, their work ethic. A person like this has a really profound, train-engine drive to just keep soldiering on. The danger here is the problem of the "weary crusader". This willpower is considerably above average ... but it's not infallible. It's not like a superhero that's always gonna come through no matter the odds. They can break down. They can lose heart.
One of the things that will constantly break them down is if people are naysaying them, and calling into question the value of their work. If they're doing a giant refactor, and people are angrily opposing it simply out of fear of change, it will break them down. As silly as it sounds, you want to coddle them — they might be unusually emotionally tough, but they are the absolute last person you want breaking down. Treat them like a "snowflake".
And yeah, I'm explicitly suggesting censorship. Don't permit negativity. Don't permit the usual cesspool OSS discussions where you have a bunch of people who contribute almost nothing bikeshedding a project to death and exercising a sort of "liberum veto" on any attempt to do new things. Just shut down those conversations — make it clear that only the people who do major heavy lifting (or really, who are discussing things "in good faith") have a voice. The whole OSS community has an unhealthy cultural fixation on "absolute free speech", and like ... I completely understand where that comes from, but it's absolutely toxic for motivation. I've observed over the years that a lot of the more aggressively successful groups at "creating things" basically just establish their own safe-spaces where people get emotionally reinforced instead of emotionally sabotaged. Occasionally their work will get exposed to the world and some toxicity from outside will leak in, but the bread-and-butter day to day experience of working on their projects has them surrounded by friendly people who believe in what they're trying to do, and encourage them to keep going.
Partly because we know we won't be there forever to keep working on it. If we feel like we're part of a group that "has our back", and will support what we built, and grow it into something, it's a million times easier to keep working on something, compared to a situation in which someone actively holds our work in spite and is itching for the first chance to tear it down the moment we turn our back, or move on, or die, etc. This is all "emotional extrapolation", but these are the sort of depressive/anti-depressive thought cycles that go through your head during a mega project, and too much of the negative side can just break people.
Frankly everything I know about software dev is geared to discourage exactly this sort of behavior.
I would not be happy to see a pull request touching the entirety of the code base for better compilation performance. Not because such a thing wouldn't pay dividends in developer performance or code readability, but because such a change is just inherently risky.
You might be able to train more people with whole program understanding (Definitely a skill that is lacking, IMO). But I doubt you could train someone to be able to "touch every file in the kernel and get it merged".
Most highschool teachers are really bad at this.
Also, you need to have gathered the skills to do what you wanna do. But you'll put in the effort yourself if you're inspired.
That'd be a challenge. He is obviously a very talented person who grew up behind the Iron Curtain and went to university just after that collapsed. The resource constraints bred creativity, a certain kind of hacker mentality. The lead developer of Crysis: Warhead and Crysis 2 (and indeed most of the former Crytek Budapest team) is another extremely talented programmer emerging from the same time. Palma sub pondere crescit!
— Ingo Molnar
http://www.ussg.iu.edu/hypermail/linux/kernel/9906.0/0746.ht... to the Linux Kernel mailing list in 1999.
So often you see people ranting about the Go team either being ignorant or scornful of PL research.
I always wonder if those same people are Agile Manifesto zealots at work.
> "The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. – Rob Pike"
> "It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical. – Rob Pike"
If you're interested in a critique that goes more deeply into Go's actual design flaws, as opposed to its deliberate design tradeoffs, I enjoy https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-....
What a great thread supporting the choices that Linus & the core team made at the time, given the hindsight we now have.
I'm not sure I would have agreed with Linus in '99 (and I was doing a Phd in the OS networking space then!).
Your bigger picture quote is great too. In academia it is easy to get yourself stuck in local minima, unable to see the better path beyond the hill of similar research surrounding you.
Edit: here's a live mirror. http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html
> platitudes hardly provide compelling evidence for ignoring 20 years
> of networking/OS research and architecting unnecessary copies into the.
> core of an OS that wants to be taken seriously...
dont forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away.
And that’s fine! It’s incredible that a collaborative effort like Linux (protected by the GPL, which forces the issue) can pull in all these big companies’ resources, given that the companies would probably prefer to have their employees work on things that only benefit them and not their competitors.
The first might produce quick, but unsustainable or less perfect results, the latter might take more time but will provide significant benefits (in this case for maintenance, time wasted/energy consumed by build pipelines on a global scale).
Large-scale changes also suffer from "too many cooks in the kitchen" and bikeshedding that can make getting the changes accepted a real slog. It is difficult to get people to buy into your vision purely with words, either because they don't see the value in it or they reflexively disagree that the benefits are achievable (sometimes motivated by their own burnout or cynical attitude).
The solution is to work on it in semi-secret until you can get things to a point where the value is demonstrated. That bypasses endless bikeshedding and pointless feedback rounds because everyone can see the value of the end state (if you've chosen something worthwhile to work on).
The same applies at 100 LoC.
And at 10.
Edit: obviously not at this scale. 2300 patches at once in a fork is a lot.
It's not "similar to a pull request", it is a pull request (or would be if this wasn't a RFC). While this one is a bit different than usual (he didn't post the full diffstat, for instance), that's the way pull requests were traditionally done, even before github/gitlab/etc existed. A traditional pull request is an email which besides the description says something like "please pull from git://... some-branch", and notice that this one has near the top "[...] which can be found here: git://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git master". The maintainer pastes these two arguments (the URL and the branch) to a "git pull" command line, so in this case it would be "git pull git://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git master".
Someone should make a tool which parses mailing lists and presents it in a Github-like frontend
Why does GitHub ignore the most important thing to look at in a PR? It’s not the tiny changes that are of top importance, it’s the big ones. The number of times I’ve been burned by trying to ctrl+f for something I’m looking for in someone’s PR (trying to see “does this diff touch X” for instance) — and getting a false impression of the code I’m reviewing — just because GitHub silently elided showing a big diff somewhere in the middle of a PR… is too high to count.
The single most important part of GitHub is the social aspect (code reviews), and it’s the part it is objectively worst at. If I just wanted to host my code, I could make a simple server with SSH accounts and let people push to it. We use GitHub because it’s supposed to be a venue for discussing complex changes, and it sucks by default at discussing complex changes.
Github's user interface was scary at first for me, much more than a plain-text mail. So was GitLab.
Some people I know have complained to me that I send plain-text mails and that nobody wants to read a wall of text. I think that's probably a user issue, and it can in turn help filter low-effort comments.
If talking about a11y, plaintext mail ought to be one of the most accessible formats: high-contrast, can be processed by multiple tools including screen readers. ASCII art trough screenreaders is probably an exception.
Now, about your specific question: there's patchwork, though I don't think that instance covers the general LKML.
SourceHut does this since it exclusively uses mailing lists.
Almost all the core maintainers are employed only to work on the kernel 9-5 outside of extraordinary situations (say a critical privesc or RCE in the kernel).
This patch set will not be ignored but it likely also won't receive review until everyone is back behind keyboards in the coming days.
Kernel devs enjoy the holiday season too!
From: Ingo Molnar
Subject: [PATCH 0000/2297] Eliminate the Linux kernel's "Dependency Hell"
25,288 files changed, 178,024 insertions(+), 74,720 deletions(-)
When will Linux move to a site like github / gitlab / something similar self hosted that supports people proposing changes without having to send every single person interested in the development of Linux thousands of emails.
And if you need email to authenticate on a website, why not just use email anyway?
But this is an interesting idea, and I'm on the look-out for ideas to replace email. I'll keep "LKML" in the back of my head as a use case.
It does support making backups of repositories at least.
>and if the host goes rogue they can make it inaccessible and make everyone lose efficiency while they swap back to the old method.
Isn't this also a problem if the person managing the email list goes rogue? You have to trust someone to host the infrastructure.
>And if you need email to authenticate on a website, why not just use email anyway?
Because Email may not be the optimal user interface for handling issues, pull requests, code review, etc.
Maybe that's your experience.
Others have used mailing lists for this purpose for decades and a lot of people prefer it.
For one, it's accessible. Every machine can do plaintext email. Every text editor can work with plaintext. It's simple. Integrate it into whatever shell/editor/scripting language you prefer.
As opposed to login into to a bespoke web interface, mainly primarily designed to be friendly to novice users.
If this in true then why do so few projects use a mailing list nowadays instead of something like github.
>Every machine can do plaintext email.
Every machine can do web browsing too. More people have used a web browser on their computer than a dedicated email client (as opposed to a web app like gmail).
>As opposed to login into to a bespoke web interface
You have to log in to email too.
>mainly primarily designed to be friendly to novice users.
What's wrong with that? Having good UX is a win. The UX for creating a new repo on github is a million times better than creating a mailing list (yes I have set up a mailing list for a project I made and no one e ended up using it except for me. Meanwhile I had much more success with getting people to join and discuss the project via Discord)
Lot's of projects use mailing lists. Lot's of projects use web-based git hosting services. Lot's of projects mix.
Why do people just use GitHub? Because it requires zero configuration and it's convenient when you're developing something yourself.
The Linux Kernel is by far the worlds largest open source project. Consider that it might have different needs.
>> Every machine can do plaintext email.
> Every machine can do web browsing too.
You're missing the point.
A web interface is more complex than plaintext emails.
To integrate into your shell environment is a lot more complicated.
> More people have used a web browser on their computer than a dedicated email client (as opposed to a web app like gmail).
I don't understand your point.
>> As opposed to login into to a bespoke web interface
> You have to log in to email too.
The emphasis was on bespoke web interface, not logging in.
>>mainly primarily designed to be friendly to novice users.
>What's wrong with that?
Fast and flexible are sacrificed. Ie. it's more convenient for the people who don't use the system all the time (people filing bug-reports) vs. the people who actually use it all the time (maintainers)
> yes I have set up a mailing list for a project I made and no one e ended up using it except for me
a tad different use-case to the development of the linux kernel it seems
> Meanwhile I had much more success with getting people to join and discuss the project via Discord
Whatever works for your project bro.
Tell that to most of my browsers which refuse to display any Gitlab content (they only ever show the side bar).
> What's wrong with that? Having good UX is a win.
What's "good UX for novice users" isn't necessarily good UX for more advanced ones.
It depends on what kind of threat you are worried about. The beautiful thing is even if LKML was abused and ruined tomorrow, nothing is lost or damaged. The emails have already been sent, and moving to a new list is more of a nuisance at this point.
If you're worried about authenticating a patch series, committers can sign their commits a number of ways.
Lastly, a large chunk of Linux development happens off-list, with subsystem maintainers building branches for Linus or Greg to pull from.
Please actually attempt the ideas you are suggesting before suggesting them.
This set of changes (and other large ones like it) is partially why the kernel needs to use email instead of <insert bloated Ruby application here>.
So’s github, anything of a non-trivial size it just won’t show by default and you need to re-request every time.
It starts struggling around the kloc scale, and the browser itself soon follows as the dom is not the neatest and you start getting tab memory above half a gig, which I assume also makes the JS start killing itself or something.
This sort of implies that the Kernel would use something centralized if it worked well, which I don't believe to be true. Using email is very much an intentional choice and a desirable solution.
The mailing list that they use now is centralized. It works by everyone sending a message to a single email address. The owner of that address then sends the email out to all of the subscrbers of the mailing list.
Having everyone who wants to upstream their patch directly to the kernel submit their work to a single website is not much different than them sending it to a single email.
This here for example is sent to Linus directly and that part does _not_ depend on the mailing list. The mailing list is just a convenience feature for people that _want_ to be aware but may be looked over/forgotten by the submitter (which still happens, and people regularly say stuff like "also CC'ing additional relevant people").
Yes you could, at least in principle, also do that via GitHub/GitLab/whatever. But a) you wouldn't gain anything (the kernel already has all the tooling it needs and it's large enough to just create missing tools, heck git was literally written for it), b) you would introduce a massive single point of failure that either i) is _not_ under control of the kernel.org team or ii) needs a massive amount of resources to host (see the fun people poke at displaying 2k commits in GitLab).
Neither seems like a good use of the kernel development resources. And if you want to contribute to the kernel... using a mailing list is an absolutely trivial problem.
Then I came to realize that my problem was not with email itself, but the way I had been using it. All GitHub, GitLab, etc do is take a decentralized platform and add centralization. I love the idea of using email now instead of a centralized issue tracker. Mailing lists become the issue tracker and those are then published on a website for others to use that want some centralized features.
A mailing list is centralized, but much less so than GitHub and is secondary in this instance.
Ingo did not send 2298 patch emails, he sent only the 0000 one, which contains the location of his public git repo for all this. People can clone this, and can even push it to a github/gitlab/etc hosted repo if they like.
I think it's natural for workflows to change overtime.
Your question is fundamentally wrong; totally bass-ackwards. The correct question is:
When will all the zillion other projects move off of GitHub / GitLab / all the other usurper sites?
it's plain-text email
the display ad on your local news website is probably wasting more resources
Think how many lines are now duplicate (and therefore need to be updated in twice as many places, and bugs introduced when a copy is missed).
Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file.
If the 100k lines were in their own subdirectory and added a major new feature, it would be worth it. But spread across the whole codebase, and introducing more duplication and chances for bugs I think outweighs a 'neater' header file system.
The 'dependency hell' is only an issue for people who only want to build part of the kernel anyway. If you're building the whole lot, you might as well just include everything and it'll work great.
That's a lot more than "shaving the build time."
As far as I understand it, the goal of this patchset isn't to improve the build time; that's just a nice consequence. The goal was to refactor the header-file hierarchy to make it more maintainable and less "brittle." Sometimes, increasing maintainability requires more code. (Almost always, if the current version is a terse mess of mixed concerns.)
Think of it this way: take an IOCCC entry, and de-obfuscate it. You're "increasing the size of the codebase." You might even be "duplicating" some things (e.g. magic constants that were forcefully squashed together because they happened to share a value, which are now separate constants per semantic meaning.) But doing this obviously increases the maintainability of the code.
I wonder what the memory savings for compilation look like? Because that's also potentially more workers in automated testing farms for the same cost.
When you're talking a project of half a million lines, sure.
The Linux kernel has around 27.8 million lines of code.
An increase of .35%
> Think how much extra stuff someone needs to skim through looking for the relevant file or part of the file.
Why add features at all? Code has a purpose. Sometimes bringing code into a static context is a net good. It was going to be generated at runtime anyway.
> If you're building the whole lot, you might as well just include everything and it'll work great.
That's not strictly true, but it's true for these features, which is a stated reasoning.
A library I'm working on is 7000 LOC which seems pretty sizable, but 0.35% of that is 25 LOC.
The Linux patch set is actually tiny
This is horribly misleading; most of these lines of code are drivers, which this patchset doesn't even concern.
It's still a massive change that only a handful of developers will ever be able to review in entirety - a fact to which the size of the project is completely irrelevant - if anything, actually, it urges even more caution, given the implied complexity. Which I believe was (at least in part) parent comment's point - given the importance and ubiquity of the Linux kernel, this may be concerning.
That said, I am very confident in the structures put in place by the kernel devs, their competence and the necessity for such a change - but trivializing a 100k LoC patchset because the project it's intended to land in is even more colossally complex isn't how I'd choose my approach.
That's not true at all, a big part of those added lines are added includes in drivers.
E.g. this commit I picked at random adds 1500 lines, of which just a few procent are in the core kernel: https://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.gi...
In Ingo's post, he points out that the main speedup is coming from the fact that the expansion after the C preprocessor step is a LOT smaller.
That's a lot of decoupling. As someone who had to go rattling over the USB gadget subsystem, I can tell you that running "grep" with "find" was the standard way to find some data structure buried in a file included 8 layers deep. Having those data structures in files actually mentioned by the file you're working on would be a huge cognitive load improvement as well as make tool assistance much more plausible.
Even if this particular patch doesn't land, it lights the path. What types of changes need to be made are now clear. How many changes are required before you see the improvement is now clear. With those, the changes required can be driven down into the maintainers and rolled out incrementally, if desired.
It has always amazed me how finding something which originally seemed a trivial little thing, usually meant going through a chain of #defines and typedefs across many header files. It's the same with GLibc, by the way. It' a bit like when you hike to a summit by following a crest path: you always think the next hump in sight is the right one, your destination, the promise land; and when you reach it, dammit, it wasn't, your goal is actually the next one. Or perhaps the next after the next. Or...
As someone who tried poking around: Oh good; I assumed I was just missing something. Alternatively, oh no; I had assumed I was missing something and there was a more elegant tool out there.
yes. and this is very, very good reason. As another poster said, you at 0.35% lines to make it compile almost twice as fast? And you're not happy about that?
> Think how many lines are now duplicate (and therefore need to be updated in twice ...
OK, how many? None! That's how many. Adding proper header dependencies to the .c module doesn't duplicate anything. Unless you think adding #include <stdio.h> in every module somehow creates unmaintainable duplication.
OK. Hrm... I think a lot, lot less is how much. That's the whole point of a major cleanup like this. Proper decoupling. Headers that you use are obvious where they belong not being brought in with some action-at-a-distance accident.
Having to recompile the entire tree every time you change a seemingly unrelated header gets old fast.
But wait, why didn't you just break the task down into sub-tasks that could be estimated in four-hour increments in the first place? Every single "scrum master" I've ever labored under has insisted that every software development task can be broken down this way.
As to maintenance overhead: it was surprisingly low overhead to keep dependencies at a minimum across upstream kernel releases - there were typically just around ~5 dependency additions that need to be addressed. This makes me hopeful that an optimal 'fast' state of header dependencies can be maintained going forward - once the initial set of fixes are in of course.
It seems like this could regress quickly just by a developer adding a "bad" dependency to some core header which happened to #include a lot of code by accident. Don't we need tools to ensure this doesn't regress? Like something which warns "this git commit increases the amount of compiled code by 20%".
Ingo is on the right track.
I’ve always thought that this stuff should have been dealt with by the GCC (in form of pre-cached header files).
So sad that it is a thorny issue to this day (for large scale projects); so glad that this issue is now alive so it can be dealt with once and for all.
I envision some type of *.ho filetype to hold these quasi-compiled headers for the next build iteration.
Is at least one GCC maintainer looking at this? Otherwise it'll likely not be dealt with "for all", right?
On the kernel side every new dependency has to be added with keeping in mind this problem as long as there's no automation to help with this it'll be a problem.
This is important, realizing that compilation units rarely collide on private namespace usage. Called 'compilation unit grouping', or 'CU grouping', I implemented this once at a customer's and reduced build time of a large C++ project by 2x.
These were files from a library that used a large number of tiny files, we prefered to embed this in our project rather than link in.
Massive build time reduction.
There is some technical name for this other than "compilation unit grouping" - I forget what
After this it should be easier to maintain out of tree stuff since it eliminates "dependency hell".
For the other humans whose brains work on the kernel- the abrupt shock of moving to a vaguely familiar but in the end quite different house. Yes, the old one was suboptimal in various ways but this new one...
Few people's brains work like Ingo's. And he has had 2 years to work and iterate through the mental model. Field day here for the Science, Technology, and Society crowd, watching how old dogs deal with new tricks.
The reason he wrote the full patch series out like this, is because he wanted to measure the speed gains. Without significant speed gains it would be difficult to convince the kernel developers to make a big uncomfortable change like this.
More importantly, because a lot of the patches involve moving code between files, or making batch changes to a large number of files at once. Evaluating these sorts of changes for correctness is a lot easier when the changes are isolated.
Moving all that header code around between files will conflict with all other code merges that touch those same lines.
Though, really, I was just saying why it may not be as bad as it sounds. Can still be bad, of course.
> - As to maintenance overhead: it was surprisingly low overhead to keep
dependencies at a minimum across upstream kernel releases - there were
tyically just around ~5 dependency additions that need to be addressed.
This makes me hopeful that an optimal 'fast' state of header
dependencies can be maintained going forward - once the initial set of
fixes are in of course.
They provide flexibility but at a cost.
I enjoyed Borland Pascal compilation speeds and clean dependencies before switching to Linux. I miss it, together with Delphi.
The gains in compilation speed (including linking) and dependency information tracking is phenomenal though. It would certainly bring C++ in line with languages like Java, Rust etc.
The CPPCon 2021 talk about this is enlightening.
If C++ gets modules, there is no reason why C can't use the same model - since the compilers will have that stuff anyway.
"Please submit the original source. If a post reports on something found on another site, submit the latter."
So I think we'll merge that thread into this one.
IMHO, much more interesting that the original article
It seems like a change this massive would have the possibility of introducing a bunch of security holes, would it not?
Think of it like changing the order around of how you cook a fine meal and putting a few tricks all senior chefs know into the recipe explicitly. No ingredients were added or removed, do poison doesn’t get accidentally added.
It sounds to me like this work is re-organizing and moving around existing code in the header files to reduce the amount of code the C compiler has to wade through during a Linux build.
So one way to verify the changes is to compare the binary files (or the resulting executable) built the old way vs the new way. In theory, they should be identical. (Or I could be misunderstanding the whole thing and you should stop reading now!)
I did something like this for a Prime minicomputer emulator project: https://github.com/prirun/p50em
It was initially coded for a PowerPC chip because that is big-endian like the minicomputer being emulated, and to be honest, I wasn't sure I could even do the emulation, let alone do it and be mentally swapping bytes all the time. After about 10 years of it working well, I spent a week or two reworking it to add byte-swapping for Intel CPUs.
The method I used was to first add macros to all memory and register references. This was done on the PowerPC build, and after each set of incremental changes, I'd verify that the executable created with macros was identical to the original executable without macros. If it wasn't, I goofed somewhere, so backed up and added the changes in smaller sections until I found the problem. I didn't have to do any testing of the resulting executable to verify thousands of code changes. Practically every line of code in the 10K LOC project was touched because of the new macros, but each macro should have been a no-op on the PowerPC build with no byte swapping.
Next I built it on an Intel CPU where the byte-swap macros were enabled and it basically worked, maybe with a tweak here or there.
As a further test, I ran the PowerPC emulator through a bunch of existing tests with tracing enabled and the clock rate fixed. This records the machine state after each instruction, and with a fixed clock, yields a reproduceable instruction trace. Then ran the same test with the Intel version and compared the instruction traces. If there were any differences, it was because I forgot to add a swap macro somewhere.
After a few months of running it 24x7 myself w/o problems, a customer did find a bug in the serial device I/O subsystem where I had forgotten to add a macro. I hadn't done any testing of this subsystem (terminal I/O on real serial RS232 devices like printers.)
If something similar is being done with these Linux changes, verifying them may not be as hard as it seems initially.
Ingo has likely done a bunch of tests himself, of course.
As someone pointed out, it’s only like 0.13% of the kernel.