
A regression is the kernel not giving the same result with the same user space - signa11
http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02474.html
======
jordigh
Mercurial, made by another (in my opinion) much more well-spoken kernel
hacker, is what really introduced me to the concept that you do not break
interface for downstream users, no matter how wrongly you may think they are
using the interface.

It’s an attitude that is difficult to convey because software developers
always want to have the freedom to “improve” their own software, even at the
possible cost of breaking something for some users (and worse, even telling
users that they shouldn’t have been doing that in the first place).

I keep going back to this blog post which I wish more people agreed with
(Steve Losh is another person influenced by Mercurial):

[http://stevelosh.com/blog/2012/04/volatile-
software/](http://stevelosh.com/blog/2012/04/volatile-software/)

~~~
acobster
Minor point, but he puts Python in with software that "almost never breaks" on
update. _Really?_ Am I missing something? Maybe he just hasn't tried running
Python 3 yet...

~~~
mikeash
Python 3 is a similar language with a similar name to Python 2. It shouldn’t
be considered an update, nor applied as one.

~~~
magicalhippo
Yet the Python 3 executable has the same name as the Python 2 executable...

~~~
computerfriend
No they don't. At least one of them has a number after it.

~~~
magicalhippo
I got both Python 2 and Python 3 installed on my machine, from the official
distribution, and both have "python.exe", no numbers.

------
__david__
Linus's followup post later on in the thread explains his position in more
detail and is a pretty good read:

[http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02487.html](http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02487.html)

~~~
piker
Interesting.

> Behavioral changes happen, and maybe we don't even support some feature any
> more. There's a number of fields in /proc/<pid>/stat that are printed out as
> zeroes, simply because they don't even _exist_ in the kernel any more, or
> because showing them was a mistake (typically an information leak). But the
> numbers got replaced by zeroes, so that the code that used to parse the
> fields still works. The user might not see everything they used to see, and
> so behavior is clearly different, but things still _work_, even if they
> might no longer show sensitive (or no longer relevant) information.

I'm not sure I prefer the described result. If my application depends on some
of those zero'd fields, it seems like that has potential to cause serious
debugging problems, as opposed to just causing an error on the parse.

~~~
chii
> If my application depends on some of those zero'd fields, it seems like that
> has potential to cause serious debugging problems, as opposed to just
> causing an error on the parse.

if your app didn't used to handle the 'zero' case (assuming zero was a "valid"
value for said field), then the app was poorly written and would'be been
broken in the first place. However, if the app _did_ handle the 'zero' case,
then the app continued to work! The end user would've been none-the-wiser
about the change (mostly), and some time later, the app could be upgraded to
ignore those known fields.

Causing a parse error would mean the app stopped working altogether. Much
worse a result for the end user.

~~~
hyperpape
There are many cases where continuing with invalid data is much worse than a
parse error. Perhaps that's less likely in this case (hopefully your credit
card payment doesn't depend on the values in /proc/<pid>/stat), but as a rule,
silently continuing with meaningless data is scary.

~~~
spdionis
Not when the application was to supposed to handle the invalid data anyway
before.

Because you had to handle the zero case before right?

The idea is that in the worst case you get an error message, not an altogether
crash.

------
bischofs
Someone needs to codify this shit for when he leaves. Its surprising that he
still needs to scream at people after two decades of kernel work. I don't have
a lot of faith that whoever comes after him will be as proud of the kernel as
he is given they just inherited most of it.

Succession is a weird concept with linux but the fact that it has always been
more stable than the other OS' is I think the main reason it is been so
successful in the internet age.

An example of a failed succession is Tim Cook; the guy is the complete
opposite of Jobs. Just take a look at the apple website nowadays. A product
guru would not sell three generations of iphones that are all cannibalizing
each other and lack any real differences.

~~~
proginthebox
We will hopefully have a new kernel written in Rust by someone new.

~~~
scott_karana
God, I hope not. Ground-up rewrites (when they even succeed) typically lose
features, performance, _and_ security, since they don't have the benefit of
20+ years of correcting mistakes. Plus you lose all your contributors.

However, Rust _is_ linker compatible with C, and can be runtimeless, so
rewriting the kernel piecemeal over time instead could be great. ;-)

~~~
proginthebox
This is what I meant. Any ground up rewrite will be reinventing the wheel and
hence too expensive.

------
bjourne
John Johansen will survive. Linus has made it incredibly damn clear what "no
user space regressions ever" mean. See f.e the similar rant from 2012:
[https://lkml.org/lkml/2012/12/23/75](https://lkml.org/lkml/2012/12/23/75) And
it is his kernel so it is his rules. John Johansen admitted he messed up
([http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02539.html](http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02539.html))
and promised to handle things better in the future so all is well.

~~~
jordigh
> John Johansen will survive.

We really need to stop excusing Linus's behaviour. Just because we may happen
to agree with what he is saying doesn't mean we also have to agree with how he
is saying it.

~~~
Karunamon
Disagree. It seems like Linus’ acerbic and direct means of communicating is
the only thing keeping garbage code out of the kernel. He’s also made his
views on “why not be nicer” well known. People need to get over it - his
project, his rules.

Plenty of other projects out there to hack on if getting yelled at when you
disengage your brain is too much. Most of which don’t have the obsession with
not breaking stuff for the end user...

~~~
stinos
> acerbic and direct

Isn't this just swearing?

> the only thing keeping garbage code out of the kernel.

I'm not sure if I'm buying that. It's possible, sure but on the other hand:
It's not because the other way (i.e. communicating in a nice way, without all
the swearing and direct attacks - which is imo not the same as direct
_communication_ ) hasn't been tried that somehow proves it won't work. E.g. I
know repos where one maintainer is kind yet gets things done, and another one
seems to have borrowed his communication style from Linus and gets less done.
Because people just give up, being sick of being yelled at in an unpolite
manner. Is it that hard to be respectful, even when disagreeing?

~~~
goialoq
Do people who don't break users and lie about it get yelled at? Why are
impolite words worse that actual harmful actions?

~~~
tertius
Though I don't disagree with Linus' words. He could convey the same thoughts
without swearing. I don't swear and I'm very capable of correcting people when
they mess up.

~~~
Karunamon
Why is not swearing a virtue?

------
delroth
As much as I think the general principle of "no userspace visible change" is
great, I can't make up my mind on whom I agree with the most in this specific
instance.

My understanding is that some distros (e.g. OpenSUSE) ship with AppArmor in a
whitelist mode where things have to be explicitly permitted by policy,
otherwise they're denied by default. AppArmor introduced a new class of
actions that wasn't there before, and with these restrictive policies it's
causing daemons to crash (because it gets auto-denied).

The whole point of these policies is to not expose new attack surface if it's
not needed by the software. It's going to break whenever AppArmor will start
supporting more checks, unless policies get pinned to a specific version of
AppArmor. But AppArmor doesn't currently support this (they refer to it as
"features ABI" in the thread).

WAI?

~~~
cesarb
It's interesting to see how it works with SELinux. The Fedora distribution
upgrades to a new kernel version often, so after a few months you start seeing
lines like these in the kernel log:

    
    
      [   14.586798] SELinux:  Permission getrlimit in class process not defined in policy.
      [...]
      [   14.586887] SELinux:  Class infiniband_pkey not defined in policy.
      [   14.586888] SELinux:  Class infiniband_endport not defined in policy.
      [   14.586888] SELinux: the above unknown classes and permissions will be allowed
    

That is, the kernel part of SELinux supports more checks, but since the policy
is old and doesn't know about them, they are allowed by default.

~~~
ChrisSD
That's not good from a security point of view but I guess it does avoid
breakages.

~~~
bluGill
No, it is neutral from a security point of view: previously you couldn't
configure the new thing at all, now you upgrade to where you can configure it,
but isn't you didn't configure it the last time you configured everything
(because you couldn't) so it default to as if you had enabled it. When you
come back latter and think about this again you enable the new configuration
with the right whitelist, but until then things work just as well as before.

------
signa11
and later he clarifies it thusly:

''' People should basically always feel like they can update their kernel and
simply not have to worry about it.

I refuse to introduce "you can only update the kernel if you also update that
other program" kind of limitations. If the kernel used to work for you, the
rule is that it continues to work for you. '''

~~~
worldsayshi
For user experience reasons that seems like a extremely sensible principle.
But doesn't this make fixing some security issues impossible?

~~~
masklinn
From the followup message Kalleboo linked to:

> There have been exceptions, but they are few and far between, and they
> generally have some major and fundamental reasons for having happened, that
> were basically entirely unavoidable, and people _tried_hard_ to avoid them.
> Maybe we can't practically support the hardware any more after it is decades
> old and nobody uses it with modern kernels any more. Maybe there's a serious
> security issue with how we did things, and people actually depended on that
> fundamentally broken model. Maybe there was some fundamental other breakage
> that just _had_ to have a flag day for very core and fundamental reasons.

> And notice that this is very much about _breaking_ peoples environments.

So breaking userland is possible if there is literally no other way, but

1\. it's the "nuclear option "of kernel changes

2\. it will be acknowledged as _breaking_ users's environments

~~~
wang_li
A number of years ago some dillhole became annoyed that strings in the kernel
inconsistently capitalized null, sometimes it was "null" sometimes "NULL". So
they submitted a patch to make them consistent. This broke the iscsi userland.
As far as I know, it was never reverted.

------
raverbashing
I'm glad Linus has this attitude and it is unfortunate it's not present in
some other places

Though _it is_ annoying if you're at the receiving end of it, own up (the
opposite of what the patch author was doing)

~~~
Cthulhu_
Annoying and hurtful, yes, but at the same time, he's not wrong - you fucked
up, you did bad and you should feel bad. The upside of this rant too is that
the author is not "fired", as such, but told to shape up or gtfo. In a lot of
companies, mistakes are accepted, fixed, and forgiven without adding a bit of
emotion to it. I think adding this kind of emotion will have a bigger impact
and will make the receiving end do better.

~~~
sepeth
How? Really? Ask that to a child beaten up by their parents in the street. I
am sure that child won't make the same mistake again. I am not interested
about if Linus is right or wrong, many other people have done that already. I
am more interested the way that he speaks. I am gonna go ahead and say it. The
way he speaks in this email is ugly, wrong, and should never be encouraged.
The language we use is important, even though sometimes it takes more effort,
love takes effort, good things take effort, and good communication takes
effort too. We should at least try to be nice. Otherwise life would be kak.

~~~
dawnbreez
Equating this email to a parent beating their child is a false equivalency.
Firstly, because Linus has not physically attacked the developer, and
secondly, because the developer--unlike the child--should know better. This is
more like Gordon Ramsay going off on a chef who has just told a customer off
for not liking his burnt toast.

~~~
sepeth
I agree with the first part, but not with the second part. I believe equating
John's behaviour to a chef who has told a customer off is just as distant as
mine.

If I was looking exact equivalent, I would just send the same link to the same
email.

I am using empathy here. I don't know you guys, you may feel ok if you get
shouted when you do wrong when _contributing_ to a project, but I would feel
bad, and as a human, I don't think my feelings are not important. If it is
coming from an important person like Linus Torvalds, this would hurt even
more, and really I don't see anything disrespectful to anyone in John's email.

~~~
dawnbreez
I would argue that the best response to an expert telling you that you are
wrong is to _fix the problem_. While your feelings do matter, the stability of
the kernel matters more, and you cannot discount Linus's message because you
think it is too emotional in tone. At the end of the day, Linus is right,
Linus created the project, Linus is leading the project, and he is using
strong language to inform you that he needs John to change his approach to
this problem for the sake of the project.

On a slightly more pragmatic note, Linus would not be angry if John hadn't
been trying to break Rule Number 1 for three weeks in a row, so an alternative
solution is to simply not break the rules. Granted, if you disagree with the
rules on a fundamental level, that doesn't work--so, third solution: leave and
start your own project.

------
megaman22
Linus is such a breath of fresh air. I'll take an honest, emotional,
impossible to misconstrue, ass-chewing over the alternatives. Too often
there's an avoidance of straightforward, blunt speech that conveys exactly
what is meant. I really have better things to do than try to parse nonspecific
beating-around-the-bush and deconstruct word-castles.

~~~
13of40
I think a good leader could get the point across without resorting to
bullying.

~~~
vorpalhex
I wouldn't call it bullying. He isn't attacking the committer personally or
due to anything outside the issue. He isn't hounding the committer.

Bluntly telling someone they really screwed up and need to stop denying it
isn't bullying. Someone being mean isn't bullying.

~~~
WhitneyLand
Dressing someone down publicly, in a tone based on anger and frustration, is a
form of humiliation, and humiliation is a form of bullying.

~~~
dmix
Brutal honesty isn't for everyone. I personally grew up with friends who would
deal out this kind of thing and I wouldn't feel bullied or humiliated.
Sometimes it got to you but, there's no way around that, but over time you
learned to develop thicker skin and bounce back without getting feelings hurt.

Especially when it's a personality style combined with the fact it comes from
a place of caring deeply about the subject... then the motivation is not just
trying to hurt the other person. This is the maturity, the point you learn to
see in people.

Clearly Linus is heavily attached to the project and the person he replied to
has been pushing an unpopular position for 3 weeks. It didn't come out of
nowhere. He's also not new to this game. He's not some new intern, who doesn't
know the deal, getting bullied by a CEO.

There has been a big culture trend to treat every small comment as potentially
devastatingly hurtful, training kids to be thin skinned, and in my opinion,
this highly disincentivizes honesty and will create a lot of inefficiency in
business. As it requires a significant time investment to make sure even
seemingly harmless statements are not 'hurtful', employing subtlety and
skirting around an issue indirectly requires significant cognitive load. In a
high volume, high pressure projects like the Linux mailing list this could
really add up and generate bad decision making, as people will rather avoid an
issue then invest in saying it carefully.

The ROI in that trend is questionable, as you can train people to see beyond
the words and look at the person's real motivation and meaning, as it is not a
given raw honesty is coming from a place of personal attack.

That said, obviously otherwise real hurtful and mean attacks, without an
underlying purpose and genuine motivation other than to hurt the person,
should never be tolerated. But sacrificing full honesty and non-heavily
filtered emotional opinions in order to never have these personal attacks is
just not worth it IMO.

~~~
WhitneyLand
Yeah but the thing is, I don't see how our points conflict.

Yes, by all means people should try to develop a thick skin and would benefit
from that throughout life. That would lead to X% improvement and Y% who still
were dealt with in a counterproductive way.

There's a wide range of capacity for dealing with this stuff. General
Macarthur was infinitely strong minded, but when he saw someone with war
induced PTSD in a hospital bed he slapped him across the face for being a
coward. I give him the benefit of the doubt that his own strength made it
difficult to see what was a much less understood thing back then. This case is
not even close to PTSD, but the concept is the same in that emotional capacity
varies widely, even when we practice resiliency.

Linus caring passionately about the project I think is irrelevant, the
question is what leadership approach yields the desired results with the least
downsides.

>...a culture trend to treat every small comment as potentially devastatingly
hurtful, >training kids to be thin skinned...this highly disincentivizes
honesty and will... >create a lot of inefficiency in business...

That's the fallacy right there. There's nothing stopping people from demanding
excellence, demanding results. Indeed, _high standards should be the bar_ ,
and if their are multiple ways apparent to achieve them, the one with the
least negatives should be applied. That's smart for both business and
morality.

>>sacrificing full honesty and non-heavily filtered emotional opinions in
order >>to never have these personal attacks is just not worth it

This is another critical point. I _do not_ advocate (and I think many others
as well) for one second that honesty, directness, fidelity, and impact of
communication be sacrificed. I think in some cases you do have to break eggs
to make an omelette. I just believe the cases where being an asshole is
required to achieve a certain level of effectiveness is in general
exceptional, and that it also wasn't the only way in this particular case.

Btw, none of this is really a critique of Linus. Yes I think he could have
done better, but there are thousands of conversations like this everyday. This
just happens to be a mini case study in leadership styles and the options for
achieving the required results.

~~~
dmix
> General Macarthur was infinitely strong minded, but when he saw someone with
> war induced PTSD in a hospital bed he slapped him across the face for being
> a coward. I give him the benefit of the doubt that his own strength made it
> difficult to see what was a much less understood thing back then. This case
> is not even close to PTSD, but the concept is the same in that emotional
> capacity varies widely, even when we practice resiliency.

I believe you mean General Patton here, not Macarthur. I know this story well
and I don't think this came from a place of personal attack either. It came
from a place of misunderstanding. This is the critical difference I'm getting
at. The personal motivation is what matters, not the lack of preemptive
filtering of the words or actions themselves.

The solution there was an evolution of the understanding of PTSD. The General
was caring deeply about such an important thing, such as not letting well-
trained soldier be a coward and hiding in the hospital when other men are
dying in the field. And he saw many young kids die and it made him raw.

It's also been said the General may have had his own PTSD and emotional
problems, and he wanted to attack the weaker man who made HIM feel that way.

Ultimately he was wrong factually, for misunderstanding the motivations of the
soldier. The soldier wasn't being a coward (AFAIK) but suffering from serious
mental trauma. If this happened in 2017 it would be a different story because
Patton should have known better and what he did was abusive in that context.
AND Patton could also have been better informed about his own risks of
behavioral problems if he had modern science and education.

Does this mean Patton's brutal honesty was the problem? Or was it the general
understanding of PTSD? I believe the latter.

Ultimately this doesn't mean Patton shouldn't have been brutally honest with
his regular troops on the battlefield. Which he was, and he was also brutally
effective at running the war too, and he was loved for it by his troops. The
guy in the hospital is a different story and should have never happened, but I
don't think I would have sacrificed Patton's honesty because society and
common sense was wrong about a few subjects.

~~~
WhitneyLand
You’re right it was General Patton,
[https://en.wikipedia.org/wiki/George_S._Patton_slapping_inci...](https://en.wikipedia.org/wiki/George_S._Patton_slapping_incidents).

However the point was not Patton did wrong, as I mentioned I believe he just
wasn’t aware.

The point was people have massively different reactions to emotional stress.
This is why two people with identical wat experience may or may not get ptsd.
Training for emotional stress has limits.

You keep saying you wouldn’t want to change the virtues of these leaders. I’m
trying to say (forget Patton) most times their virtues don’t come inherently
from being an asshole. Their virtues come many personality traits, strength,
courage, talent. We don’t have to, and shouldn’t give up thes virtues. That
doesn’t meant they can never be optimized.

------
dingo_bat
I'm always eager to read Linus's rants. I wish more of my coworkers were like
him. I hate the environment where nobody reacts strongly to anything! If
somebody fucks up hard and refuses to own up, I (or whoever's in charge)
should be able to shout at them. It's only natural.

~~~
andyjohnson0
I disagree. To me he usually comes across as an asshole, albeit an extremely
smart and accomplished one. A working environment where the person at the top
can be relied upon to react to human failings with an aggressive, sweary [1]
rant is not somewhere I ever want to work.

[1] I'm aware that use of profanity in everyday speech is common in Finland. I
don't mind swearing per se, but I find its over-use a bit tedious.

~~~
Winterflow3r
Anecdata: I grew up in Finland. It's common if you're 15-16 and in middle
school, but I've rarely heard adults swear in the workplace or in public
(unless inebriated).

If Linus wants to use perkele management, then I guess it's his project.
Doesn't mean it's the right way to do things.

~~~
dingo_bat
> It's common if you're 15-16 and in middle school

This part is probably globally true :D

------
whack
I can understand and empathize with both perspectives here. On the margin, is
it better to have consistent interfaces, or simpler implementation? This
debate reminds me of the worse-is-better design style, which is supposedly at
the root of Unix's success.

[https://www.jwz.org/doc/worse-is-better.html](https://www.jwz.org/doc/worse-
is-better.html) (copy-paste the link in a new tab. _Do not_ click on it)

 _The worse-is-better philosophy is only slightly different:

\- Simplicity-the design must be simple, both in implementation and interface.
It is more important for the implementation to be simple than the interface.
Simplicity is the most important consideration in a design.

\- Correctness-the design must be correct in all observable aspects. It is
slightly better to be simple than correct.

\- Consistency-the design must not be overly inconsistent. Consistency can be
sacrificed for simplicity in some cases, but it is better to drop those parts
of the design that deal with less common circumstances than to introduce
either implementational complexity or inconsistency.

\- Completeness-the design must cover as many important situations as is
practical. All reasonably expected cases should be covered. Completeness can
be sacrificed in favor of any other quality. In fact, completeness must
sacrificed whenever implementation simplicity is jeopardized. Consistency can
be sacrificed to achieve completeness if simplicity is retained; especially
worthless is consistency of interface.

Early Unix and C are examples of the use of this school of design, and I will
call the use of this design strategy the ``New Jersey approach.'' I have
intentionally caricatured the worse-is-better philosophy to convince you that
it is obviously a bad philosophy and that the New Jersey approach is a bad
approach.

However, I believe that worse-is-better, even in its strawman form, has better
survival characteristics than the-right-thing, and that the New Jersey
approach when used for software is a better approach than the MIT approach._

~~~
aeorgnoieang
That "worse-is-better" link redirects to a rather _rude_ Imgur image. I'm
guessing the author dislikes HN. Following the same link from a Google result
did not redirect to the Imgur image.

------
alkonaut
> "We simply do not break userspace"

Wish this would be a goal for more of the application environment on Linux
than just the kernel.

~~~
fauigerzigerk
Indeed. I upgraded a Ubuntu installation a while ago. The network just stopped
working because the developers of NetworkManager had decided that NM should no
longer manage wired network connections on some Ubuntu configurations.

That is just a crappy attitude and I have to say NetworkManager is generally
one of the buggiest and most unreliable pieces of software I have ever used.
And I'm actually making a great effort to be polite here.

------
macinjosh
Linus is right but his choice of words seems unprofessional bordering on
harassment. Almost sounds childish in parts, especially the ALL CAPS screaming
he does. Its cool to be passionate but also be civil.

~~~
arkadiytehgraet
Linus has addressed so called "professionalism" several times, one of the best
discussions is here:
[https://lkml.org/lkml/2013/7/15/427;](https://lkml.org/lkml/2013/7/15/427;)

Linus's point of view on overall "professionalism":
[https://lkml.org/lkml/2013/7/15/547](https://lkml.org/lkml/2013/7/15/547)

~~~
sameyepatch
Goddammit, I wanted to go to sleep early tonight.

------
mar77i
It's uncannily similar to him going nuclear on a guy called Mauro in 2012.
[https://lkml.org/lkml/2012/12/23/75](https://lkml.org/lkml/2012/12/23/75)

~~~
mar77i
It seems to be kind of a running gag, and I wonder _what else_ do people
imagine is what the kernel is supposed to satisfy than as many other people's
expectations as possible?

------
throw2016
The only time Linus gets talked about here is in context of conflicts. This
creates a fabricated one sided image of a person who has now given us Linux
and Git.

Its not a corporation, the tradition and history is completely different, the
levels of formality are much lower and the context and culture is completely
new.

There is no traditional hierarchy and the authoritarian structure that most
including commentators here work under in the corporate world.

No one works for Linus and he has zero power over anyone. Commentators pretend
its all roses in their world yet the corporate world is full of people who
have power over others and are actually abusive.

Linus's emails are tame stuff and those casually comparing this to abusive
behavior are simply devaluing actual abusive behavior that becomes possible
under non transparent authoritarian structures.

------
thriftwy
"the first rule of kernel development" is so awesome to have compared to what
e.g. distros do with GLIBC.

Remember when they changed behavior of memmove, which broke user code
including the damn flash player, and then blamed _users_ for that?

E.g., Drepper seems to be the person who is an opposite of Linus.

------
yeukhon
I was recludant to agree woth Linus at first. Why should we avoid break hard?
I understand we wany to avoid breaking user interface as much as possible -
who would want to use an unstable API? But why is it better off to let things
fail silently? For example he cited certain obsolete fields in /proc/stats
return zeros.

Is it because Kernel is used by so many people and projects we can’t afford
having some <insert your favorite Linuz utilit> broken after upgrading to a
new kernel. Replace Linux kernel with OSX or Android etc

Yes of course. But I feel the consequence of never break your interface and
never blame userland is to force the users to write tests that can report
changed behavior.

------
johnnyRose
Poor guy. Linus just melted his face off.

Makes me scared to even _consider_ contributing...

~~~
justabystander
Plenty of people claim that they're scared to contribute because Linus might
yell at them. That's unfounded for a lot of reasons.

TL;DR: Test your code before you submit it. Don't accept or acknowledge
patches from others that haven't been reviewed or tested. And if you make a
mistake, own up quick and he's not going to yell at you. It's pretty simple.

He will occasionally jump in and yell when people aren't accepting
responsibility for problems they caused. Especially when they won't accept
nicer, less authoritative criticism from their peers.

That's what happened in this case. Check the follow-up message in the thread
where Linus explains his dissatisfaction
([http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02507.html](http://lkml.iu.edu/hypermail/linux/kernel/1710.3/02507.html)).
The contributor who caused the regression was caught denying that there was
even a regression. His patch had already long been identified as the culprit
and reverted. But over the course of _three weeks_ he was insisting that it
was not a kernel regression. Essentially he dug in for 22 days saying that it
wasn't his problem. And he wasted tons of valuable contributor time where
people were nicely telling him that, yes, it was indeed a regression.

Multiple people were telling him about the "don't break userspace" policy and
how it was a regression. And yet that contributor was still deflecting
responsibility for the problems caused. But the problem is that this patch
would leave the distros in the lurch - scrambling to update their
configuration before the kernel breaks behavior. And that's without any ETA on
their "long term ... flexible solution"
([http://lkml.iu.edu/hypermail/linux/kernel/1710.3/00380.html](http://lkml.iu.edu/hypermail/linux/kernel/1710.3/00380.html)).
Thousands of people with varying levels of technical competency would have had
their systems break in unexpected ways.

Quite frankly, he earned the dressing down. And plenty of people tried to be
nice to him about it. It just didn't take hold in his mind until Linus dropped
in to rudely remind him that even "correct" behavior is a regression if it
breaks userspace. Also note that Linus said the issue had been escalated to
him. Someone explicitly brought this to his attention because there was a
distinct lack of progress due to CYA responsibility deflection.

------
eeks
Ahh, nothing like a good rant with my morning coffee !

------
leemike
Many good summaries of the Linux kernel userspace compatibility policy are
available. Search for them.

------
eutropia
Torvalds gets upset about people messing things up because he cares. He
reminds me of Gordon Ramsey, who also gets a bunch of shit for being an
asshole to people who ruin food in his kitchen -- but Ramsey has more Michelin
stars than anyone else...

------
andromaton
I read Linus' reply and it's not clear to me if he is ok with the regression,
is not ok with the regression, or even if he considers it a regression. I wish
he'd stop trying to be so diplomatic.

------
jokoon
Eli5, What is kernel regression? Does he mean breaking backward compatibility?

~~~
santoshalper
A change was made to the kernel that broke some applications - this is called
a regression (a bug introduced in an update). The developer who made the
change that caused the bug in the downstream application tried to argue that
it was the application's fault, not the kernel.

Linus then calmly and politely reminded him that the official policy of kernel
development is that they never introduce changes that break applications.

~~~
Beltiras
You only say "calmly and politely" because Linus is a known pottymouth and
frequently "tears people a new one". An email like this might get you demoted
or fired in some workplaces.

------
snvzz
If the interface was simple, such as seL4's, we wouldn't have to see it become
a clusterfuck which, to maintain userspace compatibility, can't be fixed.

It will only get worse over time.

------
fareesh
Linux is what it is because of the passion and principle that fuel rants like
this. I hope he continues to be just the way he is, and never gives an inch.

------
drngdds
What a child. There's a difference between being direct and telling your co-
collaborators that their opinions are "fucking idiocy."

------
butterisgood
Same view as EVERY kernel regression it seems.

------
danschumann
What does "regression" mean in this sense?

------
dis-sys
glad that Linus is still 100% passionate on _his_ kernel. I will start to
worry for the future of Linux when Linus stops doing such rants.

------
sunstone
Yikes, Linus on a bad day is not a pretty sight :)

~~~
d0g
It's the best sight there is . Got to love the man .

------
tuananh
and here in node.js land, we have this beautiful thing call Code of Conduct /s

------
jubalfh
the orange site truly is a self-perpetuating Stockholm syndrome chamber

------
oinkush
That's bonkers.

This sounds like a reasonable definition, but quickly it turns out to be
completely insane.

Lets start somewhere far away from Linux, as far away as possible.

Windows XP SP2 blocked access to raw sockets for non-administrators. It also
blocked access to \Device\PhysicalMemory to all user-mode code, regardless of
the user running it. Linus would say this is a regression.

Fine, lets try something else.

I wrote some C++ code that violated the strict aliasing rule. It worked for me
15 years ago when compilers weren't optimizing so aggressively. I tried to
compile it using the latest GCC and got strange results. Should I file a bug
with GCC? Sounds like a bug according to Linus's definition. It used to work
and now it doesn't. Who cares that I violated the rules in the first place?

"Now, hold on" I head you say. "We were talking about kernel and user, not
compiling C++ code. That completely different!"

Well, sure, it's different. But it's also similar. I maintain that the point
is the same, but lets go back to Windows for a second.

Lets say we have a program that used to parse the PEB.Ldr structs for whatever
reason. In Windows 8 they went through a major change. This program no longer
works. Is this a regression?

Raymond Chen has had to deal with this nonsense enough. For example:
[https://blogs.msdn.microsoft.com/oldnewthing/20031223-00/?p=...](https://blogs.msdn.microsoft.com/oldnewthing/20031223-00/?p=41373/)
And his blog is full of example of dumb programmers counting on internal,
undocumented, subject to change and actually changing structs.

If you, the reader, think you're very clever and say it's dissimilar because
the LDR structs are user- rather than kernel-owned and -managed structs I give
you the same example with EPROCESS. Take a look:
[http://terminus.rewolf.pl/terminus/structures/ntdll/_EPROCES...](http://terminus.rewolf.pl/terminus/structures/ntdll/_EPROCESS_x64.html)

The point is that "breaking" code that does things it wasn't supposed to to
should be - and can reasonably be - considered a regression.

The very first example - with raw sockets - may be considered a regression,
though I prefer the term "breaking change", since "regression" is a form of a
bug, and this is not a bug, even if you think it's a bad policy.

The other examples don't even fall under the "Breaking change" the category.
Nothing THAT WAS GUARANTEED was changed. If you counted on things that nobody
promised you that would remain the same to actually remain the same that's
your problem.

And back to Windows for a moment. Lets say someone uses the
UpdateProcThreadAttribute API to set the mitigation policy on a child process
it creates. Instead of only setting the bits he want (like
PROCESS_CREATION_MITIGATION_POLICY_FORCE_RELOCATE_IMAGES_ALWAYS_ON to force
ASLR on all modules,
PROCESS_CREATION_MITIGATION_POLICY_WIN32K_SYSTEM_CALL_DISABLE_ALWAYS_ON to
block win32k calls and
PROCESS_CREATION_MITIGATION_POLICY_EXTENSION_POINT_DISABLE_ALWAYS_ON to
prevent injection by the system of a couple sorts of plugins) he goes on to
pass 0xFFFFFFFFFFFFFFFF. He wants all the possible mitigations! He's that
smart!

Then comes along Windows 10 with its binary signature policy mitigation and
(0x00000003ui64 << 44) is interpreted as
PROCESS_CREATION_MITIGATION_POLICY_BLOCK_NON_MICROSOFT_BINARIES_ALLOW_STORE.
His child process can't load any of his DLLs. Clearly a regression under the
Linus rules. Obviously not a regression.

From what I understand the situation is very similar to this. I'm quoting from
John Johansen message on the thread:

It is entirely possible to use the 4.14 kernel on suse without having to
modify policy if the policy version/feature set is pinned. However this is not
a feature that suse seems to be using. Instead suse policy is tracking and
enforcing all kernel supported features when they become available, regardless
of whether the policy has been updated.

The clever folks at SUSE decided to enable all the
protections/mitigations/features/whatever of AppArmor, including those that
didn't even exist when they wrote this stupid policy. It's completely
reasonable to think that a program restriction infrastructure's new feature
won't restrict programs more. Sure. If it does that's a regression.

------
peterwwillis
_Slow clap_

------
draw_down
He is right. I’m sorry if he says “fuck” too much or whatever.

------
Nokinside
Linus being salty is like soap opera for HN.

Pleasant and good mannered application programmers gasp when real men (aka
systems programmers) communicate at each other.

------
orcdork
In any other kind of social/business arrangement he would be politely but
firmly escorted out, possibly by security, for this kind of behavior. Being
right or wrong is irrelevant.

~~~
Cthulhu_
Even when he's effectively CEO, visionary, and end-responsible for the
business? He's the Big Boss of Linux, there is nobody above him in the chain
of command. You're used to traditional businesses when there is no single end-
responsible person, and when there is a public face for the company (like
Jobs, Gates, etc), there is still a group of faceless shareholders above them
that can choose to remove them. Not in this case. I like it myself, a
dictatorship is a much more effective form of managing a software product.

~~~
dingo_bat
And I'm pretty sure Gates and Jobs were similar in their time.

~~~
gus_massa
For example: "My First BillG Review"
[https://www.joelonsoftware.com/2006/06/16/my-first-billg-
rev...](https://www.joelonsoftware.com/2006/06/16/my-first-billg-review/)

~~~
dingo_bat
That was a very good read. Kinda sad at the end.

