
“Because I'm a bastard, and proud of it” (2000) - unusximmortalis
https://lwn.net/2000/0914/a/lt-debugger.php3
======
hoodoof
It's possible that it might be appropriate for the leader of the Linux kernel
but for other programmers and leaders - don't model your own behaviour after
this. You're not Linus. You're not Steve Jobs either - also known for his
abrasive style.

Firm but fair kindness, diplomacy and a lead-by-example approach are the keys
to leading a technical development effort over the long term. Do make tough
decisions, do hold people to account, do work hard, do expect the best from
other people, do get rid of problem team members and do address under-
performers, but all that can still be done with patience, kindness and
diplomacy.

I made the mistake in my early career years of modelling my leadership style
on Bill Gates also known for the same abrasive, no holds barred style. It
doesn't work.

~~~
golemotron
People go out of their way to condemn his style. The reason why is because it
is the complete antithesis of PC culture. It wouldn't be an oversimplification
to call his leadership style a "Dad" style: "it's not about you, it's about
the system" versus the "Mom" style that makes nurturing paramount "if we make
everyone feel good we can all win."

Both work. No _seriously_ both work. It's just politically incorrect to run
Dad-style projects or companies now.

~~~
vezzy-fnord
I like how everyone here is discussing the rhetorical style and ignoring the
fact that his rejection of kernel debuggers is batshit insane and idiotic,
which is the main topic.

~~~
edejong
He's acting as a teacher in this email (like in many situations) and trying to
explain that, in a deterministic and transparent system, the usage (and
necessity) of a debugger is a clear signal of the inadequacy of the design. If
you have to use a debugger, you should go back to the drawing board! You've
made a fundamental design flaw in your abstractions and following a poor
instruction pointer will not help you find that flaw.

It's kind of ironic to see that, nowadays, with the increased necessity for
concurrent and multi-agent systems, a debugger becomes even more obsolete and
cumbersome, since you're often fighting a race-condition or some kind of
resource starvation.

Personally, I see his rhetoric in this email as highly intelligent community
leadership in which Linus is arguing for better systems design (inside, or
outside of the Linux kernel).

~~~
vezzy-fnord
And yet, hardened Linux developers like Al Viro will rant all about the awful
systems design inside and outside of the Linux kernel constantly (cgroups and
udev are frequent targets of his). We can thus extrapolate that a lack of a
kernel debugger has not reduced leaky abstractions in any measurable way. At
best we can only speculate in counterfactuals.

Solaris has a kernel debugger and a more concise architecture.

~~~
edejong
Linus argument is that use of debuggers may contribute to bad design, but he
does not prohibit their use. As Linus was not the core contributor to cgroups
(Google was) and kernel debugging certainly was available when cgroups was
first implemented, I would say the opposite might actually be true.

Personally, I think the UNIX OS design is a poor fit for the requirements of
Docker et al. The problems with cgroups should be a signal that it breaks the
fundamental design assumptions of Linux (doing fundamental resource separation
and multi-ring security in a monolithic kernel).

~~~
vezzy-fnord
Actually, I agree with most of this. However, Linus letting in the cgroups
branch instead of vetoing it proves the opposite, unless the presence of some
kernel debugging (nothing close to kmdb, I'd think) depleted Linus' skills
somehow.

------
luso_brazilian
Considering the date (6 Sep 2000) this should have the [2000] tag for context.

~~~
davesque
Yes, this. The culture of software development was different back then. My
opinion of this little snippet changed a lot when I saw the date.

------
dmethvin
One of the insidious things about debuggers is that they let you be lazy when
reasoning about the code you've written. When you _don 't_ have a debugger,
you need to be more careful about what you do and really understand the
complexity of your code, because a misstep when writing the code can take
hours to remedy. With a bunch of safety nets at hand, you feel like you can
just blow out some code and work it out via the compiler and debugger later.

There's no doubt that better tools have let us be more productive, but in the
process we understand the code we write less deeply than we otherwise would,
because we are required to spend less time contemplating it.

~~~
overgard
I've heard this argument a lot, but I've never really bought it. It implies
most bugs can be avoided by being super careful. Most bugs aren't a result of
carelessness, and if you spend 20 hours staring at each line of code you
write, I guarantee you'll still write software with bugs. The only thing the
debugger changes is how fast you find it.

And I say this working in an area where debuggers either kind of suck or are
limited (graphics programming), so I have plenty of experience working on
complex systems with no debugger. Believe me, I do this exact painstaking
thinking when I write a shader (no break points or even print statements), and
yet the shader code I write is about the same quality as any other code I
write, it just takes longer.

~~~
Retric
At one point I worked on a large, complex, and legacy code base averaging less
than 1 bug per year without a debugger. Granted, now days you can take the
debugger from my cold dead hands, but I also don't really understand the code
nearly as well.

------
thebiglebrewski
Lol if everyone did open source like Linus, we'd never get any new
contributors. I have no idea how this even works in their context either

~~~
gnaritas
> Lol if everyone did open source like Linus

We'd have less, but much better software.

~~~
vezzy-fnord
I didn't know the better the software, the more primitive the development
tools.

~~~
gnaritas
A debugger is a thinking crutch, yes, it's easier and more productive to just
see what it's doing; but to fix something without a debugger, you have to
understand it better. That's his point, and he's right. Quite simply, the
easier you make something, the more people who are less intelligent can do it
and the result is more but less quality software. Examples abound, but every
attempt to make something easier results is people producing more shitty stuff
with pointy clicky interfaces.

~~~
vezzy-fnord
Again, this would require us to reject most advances in software after 1968 if
we are to apply the logic consistently. Go back to single-tasking,
cooperatively multithreaded systems with static file allocation tables and no
access control. Actually even the Multics of 1968 was far more advanced than
that.

Fact is, without debuggers, most highly subtle war story bugs would go
unsolved. Logic analyzers, tracers and ICEs are debuggers, too. "Debugger"
doesn't just mean gdb. Not that there's anything wrong with it.

~~~
gnaritas
As I said, we'd have "less" but better software; that doesn't preclude we'd
not have some things we have today nor does it imply better software can't be
built with debuggers; it only means we'd have fewer less capable developers
making software and thus the overall quality of what we would have would be
higher than the overall quality of what we have today given how much shit code
exists.

There's a ton of programmers who can't code without a debugger because they
don't really understand what's going on any other way than line by line. Those
people would be filtered out. Debugging doesn't require a debugger, it's just
the easy way to do it.

~~~
vezzy-fnord
We'd certainly have less. There would be no reason to presuppose that it would
be better:
[https://en.wikipedia.org/wiki/IEFBR14](https://en.wikipedia.org/wiki/IEFBR14)

The key to better software is learning from the research literature, not
thinking that you're improving because you write nothing but ASM macros in
ed(1) in a TTY emulating a VT100 from the 70s.

~~~
gnaritas
> There would be no reason to presuppose that it would be better:

Sure there, it's been said multiple times; those still writing software would
be the smartest around. That alone is reason enough to think the overall
quality would be higher; if you don't believe that, then imho you don't know
what overall (average) means. You're not listening, you're focusing on
outliers and I'm taking about the mean. It's a valid logical inference that if
you weed out less intelligent developers by making developing harder, that the
average quality goes up; not the absolute quality, the average, mostly because
you'd be weeding out tons of bad developers who write tons of bad software.
This really isn't debatable, it's logic. Weeding out the bad raises the
average quality, there's nothing to argue here, that's just a fact.

------
outside1234
This seems like one of those false "tyranny of the OR" logic bombs: we can
have 1) high quality software OR 2) a kernel debugger.

It seems to me that these two things are not actually logically connected and
we can have both: underthought code is a unrelated problem to a debugger.

Its like saying you can't build world class woodwork if you have a router.
Instead you must only have awls so you have plenty of time to think about it.

It strikes me that simply means there will be less software in the world at
much higher dev cost and that isn't a good thing necessarily.

~~~
VLM
A better woodworking analogy (or parody) would be:

Gentlemen, I am well aware that many woodworking shops have a FooBar2000 scrap
wood burner. This shop will not. I am not saying woodburners are always bad,
or the FooBar2000 is individually bad. Or pretending that scrap never happens.
Why I enjoy a warm toasty winter night in front of a fire like the next lad.
However having a scrap wood burner implies we create scrap that needs burning.
And I will not tolerate that in my shop, nor encourage other shop owners to
recklessly generate scrap.

Gentlemen, here we make keels of ships. The entire ship relies on this keel
never breaking, creaking, cracking, groaning. Upon those ships... Lives
depend. Businesses depend. Governments depend. And I will not tolerate scrap
as a boat keel and all the destruction failure can cause. Therefore given that
we can't afford to create scrap, we will work slowly and methodically and
intelligently so as to not create scrap, and thus have nothing to burn in the
FooBar2000 scrap wood burner, thus we do not need one. In the unlikely event
that I or someone else make a keel that needed some scrap to be disposed of,
well, if it tests good, thats OK. But I'd never tell a dude setting up his own
shop that if he's going to build boat keels he badly needs a FooBar 2000 scrap
wood burner because he'll probably screw up a lot, because if he's screwing up
a lot, I don't want the world depending on his boat keels and I won't accept
that caliber of workmanship in my shops nor stamp my name on it.

Gentlemen, you may not like my opinion about your deeply held beliefs about
woodworking. I sympathize. I don't agree with everyone else either. Thus I'll
give you a psychological outlet by talking like a pirate for a bit now, and
making light of your mother's parentage and something about elderberries. Thus
if you're insulted by my technical evaluation of your woodworking skills, you
can tell everyone else you're actually insulted by my talking like a pirate.
Then everyone goes home happy, and the world gets safe strong indestructible
boat keels.

~~~
unusximmortalis
LOL

------
unusximmortalis
I love Linus point of view here because although he is using words which can
have negative connotation to describe his position here ("I am a bastard", "I
do not care", etc), he is actually looking to steer the community towards the
greater good.

~~~
nickbauman
I'm not a kernel developer but his rant strikes me as very platonic, and not
in a good way. Platonic in the sense that real knowledge requires turning away
from the cosmos and "deeply introspecting" to obtain a "gnosis" of sorts.
Plato, for all his contributions, was at heart a religious ideologue closely
aligned with the corruption of the Pythagoreans that came before him. Reality
was "the cave" and the gnosis was leaving the cave.

Full disclosure: I don't use debuggers myself, partly because it isn't
practical (I spend most of my time in Go these days, rather unwillingly I
might add, and Clojure. Both languages aren't terribly friendly to step-
debugging.) but when I need one I really, _really_ need one.

~~~
tremon
> Plato [..] was at heart a religious ideologue

Which religion would that be?

~~~
nickbauman
I don't know it by name but this quote reveals it pretty soundly: _" We shall
approach astronomy as we do geometry, by way of problems, and ignore what is
in the sky."_ (Plato's _Republic_ ) This kind of thinking is likely the same
sort of thinking that got Kepler off the track trying to force the Pythagorean
solids into planetary orbits in his _Harmonices Mundi._

(maybe call it "Pseudoscientism" a belief that merely scientific-looking
commonalities are in fact universalities)

~~~
tremon
I am unclear how "We shall [..] ignore what is in the sky" translates to a
religious ideology. It sounds to me like Plato is arguing against the Greek
tradition of trying to explain a natural phenomenon by tying its existence to
a god, and arguing for (what we now call) scientific rigour.

Can you clarify what you mean with "this kind of thinking"?

~~~
nickbauman
That tradition was not at issue with Plato and his ilk.

If you can agree that religion is merely _institutionalized_ ideology, I think
by contrast you could look to Aristotle, whose lifespan intersected with
Plato's, who took the opposing view: Describing what he has observed about
bees, he writes "the facts have not been sufficiently ascertained, then we
must trust perception rather than theories."

The Greeks formally established the mechanics of inductive reasoning as the
formulation of truths from _empirical evidence._ Plato, for all his gifts,
stands firmly in contrast to this. Ideology is fundamentally about turning
away from evidence and making the world fit into your polemic.

~~~
tremon
Actually, their lives intersected, not just their lifespans: Aristotle studied
under Plato.

 _If you can agree that religion is merely institutionalized ideology_

I do not fully agree, as not all ideology would qualify as religion, even if
institutionalized. For example, capitalism is not a religion yet it is both an
ideology and thoroughly institutionalized. I would agree that religion is a
_form_ of institutionalized ideology, but not _merely_.

But I (think I) understand what you're saying. You're saying that because
Plato considered fundamental truths unknowable, he made his vision of the
world subject to dogma concerning the interplay between the unknowable and our
reality. That is certainly true, but I'm unsure how much prescriptive dogma is
in Plato's works.

Regardless how his theories have been adapted by Christianity later, for as
far as I know Plato himself:

\- never declared empiricism as futile,

\- did not give his "Shapes" any agency in our reality,

\- did not create his own narrative of good vs evil.

In fact, Plato was quite direct in his approach to morality: people should do
"good", because doing good things would bring happiness, and doing bad things
would create unhappiness. That is a quite utilitarian approach to
civilization, and I would not call this a religious stance. Plato did consider
the soul to be immortal, but he also said that learning would begin anew in
the next life, so I don't think he subscribed to any form of afterlife
judgement.

~~~
nickbauman
I understand "the cave" in Plato's _Republic_ to be the world as we understand
it through the senses. (AKA "the natural world"). Leaving the cave is going
into a deeper "gnosis" that, perhaps, resembles pure mathematics, or something
else, he doesn't make clear, but it was certainly mystical and thereby
ultimately "occult" (or fundamentally "hidden").

Consider this tradition survives even in Newton, who was an avid _alchemist,_
wrt The Philosopher's Stone and obsessed with the numerology of the dimensions
of Solomons Temple even as he worked on his _Principia._

------
seivan
I like his vulgar style, but that's pretty much irrelevant here. How can one
think debugger is a bad thing?

I feel kinda incompetent as I am missing something here, does the inclusion of
a debugger, even when not in use hamper performance or anything in particular
in anyway?

Is it the same type of debugger, where one could step through code and inspect
objects or is this something completely different when it's related to the
kernel?

~~~
unusximmortalis
good questions, however, the answers are all in his email :)

~~~
seivan
I guess I was shooting for more technical answers related to e.g security,
performance or etc.

------
pklausler
About the time this post was written, I ported the Linux kernel and basic
userland to what was then a brand new instruction set and system architecture
(the Cray X1 and its successor). It was much easier to do this in a simulator
environment with breakpoints and tracing than it would have been on real
hardware, had I even had any. Is that a debugger? Fine. I think Torvalds is
off base here, trying to blame crutches for somehow causing lameness.

------
overgard
This is such a dumb thing to be proud of. Would you want to hire a contractor
who says "I'm not going to use a hammer because this rock I have works just
fine for driving nails! Sure if you use a rock maybe building a house is
slower and you smash your fingers a lot, but I wouldn't want to work with
builders that take the easy way out!"

~~~
falcolas
A better, though still poor analogy is to say "the house keeps falling down
when u build it, so I'll use an ultrasound device to inspect every nail as its
pounded in to find the one that fails."

Linus, in his own way, is pointing out that an ultrasound will not identify
that your house is missing a wall.

------
irascible
This rant overlooks the use case of stepping through new code in a debugger
regardless of whether it actually works or crashes. Inspecting intermediate
values.. making sure execution follows the right paths etc. Debuggers aren't
just for bugs.

------
pliu
I wonder if Linus has changed his opinion, 15 years later.

~~~
gdubya
I hope not!

------
paxcoder
If you want freedom and manners, don't follow Linus.

~~~
seivan
Maybe I don't want your definition of manners, the same way I don't want
Westboro/ISIS definition of religion. Don't read too much into vulgarity.

Linux is vulgar, but that's pretty much it. Doesn't mean he's a bad person.
His opinion can be bad without resorting to bashing his personality style.

~~~
paxcoder
Definitions are only useful when agreed upon. Being vulgar and unpleasant
towards people means Linus(with an s) has bad manners. You can choose to
tolerate that for some reason, but you cannot expect it of other people.
Instead, it is expected that people moderate their own behavior.

P.S. Ask yourself why you have double standards with regards to "bashing".

~~~
seivan
What double standards? I think you're reading too much into the word "bashing"
\- proving my fucking point.

Exceptional.

~~~
paxcoder
I'll be curteous and elaborate: The double standard is allowing Torvalds to
bash, but disallowing "bashing" his bashing.

I refrained from guessing the cause of this before, but I believe it is you
being a fan of Torvalds' and/or being sympathetic to his unwilingness to
moderate own behavior.

Please address my point about definitions or explicitly introduce a new point
if you really want to argue the other thing further.

~~~
seivan
Because it's an easy way to get off topic. You're mixing his personality style
with the content. I'm saying ignore anyone's personality style and focus on
content.

Be the change you want to see. Going off topic to bash his personality style
doesn't really add anything to the discussion of Kernel debuggers.

You're right with me being sympathetic, I don't hide that but it's not
relevant to my point. Thanks for being curteous.

The double standard is how we attack Torvalds for the same thing radical
feminists say on a daily basis. You ever interacted with Shanley?

I don't care about their personality style. They're both abrasive, I just
think Torvalds has actual value and skill sets that I could only dream of. The
other one... Not so much.

~~~
paxcoder
That's a different point. What you were doing was defending the personality
flaw.

Going "OT" was due to the title. I thought it was that post where Torvalds
obstinately defends his rudeness. Basically the man's lack of style almost
prevented his point from being communicated. So even digging for points which
I did and do* only went so far.

*Mind you, I've found sifting through noise to have diminishing returns. People often use cuss words instead of arguments. Strong points do not need strong language.

------
vezzy-fnord
What the fuck? Everybody here is praising Linus for some of the most perverse
and regressive logic imaginable because they enjoy his supposed bravado.

What nonsense. You can apply his exact same logic about kernel debuggers to
reject all advances in software made after 1968. If only all programmers used
nothing but DOS and ASM on 4-bit CPUs, we would finally be liberated. Finally
let "Darwinism in software development" take its course.

(Of course, I'm sure most people commenting here have never hacked on anything
more sophisticated than a V7 Unix derivative.)

~~~
twic
Linus's argument here isn't just wrong, it's _bloody obviously_ wrong. And not
for the first time. Does anyone remember the Git generation number debacle? Or
the business about wanting fixed hardware interfaces to video cards?

Linus is an uncommonly gifted low-level x86 hacker. But his technical judgment
is mediocre. I just don't get why he's held in such esteem.

------
moron4hire
>> And quite frankly, I don't care. I don't think kernel development should be
"easy".

Aaah, the myth of the Protestant Work Ethic.

~~~
VLM
Possibly the part you misread, or Linus being a Finn, he might have poorly
expressed in English, was the "start being careful early rather than late".
The whole essay hinges on that.

Look we're all devs here, and everyone who's written more than fizzbuzz has
run into judgment issues about speed. A military campaign analogy fits really
well. You can advance slowly, defensively, without possibility of failure or
accident, with full knowledge of every little step you take, every footfall of
every process is understood and safe. In which case your code will never crash
and people can run nuclear reactors with it and coincidentally a debugger
would be a complete waste of time because it'll never be needed. Or you can
ambitiously run ahead in a reckless offensive charge, probably right into a
swamp or ambush or minefield, and then you'll need a debugger to pick and prod
your way out of a self caused disaster. That obviously requires a debugger. It
also results in code that possibly thru sheer luck didn't crash into the
debugger being shipped, and then killing people when the nuclear reactor
controller crashes or whatever. There are whole swaths of logical errors and
race conditions and spec problems that debuggers won't help with but sitting
there and thinking hard will avoid, and nobody wants to find all those
mistakes the hard way in shipped deployed code, and then Linus has to find
what should have been an obvious bug had the dev slowed down and thought a bit
first, etc.

If you needed a debugger because you were too lazy to check a memory reference
before you freed it or whatever, that's probably the quality level of code
that also "forgot" to check for trivial buffer overflows or null termination
of strings or just basic foolishness. He just doesn't want it.

Note that this is ultra low level kernel code that entire systems are based
on. You have to be more paranoid and slow and methodical than everyone else
with this kind of code. Probably the completely wrong attitude for someone who
just installs wordpress, etc.

Note that he expressly denies the work ethic... if you can send him production
quality code despite having used a debugger and despite having intentionally
avoided thinking hard, if its none the less good he explicitly says he will
take it... he (and I) just think it very unlikely that situation would occur.
If he demanded the work ethic he'd be insisting on testimony that you
meditated on that function for a minimum of two minutes per line or something
equally effort based rather than achievement based.

~~~
moron4hire
It is impossible to plan in the way you're suggesting. You cannot ever have a
perfect view of the road ahead of you. What you're suggesting is that having a
policy of taking a look around you to see if your mental model actually
matches reality is being "lazy".

------
draw_down
What I don't get is why he chose to live in Portland. People here are... not
really like that at all. It must drive him nuts.

~~~
mr_tyzic
Probably because he isn't really like that.

