```A year or two after I'd joined the Labs, I was pair programming with Ken Thompson on an on-the-fly compiler for a little interactive graphics language designed by Gerard Holzmann. I was the faster typist, so I was at the keyboard and Ken was standing behind me as we programmed. We were working fast, and things broke, often visibly—it was a graphics language, after all. When something went wrong, I'd reflexively start to dig in to the problem, examining stack traces, sticking in print statements, invoking a debugger, and so on. But Ken would just stand and think, ignoring me and the code we'd just written. After a while I noticed a pattern: Ken would often understand the problem before I would, and would suddenly announce, "I know what's wrong." He was usually correct. I realized that Ken was building a mental model of the code and when something broke it was an error in the model. By thinking about how that problem could happen, he'd intuit where the model was wrong or where our code must not be satisfying the model.
Ken taught me that thinking before debugging is extremely important. If you dive into the bug, you tend to fix the local issue in the code, but if you think about the bug first, how the bug came to be, you often find and correct a higher-level problem in the code that will improve the design and prevent further bugs.
I recognize this is largely a matter of style. Some people insist on line-by-line tool-driven debugging for everything. But I now believe that thinking—without looking at the code—is the best debugging tool of all, because it leads to better software.'''
But when I’m working on large software projects or certain external libraries I tend to encounter bugs or design issues where I realize I made the wrong assumptions about how someone else’s code works in the first place, and a good debugger is very useful in those cases when the problem changes from debugging your own logic to reverse engineering someone else’s.
The first point your code sees uncertain data is the point that it needs to clarify what it has.
The only way you get the errors you're talking about is if you ignore the above practice.
Garbage in. Garbage out.
For code i write, there is usually only one option for what is in a variable, the type of the data i out in there. This is true for dynamic languages as much as static.
In some situations I will also allow a null value. All that means is that there was no data, and no default is wanted. Usually, I want a default.
Closing off all entry points for uncertain data you need to make assumptions about is the first port of call when dealing with other people's or legacy code. It's the way to reason about code without a debugger.
If you can't reason about code without a big question mark above every piece of data you need a debugger.
If your mental model of how the code that you're interfacing with works is wrong, then it won't help you. Your data validation will error out and you might find that it was too much too early. Not only do you have logic based on flawed assumptions but check code as well.
Understanding the code base you're programming against, that's a skill that can be improved but hoping to guard against all misunderstandings is probably unrealistic. Given that, following a trace may help you identify disagreements with your model more quickly.
I am saying you don't need a debugger in the scenario above because the only time you need it in that case is if your inputs can't be trusted.
If you control the domain, the only way you can't trust your input is if you fucked up.
The answer to that isn't "oh now I need a debugger" the answer is to go clean your code.
While I'm here, I'd also like to point out this isn't a "I'm good and your shit" thing I'm describing, this is my day job. I make lazy crap code to get things done, then I go in to modify it and find I can't reason about it, so I go and clean up the mess.
Maybe I made it, maybe I didn't, it's besides the point, you have a mess, clean that first, then reach for a debugger.
Hell reach for a debugger to help clean up if you need to, just don't sit there and tell me you need a debugger because code inherintly needs to make assumptions about what is in a variable. It doesn't. If it does, it's a code smell.
> So many errors boil down to assumptions about what is in a value.
Only avoidable errors. They should not be dictating your tools or your language.
Not making mistakes certainly saves time.
I often use a debugger to inspect variables and check my assumptions about the application state at that point.
Good programmers are humans and make mistakes, of course. However, that doesn't mean good programmers don't exist or that people can't vary a lot in skill or average quality of output.
She is correct that starting with some notion of "good people" is no silver bullet. But the conclusion that "we all suck", I hope that is hyperbole, because it isn't true. (Or maybe: everybody sucks a little bit differently.)
I am the first person to understand developers are far from perfect, my self included. But it's just code, you can refactor and learn from mistakes.
The thing I don't buy is that this stuff is difficult or that you can't expect good behavior from professionals.
Avoiding side effects, or pushing them out to the edge where reasoning about them is clear is something you should have been taught when you got your expensive piece of paper.
Likewise, avoiding, mutation, over abstraction, early optimisation, and variable reuse are all really basic shit you need to understand, if you're not hiring for these basic qualities, what the fuck are you hiring for? Good looks?
An example inconvenient debugging story: https://mjg59.dreamwidth.org/11235.html
(I don't think DMA triggers hardware watchpoints, but if you set a hardware watchpoint and an an address changes value without hitting a watchpoint you know that something funny is going on)
This. A few days ago, I spent a couple of hours trying to get a Jest+Enzyme test to open in chrome inspector, (apparently there's a bug which causes debugger statements in Jest tests to be ignored), because I hit an edge case bug in a method in the Enzyme library. If I had been able to step through it in the debugger, it would have taken me minutes to figure it out, instead I spent a couple of hours going through the codebase and figuring out where to put the log statements.
But when everyone starts taking this view on it, the result is that changes get made tactically based on what makes your task work. With no understanding of the overall vision. The result undermines the integrity of the system, and makes debuggers ever more necessary going forward.
This is a good point to hold in your mind as you read or re-read Programming as Theory Building by Peter Naur: http://pages.cs.wisc.edu/~remzi/Naur.pdf.
I guess the pie in the sky vision is that tactically making your task work somehow becomes harmonized with the overall vision. Extreme Programming was supposed to do this through the high information exchange of pair programming, the constant refactoring, and the practice of there only being 7 or so large scale patterns for the whole of the application.
The way this works in most of the real world, is that it's supposed to work like this, but there's no pair programming, and you never get enough time to refactor.
I rarely actually fire up the debugger, and I try to refrain from gratuitous prints (though I use them in emergencies or if a debugger isn't convenient).
I think this makes me a better programmer, though that's really hard to tell objectively.
You can pry my debugger from my cold-dead hands, but I don't even know how to step through code in my favorite debugger.
In my opinion, the only right way to fix a bug is to build a mental model of the code, and a debugger is a massive force multiplier in doing so. For any code that wasn't written by me in the past 6 months or so, the code itself is a mystery and while reading the code is a big aid in understanding how it works, it can also mislead you. In particular there is a class of bugs that are when how the code actually works diverges from how the author thinks the code works. A good author will structure the code to guide a reader into how the code works, but when the author was mistaken, this can be very misleading (in particular I will actively avoid reading comments when I know there is a bug in the code, because comments are the one part of code that are never tested).
Another way of putting it: The source code is very good for telling you how the program is intended to operate, and a debugger is very good for telling you how the program actually operates.
1: I'm sure it's listed how somewhere here, but I've never felt the need for it: https://common-lisp.net/project/slime/doc/html/Debugger.html
1. Instrument the code in one way or another. At the simplest level, many debuggers can log each function call.
2. Change the definition of functions while the system is running. Most highly dynamic languages will let you do this. I'm told that there are C IDEs that can do this too though (modulo inlining anyways).
3. Change the timing characteristics of the program; if a race condition is suspected, ordering can be forced through the use of thread-local breakpoints, for example.
4. Inject specific data. Have a function called under normal operating conditions and modify some or all of its parameters. Think instant unit-test, but no need to mock anything.
If your lisp supports it, pressing 's' in sldb should do it.
Personally I _love_ debuggers when coming to new codebases. I view code in the process of execution as the natural state of a codebase. Instead of hoping around through source code by hand, why not step into functions, continue execution, and let the program flow do it for you? Most code only has small parts that are important and with debuggers I can usually find those parts right away.
I don't blame Torvalds for not wanting to use a debugger. He doesn't like them and doesn't want to support them. That's his choice. But I find it odd to just categorically dismiss them.
One of my favorite bugs I ever introduced was typing 0 instead of O in a variable name. Review your mental model all you want, a debugger is going to be a lot more useful in sussing that kind of thing out. Even just being able to see you have two variables on the stack RockOn and Rock0n will pretty much save you.
I rather suspect that Torvalds' beef with using debuggers is that so many engineers get lazy and begin to use them as a substitute for thinking things through.
I helped found a Mathematics of Finance masters program at my university, teaching a numerical methods course before we could hire specialized staff. The students got stronger each year. My last year at it, three very strong students were trying to implement a research paper, tying together all their work to strengthen their resume. They had an impression of me as hot-shot C programmer from a computer algebra system I had written (everything is relative) and they sought my advice when their code wouldn't work.
I was in deep inner cringe, aware of their hefty tuition, how I truly didn't understand a word of what they were saying. I had to think of something to say that would get them to leave my office, apparently satisfied. Then I heard their voices catch as they apologized for introducing a fractional time step as their algorithm shifted phases.
"No! Always a bad idea! Here's another way to do that. I don't know if that's your bug, but..."
The email that evening was profusely thankful, warning me they'd be back the next day with their new bug. Launder, rinse, repeat.
While I'll probably never hang a shrink shingle in Silicon Valley, as tempting as the idea is, one can also listen to one's own voice catch. Ever notice how everything that can go wrong cooking was actually anticipated and ignored at the time? Mathematical research teaches one to listen to the faintest anomalies. It's the universe whispering truths.
Pike is conflating the diagnosis process with the quality of the chosen solution. In practice, they are often unrelated.
A good debugging tool will not only show you the state of the system when the bug happens, but help you understand the execution path that formed that state. (If your debugger doesn't do that for you, then you should either get better at using it, or find a better debugger.)
A debugger _doesn't_ tell you how a problem should be solved; that's a separate decision which often has many other factors beyond "improve the design to be more future-proof".
The story also implies that it should be enough for the programmer to explore the mental model they currently have. But if I have a problem that needs debugging, and a minute's thought isn't enough to reveal it, then it's usually because that mental model _is wrong_. I've spent enough time dealing with engineers who furiously insist that a system shouldn't be doing something that it clearly _is_ doing (including myself) to know that your mental model can be just as much of a barrier to debugging as it is an aid.
So, as often with arguments about whether you, a smart engineer, can solve all problems with pure reason: No, you can't. There are often things that you don't know, and things that you don't know you don't know, and getting external assistance is often a huge time- and embarrassment-saver.
You've got to turn off the targeting system and start thinking about problems holistically. Programming is not yet a solved problem and the really valuable element in the equation is a human's ability to reason and predict by aggregating massive amounts of information.
1. Write down the problem
2. Think really hard
3. Write down the solution
I often debug in the same way I design software. Step away from the computer and write something down.
It's probably a personal thing, but I solve problems much faster with a pen in my hand as opposed to a keyboard at my fingertips.
Some, things to note. I've worked on code bases that were in the 10s of millions of lines of code (healthcare industry). I still use this technique, but with the understanding I may never have a complete model of a code base that size.
It only makes sense to me, although it does require you to actually understand what the code is (trying) to do.
I do sometimes use the standard debugging techniques, including debuggers, logging, etc. But I end up doing that only when I'm working with code that I don't fully understand, or if I'm totally at a loss as to what's going on. I'd say that covers 10-20% of cases.
Of course you can verify these invariants with a debugger. But when you hit a bug and immediately start digging in like Pike says, that tends not to be what you're thinking about.
Debuggers are great for putting in the minimal hack in a production codebase that doesn't disturb the code. But they're generally bad for developing software, because most (all?) software should have some invariants that allow the Thompson style of debugging.
You step back, think about what invariant could have been violated to cause the bug, put in a print statement somewhere to see if it was violated. Then you take another step back to think about how to fix it, possibly with a fundamental change to the data structures. That is, you design your software differently to avoid the bug, rather than just putting in the minimal hack.
The other problem with debuggers is that once you've used one and modified the code, then everybody who modifies the code later probably has to use one too. The code has become subtler, but not in a good way. This makes development slower.
I think it also relates to another thing Linus said :
Bad programmers worry about the code. Good programmers worry about data structures and their relationships.
It's easier to diagnose problems in data structures (by printing them) than to diagnose problems in control flow (which is greatly aided by a debugger).
One weakness of debuggers is that they don't print things in application-specific formats. You have to write debugger plugins, and those aren't set up for most projects, and they don't behave the same way on all platforms, etc.
So basically having a bug that you can't figure out without a debugger is a sign that you may be using too much CODE and not enough DATA. It's a smell that indicates a problem with the software design.
Of course, in the real world, sometimes you have to swoop in with a debugger and fix something. But I do think that there is something wrong with living in a debugger for months on end (which I have experienced on one dev team.)
As you can see, a number of invariants tie the different record fields together. Maintaining such invariants takes real work. You need to document them carefully so you don't trip over them later; you need to write tests to verify the invariants; and you must exercise continuing caution not to break the invariants as the code evolves.
It's also covered here under "make illegal states unrepresentable".
The author is advocating OCaml, which allows you to encode such invariants in the type system.
But 99% of state machines are not written in OCaml -- more often it's C or C++. For example, the entire Linux kernel :-/ So basically I'm saying rather than poking and prodding at the code with the debugger and flipping variables, you can think of your whole program as a state machine, and the state machine has invariants like the ones listed:
last_ping_time and last_ping_id are intended to be used as part of a keep-alive protocol. Note that either both of those fields should be present, or neither of them should. Also, they should be present only when state is Connected.
For another example, the "lossless syntax tree" data structure I use for my shell has a number of invariants:
I use the same data structure for generating errors and translating between two languages. So it really has to have some higher-level properties rather than just setting and getting variables.
In C# they have something called "Red-Green Trees" that are similar:
In summary, your debugger is not going to show you which nodes are red and green in this sense! It has no idea about such things. IMO it's better to think at a higher level when you can.
(Note that red-green trees are just a name for an application-specific concept they made up. I t has nothing to do with red-black trees, which are a generic data structure. Even so, your debugger also has no idea about the invariants in a red-black tree implementation either!)
Maybe a simpler way to put it:
- the assert() statement in C and Python checks invariants at runtime. There's something of an art to doing this; I think it's covered in books like "Code Complete". You can grep a codebase for assert() and see what invariants the author wants to maintain.
- I'm not sure if they still teach "loop invariants" anymore, but it basically means "something that's true at every iteration of the loop", regardless of whether it's the first iteration, last one, in the middle, etc. If you have off-by-one errors and you stick in a -1 to fix it, that's a sign that you could think more rigorously about the loop.)
Invariant means "a thing that's always true, regardless of the values of the variables." It's a higher-level property than what the debugger shows you.
Survival of the fittest sure, but when the requirements to change are hard because of high technical accomplishment and a high standard vs grossly complex are two different things.
I like Linus and his willingness to say what he thinks but sometimes he really is just an asshole from the old guard.
It makes for an impressive story and establishes the speaker as a jedi master.
It's also mostly bollock. Every one will have epiphanies. I sometimes work on a problem for a while, go to the loo and have the solution come up. It's the well-known phenomenon of stepping away for a while and letting teh brain unfocus from the details.
Yes, it works when the problem was something fundamental or architectural. It rarely does when the problem was using the wrong variable, or a test that was inverted. These are easier to find with traces or a debugger.
But dissing the low-level practices is so much better at getting that alpha status...
Always difficult to tell what the cost of this was. What features did we not get because developers were put off? What problems did it prevent? How did this affect downstream Linux ecosystem development and culture?
Re: use of debuggers, I've found them most useful when dealing with other people's code; you can short-circuit a huge amount of "where is the code that does this" or "how did I get here" by just setting a breakpoint and getting a stacktrace.
And in other cases, the debugging statements have to be present in the code to even get to step through the program, and their concerns revolve around code management when tools that have code inserts like that.
But it would be nicer if they wrote their stuff like their use cases weren't the only ones, or the most important. What features have we lost because they couldn't think past themselves?
For example, I would think that the best Java engineers are probably much more productive than the best C++ engineers. However, the market is probably full of poor Java engineers that the average Java engineer is probably a lot worse than the average C++ engineer.
C++ is such a complicated language that if you are a professional C++ engineer and still employed, you must be fairly skilled. So, for example if you start a tech company or an open source project it might be better to choose C++ as the implementation language even though Java might make you more productive.
Funnily enough, Linus does not apply this logic to C vs C++ debate and has come to the opposite conclusion : http://harmful.cat-v.org/software/c++/linus
I don't know whether this is his goal (he does make it sound like it), but I think the consequence is another: In order to manage not having a debugger, things are by necessity kept simple. It reminds me of a quote from Brian Kernighan:
> Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?
Since simplicity has other positive consequences apart from ease-of-debugging, there might sometimes be advantages to not using the fanciest tools all the time.
By taking twice as long to debug it ?
Nah. Many of the C++ gigs I've had were at places which only used fractions of the language. There are a lot of bad C++ programmers. I'm not disagreeing with your overall point about ratios though.
I don't think any project or person uses or even knows the entire language. I think even Bjarne himself said that no human can possibly know all of C++.
Seriously, in every C++ project I've seen, it seems like whoever's in charge picks some subset of the language to confine themselves to, and uses that. One of C++'s strengths is that you can do this; it's adaptable to your project and style.
Deliberately keeping a working environment hazardous and unsafe doesn't give you a breed of superhumans that never let accidents happen, it just leads to a lot of unnecessary accidents.
I'm pretty sure this isn't true. Averages can be skewed by outliers. It's likely that most people are better/worse than average. Consider how the average life expectancy was horribly skewed by infant mortality.
The "Blub Paradox" argues you should hire developers using the most productive tools, even if those tools are unpopular, because that naturally selects for the best developers.
You are arguing that you should hire the developers using the least productive tools, because if they can get anything working at all with those tools, they must be pretty talented.
So there are points in all 4 quadrants I think.
LISP/smalltalk: PG-productive, not difficult
Java: PG-unproductive, not difficult
C++: PG-unproductive, difficult
(Forth? Haskell?): PG-prodictive, difficult
The "Blub Paradox" is the claim that you can only understand how powerful languages are in terms of the powers the languages you know have. So if you understand Python and C++ and not Lisp, you are unable to estimate the power of Lisp.
While I didn't get this opinion from my read of Linus's post, Linus seems to want to make things unnecessarily hard and somehow fails to see how that could be bad because of lack of arguments while simultaneously claiming making things easy would just be bad outright with little supporting arguments for it.
I was under the impression that software should be made as simple and easy to understand as possible. So for example, should I make some esoteric bit manipulation to multiply a number by 2^n or should I just use a library function? It may make sense if performance and memory are limited, but that's about the only reasons I can think of. Otherwise, you should have the code explicitly state your intent. It makes me think the kernel is a garbled mess they way its talked about in that post with not very easy ways of maintaining it.
Most commercial projects I've worked on really were a "garbled mess" by comparison, and far less maintainable.
It seems like he'd exactly agree. He goes on to say he doesn't want to add all those features, he'd rather go slower and his biggest job is to say "no" to new features.
I guess, my original point is, there will be "average" developers committing to the kernel just by statistics alone.
Obviously there's still a lot of greenfield C++ work out there too, but I would expect that it's no longer the lion's share (C# is very much getting to this stage of it's life too).
I would guess that the top-level C++ folks have by-and-large moved on to those greenfield projects naturally as it's where their skills are most required, and where the most money is available to pay for them (e.g. fintech).
I wish we moved to Rust (or even Ada), because most of the bugs I see occurring couldn't happen there. But embedded compilers are usually lagging a bit.
Given that C# is the go-to language for new development on Windows, and with Microsoft now officially supporting cross-platform C# and .NET, I find that very hard to believe.
What’s the trajectory of - say - new Github projects that use C#/.NET?
It's still used in a lot of new embedded work. It's still the best option for a lot of HPC and numerics. Some of this work ends up being targeted mostly for inclusion in another language runtime, sure, but it's still new work.
> And quite frankly, I don't care. I don't think kernel development should be "easy". I do not condone single-stepping through code to find the bug.
Reminds me of this quote from the introduction to Log4j docs :
> As Brian W. Kernighan and Rob Pike put it in their truly excellent book "The Practice of Programming":
>> As personal choice, we tend not to use debuggers beyond getting a stack trace or the value of a variable or two. One reason is that it is easy to get lost in details of complicated data structures and control flow; we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places. Clicking over statements takes longer than scanning the output of judiciously-placed displays. It takes less time to decide where to put print statements than to single-step to the critical section of code, even assuming we know where that is. More important, debugging statements stay with the program; debugging sessions are transient.
The above quote made me reconsider my intense debugger usage that I had fallen into at the time. On one hand you can't take all your cues from authority, but on the other hand, what K&P said sounded like it might make sense. So I tried using the debugger less and logging more, and over time I think it's made me a better programmer. I think the key statement is:
> we find stepping through a program less productive than thinking harder and adding output statements and self-checking code at critical places.
which Linus also echoes:
> I happen to believe that not having a kernel debugger forces people to think about their problem on a different level than with a debugger. I think that without a debugger, you don't get into that mindset where you know how it behaves, and then you fix it from there. Without a debugger, you tend to think about problems another way. You want to understand things on a different _level_.
All that having been said, surely there is still great value in applying a similar amount of consideration as to where to put the breakpoints as you would in determining where to put a print statement! (edit: and also in thinking hard about what new values to inspect if your initial guess is not borne out)
This is why I love debuggers. Well, the Chrome debugger. But I find that this situation is less common or doesn't happen at all when I'm writing Java with unit tests. It seems like dynamic environments like the browser are more ripe for debugger use as you describe. But then again, I've met some incredible developers who made really great browser UI (ie the RethinkDB UI) who told me they've never used the browser debugger.
A lot of the hard problems I've debugged have been the interaction of multiple state machines. In these cases I needed to see the forest of interactions rather than the trees of instructions. On top of that, I felt it was faster with getting more information up front (the entire log) rather than identifying what I need to look at, starting the program with debugger attached, setup break points, get more information, and repeat (yes, a lot of my problems required relaunching the process or even rebooting so I could start from a known good state, yay hardware drivers).
For the book Kernighan and Pike wrote, The Practice of Programming, the canonical abbreviation is TPOP.
Sorry about the pedantry.
I remember being scared and nervous looking at a large codebade and single stepping through the code. I did not know how else to understand the code.
You mention that one needs to think at a different level. Could you offer pointers as to what that level could be. I lost my job as a developer and I am trying to get a developer position. However I am absolutely scared of looking at large codebase and not being able to understand it and later not being productive.
Single stepping through code for me has also been a great entry point into large code bases. I think what happens over time, especially if you look into the internals of open source project and libraries, is you lean to identify what kind of thing you're looking at: like, Is this a thing that's driven by an event loop? Is this an event driven system that calls some pipeline of functions through the stack? You begin to see that people generally make things in only a handful of different ways, and so it's easier figure out how things are connected. But for me it took a lot of stepping through code to figure out a lot of these things, so I'm not sure either how else you can go about it.
It is true you ought to build up a model of what code is doing to work on it properly, but I find trying to just read the code that I'm not familiar with to build that model is often both slower and worse than using a debugger to walk through it. Slower, because I'm sitting on the outside trying to stare in at a dynamic process based on my radically incomplete understanding of its static description and the stored state in my rather meager head (I think my raw ability to hold state in my head is probably a bit below average in the programming domain, as evidenced to me by the behaviors I've developed to compensate and other people's lack of need for them). Worse, because when simply reading code it's easy to miss the "validate_input(input)" call that in fact turns out to invoke a multi-thousand line subsystem that doesn't just validate the input, but also statefully mutates it based on all kinds of things and is actually 60% of the system despite its seemingly innocuous name. Stepping through can fix both problems much more quickly, and I'm not being paid to "not use a debugger".
I don't disagree that pervasive, continuous use of a debugger can afford bad habits, but the best answer is to simply not develop those bad habits, and then still use it when it is helpful... because there are days when it is very very helpful.
(Highly recursive code is an exception because Ye Olde Printe Statementse really break down at that point, and even if you make them work it can still be a lot easier to understand it from the inside than the outside, after the fact. I don't just mean "recursively folding on a list", I mean code that does things like walk heterogeneous trees for a DOM tree or a compiler AST and does a whole bunch of inter-related recursive stuff. Sometime you just have to admit your brainstack has overflowed and call in the tools. Best to avoid writing that code at all if you can, of course, but there's times when it is not avoidable.... because that's where the superpowers come from: https://steve-yegge.blogspot.com/2007/06/rich-programmer-foo...)
Stupid question - did their debugger not have breakpoints? Not that that would make using the debugger superior in all cases, but would eliminate time wasted "single stepping to the critical section".
I do agree with their overall point.
As for logs, those are a necessity and should be in every complex program not just because of exceptions, but because of program logic that you implement and want to know where you are, and were, while the program is in use.
Tracing is nice when it is a feature built into the language implementation (i.e. the compiler or the interpreter).
Debuggers are useful when you really have no fucking idea what's going on.
Thinking is still required, but I don't recommend relying on it, since it's typically what gets you into this whole mess in the first place. At the very least, don't rely on thinking to guess what the computer is doing, when you can use the debugger to find out for certain.
Even when reverse engineering something it's best not to get bogged down in the instruction-by-instruction details and instead try to follow the general program flow and reason about how you would do something if you were in the programmers' shoes.
However, I did have one job working on an embedded device with no console at all, and a JTAG debugger was invaluable there. But I would have preferred to have a console and be able to do printf debugging, as for easier bugs it's just a lot quicker.
But, those times when I had to use it, it saved me probably thousands of hours of work - especially when debugging complex software which I didn't write myself.
And nobody has explained to me why these are _bad_ things.
Uhh, you're dealing with a largely voluntary unpaid workforce that requires a lot of effort to become proficient, and by definition is not user facing.
Any barrier to entry you throw up is going to decrease the viability of the project and have knock on effects years into the future.
Are you, though?
"While many people tend to think of open source projects as being developed by passionate volunteers, the Linux kernel is mostly developed by people who are paid by their employers to contribute."
And even if this were true, Linux is no ordinary FOSS project at the risk of losing "viability", with what I imagine is an absolute minimal amount of casual contributors which would be turned off by this sort of supposed "barrier to entry".
Very much, yes. There are still so many bugs and the monolithic nature which forces driver to be integrated in the kernel ... requires so much more people at least to understand the kernel better.
Linus could still have his elite darwinist selected core hacker group, but for the whole linux eco system it is very bad that the barriers for the kernel are so arbitarily high.
Isn’t that a result of the intentional non-guarantee on not maintaining abi compatibility? Otherwise you end up with microsoft’s much more difficult to fix situation (maintain abi, difficult to update apis in general, but vendors can trivially produce them without a concern).
Im not sure how microservice architecture or something would sidestep the issue; its more of a political question of who's in charge of maintaining the drivers
And even more stupid to set the bar then artificial high, when you don't have enough elitist, ideological pure enough hackers to do that.
It’s a pretty clearcut tradeoff, and they apparently have managed the manpower for it thus far... I don’t know why you need “elitist, ideological hackers” to support this strategy
I don't do kernel development, most of my stuff is distributed systems / networking / devops type stuff. In this realm I feel like debuggers are almost an antipattern because:
1. Distributed systems with a lot of nodes means you might have to attach debuggers to everything, and that's just too hard to control.
2. Anything with networking, like web services, distributed systems, protocols, etc. will likely time-out the connections while you step through line by line. Once your TCP socket gets closed on you from below or the other side timing out waiting for a response, then you have to set up your debugging environment all over again.
3. Most of the real hard bugs I have to deal with usually involve timing or race conditions. This means that I might not be able to reproduce them in a debugger, and sometimes even the act of attaching a debugger makes the timing different enough to not repro the bug.
4. In production of these systems, most times you can't just hook up a debugger and basically shut down the system while you try to diagnose a bug. I collect what info I can from the logs and move on. In this situation, you need great logs. Great/useful logs are built over time, by adding log statements to the right places. So for every bug I diagnose, I keep the logging statements in I used for debugging (although I usually don't leave them emitting messages, because I have log levels).
Logs give a timeline of change.
I had this idea of console.snap but never got around to it. The idea is that it would not only capture the stacktrace, but a shallow copy of scopes at every function. You can query the snap to see how a variable changed, or find what set of variables caused something else to change later on.
I feel like this is the holy grail of debugging. Smarter logs that you can reason with and do time travel analysis.
I never got around to doing it, but now working at an analytics company, I see it being very valuable as it saves so much guess work.
Debuggers actually are often really good for that sort of thing. If you can break the code when the race happens and start looking at the stack and execution trace finding what happened is usually trivial. The big deal is by understanding very clearly what happened you know your fix actually fixes the problem.
Of course understanding the root cause is the key to fixing any issue.
What's interesting to me is that a man could reach the age of 31 and still be talking like this. It's teenage language.
At the same time he's also so clearly in command of his domain.
I think expertise and power in one domain can really stunt your development in others.
It's machismo. We don't hear very much of this any more from adult professionals in Western desktop jobs, but it's definitely out there. Even in the tweets of certain high controversy members of government.
On the contrary, I find that a lack of debuggers makes me lazy when performing this step: I’ll go “yeah, these lines of code remove the item from the list”, but then having no way to check that I get burned later when it turns out it actually did something subtly different than what I expected. I think he is right, though, that debuggers are not always the right tool for the job; occasionally they cannot produce relevant output (there might be too much too look at, of the part you are looking at is difficult to reproduce), so having these “big picture” skills are quite useful.
Every kernel problem I have ever faced down was an intermittent problem involving some sort of race condition. A debugger doesn't help you there because:
1. Timings might be changed so you can't (possibly) reproduce the bug in the debugger, or
2. The race condition involves some rare events that might turn up every week on a server but that could take years to find in the debugger so you can't (practically) reproduce the bug.
In applications work (with say Python or Java), however, I am strongly against debug
printf's and any temporary changes to the source code which are motivated by the needs of debugging.
Typically you wind up with the code (somewhere) in a state having some temporary debug-related changes and an in-progress fix. It might be checked in and pushed because
that is how you get it up on the test server.
These temporary changes have a way of becoming permanent, thus somebody tells your company that there is something at the bottom of the home page that shouldn't be there and you realize that it had been there for two years without anyone noticing.
Yes you can push back against that with code reviews and process, but wouldn't you rather use that spray can of management on real problems instead of avoidable problems?
Thus the debugger is great if it means you can eliminate the use of temporary debug code changes (e.g. not the in-progress-fix)
I also like developing unit tests in the debugger in Java and C#. You edit this, you edit that, you can look at data structures at any point in time... It's a lot like using the REPL in Python.
Where I have seen debuggers being useful at every point in the stack from kernel to compiler to loader and beyond is with ports. If all you are doing is running some existing well understood code on different hardware then being able to poke at the internals when things go wrong can often help find the cause quickly.
Debuggers are nice for big apps that you don't truly grasp the scope of, I know debuggers have saved my bacon several times when dealing with quirks/bugs of UI frameworks. Just being able to inspect variables and check the call stack is a great way to find out why/when shit hit the fan.
Hard to take this post seriously because he offers no clue as to how he actually finds the bug (being "careful" is ludicrously vague). That would be an interesting read.
If you're getting unexpected output without syntax errors then it's due to something along the way being set to the wrong value. Print statements are very helpful for uncovering that.
As soon as you see where things are going wrong, you know exactly what needs to be changed to fix it. With a bunch of print statements you can see the state of the system in multiple spots at once.
Plus with print statements you have the option of keeping them around all the time but tucked behind a DEBUG log level. In development, you can often take the guess work out of things if you're always in a position to see the state of your app at various steps. It can speed up development by preventing bugs / unexpected output before it happens.
With a debugger you can see the state of the system in multiple the spots at once, except that you don't need to guess in advance which spots you will care about, and you don't need to recompile / re-run every time you guess wrong
Yes, I used to do that all the time. It was excruciating, only I didn't realize it.
Then I started actually learning how to use a debugger. Efficiency in uncovering bugs went through the roof.
Plus, debugging is way less useful for web development where you have a bunch of different levels of technology at play. Also, most editors can't even be configured to work nicely with Docker.
And it's important to believe that bug has really happened. To never even think that the error is impossible.
He also hints about guessing the cause through the higher level instead of going from the lower level with the debugger.
It's always the two polarities: understanding the program vs focusing on tiny details of the bigger picture. For me, the debugger is THE tool for understanding the bigger picture!
Example of a concrete problem that I solved with a debugger today: I was populating a list in the UI, yet the elements were constantly disappearing. Why were they disappearing?
It's a JavaFX where the UI elements are bound to observable lists.. So I put a change listener on the list and a breakpoint inside it. The breakpoint had been hit a couple of times and one of these times I saw a piece of my code in a remote location  that was clearing the list! Root cause found, problem solved.
EDIT: could I have logged the stack trace rather than using a breakpoint? Yes, but the stack trace printout is far less actionable (harder to pinpoint something interesting, can't inspect variables, etc.) than an interactive stack trace in the IDE while the program is suspended.
 Why remote location? The UI is the frontend that communicates with a backend. The UI is rather strongly separated from the backend bridge; the bridge also runs in separate threads. So the bridge takes care of the back-end, receives messages and interprets them and updates the elements (data model) observable from the UI.
So my impression is that people who talk down debuggers simply haven't learned how to use them effectively.
Making good GUI is hard, and goes against terminal-centric Linux culture. Debugger needs IDE integration, and integration again goes against the culture, see "doing one job well" mantra.
When I program for Linux, I don't like debuggers either. Using gdb sometimes but it's not too useful.
When I program for Windows, visual studio debugger is awesome.
Are people - especially young and impressionable developers - able to separate the two sides of the coin, or does it serve to plant a seed of normalisation for this kind of communication?
As a young coder, I remember reading these linus oldtime posts as an exhilarating breath of fresh air. Everyone around me was all-in about "object orientation" and similar bullshit. When I say everyone I mean it: there was not a single person that suggested even a hint of hypothetical disagreement on software engineering mantra. Thus, it was refreshing to read these posts and realize that there existed, after all, another side of the coin.
We should not be afraid that young and impressionable people see "all sides of the coin". We should be afraid that they only get to see one side.
Somehow, the Go language is doing it, eschewing traditional OOP in favor of a get shit done fast and a little dirty approach. It's not even because of the charm of an individual either.
Then again, before Go there was JS as the big opponent to rigid OOP. I still don't understand why they felt like they had to add half-baked classes / OOP to it.
Keeping data and functions together can be nice. In some rare cases inheritance is almost OK. But still don't understand how it all went nuts and they started writing, say, parsers in that style...
But it's always nice to hear that there are other people having similar opinions. Especially if it's people like Linus.
Bit of a blanket statement. I enjoy it much more than the little Node I investigated for work.
The lack of functional programming though, yeah. I feel like it might be due to GC optimisations as well, however. From what I understand, the reason the GC is so fast is because such a style of programming is heavily discouraged.
I'm another one that prefers putting a bullet on my foot than putting it on my head.
That one I'll never forgive :-)
I'd say we should. For me his apologizing was more cringe worthy than anything in these emails.
Basically caving in to the American (and some Europeans by mimicry) culture of and "shame! shame! shame!"-style demand for public apologies.
We need more people standing up to what they see as BS in tech. And if the use strong language doing it, that's in a great tradition too.
>Are people - especially young and impressionable developers - able to separate the two sides of the coin, or does it serve to plant a seed of normalisation for this kind of communication?
Yeah, as if we suffer from too much candor and strong words in today's tech landscape.
It's more colorful. It drives the point harder.
It's also more healthy and the evolutionary smart thing to do:
From social change (e.g. "look at these backwards bigots"), to running a tight crew ("wtf Kowalski, you stupid! You nearly got us all killed"), deriding others can just as well have positive effects.
Then there's the fact that some things naturally and inevitable deserve derision.
Not all ideas are derision proof.
Let's have Theranos or Fyre festival or non-vaccination as an example that we all agree on.
A healthy society should include a healthy dose of derision for derision worthy ideas.
(And if you think my argument above is stupid and deserves derision, then you've just made my point!)
>Of course, I'd also suggest that whoever was the genius who thought it
>was a good idea to read things ONE FCKING BYTE AT A TIME with system
>calls for each byte should be retroactively aborted. Who the fck does
>idiotic things like that? How did they noty die as babies, considering
>that they were likely too stupid to find a tit to suck on?
Surely there are better ways to express a point without going at great length attempting to offend someone else. If you believe this sort of abusive attitude is acceptable, then why shouldn't people provide feedback with their fists as well? It can carry a similar degree of persuasion.
What if the purpose is to discourage the people making those mistakes from contributing to the project?
And let us also gently place everyone's thumbs into their mouths while they are asleep because it feels cute and cuddly.
I would not accept shit from people just because they have some authority in some place.
I just read it and it seems pretty tame to me. It's direct but if people get offended by this these days, by gods, we have a problem. Sometimes growing a thicker skin is the answer, one can't put the burden all on the messenger.
I mean, there's some Linus e-mails (and also Erik Naggum posts on Usenet) that I thought were awesome when I was young and impressionable but that I now think are over the line. But this specific example, if that's not possible... phew.)
(I also get the impression some commenters here do not realize what "fsck" refers to.)
Some people are offended by this other side of his style, yes. TBH, I don't understand why was it such a big deal. He never spoke about non-technical things in that way. Meanwhile, his technical opinions always were clear, sound and to the point.
> This is a much better email. It has 43% as many words, but loses none of the meaning. It's still forceful and unambiguous. With fewer words, it's easier for someone to absorb the core message about unthinking deference to standards.
> It also doesn't berate anyone, building a needlessly antagonistic culture around the project. Writing this email instead of the original email doesn't require any extra work, and will save mileage on Linus' (or your) fingers besides.
> If you read apologists commenting about this email, they often completely miss Linus' actual point. He's fine with the change; he doesn't like the justification, which uses an appeal to authority. The very people who apologize for the useless bloat miss this because of the bloat!
> These people missed the signal because it was buried in noise. But most of them go on to argue that the noise is boosting the signal; that ranting is good because it drives the point home. But they didn't even detect the signal because they were so focused on defending the noise!
Well, they know there are hundreds of millions of others who use their work, thousands who worked with them, and a trillion dollar industry built on their labor and effort, and that they have spoken their mind, with no BS, and no fake niceties like it's the norm.
I guess they can always take comfort in that.
Others don't want certain kinds of people to "want to talk to them" in the first place, so they are non-polite towards their kind on purpose.
And a final similar case is wanting people to talk to you only if they can stand that level of discussion/no BS talk/bluntness/sense of humor/profanity/etc. So that you can be yourself among them, and they can be themselves with you, with no facades.
I, for one, pretty much prefer being spoken to that way, instead of a sanitized corporate newspeak that means exactly the same thing but uses "professional" words.
I think people mean well when they talk about empathy and kindness, but those concepts assume that the only reality is how we feel and speak about things, when generally it's the underlying reality that is painful or harsh. And when you have to deliver a product, you can't avoid that reality.
I try to address it by frequently talking about my own fuckups and thus making it easier for junior devs to understand that failing is something we all go through.
But nothing can change the fact that it hurts to invest deeply in something and see it fail. We can't make pain go away, so it's better to just get it over with and learn to adapt to it and be resilient.
As a younger developer - what matters to me most is the constructive criticism. I don't want to have it phrased nicer(nor harsher) than needed - i just want it to be down to earth and constructive.
Personally i think when projects grow, there is a point after which you cant really reason that much about what is going on, just have to debug it.
The quality and integration of the debugger alone made Visual Studio head and shoulders above most development tools for other environments.
You need to know the person. In my experience most of them are very nice guys.
It's just that when they think something is not that great they will just say "it's shit".
I was told many times my code was shit by these guys at the time and I didn't take it as a personal insult but as constructive feedback. They liked that I took it that way and we ended up good friends.
People should see through the words before being offended.
I once worked in a place where a new trainee engineer was struggling with a piece of work. There were a few reasons for this that weren't their fault (not a lot of explanation of the requirements). Nonetheless, the work wasn't getting done as fast as the PM wanted.
Now, the PM in this case had moved across to software development from a post-graduate background in teaching, which I only point out to explain that this particular PM considered themselves something of a big deal.
(Having moved around as an independent contractor I'm fairly used to seeing small fish in smaller ponds that think they're the office genius, but this was next-level conceit.)
Anyway, they decide to let this engineer go one morning. Their prerogative I guess, but the PM shares with everyone later on that same day that they were able to steer this person in the right direction on the way out the door by kindly explaining to them that it hadn't worked out because "they didn't have the natural aptitude for engineering and would be better served pursuing a different career path".
Most of my gigs are in a fairly small city with a smaller community of engineers, so I'm used to encountering people a couple of years down the road. Sure enough, I never saw this person around again in any of the meetups or in any other gig, so I assume they left the industry based on this manager's feedback.
So yeah, bastards abound.
Nonetheless, that would require this person to be an arbiter of the software industry.
Given that you're on HN, and therefore very likely in the industry yourself, does the idea of someone you've never heard of deciding who does and doesn't belong doing this work sit well with you?
If so, yikes.
I wouldn't trust Linus Torvalds, Bill Gates, Dennis Ritchie (may he rest in peace), or whoever else to assess whether or not someone has a place in the software industry - so the idea of a mid-level manager in a nameless tech department doing so is utterly ridiculous to me.
I'm not offended when developers say that, I just observe that they are unable to build and express a reasoning, and that worries me about their abilities as developers.
It means the contribution isn't up to the standards and therefore can't be integrated, and thus informing the contributor that he needs to improve his work in specific aspects in order to avoid similar problems.
> What is the constructive part ?
You're focusing too much on irrelevant aspects of the communication (i.e., which and how a word was used) and in the process ignoring (intentionally or not) what was actually said. Typically linus includes specific comments on the problems present or caused by a contribution, whether in code or the development process. In the very least it's very easy to understand that this sort of process works through negative feedback.
Linus doesn’t just say “it’s shit” though - he says “you’re shit”. That’s the problem.
But let's face it, too many people in this industry think that they are geniuses and take it very badly when they are brought back down to Earth.
The guy has to sort so many contributions that at some point it's not crazy to shut some people out. The effect of Sturgeon's Law on someone a but rough like Linus...
I don't know if he did any development other than committing patches/merging-rebasing branches after the collaboration started, but I'm guessing it was rare.
But for actual debugging, especially for code you're not familiar with, they can save huge amounts of time.
Some more notes on this at:
But the debugger is nice for a certain set of problems. If you get a (java) lambda statement that is going fubar, it's nice to attach a debugger and just analyze the lambda flow to see how each statement effects the data.
One of the projects that I worked on, we'd often repeat that we had to look for the _root_ causes of mistakes. Which is done without a debugger, it might just be a conceptual error. e.g: if we had a nullpointer, we could just add the simple null-check. But it'd be more interesting to reason about if it should even be possible for that variable to be _null_ anywhere in the code. So we could rewrite it in a way that it can't ever be null and avoid the check.
Anyway, you probably should use a debugger , but it's just one tool of many.
A lot of the code can run in the build environment, supported by dummy components, without requiring to boot a VM. A lot would need that VM, and the VM would need to be able to emulate a lot of hardware for the tests to be comprehensive. OTOH, this emulation code (or the config files that would set it up) would serve as always verifiable definitions of how the hardware is supposed to work. And dummies and emulation could be automatically validated against actual live hardware when it's available.
The debugger generally sucks because by the time you have to use it the whole system isn't functioning properly.
If you can just find out WHICH component broke you can rip out that component and build a test for it or expand upon an existing test.
It usually turns out some input for a function broke the range and domain of the function and just adding another test for it and then fixing the tests fixes the problem.
The additional benefit here is that not only do you not need to use the debugger but since its now a unit test and in your code you can't ever break that way again.
A lot of people joke about how they don't write unit tests because they're trying to save time.
In my experience, in any significantly complicated code base something massive will break and knowing exactly WHEN it broke and in what commit can dramatically lower the time needed to fix the problem.
I'm a bastard. I have absolutely no clue why people
can ever think otherwise. Yet they do. People think I'm
a nice guy, and the fact is that I'm a scheming, conniving
bastard who doesn't care for ...
These threads are always a bit disconcerting as people get so sensitive and personally offended by them. And invariably despite the astonishing success of Linus' leadership, we'll just throw that all aside and posit that if it were a committee of equals who treated each other with love and admiration it would be as successful.
Or maybe it would be an abject failure, which odds tell us it would have been?
I mean compare Linux to the Java ecosystem, which is similar in scale (billions of installs / dependencies) but run by commission. How is that working out? Less hurt feelings I guess but it's not making any significant advances either and has been stagnant for years.
But does it work for the software he’s responsible for, or does the software he’s responsible for work in spite of his behavior?
How is that working out? Less hurt feelings I guess but it's not making any significant advances either and has been stagnant for years.
Commissions that don’t hurt people’s feelings make something stagnant? Could there be other forces at work to explain Java’s stagnation, other than a commission that doesn’t hurt people’s feelings enough?
Not a chance. All of Java's roadmap discussion revolves around dealing with legacy issues, and they even had an entire major release just trying to refactor the language into modules because the core was so bloated.
You can't blame lack of funding, or that it's a niche product or anything like that.
You can't blame lack of talent, because there were a ton of very smart people working on it.
You can't blame novel ideas, because very little in Java was unprecedented.
Java's legacy traces inexorably back to:
a. features that no one could say no to
b. badly implemented features
he's aware of it.
That said, a behavior can be both cringe-worthy, and also productive. I can think of certain comedians, talk-show hosts, etc, who manage both.
Do you think the statement is untrue?
What makes you believe that he tries to convey "how dominant he is" instead of just stating a fact?
Why would it be a problem that he has a personality disorder?
 I believe that Linus himself aknowledged that when the CoC was added to the kernel documentation.
why do you find it cringe worthy?
Why would it be a problem that he has
a personality disorder?
Ah yes, the tried-and-true arrogant neckbeard stance that so defined unix interfaces for decades: If it was hard to write, it should be hard to understand.
But we've already proven that good UX is a good thing. In fact, UX has been so front and center for so long on HN, I'm rather surprised that people still cling to the old ways.
We are not talking about some throwaway web app. We are talking about the Linux kernel. Something that has been used for decades and is currently used on millions of devices every day. From literally computers that power spacecraft down to mobile devices.
UX certainly has its place, however I think changes and additions to the kernel should be carefully considered. If excluding a debugger excludes some careless programmers then maybe that is a good thing.
Years ago at school my group wrote a ray tracer from scratch, after a few months we noticed that the vector class had an inverted scalar operation. The program worked, because we had inverted the entire vector space as we had just fixed the code until it worked.
After fixing the vector class most of the other formulas in the program started to look more reasonable :)
> And quite frankly, I don't care. I don't think kernel development should be "easy".
Previous discussion on HN from 2015:
I personally do not use debuggers, but I can see why they might be used, especially to step through someone else's convoluted code.
Such honesty. I think that jerks who scheme and hurt individuals for the common good are the best kinds of jerks and the world needs more of them.
Also, his points make perfect sense for Linux Kernel development. It shouldn't be easy to change.
Stability is by far the most important feature. Even excellent developers don't cut it for the Kernel; you need complete freaks of nature.
It's impossible to even wrap one's mind around the sheer number of systems that depend on the Linux kernel.
Kernel development cannot be slow enough. Most of the new code should be thrown away without batting an eyelash.
In fact, every line of code merged should be so good that it should deserve an international conference dedicated to it and the author should get a medal.
Heck, every line of code should have a religion built around it; complete with churches, priests, a pope, schools, etc...