
The Debugging Mindset - jodooshi
http://queue.acm.org/detail.cfm?id=3068754
======
nostrademons
The scientific-method-like "general approach to debugging" that the author
describes in the last third of the paper works, but it's pretty inefficient.
The number of possible hypotheses that may cause an observed bug is very
large; if you test each individually, you may have to go through many
permutations, and you're not systematically gathering data that may help you
narrow the search space.

Rather, I've found "divide and conquer" debugging to be much more effective.
You should know (based on your domain knowledge of the program) the rough
sequence of operations between the input and output of your code; if you
don't, try single-stepping through it or adding log statements. Pick a point
approximately halfway across that computation, and inspect (via debugger,
println, or unit tests) the state of the program at that point. Does it match
your mental model? If so, the bug is between there and the output; repeat the
process on the latter half of the code. If not, the bug is between the input
and your intermediate point; repeat the process on this subset. As an added
benefit, you can often find and fix other, unrelated bugs that may not be
symptomatic yet but are silently waiting to trigger strange results.

This process is O(log N) in the size of the code path, while the "formulate
hypothesis and test it" approach is potentially combinatoric if the code
interacts with itself.

~~~
DrJokepu
Also, I observed that as you get more and more experienced, intuition
("hunch") works surprisingly well. It's not always reliable and sometimes it's
just plain wrong though.

~~~
brlewis
From the article: "Intuition can be an effective strategy for debugging but
requires extensive experience and, when used as the only strategy, leaves the
programmer unprepared to handle new, unfamiliar bugs."

On the other hand, I don't see how you can keep experience and intuition from
being a factor in any instance of the scientific method.

~~~
philipov
It sounds like a moot point. It's not possible to acquire the extensive
experience necessary to have intuition without having another strategy in the
first place. Intuition is the long-term consequence of employing a strategy:
no longer having to step through the same sequence every time.

~~~
dhodell
Exactly this.

------
cubano
I have always used a step-debugger for debugging and personally find the
process somewhat thrilling, especially when debugging other peoples code and
learning all the tricks and techniques that they are using to get the jobs
done.

I don't know how much time and effort I have saved over the years using tricks
I gleaned stepping through code written by people much smarter than I.

There is simply nothing that comes close to learning and debugging code line-
by-line, examining variables and taking branches in real time. In fact, how
people can debug large existing codebases _without_ a debugger is beyond me.

Of course, nowadays in modern webdev it seems that most developers no longer
use step-debuggers and, in general, want to rewrite the app in whatever stack
will look best on their resume and/or sound impressive in the pub or gym
afterhours.

I long ago gave up poo-pooing the importance of social signalling within the
young-ish developer community, something that really did not exist when I was
20-something as programmers had few choices about the stack.

I really love the jist of the OP, and agree that the misapplication of the
programs mental model is the underlying cause of almost all bugs. This mental
model is what I'm always trying to load fully into my brain as, IMO, true
productivity occurs when I have it fully loaded and I am confident that I am
covering all the edge conditions.

 _Debugging must not be an afterthought in educating; industry must stop
insisting that bugs be interpreted as failures of individual programmers ..._

Funny...I've really never separated debugging from developing as they always
go hand in hand, and certainly have never thought that bugs signify "failure".
In fact, to me it represents progress...

~~~
coldtea
> _Of course, nowadays in modern webdev it seems that most developers no
> longer use step-debuggers_

You make it sound like step-debuggers were the traditional, established thing.

Whereas most programmers back in the day would get by with some carefully
placed print statements (based on specific intuition) instead of stepping
around to see what goes on.

"The most effective debugging tool is still careful thought, coupled with
judiciously placed print statements." (Brian Kernighan).

~~~
nostrademons
I've used both, and still do use both in concert with each other.

The debugger is a tool for gaining that intuition about how the program
operates, and carefully-placed log statements are a tool for cutting the
search space of where you need to place breakpoints and single-step.

~~~
arethuza
"carefully-placed log statements are a tool for cutting the search space!

Indeed, inevitably serious problems tend to be where there is no or poor
logging :-)

------
edejong
Of course, Dijkstra had his view on it. I try to adhere to his position and
prefer proof by type-checking over proof by single example.

"A common approach to get a program correct is called "debugging" and when the
most patent bugs have been found and removed one tries to raise the confidence
level further by subjecting the program to numerous test cases. From the
failures around us we can derive ample evidence that this approach is
inadequate. To remedy the situation it has been suggested that what we really
need are "automatic test case generators" by which the pieces of program to be
validated can be exercised still more extensively. But will this really help?
I don't think so." [1]

Linus (used) to have a similar position [2]

Debugging hides the underlying architectural problem: to either communicate or
abstract the algorithm in such a way that it can be understood by the
programmer. If something needs debugging, not the bug needs to be fixed, but
the architecture to understand the bug before debugging.

[1] Edsger Wybe Dijkstra - EWD303.
[https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...](https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD303.html)

[2] [http://lwn.net/2000/0914/a/lt-
debugger.php3](http://lwn.net/2000/0914/a/lt-debugger.php3)

~~~
dhodell
Thanks for this.

This seems to be basically what I'm saying through most of the article. Bugs
occur because of a gap in a mental model. To fix the gap, we need more
understanding. I'm not sure I would say that's necessarily a design or
architecture flaw in the software.

Unfortunately, proving things correct in most popular languages is a Very Hard
Problem. I do address that to some extent in the article, though. In
particular, the sorts of languages that end up being easy to prove correct
tend to be difficult for people to learn. Therefore, people are likely to
either learn something easier, or they are likely to write code that is
"correct" or "safe" but still has logic errors. I think that there are other
many valid reasons to write software, and not all of them require correctness
to the extent Dijkstra would like.

But I think that's a pipe dream anyway. For one, most software is not written
in formally verifiable languages, or formally verifiable dialects of popular
languages. Porting all software would be impossible. Second, who defines
"correct"? What about bug-compatible software, where the definition of correct
is imitating something else that is known to be incorrect? What about
ambiguity in specifications like JSON, or undefined behavior in C (some of
which can only be detected at runtime, because we don't actually have infinite
tape in our Turing machines)?

What happens when there's a bug in the runtime of your formally verifiable
language? What happens when there's a hardware bug? I'm not saying these are
necessarily common, but regardless of your software environment, you're
running on something else that can have the same bugs. People make mistakes,
even when proving things. Math is great, but people make mistakes, and our
tools are still bounded by the laws of physics.

No matter which way I look at this thing, I keep coming back to teaching and
understanding as the only generalizable and approachable ways to address the
problems bugs. (Whether that approach is taken by proving correctness of some
implementation, or post-hoc with testing and tools.)

~~~
edejong
Thanks for a great and insightful article and answer. Views like yours should
be shared by a wider audience, since it accentuates a section of software
engineering that is rarely discussed: understanding and adjusting the mental
model of programming (or should I say: systems design?)

Applying formal methods does not automatically entail that a _system_ is
proven to be correct, which would be an exaggeration. Proponents of formal
methods believe it helps to ascertain certain robustness and reliability
aspects of a system. For example, I could prove a Paxos implementation
correct, given the specification of a Paxos algorithm, but still use it
incorrectly in a larger system, or vice versa.

Correctness is a rather vague and abstract concept and quite distinct from
proof. Using a type system, we could prove that, given certain pre-conditions,
large state-spaces cannot be visited. Whether it will behave correctly within
the reduced state-space, the programmer can only hope. Even formal languages
such as Coq rely on correct execution of the assembler, CPU and system
environment.

I think what Dijkstra was aiming for was not a replacement, but more a
grounds-up reeducation of the field. And to a certain extent this is currently
happening. Functional languages and techniques are everywhere. I hear people
talk about monads, monoids and functors, others embrace reactive programming,
which is the coinductive branch of programming. Coq and Idris are finally
reaching a larger audience (but still small). However, the larger change is
slow and requires a generational paradigm shift.

It is exactly the thing you are addressing in your article: do the users of a
tool believe they are changeable and changed by the tool (Heidegger)? If they
do, they also understand the need for self-protection and focus.

Teaching and understanding are indeed the underlying pillars if we want to
bring software engineering to a higher level. I think the order in which
material is taught greatly influences the mental model of the student. Making
multiple iterations, one could start with concepts from category theory and
work down with for example Coq, Haskell, LLVM / Assembler, Machine
Language/system design, CPU Micro-ops to Digital Electronics. This would lead
to view the concept of an LLVM as an implementation artefact of Haskell (which
in itself can be expanded to other languages). The mental map becomes
relatively machine and system independent. Unfortunately, most programmers
nowadays start with an easy language that gives instant reward. It is made
difficult for the aspiring programmer in this position to understand the
concessions made.

~~~
dhodell
Thanks for clarifying, I agree with all of this, and they're great points!

------
jgrahamc
“ _By June 1949, people had begun to realize that it was not so easy to get a
program right as had at one time appeared. It was on one of my journeys
between the EDSAC room and the punching equipment that the realization came
over me with full force that a good part of the remainder of my life was going
to be spent in finding errors in my own programs._ ”

[https://research.swtch.com/discover-
debug](https://research.swtch.com/discover-debug)

------
unexistance
"Everyone knows that debugging is twice as hard as writing a program in the
first place. So if you're as clever as you can be when you write it, how will
you ever debug it?"

Timeless

If you have to use 'clever' hacks, the least you can do is justify it in the
comment

~~~
blauditore
And verify it with good tests. It sometimes happens that a certain piece of
code is inherently complex, due to a complex underlying algorithm. But often,
test cases are still easy to define, so it makes sense to heavily make use of
them.

For example, sorting algorithms may be somewhat complicated in implementation,
but it's trivial to come up with some sample before/after lists.

------
theaeolist
Debugging an API, which may have obscure corner cases or usability quirks
seems appropriate to me. "Debugging" an algorithm which has inherent logical
flaws by attempting to patch it up with case-base reasoning and exceptions
(logical, not runtime) is a road to disasters. I have seen too many devs
trying to hill-climb their way out of a logical hole and the result is
invariably a mess of a program which is still wrong.

Also, in case of concurrent and parallel code debugging is basically useless
since realistic timing conditions are not reproduced.

I personally prefer logging, which gives a faster and more targeted trace of
program behaviour than step debugging.

------
_nalply
My secret sauce to debugging is formulating hypotheses about what went wrong
then instrument the software to validate the hypotheses. By instrumenting I am
talking about manipulating running software to give me information.

Sometimes the manipulating step is made unnecessarily hard, and the trick is
to force the hand. I see myself an inquisitor of software and use programmatic
torture to get what I want to know. That's the central attitude to debugging.

The devil is in the details, there are many possibilities to get information:
using a runtime debugger, insert output statements, read the logs, use
LD_PRELOAD, use strace, and so on.

------
sleepychu
My favorite debugging trick (when faced with an 'impossible' bug) is to state
my assumptions, it's usually clear when they're all laid out which ones are
conflicting.

~~~
mrSugar
Sounds like a variation on Ayn Rand's "contradictions do not exist, if you
think you see one, check your premises, some of them will be wrong".

~~~
eriknstr
Is that quote intentionally saying that you have contradicting premises, and
by extension is the statement in that sentence that contradictions do not
exist contradicted by the later part of the sentence, or is there something
about the sentence that I'm not understanding?

~~~
mysterydip
It reads like that at first. What it's saying is for example: Assume x is 5.
Assume function TimesFive multiplies things by 5.

When we do TimesFive(x) and my answer is 50 instead of the 25. I check the
TimesFive function and it's just "print x*5". But how is that possible? 5x5 is
25!

The problem in this case is my assumption that x would be 5 was wrong.

I've had this issue before in my code (though not as trivial an example of
course), but it's always a good reminder of something to check when
troublshooting.

------
sharpercoder
>Are the arguments to memcpy in the order of source, destination, length; or
are they destination, source, length? Is strstr haystack, needle; or is it
needle, haystack? Many individuals develop a habit of consulting references
(such as system manuals) as soon as the question comes up.

I've always found this particular problem solveable by allowing programming
languages having syntax in the form of

    
    
        public void CopyFrom(int address)To(int address)OfSize(long size) { // impl }
    

instead of

    
    
        Copy(int fromAddress, int toAddress, long size)
    

Are there good reasons why languages haven't adopted this syntactic nicety?

~~~
xgbi
Obj-C uses this. C, on the other end, allows prototypes to be defined as:

    
    
        void* memcpy(void*, void* size_t) // no argument name
    
    

Which is the worse IMHO. The autocomplete cannot event tell you which is
which..

~~~
nostrademons
And Swift, and Smalltalk. 3 generations of that one family.

It leads to very verbose code, which is perhaps why it hasn't caught on in
other languages. It also can be surprisingly clumsy when a method is called
from multiple call sites (which may apply it to different parameters in subtly
different contexts, which can make the original phrasing awkward), or when a
method is passed around as a first-class value.

------
sn9
Udacity actually has a course on systematic debugging taught by the creator of
DDD.

The course covers the importance of the scientific method and hypothesis
testing, divide and conquer approaches, etc.

------
noyeda
Nice article. Thought I'd note something:

> Software developers spend 35-50 percent of their time validating and
> debugging software.1

This is a strong claim, so I looked at the cited source, given the unfortunate
experience of having seen many strong claims backed by weak underlying data.

> The research phase will predominantly comprise of short interviews with
> approximately 10 to 12 organisations that compile code on the LINUX
> operating environment. These organisations will also fill in the Cambridge
> Venture Project (CVP) survey to help quantify how much time they spend
> debugging with and without RDBs. Broader research in the form of the CVP
> survey with potential users of reversible debuggers will be conducted to
> gain insights on how much time is currently spent debugging.

Further down:

>49.9% Programming time spent debugging __

> __According to 54 questionnaire responses to the CVP survey and 11
> interviews, based on the question ‘Of the time spent programming, what
> percentage of your time is spent on the following: (1)fixing bugs (2)making
> code work (3)designing code (4) writing code.’ (1) and (2) then were grouped
> as debugging.

While this might be indicative of a widespread phenomena, it doesn't seem to
be enough evidence to support the factual and strong claim of the first
sentence in the ACM article. There certainly isn't enough information in the
cited source alone to decide (and given that, is this really sufficient as a
source?).

Perhaps I missed further evidence?

That being said, I won't throw the baby out with the bathwater. Debugging is
clearly a time sink in industry, regardless of what the actual percentage is.

I enjoyed this piece overall. I was previously unfamiliar with some of these
terms -- specifically incremental vs. entity mindsets -- and it shed some
light on work issues I've seen (coincidentally in the debugging technology
space). Being someone with an incremental mindset and having worked for
someone with an unjustified entity mindset, I found the environment suffered
from almost all the problems (and more) you described associated with the
latter mindset. Granted, this is anecdotal.

I quite strongly agree that improving the process and mindset behind problem-
solving results in a significant improvement to specific skills like debugging
software. It's refreshing to see an article like this instead of the many
others hyperfocused on specific, less-abstract techniques (and widely-
applicable abstract techniques/processes are the most effective use of one's
learning time).

~~~
dhodell
Thank you for the feedback. I chose (for better or worse) to cite the most
recent article as opposed to maybe the least controversial one. Other papers
I've read have suggested similar ranges; nobody presents a stddev so it's hard
to compare. And how you classify "debugging-related tasks" is kind of
subjective, as you correctly point out. So linking other papers might or might
not be useful anyway.

In "A framework and methodology for studying the causes of software errors in
programming systems" (Ko & Meyers 2005), they cite a NIST publication as
saying 70-80% of time is spent debugging. I parroted this off in a talk once,
and then I looked at the same publication ("The Economic Impact of Inadequate
Infrastructure for Software Testing" RTI 2002) and the only 80% I found was
time spent in testing software in the early days of software engineering. In
the 1990s, it finds coding and unit testing to comprise about 30% of the time.
This is as far as I can tell, at least similar to the (1) and (2) in the CVP
survey. But again, indirect comparisons, who knows?

In tables 6-4 and 7-6, the RTI study finds different numbers for time spent on
bugs, and different frequencies of where bugs end up being discovered.
Whatever the rate, it is clear that post-production discovery bugs are the
hardest to fix, and I think this is because you only ship code when you have
high confidence in its correctness.

I think this is a hard measurement to make. It's hard to find participants,
hard to guarantee they have the same idea of what "debugging means", hard to
quantify. So thank you for calling me out on this.

But I think other studies have evidence that this isn't too far off from a
correct measurement. For example, Gugerty and Olson in 1986 did some novice vs
expert study at debugging LOGO programs. (I don't like the Dreyfus model for
classifying debugging, but whatever.) They found that novices took about 18
minutes to fix bugs, testing an average of 3.6 hypotheses per program, with
the first hypothesis being correct only 21% of the time, and re-executing code
every ~3 minutes. Experts took 7 minutes, testing an average of 1.6 hypotheses
per program, with the first hypothesis being correct 56% of the time, re-
executing code every 2 minutes. So guesses are only right maybe half the time
if you're really good.

Basically any novice vs expert study compares how much time people spend on
the problem (they're trying to reduce debugging time), but it's hard to
extrapolate this into percentage of development time. For example, some people
spend 100% of their time debugging because they're on QA or sustaining
engineering teams.

To be quite honest, there _fundamentally is not_ enough research into the
practice and pedagogy of debugging. I have about 60 papers in my "Debugging
Papers" folder that I've read over the past year, and basically everything up
until the Dweck-inspired research in 2008 isn't great. There are only 9 papers
in all of computer science that cite Dweck that I've found as of maybe 6
months ago (I'm not a researcher and I don't look that often). From memory,
two of them were recommending further research based on her work, maybe five
of them were from people in CS departments helping Dweck create games and
programs to perform her research, and then only two were actually applying her
work to debugging.

I would agree with you if you said this needs more research, and that the
claim isn't strongly supported by the provided evidence.

I greatly appreciate your feedback.

------
filleokus
I think it's fascinating to see how my friends at university (who previously
had no programming experience) have started to "get" debugging and the
benefits of using something like a step-debugger. It took several programming
courses before they started to actually debug the code instead of just reading
it and trying to figure out what happens. I mean, reading the code is also an
integral part of debugging, but often times they said things like "fooBar
_should_ be true, so we _should_ enter this conditional". Then it of course
turned out that fooBar was in fact not true because of some logic bug or
something they didn't even consider.

~~~
hamstercat
I had the same experience at University. We didn't learn how to use a debugger
and rather relied on logging data to see the flow of the program. That, plus
the fact that our code was a big mess since we were just starting didn't make
it easy to find the more obscure bugs. I was really impressed the first time
my friend fired up Netbeans and showed me that how he could find the value of
the variables at runtime, at any point in time of the execution.

Today I find it unacceptable when I'm in a situation where I can't debug a
program I'm working on, it just seems like a waste of time.

------
mattmanser
I personally found this to be a poor article on the subject. There's very
little actual content about debugging, he spent more time talking about
theories of intelligence than debugging.

I've recently been thinking about this, thinking about making a course. It's
my personal experience that his approach is flawed. Fundamentally, step one is
not develop a theory. That is the worst thing you can do, it can waste a lot
of time. I still sometimes guess what the problem is, but all I really use
that for today is where to _look_ and that's sometimes a waste of my time as I
skipped step 1.

Step 1 is recreate it. No theorizing, no thinking, no faffing around. Can you
recreate the bug exactly? If you can't, find out why. Talk to the bug
reporter, find out what they were doing, this is a people skill you need to
learn as a programmer. Maybe you need access to the live data to see the state
an object is in. There's only a tiny percentage of bugs (usually race
conditions) that can't easily be recreated. If you can't recreate it, that's
when you turn to logging to try and catch the conditions to recreate it, but
logging is the last thing you should try and I rarely ever have to add it
(we're talking 1% of bugs). I often see junior programmers littering code with
log statements when trying to debug, I feel this is a bad method, but can
understand the temptation, especially when they don't really understand what
the code is doing (an understandable position I was in in my earlier days).

Step 2 is isolate it. If the code is simple and it's obviously one method, no
further work needed. If the code is complex, you need to really isolate the
exact problem. Say you've got the wrong value displaying for a basket total,
where is that coming from? Is it not updating, or is the total function the
problem? Isolate the exact step in the process that's going wrong. Sometimes
it's hard to isolate and logging is needed again, but same advice as before,
logging is a rare necessity.

Step 3 is make it easy to run a recreation. If you can run it in 10 secs or
less, that's great. If it's one method of a complex web call, make a test
endpoint that only calls that method that's failing and exactly recreates the
conditions. Then you can run it quickly without all the cruft. In an extreme
case, this can be taking the code completely out of the context and putting it
in a tiny standalone app.

Step 4 is read + understand the code. Is there an obvious logic bug? Don't
jump to conclusions. Still don't get what's going on? Step through the code,
breakpoints. Understand what's going wrong. Breakpoints and stepping is fine,
inspecting variables, etc.

Step 5 is fix it. This should be fairly simple now. This is also the time to
think about the future. Is this part of the code a constant headache? Is it
buggy because it's over-complicated? It might need a bit of a refactor to make
sure this doesn't happen again. This is a judgement call, unnecessary
refactors are expensive, might even introduce new bugs, but over-complicated
code will lead to more bugs. It not being written in your favourite style is
generally not a legitimate reason to refactor.

Step 6 is testing the fix, recreating the conditions from step 1. Don't rely
on QA if you have them. Ensure you've done your job.

(Step 7 get rid of your logging statements + test it again! You just changed
the code)

Now you can skip steps, be far less formal about it. Sometimes it's obvious.
Sometimes it's hard and you may need to iterate over 4/5/6 a few times. But
never, ever skip steps 1 + 6. Recreate it. Test your fix. Because if you skip
those steps, and you "fix" it, you don't actually know it's fixed.

~~~
Sean1708
I 99% agree with you, I would 100% agree with you if you put step 6 between
steps 3 and 4. In my opinion you should always (where possible) be able to
deterministically and _automatically_ recreate a bug before you even think
about fixing it, the automatic part being your test. I say this because human
nature being what it is you will forget how you recreated the bug, or your
recreation will have a bunch of extraneous stuff in that really isn't needed.

~~~
AstralStorm
Unless you can prove the kind of nondeterminism which triggers the bug.
Sometimes it is much harder to write a testcase to reproduce a
nondeterministic bug than proving formally that it isn't possible after the
fix.

------
rdtsc
To make debugging easier you have to pick tools (languages, IDEs,
methodologies) that make it easier. In other words you optimize for debugging.

For example, I find Erlang code very easy to debug. There are few reasons for
that:

* Built-in runtime tracing is easy. Can remsh into a VM node and trace any function at runtime.

* Functional style with immutable data and variables make it easy to figure what is changing in the code. Because of immutability the state is always updated explicitly. That makes it much easier to understand what is happening for me that figuring it out than having a class inheritance hierarchy with methods overrides and such in an OO language

* Code hot-patching makes it easier to test and iterate to zoom on in a bug. With care it can be done in production. You see strange failure you never saw before and which you had an extra log statement in there? That's easily done.

* Language is simple and self-consistent. That means there is less chance someone used some obscure new feature I haven't heard about. This is a bit like C vs C++. C is very simple at the language level. You can still have hairy code with triple pointers, funky dispatch tables with Duff's Device thrown in there but at least you can read through it. Looking at C++ code I sometimes couldn't even parse what was happening because of a language feature (so they overrode the operator some place, and using a templates, ...). Or someone created a macro and then everything is more interesting all of a sudden.

* Process heap isolation. When something happens with one process, I know for a fact that it hasn't scribbled on memory of other processes. I can restart that process if needed and test my hypothesis of what I think is wrong, often without affecting the rest of the system (even while it is running in production).

* Sane concurrency primitives: Having the ability to create hundreds of thousands of processes with isolated memory often mean a smaller impedance mismatch with your business code so you have less total code to write. The less code your write, the less code you have to debug.

Anyway the point being, debugging should be a trade-off in picking your tools.
People don't think about it much and focus on other things first. But it will
be something that over time will make a tremendous difference.

> Virtual machine-based languages, interpreted languages, and languages with
> runtime environments encourage users to view the execution environment as a
> black box. The goal here is to make programming easier by reducing the scope
> of the mental model the programmer must maintain. When bugs occur in these
> execution environments, you're left with a complete gap in understanding.

It can be the opposite as well. Having a VM with good tracing and debugging
facilities is very useful. Often you can do things wouldn't be able to with
just plain compiled code.

------
jerianasmith
Investigating is essentially a space particular term for critical thinking.
Bugs are depicted as word problems.Since mental models are approximations,
they are in some cases inaccurate, prompting unusual conduct in programming
when it is created on top of flawed suspicions.

