
Learning from Your Bugs - henrik_w
https://henrikwarne.com/2016/04/28/learning-from-your-bugs/
======
hankerfaster
We do something similar. Any bug that is submitted by a customer (rather than
QA, or a trial team, or other developers) is analyzed. 50% of our bugs are due
to poor requirements. 25% are due to poor design. The other 25% are just
coding problems.

Most of the requirements problems derive from lack of understanding about the
tradeoffs in the current system or lack of understanding what the customer
actually wants.

Most of the design problems are a result of excessive complexity. People often
forget to simplify and refactor prior to adding a new feature. They end up
having to keep a larger system in their brain when designing than they
otherwise need to. Therefore they miss interactions that exist or forget to
handle them in the new design.

Most of the coding problems would have been caught had unit tests been clearer
and closer to 100% branch/code coverage. It is amazing how many bugs lurk in
the last 5% of testing.

It is actually quite fun to tell a product manager that they are the cause for
a bug :) Usually it is the developer that

~~~
chriswarbo
> Most of the coding problems would have been caught had unit tests been
> clearer and closer to 100% branch/code coverage. It is amazing how many bugs
> lurk in the last 5% of testing.

Presumably, coverage starts with the low-hanging fruit (simple, linear code;
heavily-exercised code like startup routines and main loops; etc.) and grows
to encompass ever more unweildy, smelly, edge-case parts of the codebase.

Hence, by increasing code coverage, we're boiling off the solvent of run-of-
the-mill code, such that what remains uncovered gets an ever stronger
concentration of hackery, unintelligibility and black magic. By 95% coverage,
the bits remaining are those parts people don't want to touch, so I'd expect
the concentration of bugs to be highest there.

:)

~~~
hankerfaster
precisely! As programmers that is what we care about.

But really, at that point the business often doesn't care. Not when a
requirement bug could cause months of lost engineer time.

Our experience is requirement bugs are much more costly than a missed unit
test resulting in a bug.

------
dfan
I do this with chess. I take positions from tournament games in which I made a
correctable mistake (generally not positions in which there was just a lot of
calculation to be done and I wasn't up to the task) and add them as flashcards
to my spaced repetition system.

It is true that software engineers often treat bugs as unfortunate but
unavoidable noise in the system; when one pops up, you fix it and move on. But
it's really worth thinking them as an inherent part of the process of software
development, one which you can improve and learn from, just like anything
else.

------
jondubois
I never bothered to write down details of bugs... I rarely learn a lesson from
any one specific bug - Lessons are learned by being exposed to a wide range of
different bugs over many years. Eventually it becomes gut instincts.

For example today I was debugging an issue with Kubernetes (it was a really
cryptic error about client-server not being able to negotiate a protocol or
similar). I had never used Kubernetes before and I couldn't find anything on
Stack Overflow.

The first thing I did was check the client and server versions (to see if they
matched) - They did so that wasn't the issue. So then I had a strong 'gut
feeling' that it was related to the client configs.

In this case, my second hypothesis turned out to be correct. 5 years ago I
would probably tested my way through 20 different hypotheses before finding
the problem.

To summarize my debugging strategy, it's simple; just pick all the low-hanging
fruits first. The hard part is knowing what a low-hanging fruit looks like.

~~~
MattGrommes
This is basically intuition, a base of experiences that leads to these "gut
instinct" decisions and feelings. Doing a journal like the author describes
could actually help with your intuition because you're adding more specific
information from each bug into your mental storage. Now you're getting a
cursory level of data from each individual bug which leads you to think each
one is less important. It's like skimming 3 books on a subject instead of
close reading 1.

I'm just thinking out loud because I do what you describe here but I think the
journaling idea could be valuable.

~~~
vitd
FWIW, journaling has been useful to me. I've run into bugs and thought, "I
know I hit something just like this before. Oh wait, let me look through my
debug journal and see if there's something in there!" And often there is. It
saves me having to do web searches or guess and use hunches. I can always fall
back on those things if it's something I haven't seen before or is manifesting
in a different way.

I'd go even further and say that I've often wasted time chasing down a hunch
that was completely wrong.

~~~
jondubois
This has happened to me several times as well (encountering a problem which I
felt like I had encountered before on a specific project).

It's a good memory exercise to try to remember when was the last time I hit
that bug. At the beginning, I wasn't very good (and I couldn't remember the
solution of the earlier problem - a journal would have been handy at that
point) but after doing this for a while, now I almost always remember the
solution to various regressions I encounter.

------
jesserayadkins2
I have a similar process myself. What I find to be helpful is to ask myself
what, in particular, went wrong. If I have a handful of cases where "some api
was used wrong", then I take that as a sign of a need for refactoring to make
the callee easier to use.

Another that I like to ask myself, is how I can use the compiler and its type
checking to my advantage. As in, how can I make it more difficult for myself
to make this bug again in the future?

But I agree with the author's point: Nearly every bug is an opportunity to
learn. You should not just fix bugs, but also to understand why they happened.

------
gitmagic
I'm kind of doing the same but instead I write my analysis and notes in the
pull request that fixes the bug. I think it's a great way for others to learn
too :)

------
another-dave
Sounds quite interesting to see how it will build up and to spot any patterns
emerging in the results.

Will give it a go over the next while. Cheers for sharing!

------
gumby
This is a clever and straightforward technique.

