

ASK HN: What are the most costly bugs you have fixed? - geekytenny

Coding is a lot of fun + debugging. So as encouragement to starters what are the most costly and maybe the toughest bugs you have had to fix?
======
btilly
The two most costly for me were both email bugs.

One defaulted people's email preferences to off when they signed up. The other
was a poorly chosen subject line that caused all the wrong people to read one
of the most important emails that company had.

I can't give out an exact dollar figure for either. But both were in the 7-8
digit range. And both took a surprisingly long time to track down because
everything _seemed_ fine. And so the problem sat out there. And even once we
knew that there was a problem, the problem was only discovered by accident.
(But having been through it, twice, I now know that if business just starts
drifting downwards, double check what is happening with email. Again.)

By contrast the obvious disasters that I have seen have cost much less.
Surprisingly so. The database crashed without a proper backup taken in the
last 2 months? OK, we're out of business for a week and a half while we replay
all of the logs, but then the email blast we sent saying we were back up
attracted more than enough business to make up for the time lost.

And the toughest bugs? Surprisingly cheap. It took literally months to track
down why we had to do a careful dance bringing up Apache or else nothing
worked. Yet despite months of grief the actual cost in lost business was a few
hours of downtime at off hours. I'd be surprised if we lost 5 figures on that.
And the memorable one where you couldn't fill in a particular form if your
screen was the wrong size? I'm pretty sure we lost $0 on that. (The bug, for
the curious, was an invisible div that was in front of the window. If the
window geometry placed it over the form, it became rather hard to fill out
said form.)

It is a hard lesson that what developers care about and what businesses care
about are only tangentially related.

~~~
geekytenny
Boy! what an experience you have had, thanks for sharing.

