
What does $1265 of bugs look like? - cperciva
http://www.daemonology.net/blog/2011-08-26-1265-dollars-of-tarsnap-bugs.html
======
ralph
I've spent a while looking for tarsnap bugs; to date I count 86 email threads
with Colin to report my findings. There are a couple of things I think are
pretty good about the scheme Colin has devised.

Unlike some other bounty schemes, there are small bounties available. One can
look for ages without finding a $50 bounty, so long that boredom would
probably set in and the hunt would be abandoned. But coming across the odd $1
or $5 bounty keeps interest going at small cost to Colin and still improves
the overall source quality. It's almost like a bit of a game, a treasure hunt,
where small pickings on route keep one's hopes up of rich pickings down the
road. A game that can be set down and picked up later as time allows.

Secondly, Colin was very fair in judging the bounty to be awarded. It soon
became apparent that I could trust his judgement; whenever I thought a bug was
undervalued another would come along that might be overvalued. On mentioning
this to him, it turns out it was sometimes deliberate; he'd be wavering and
thought it fair to even out which side of the boundary the bug fell.

As for the OCD, I boggle a bit at some of those style bugs and don't think
they were all me! :-) I do recall sending in some trivial thing suggesting it
wasn't bountiful but perhaps should be fixed anyway; Colin generously still
gave a dollar. I thought it worth reporting even without expecting a bounty
because I think anything that interrupts the flow of the reader, causes them
to pause, irked at a possessive "its" misspelt as "it's", detracts from the
odds of them spotting something more serious.

Overall, a well crafted and executed bounty scheme. In some ways I'm surprised
other companies don't do something similar, even if it was with closed source
under an NDA. If the feedback is prompt then the bounty hunter can decide to
stop if the bounties aren't awarded fairly enough in his opinion, or the
company can cancel, perhaps having shelled out their budget or unhappy with
the quality of the bugs, without the hunter having wasted much time since the
last bounty.

------
psykotic
Cool experiment!

I found it amusing (and unsurprising) that the distribution of reported bugs
matches what you would find in code reviews within most companies: a small
handful of genuine issues buried in a giant pile of superficial nitpicks.

Before anyone interjects, no, I'm not saying misspellings and stylistic
inconsistencies don't matter. They matter to me a great deal--I am crazily OCD
about that shit. Seeing 'shear' misspelled as 'sheer' throughout most of
Unreal Engine 2 and 3 without being able to fix it (there were too many
expected breaking changes to licensee code for such a cosmetic change to make
sense) annoyed me like you wouldn't believe. But when I force myself to take a
deep breath and consider the matter more rationally, it's clear that such
issues are far down the list in importance. While I'm reviewing someone else's
code and my OCD-ish code aesthetics sense is stroked, I always try to remind
myself of that.

~~~
cperciva
Yes, this is roughly the distribution I expected. Maybe a few more spelling
mistakes than I expected, but not as many style bugs, so the "harmful bug" /
"harmless bug" / "cosmetic bug" distribution matched reasonably closely.

But I wouldn't call the harmless bugs "superficial nitpicks". Most of them
could become problems later -- if other code changes result in the functions
they are in being called with different parameters; or if an error is no
longer treated as fatal but instead Tarsnap continues and retries the failed
operation; or with more aggressive compiler optimizations.

Better to fix the bugs before they cause problems than to wait for them to
cause problems.

~~~
psykotic
Oh, I meant the cosmetic bugs specifically. And to be clear, I'm not saying
they shouldn't have been reported. Issues like that _should_ be fixed. I just
found it all vaguely amusing for some reason because unless I keep myself in
check I'm so much that grammar and spelling and formatting guy when it comes
to code reviews.

~~~
cperciva
Ah, ok. My original motivation for offering bounties for those was as a proof-
of-work for code reading -- I figured there would be enough of those that
anyone who spent a while looking at the code would pick up a few, even if they
couldn't find any "real" bugs.

The fact that it satisfied my OCD is just a side benefit. ;-)

------
thaumaturgy
You have an astounding dedication to your craft.

~~~
cperciva
It makes me sad that this is considered abnormal, never mind astounding.

~~~
jgrahamc
I agree with that sentiment.

I have a piece of C code that I license to companies (it does multi-label
classification of documents---mostly email) and I have very good test coverage
and have been obsessive about its design and operation. Since it does a lot of
low level C tricks for speed I wanted to be absolutely sure that it wouldn't
crash, and it has to work on Windows (32 and 64), Solaris, HP/UX, AIX, Linux,
... The bottom line is that being obsessed about the stability of the code has
meant that in the five years it's been shipping not a single client has
reported a bug and I have never heard of the code crashing.

This actually turned out to be a problem for me, because I had a hard time
getting some people to renew maintenance on the code because it never failed.

~~~
TheAmazingIdiot
I deign to mention it, but if you were to add in a "bug" that would safely
'crash' at sparse but random-ish intervals, you could charge for maintenance.

Maintenance would consist of lowering the multiplier of "crash" occurance. So
your Program is always improving. I see little difference between this and the
way hardware makers cripple lower end hardware

You can sell maintenance then. Slimy, i know. But business is business.

~~~
jacques_chester
This tactic might exist in the wild, based on some stories I've seen on
DailyWTF, but I would be _amazed_ if it didn't result in lawsuits. One of
these days those "no liability" clauses are going to be removed by legislation
or a blockbuster court case and whole sewerage plants are going to smash into
fan factories.

------
olliesaunders
I hadn’t ever encountered this idea (bug bounties) before and I think it’s
great. Only thing that bothers me is that, if you were to run the scheme for
long enough it might start to become a victim of its own success: wary of the
bug hunters your programmers would be so much more careful (think especially
about the harmless and stylistic stuff) that there wouldn’t be any small
things available to find any more and bug hunters would get bored and give up.

To resolve this you could only conduct the bounties periodically so there’s
always enough of a backlog of bugs to be found.

------
mhlakhani
I don't know about the others, but I think the bug bounty is a good idea. I
mostly went over the code so I could learn something about good C coding
conventions. Plus I wanted to try out tarsnap without having to go through the
hassle of getting access to a credit card; the bug bounty is pretty helpful
for that use case.

------
ph0rque
Hmmm, cperciva mentions 211 bounties, but 53 + 70 + 60 + 11 = 194 bugs. What
were the other 17 bugs; duplicate reports?

~~~
cperciva
It was _about 70_ style bugs, not exactly. A few bounties covered
documentation, and I awarded one to someone who told me how to fix some CSS on
the website.

~~~
ralph
I came across a comment on the CSS in the HTML of the site offering a bounty
for the fix, realised I didn't know the answer, and pointed it out to a LUG
friend of mine who I thought would. He did and got the sole $5 bounty. :-)

------
LeafStorm
I clicked on the link, and it displayed the "Aw, snap! Something went wrong
while displaying this Web page." page. I suppose that $1265 of bugs could look
like that.

