Hacker News new | past | comments | ask | show | jobs | submit login
What does $1265 of bugs look like? (daemonology.net)
211 points by cperciva on Aug 26, 2011 | hide | past | web | favorite | 31 comments



I've spent a while looking for tarsnap bugs; to date I count 86 email threads with Colin to report my findings. There are a couple of things I think are pretty good about the scheme Colin has devised.

Unlike some other bounty schemes, there are small bounties available. One can look for ages without finding a $50 bounty, so long that boredom would probably set in and the hunt would be abandoned. But coming across the odd $1 or $5 bounty keeps interest going at small cost to Colin and still improves the overall source quality. It's almost like a bit of a game, a treasure hunt, where small pickings on route keep one's hopes up of rich pickings down the road. A game that can be set down and picked up later as time allows.

Secondly, Colin was very fair in judging the bounty to be awarded. It soon became apparent that I could trust his judgement; whenever I thought a bug was undervalued another would come along that might be overvalued. On mentioning this to him, it turns out it was sometimes deliberate; he'd be wavering and thought it fair to even out which side of the boundary the bug fell.

As for the OCD, I boggle a bit at some of those style bugs and don't think they were all me! :-) I do recall sending in some trivial thing suggesting it wasn't bountiful but perhaps should be fixed anyway; Colin generously still gave a dollar. I thought it worth reporting even without expecting a bounty because I think anything that interrupts the flow of the reader, causes them to pause, irked at a possessive "its" misspelt as "it's", detracts from the odds of them spotting something more serious.

Overall, a well crafted and executed bounty scheme. In some ways I'm surprised other companies don't do something similar, even if it was with closed source under an NDA. If the feedback is prompt then the bounty hunter can decide to stop if the bounties aren't awarded fairly enough in his opinion, or the company can cancel, perhaps having shelled out their budget or unhappy with the quality of the bugs, without the hunter having wasted much time since the last bounty.


Cool experiment!

I found it amusing (and unsurprising) that the distribution of reported bugs matches what you would find in code reviews within most companies: a small handful of genuine issues buried in a giant pile of superficial nitpicks.

Before anyone interjects, no, I'm not saying misspellings and stylistic inconsistencies don't matter. They matter to me a great deal--I am crazily OCD about that shit. Seeing 'shear' misspelled as 'sheer' throughout most of Unreal Engine 2 and 3 without being able to fix it (there were too many expected breaking changes to licensee code for such a cosmetic change to make sense) annoyed me like you wouldn't believe. But when I force myself to take a deep breath and consider the matter more rationally, it's clear that such issues are far down the list in importance. While I'm reviewing someone else's code and my OCD-ish code aesthetics sense is stroked, I always try to remind myself of that.


Yes, this is roughly the distribution I expected. Maybe a few more spelling mistakes than I expected, but not as many style bugs, so the "harmful bug" / "harmless bug" / "cosmetic bug" distribution matched reasonably closely.

But I wouldn't call the harmless bugs "superficial nitpicks". Most of them could become problems later -- if other code changes result in the functions they are in being called with different parameters; or if an error is no longer treated as fatal but instead Tarsnap continues and retries the failed operation; or with more aggressive compiler optimizations.

Better to fix the bugs before they cause problems than to wait for them to cause problems.


Oh, I meant the cosmetic bugs specifically. And to be clear, I'm not saying they shouldn't have been reported. Issues like that _should_ be fixed. I just found it all vaguely amusing for some reason because unless I keep myself in check I'm so much that grammar and spelling and formatting guy when it comes to code reviews.


Ah, ok. My original motivation for offering bounties for those was as a proof-of-work for code reading -- I figured there would be enough of those that anyone who spent a while looking at the code would pick up a few, even if they couldn't find any "real" bugs.

The fact that it satisfied my OCD is just a side benefit. ;-)


Well, the funny thing about cosmetic bugs is that they tend to cluster with other, more subtle bugs (in my experience, at least, I'm more likely to find substantive errors by concentrating on sections of code near the obvious cosmetic bugs).


http://en.wikipedia.org/wiki/HTTP_referrer

The misspelling referer originated in the original proposal by computer scientist Phillip Hallam-Baker to incorporate the field into the HTTP specification.[1] The misspelling was set in stone by the time of its incorporation into the standards document Request for Comments (RFC) 1945; document co-author Roy Fielding has remarked that neither "referrer" nor the misspelling "referer" were recognized by the standard Unix spell checker of the period.


You might find Watts Humphrey's book A Discipline for Software Engineering to be an interesting read. In it he describes a scheme where you track all your mistakes, no matter how tiny, and patiently develop ways to prevent them from happening in future.

edit: Look, I know that Humphreys is the agile bogeyman, but why the downvote? Engage, don't merely click an arrow.


You have an astounding dedication to your craft.


It makes me sad that this is considered abnormal, never mind astounding.


I agree with that sentiment.

I have a piece of C code that I license to companies (it does multi-label classification of documents---mostly email) and I have very good test coverage and have been obsessive about its design and operation. Since it does a lot of low level C tricks for speed I wanted to be absolutely sure that it wouldn't crash, and it has to work on Windows (32 and 64), Solaris, HP/UX, AIX, Linux, ... The bottom line is that being obsessed about the stability of the code has meant that in the five years it's been shipping not a single client has reported a bug and I have never heard of the code crashing.

This actually turned out to be a problem for me, because I had a hard time getting some people to renew maintenance on the code because it never failed.


This is why I've never believed in the "charge for support" model for funding open source software -- it creates a financial incentive to have hard-to-use bug-ridden code.

With Tarsnap, my incentives are perfectly aligned: Writing good code results in happy Tarsnap users, which results in more Tarsnap users.


Well, "charge for support" works for amalgamations of code like Red Hat, but I agree it's probably not ideal for individual projects (at least small projects). I prefer offering services related to the code -- not only bugfixes for cash, but also new features and installation. CodeWeavers develops WINE this way and I think they're doing pretty well.


Just curious - did the fact that it was hard to get people to renew maintenance contracts have any effect on your code quality at all?


No, I take bugs personally. If you actually take bugs personally they hurt and you don't want to have to deal with them. Thus you make them go away.


For literally decades I have advocated this attitude. Time and time again my efforts are beaten back with "Bug-Free doesn't sell" and "Bug-Free doesn't pay" and, most insidious, "Bug-Free is impossible."

This last is then followed by

* You have to verify the compiler,

* You have to verify the hardware,

* You have to verify that there wasn't an error in loading the code,

* You have to use radiation-hardened hardware

* ...

and so on.

They miss the point!

Bugs in your code are way more likely than any of the other problems, and they matter.

I've generally given up, but at least it's nice to see the attitude here not being instantly shouted down.


I'm with you on "Bugs in your code are way more likely than any of the other problems, and they matter." This is very true. A while back Eric Raymond had a post called "When you see a heisenbug in C, suspect your compiler’s optimizer" (http://esr.ibiblio.org/?p=1705) which I felt was very wrong and responded with "A bad workman blames his tools" (http://blog.jgc.org/2010/02/bad-workman-blames-his-tools.htm...).

As might have been predicted the bug that Raymond was talking about wasn't caused by the optimizer at all (http://esr.ibiblio.org/?p=1705#comment-248328).

It's depressing to think that when your code works, it's almost always your fault.


I think I got my attitude from finding a bug in a major production compiler. I sent in my snippet to demonstrate the problem and got very short shrift from the company.

And they were right - my code had bugs in it.

So I recrafted my code and effectively, informally proved it to be correct. Then I sent it in again and got a formal acknowlegement that I had, indeed, found a bug.

Over the next 2 years I found 5 bugs in three different compilers, but in the first few cases I spent ages crafting my examples to prove the problem wasn't my code. Then it dawned on me, why not right the code to be correct in the first place, then I wouldn't have to re-write it when I found bugs that might be mine, and might be someone elses.

My productivity improved dramatically, and I was hooked.


Indeed. My C++ teacher's mantra for students was "the compiler is always right".

Obviously, this false, but until you are smart enough you'll never know when the compiler is wrong.


Interestingly, ESR himself says "don't claim you have found a bug unless you are very, very sure of your ground": http://catb.org/~esr/faqs/smart-questions.html#id478549


Can you transfer this attitude to, say, a larger piece of software? I'm thinking of working on an "enterprise"-level project, where requirements may be vague, or timelines constrained.

I'm not disagreeing with your premise, but I find my attitude can be different depending on the scope of the project. For example, if I'm working on a smaller project, I find I can focus on the overall quality and strive for "no bugs." But on a larger project (often behind schedule and over-budget, most often not because of the dev team, directly), there are lots of edge cases that get ignored, at least during the first release cycle.


I deign to mention it, but if you were to add in a "bug" that would safely 'crash' at sparse but random-ish intervals, you could charge for maintenance.

Maintenance would consist of lowering the multiplier of "crash" occurance. So your Program is always improving. I see little difference between this and the way hardware makers cripple lower end hardware

You can sell maintenance then. Slimy, i know. But business is business.


This tactic might exist in the wild, based on some stories I've seen on DailyWTF, but I would be amazed if it didn't result in lawsuits. One of these days those "no liability" clauses are going to be removed by legislation or a blockbuster court case and whole sewerage plants are going to smash into fan factories.


Using the car analogy, crippling hardware to make it slower is an equivalent of putting speed limiter on an engine (admittedly for different reasons) which still makes it fully operational. Purposefully introducing bugs is like making airbags go off at random times so you could charge for service. Nice business practice.


Let's say an admirable dedication to your craft, then.

I'm definitely glad you put out these bounties - you set a very good example for us all.


I hadn’t ever encountered this idea (bug bounties) before and I think it’s great. Only thing that bothers me is that, if you were to run the scheme for long enough it might start to become a victim of its own success: wary of the bug hunters your programmers would be so much more careful (think especially about the harmless and stylistic stuff) that there wouldn’t be any small things available to find any more and bug hunters would get bored and give up.

To resolve this you could only conduct the bounties periodically so there’s always enough of a backlog of bugs to be found.


I don't know about the others, but I think the bug bounty is a good idea. I mostly went over the code so I could learn something about good C coding conventions. Plus I wanted to try out tarsnap without having to go through the hassle of getting access to a credit card; the bug bounty is pretty helpful for that use case.


Hmmm, cperciva mentions 211 bounties, but 53 + 70 + 60 + 11 = 194 bugs. What were the other 17 bugs; duplicate reports?


It was about 70 style bugs, not exactly. A few bounties covered documentation, and I awarded one to someone who told me how to fix some CSS on the website.


I came across a comment on the CSS in the HTML of the site offering a bounty for the fix, realised I didn't know the answer, and pointed it out to a LUG friend of mine who I thought would. He did and got the sole $5 bounty. :-)


I clicked on the link, and it displayed the "Aw, snap! Something went wrong while displaying this Web page." page. I suppose that $1265 of bugs could look like that.




Applications are open for YC Winter 2020

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: