> For classes of bugs intended for all or a wide variety of engineers on a given platform, we have gravitated toward a "diff time" deployment, where analyzers participate as bots in code review, making automatic comments when an engineer submits a code modification. Later, we recount a striking situation where the diff time deployment saw a 70% fix rate, where a more traditional "offline" or "batch" deployment (where bug lists are presented to engineers, outside their workflow) saw a 0% fix rate.
> The response was stunning: we were greeted by near silence. We assigned 20-30 issues to developers, and almost none of them were acted on. We had worked hard to get the false positive rate down to what we thought was less than 20%, and yet the fix rate—the proportion of reported issues that developers resolved—was near zero.
> Next, we switched Infer on at diff time. The response of engineers was just as stunning: the fix rate rocketed to over 70%. The same program analysis, with same false positive rate, had much greater impact when deployed at diff time.
I am the Sourcegraph CEO, and we've long been aware of tools like this at Facebook, and Google's similar tool (described in another ACM article, https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...). We are working on bringing static analysis into the dev workflow, and letting you run campaigns to apply static analysis fixes across your entire codebase. Check out https://about.sourcegraph.com/product/automation/ for info and screencasts to see what this looks like.
The root of that shock was the human factor. Us devs aren't machine parts, but our essential job is to do stuff and fix the inevitable bugs. The fact that presentation of bug reports could change things so radically was an eye opener, and one I'll generalise and remember.
Then again I've never worked on huge codebases, and I assume the devs there were top engineers so I'm not attributing this to laziness or indifference - but still, amazing and not in a positive way! I'd just assumed static analysis = bugs found = bugs fixed = better code; end of story.
if someone comes to you with a bazillion issues that you cannot possibly fix in the X hours you have during the day what are you going to do? you're going to prioritize and you're going to fix the most important bugs, right?
if you're developing a new feature and things are pointed out during the code review (bots or no bots) what are you going to do? fix it, right? you want to ship whatever you're doing with high quality.
so the problem outlined below is a false one. you can never fix all the possible bugs if the codebase if nothing but trivial.
That doesn't make sense. Presented one way, the bugs were fixed to a 70% rate. Presented another way, 0%.
Presumably the bugs in each case were about equivalent in their severity (else the conclusion is meaningless) so severity was not the driving factor in what got fixed.
> you can never fix all the possible bugs if the codebase if nothing but trivial.
That's a non-sequitur. No-one suggested they could fix all, or should try. It was about known bugs presented to them.
If it's a comment on a code review for code I am already working on, then of course I want my new or changed code to be up to date with standards and avoid error prone patterns, etc.
However that does leave a major question for me; if devs are triaging their own bugs to this extent (70% vs 0% fix), is this safe, because those bugs will relate ultimately to business requirements related to uptime and correctness.
It's not perfect, but the smaller scale reporting is easier to deal with, fix, add suppression rules, etc. Sounds obvious, I suppose.
Duh? If I am assigned a low priority bug, I will put it on the bottom of my todo list. My manager has the ability to reprioritize items on my todo list, but that doesn’t seem like that was done in this case. If you want people to fix your low priority bugs, just filing them and praying isn’t going to work.
The obvious advantage inspections have over diff-time tooling is that detectable issues are often fixed immediately when a developer is most familiar with the code in context (edit-time), which translates to improved productivity. Additionally since most inspections provide quick fixes, they not only save time, but also provide code consistency regarding how issues are resolved.
I realize FB uses languages other than Java, so I wonder if their static analysis investments would be better served focused more on building/buying better IDE tooling?