

Big discussion: Do Static Source Code Analysis Tools Really Work?  - Tatyanazaxarova
http://developers.slashdot.org/story/08/05/19/1510245/do-static-source-code-analysis-tools-really-work

======
tzs
Carmack wrote about static code analysis a few months ago:
<http://altdevblogaday.com/2011/12/24/static-code-analysis/>

It works.

------
seclorum
I've used Coverity in the past to radically increase the quality of a large
SIL-4 code-base. (Safety-critical code for a major industrial transportation
systems provider.)

Without question, it helped my project gain a large step in terms of quality
and functionality and was definitely worth it in the context of the regular
costs of the project. I won't say the codebase became 100% bug-free, but I
will say that it helped us rapidly increase the quality of our code in ways
that could only have cost us a lot of time with testing, testing, testing ..
and more testing. It found bugs that were extremely esoteric, but
nevertheless, extremely important to eradicate, especially in a SIL-4
codebase.

My only desire is that these powerful tools become more widely available
outside of the safety-critical scene. It would be wonderful to have this sort
of power for some of my own smaller, less critical projects - it definitely
makes one a better, more careful, programmer.

------
twiceaday
If the tools were completely free and easy to integrate would there even be a
discussion?

~~~
wink
Yes, because you can be swamped in false positives so much that it's easier on
time, money and nerves to just not use the tool. And this is a general problem
in static source code analysis, not just the "bad tools".

I had this awesome example in one book about metrics where the static analysis
on some 100k LOC project produced output with the same line length as "War and
Peace". Good luck sifting through those.

But I'm absolutely not saying you shouldn't use them, but as some commenters
in the original post pointed out, you need to check if they produce sensible
output for your project or if they can catch some really nasty bugs, if it's
worth the additional work.

------
moldbug
Great comment from the Slashdot thread (direct link:
[http://developers.slashdot.org/comments.pl?sid=557384&ci...](http://developers.slashdot.org/comments.pl?sid=557384&cid=23464230)):

My experience has been that while in the hands of people who know what they're
doing, they're a nice tool to have, well, beware managers using their output
as metrics. And beware even more a consultant with such a tool that he doesn't
even understand.

The thing is, these tools produce

A) a lot of "false positives", code which is really OK and everyone understand
why it's ok, but the tool will still complain, and

B) usually includes some metrics of dubious quality at best, to be taken only
as a signal for a human to look at it and understand why it's ok or not ok.

E.g., ne such tool, which I had the misfortune of sitting through a salesman
hype session of, seemed to be really little more than a glorified grep. It
really just looked at the source text, not at what's happening. So for example
if you got a database connection and a statement in a "try" block, it wanted
to see the close statements in the "finally" block.

Well, applied to an actual project, there was a method which just closed the
connection and the statements supplied as an array. Just because, you know,
it's freaking stupid to copy-and-paste cute little "if (connection != null) {
try { connection.close(); } catch (SQLException e) { // ignore }}" blocks a
thousand times over in each "finally" block, when you can write it once and
just call the method in your finally block. This tool had a trouble
understanding that it _is_ all right. Unless it saw the "connection.close()"
right there, in the finally block, it didn't count.

Other examples include more mundane stuff like the tools recommending that you
synchronize or un-synchronize a getter, even when everyone understands why
it's OK for it to be as it is.

E.g., a _stateless_ class as a singleton is just an (arguably premature and
unneded) speed optimization, because some people think they're saving so much
by a singleton instead of the couple of cycles it takes to do a new on a class
with no members and no state. It doesn't really freaking matter if there's
exactly one of it, or someone gets a copy of it. But invariably the tools will
make an "OMG, unsynchronized singleton" fuss, because they don't look deep
enough to see if there's actually some state that must be unique.

Etc.

Now taken as something that each developper understands, runs on his own when
he needs it, and uses his judgment of each point, it's a damn good thing
anyway.

Enter the clueless PHB with a metric and chart fetish, stage left. This guy
doesn't understand what those things are, but might make it his personal duty
to chart some progress by showing how much fewer warnings he's got from the
team this week than last week. So useless man-hours are spent on useless
morphing perfectly good code, into something that games the tool. For each 1
real bug found, there'll be 100 harmless warnings that he makes it his
personal mission to get out of the code.

Enter the snake-oil vendor's salesman, stage right. This guy only cares about
selling some extra copies to justify his salary. He'll hype to the boss
exactly the possibility to generate such charts (out of mostly false
positives) and manage by such charts. If the boss wasn't already in a mind to
do that management anti-pattern, the salesman will try to teach him to. 'Cause
that's usually the only advantage that his expensive tool has over those open
source tools that you mention.

I'm not kidding. I actually tried to corner one into;

Me: "ok, but you said not everything it flags there is a bug, right?"

Him: "Yes, you need to actually look at them and see if they're bugs or not."

Me: "Then what sense does it make to generate charts based on wholesale
counting entities which may, or may not be bugs?"

Him: "Well, you can use the charts to see, say, a trend that you have less of
them over time, so the project is getting better."

Me: "But they may or may not be actual bugs. How do you know if this week's
mix has more or less actual bugs than last weeks, regardless of what the total
there is?"

Him: "Well, yes, you need to actually look at them in turn to see which are
actual bugs."

Me: "But that's not what the tool counts. It counts a total which includes an
unknown, and likely majority, number of false positives."

Him: "Well, yes."

Me: "So what use is that kind of a chart then?"

Him: "Well, you can get a line or bar graph that shows how much progress is
made in removing them."

Lather, rinse, repeat, give up.

Or enter the consultant with a tool. Now while I'll be the first to say that a
lot of projects do end up needing a good consultant to get the code to perform
right, I'll also warn about the thousands of parasites posing as one. There
are more than enough guys who don't actually have any clue, and all they do is
run the code through such a tool, and expect miracles to happen. They don't
see the big fat SQL query that causes the delay, or the needless loop it's in,
but they'll advise blind compliance to the 1 microsecond saving tips of their
tool. And of course, again, try to justify his fee by reducing such a total of
harmless warnings.

And I guess the moral of the story is that you also need to train the people
to understand those tools, and understand what they can ignore and what not.
It might mean a bit more expense than just downloading the tools. Especially
training some managers what _not_ to do with them.

------
funkah
They do. Really.

