
John Carmack on Static Code Analysis - niyazpk
http://www.altdevblogaday.com/2011/12/24/static-code-analysis/
======
pnathan
A couple things:

\- Competitors with Coverity are CodeSonar[1] and Klocwork[2]. I've not seen
Klocwork output, but CodeSonar and Coverity are in the same area of quality,
with differing strengths. I can not recommend static analysis highly enough if
you have a C/C++/Java/C# database. It's very expensive (well into five figures
according to Carmack), but how expensive is a bug? What if you have your
entire codebase checked daily for bugs? Consider the effect on your quality
culture. :-)

\- The fact that you are paying "well into five figures" for a tool that
essentially covers up design deficiencies in your language should start
sounding alarm bells in your head. The proposition more or less goes, "To have
reliable C++ code in certain areas, you need a static analyzer; to gain that
same advantage in Haskell costs you nothing more than GHC". Of course Haskell
doesn't have certain C/C++ capabilities; but it's worth meditating on for your
next application, particularly if bugs are more important than performance.
N.b- I don't know the ML family enough to say one way or the other in this
regard. :-)

[1] <http://www.grammatech.com>

[2] <http://www.klocwork.com>

~~~
timtadh
On the Java end (all though these tools also do C/C++ and other languages
maybe even .NET but their speciality from my understanding is Java) is Fortify
[now owned by HP] and IBM AppScan Source [previous Ounce from Ounce Labs].

Both produce high quality output in their domain. Fortify is widely use in the
AppSec world.

~~~
johnyzee
What do they give you that you don't get with open source tools like Clover,
Findbugs and Checkstyle?

~~~
timtadh
Accuracy. Detail. They have better analysis engines, which do more complex
types of program analysis. The static program analysis field is incredibly
deep it takes a lot of engineering to make a world class analysis engine. Just
like it takes a lot of effort to make a world class optimising complier. (eg.
the Intel C Compiler _still_ out performs GCC despite monumental effort on
GCC's side. It takes a long time to build this stuff)

They also have better rules sets from what I understand than the open source
tools.

~~~
mzl
While the Intel compiler may be better in some/many cases, it is not
universal. In my experience (a couple of years ago, C++ code, high-performance
requirements), the Intel compiler was quite a bit worse than GCC for code with
a lot of branches and abstraction layers.

The Microsoft compiler was way better than both. Around 10% faster run-time
IIRC.

------
santaragolabs
So I've dealt with dozens of Fortune-100 companies implementing and using
static code analysis tools. They can and will help but in general I feel that
these tools are not much more than the code-equivalent of the syntax- and
grammar- checker in your word processing software.

I've been doing manual code reviews for a living now (mostly security related)
for roughly 3 years now and while I get assisted from time to time by code
analysis tools I still find heaps of bugs not caught by any of the tools
mentioned by Carmack. The biggest issue for a development shop is to properly
integrate these tools and to not overwhelm developers with too much false
positives.

I've had cases where a developer got a 1500 page PDF spit out by one of these
static analysis tools. After spending two weeks going through everything the
developer ended up with 50 pages of actual bugs; the rest were describing
false positives. Then I got on-site and I still logged dozens and dozens of
security-related bugs that the static analysis tools failed to find.

Edit: also consider that one even needs a SAT solver to even do proper C-style
preprocessor dependency checking. A lot of these code analysis tools are being
run on debug builds only and then there when the release build is being made
these tools are not being run meaning they fail to catch a lot of issues. It's
insanely hard to write proper code analysis tools and static source code
analysis tools which do not integrate with the compilation process I wouldn't
trust at all.

Nowadays with clang there are very nice possibilities for someone to write
your own simple checks and integrate them into the build process. But even
clang doesn't expose everything about the preprocessor that you might want to
have from a static code analysis perspective.

~~~
btilly
Beware the false false positive!

A couple of years ago I ran across an article from Coverity about the
challenges that they had getting people to use their tool. One of the most
interesting was that the tool would find real bugs, but the developer wouldn't
understand that the bug actually was real, and would say, "This tool is crap."
Then they would not get a sale.

They had this problem particularly strongly with race conditions. There are a
number of checks that they took out because, even though they were finding
real bugs, developers were convincing themselves that the bugs were not real
even though they really were.

It really does not help that the developers who are asked to evaluate are
likely to be the same people who made the mistakes in the first place, so all
kinds of defensive behavior are to be expected.

~~~
eru
You are probably taking about this article
([http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-
lin...](http://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-
code-later/fulltext)). A very good read, indeed.

~~~
btilly
I was, and it was. Thanks for finding it again.

------
stcredzero
_> We had a period where one of the projects accidentally got the static
analysis option turned off for a few months, and when I noticed and re-enabled
it, there were piles of new errors that had been introduced in the interim.
Similarly, programmers working just on the PC or PS3 would check in faulty
code and not realize it until they got a “broken 360 build” email report.
These were demonstrations that the normal development operations were
continuously producing these classes of errors, and /analyze was effectively
shielding us from a lot of them._

Something which corroborates this: When penetration testers break into
systems, they're often using new 0-day exploits. Think about that. Most of
today's software development practice produces such a steady stream of low-
level bugs, that penetration testers can _assume_ that they're there!

 _> Trying to retrofit a substantial codebase to be clean at maximum levels in
PC-Lint is probably futile. I did some “green field” programming where I
slavishly made every picky lint comment go away, but it is more of an
adjustment than most experienced C/C++ programmers are going to want to make.
I still need to spend some time trying to determine the right set of warnings
to enable to let us get the most benefit from PC-Lint._

This could be encouraged using game dynamics. Have a mechanism where a
programmer can mark parts of the codebase "green-field." A programmer's
"green-field score" consists of the number of lines of green-field code (or
statements, whichever lousy metric you want) that he's successfully compiled
with no warnings whatsoever. Combine this with random sampling code
walkthroughs, which has many benefits but will also catch boilerplate, auto-
generated, or copy-paste programming by a "Wally" who's trying to "write
himself a new minivan."

------
hsmyers
Having used PC-Lint almost all the way back to it's origins, I can testify to
just how scary it is to run this on your code. Code you wrote as well as code
written by teammates. In self defense, you HAVE to spend time tuning the
system in terms of warnings and errors---otherwise you drown in a sea of
depressing information. I liked John's comment about attempting 'green field'
coding. It is a tremendously valuable process given the time. Great article,
definite thumbs up.

------
js2
(2011) Original HN discussion <http://news.ycombinator.com/item?id=3388290>

------
egonschiele
I've been wanting something like this for Ruby for some time now. Since it's
dynamically typed and ridiculously easy to monkey-patch, Ruby is a much harder
challenge than C++. The two best efforts I have found are Diamondback Ruby
(<http://www.cs.umd.edu/projects/PL/druby/>) and Laser
(<http://carboni.ca/projects/p/laser>)...but they mostly try to add static
type-checking to Ruby code. After looking at these I implemented a contracts
library for Ruby (<https://github.com/egonSchiele/contracts.ruby>) to get
myself some better dynamic checking. The next step is to use the annotations
for the contracts library to do better static code analysis. One thing I'm
working on is generating tests automatically based on the contract
annotations. But I've got a long way to go : ( If anyone knows about other
projects that are working on static analysis for Ruby I'd be very interested
in hearing about them!

~~~
pjungwir
Looked at your Contracts library for Ruby. I was just thinking yesterday about
starting a project like this! Some thoughts:

\- I like that the contracts are right there in the code and not in a separate
file.

\- The actual Contract methods only run once upon loading each class, so they
don't cause a performance hit. I'd like a way to make them all noops (so they
don't add any instrumentation) when running in "production".

\- My thought is that if you can define pre- and post-conditions as you're
doing, then you can cut out a lot of the checks in your tests, so that as long
as the tests exercise the code, you can rely on the post-conditions to do
quite a bit of the validation. Plus you're validating outputs all over the
place constantly, so you'd probably catch edge cases you never even expected.

\- I'm disappointed that your system seems to offer little more than type
checking (unless I'm mistaken). I'd like something that gives me true pre- and
post-conditions: basically passing a block of Ruby code for each one, with
common idioms abbreviated so I don't need to write a whole block if the
conditions are simple.

Good luck on the project!

~~~
egonschiele
Great thoughts!

> I'd like a way to make them all noops

Sure! There's a default failure callback that gets called when a contract
fails. You can just clear this callback in prod:

    
    
        if ENV == PRODUCTION
            class Contract
              def self.failure_callback(data)
                # blank
              end
            end
        end
    

> I'm disappointed that your system seems to offer little more than type
> checking

Nay! You can write your own contracts to do anything you want. So yes, lambdas
are allowed in contracts:

    
    
        Contract lambda { |x| x.is_valid_email? } => Bool
        def save_email(email)
          # save an email
        end
    
    

> with common idioms abbreviated

I provide some common idioms already, but you can make your own abbreviations:

    
    
        ValidEmail = lambda { |x| x.is_valid_email? }
    
        Contract ValidEmail => Bool
        def save_email(email)
    

Let me know if you have any other questions :)

~~~
pjungwir
Thanks for the clarifications!

------
zwieback
The article mirrors my recent experience 100%. We've got a Coverity license
and I've started using it recently. Luckily, our code base is relatively
small, it's straight C and embedded (no mallocs, no OS). Even in this
extremely simple environment it's shocking how many errors Coverity can ferret
out.

The false-positives are a problem and the general advice to get started is to
initially ignore all existing bugs and focus on avoiding adding new bugs.
Then, when you get the hang of writing code that passes the checks you go back
and look for the worst of the older bugs, etc.

------
gaius
_Microsoft feels that game quality on the 360 impacts them more than
application quality on Windows does. :-)_

Many a true word spoken in jest!

~~~
cbs
Carmack is no doubt familiar with Microsoft's game cert process, it probably
wasn't in jest at all.

It does impact Microsoft more, because Microsoft has taken on responsibility
for games to be good (well, meet certain baselines for various definitions of
good) through their TCRs.

~~~
gaius
'Twas his smiley on the end, I didn't add it.

------
estebank
It's a great article by an insightful individual.

If you haven't read it, do so.

You can read further discussion on this 270 days old article at
<http://news.ycombinator.com/item?id=3388290>

------
cpeterso
FindBugs [1] is a great code analysis tool for Java. It's free, open source,
and supports plugins for writing your own checks. The FindBugs site reports an
interesting story from a Google test day:

"Google held a global "fixit" day using UMD's FindBugs static analysis tool
for finding coding mistakes in Java software. More than 700 engineers ran
FindBugs from dozens of offices.

Engineers have already submitted changes that made more than 1,100 of the
3,800 issues go away. Engineers filed more than 1,700 bug reports, of which
600 have already been marked as fixed. Work continues on addressing the
issues raised by the fixit, and on supporting the integration of FindBugs into
the software development process at Google."

[1] <http://findbugs.sourceforge.net/>

~~~
jadc
For Java code, I am a big fan of Sonar Source [1]. It is open source, is able
to leverage checkers such as FindBugs, and has integration with code coverage
tools such as Cobertura. I have found the clean dashboard to be a great boon,
and have never felt intimated by the reported warnings like I've been with
Coverity et al.

[1] <http://www.sonarsource.org>

~~~
npsimons
As long as we're on an open source kick here (since I prefer open source over
proprietary):

<http://checkstyle.sourceforge.net/>

<http://cppcheck.sourceforge.net/>

[http://google-
styleguide.googlecode.com/svn/trunk/cpplint/cp...](http://google-
styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py)

<http://www.dwheeler.com/flawfinder/>

<http://www.dsm.fordham.edu/~ftnchek/>

<http://community.haskell.org/~ndm/hlint/>

<http://jlint.sourceforge.net/>

<http://search.cpan.org/dist/Perl-Critic/bin/perlcritic>

<http://deployingradius.com/pscan/>

<http://pychecker.sourceforge.net/>

<http://pypi.python.org/pypi/pyflakes>

<http://www.logilab.org/857>

<http://packages.debian.org/squeeze/rats>

<http://www.splint.org/>

<http://www.incava.org/projects/303902593>

<http://packages.debian.org/stable/devel/icheck>

Please note: I have not used all of these; one day I was bored and decided to
see how many code checking tools I could find packaged for Debian and put them
all in a script with the warnings cranked to the max and make that part of my
default nightly build and test script. Of course, you can also crank up
warnings on your compilers and interpreters as well (-Wall -Wextra, python
-tt3 with warnings.simplefilter("always"), perl -tTW with use strict and use
warnings, etc)

------
quaunaut
I'm a web developer interested in diving into graphics programming sometime in
the next year, but this made me stop and wonder:

> If you aren’t deeply frightened about all the additional issues raised by
> concurrency, you aren’t thinking about it hard enough.

Why exactly is that?

~~~
amackera
The error conditions that cause bugs in concurrent code are difficult to
replicate since concurrent code can have so many additional states of
operation. It's also harder to program, especially in languages / environments
that don't explicitly encourage it right from the beginning.

~~~
prodigal_erik
It's important to remember that some of the possible states arise from the
memory model rather than what your code intentionally does. Say a producer
thread fills a buffer and queues work, and a consumer thread receives the
queue entry and reads the buffer. Does the consumer read any or all of what
the producer intended? It depends on whether they were scheduled in different
cores, and how full their caches were, and whether there was a memory barrier
that's global (which is slow and not always needed) or only applies to the
queue itself. Good platforms trade away just enough performance to guarantee
correctness here.

------
jaimefjorge
This article is one of the reason I created my tool: <http://www.qamine.com>

Qamine integrates directly with github and is designed to be used by small and
medium companies that cannot afford those expensive tools.

~~~
Darmani
Your tool looks pretty cool and useful, but it seems quite a bit different
than tools like Coverity and Astree which reason about the behavior of
programs under all possible inputs. Coverity can detect race conditions. Your
tool looks like it's using technology closer to that of Refactoring Crawler,
which was a quite impressive achievement, but not a static analysis tool.

------
dexen
Dear Lazyweb,

the `Controler' part of my main codebase consists of interwoven PHP and MySQL.
Is there static analysis tool that understands both, one in relation to the
other?

~~~
Darmani
The closest thing I'm aware of is a tool by the University of Tartu which does
this for Java, described in the paper "An Interactive Tool for Analyzing
Embedded SQL Quries." It can find the SQL errors (misspelled "name" and
missing space before "AND") in the following code:

    
    
        public Statement ageFilter(int minimalAge) throws SQLException {
          Connection conn = getConnection();
          String sql = "SELECT name FROM people WHERE nmae IS NOT NULL";
          if (minimalAge > 0)
            sql += "AND (age >= " + minimalAge + ")";
          return conn.prepareStatement(sql);
        }
    

There are a lot of really interesting challenges in doing this. Their tool
uses a powerful technique called "abstract parsing" which can analyze all
possible parse trees created by running the SQL parser on a string produced by
Java string operations. It's pretty impressive what modern static analysis can
do, and just how weak the linters we've gotten used to are in comparison.

------
mrich
For C/C++, also try just compiling with clang. It has great diagnostics. Also
it has the static analyer whose C++ support just improved greatly in trunk.

------
nfriedly
Funny timing, I just got jslint turned back on in our build today! (well,
jsHint now due to the 'for(var i=0...' failing even with -vars enabled, but I
digress...).

Another dev and I spent literally the entire day fixing issues - and we had
jslint running on every checkin until a few months ago!

But, it was worth it. It feels great to know that those bugs won't happen
again without a failing build :)

~~~
ahoge
>due to the 'for(var i=0...' failing even with -vars enabled

That's because `for(var i=0...` is completely bogus. What if you have two of
those loops? Are you going to put `var` in the first one, but not in the
second one? Are you going to use 'i' in the first one and 'k' in the second
one?

What if you need some refactoring and those two loops switch places?

It's fine if you like Java, but JavaScript is different. When in Rome, do as
the Romans do. I also wish there would be block scope, but merely pretending
there is doesn't make it so.

It's better if you declare all of your variables at the very top of the
innermost function, because that's where they effectively are (thanks to
hoisting). It's better if your code reflects how it's actually working,
because that will always make it easier to comprehend.

Once there is `let`, you should switch to Java's "declare on first use" rule
and never use `var` again.

~~~
Joeri
What is the bug that can result from the "for(var i=0)" pattern? None, beause
browsers ignore the second var for the same variable and do te ituitively
right thing.

What's the bug that results from the "var i; for(i=0)" pattern? Someone
refactors the "var i" away and you end up with a global var.

So, really, while the pedantic answer is to forbid the pattern, the pragmatic
answer is to recommend it.

~~~
ahoge
>What's the bug that results from the "var i; for(i=0)" pattern? Someone
refactors the "var i" away and you end up with a global var.

Wrong. JSLint will complain and then you'll fix it. That's why you use JSLint.
It can find stupid things like this easily and accurately.

------
rabidsnail
Has anyone tried running their commits through CRM114, marking removals in
bugfix commits as "bad"?

~~~
eru
You could also try to train some statistical model and see whether you can
find some common ground in commits that introduce bugs.

And then fire that person. (Or rather, see if big commits introduce more bugs
than short ones, or whatever else you might find.)

------
Flow
How good is the code analysis in IntelliJ compared to these tools?

~~~
calpaterson
Code analysis is nice but stand alone static analysis tools are a lot more
advanced. Even the free ones, like FindBugs catch a lot of issues.

------
Havoc
Pretty sure I've read this exact thing ~1 year ago.

~~~
samspot
Yeah it is old, you can check the article date.

------
seanalltogether
It's too bad he didn't post any samples, I'd love to see what kind of common
mistakes he caught so I can avoid them myself.

~~~
glaze
PVS-Studio's author has a blog where he lists examples from open source
projects: <http://www.viva64.com/en/b/>

------
speg
Any recommendations for tools in web development? I use JS Lint, but what
about server side languages?

~~~
Too
For PHP: phpLint[1], almost makes php into a type safe language.

For .NET you have Code Contracts[2], this requires you to annotate pre- and
postconditions for your functions but after that it's for example impossible
to pass a possibly null into a function without compile time error. It can
even do range-check of numerics after calculations etc.

[1]<http://www.icosaedro.it/phplint/>

[2]<http://research.microsoft.com/en-us/projects/contracts/>

~~~
nahname
ReSharper is the best lint tool you can get for dotnet development. It
literally makes you a better programmer that writes cleaner code. It really
sounds like JetBrains needs to make a similar tool for C++

~~~
adamzochowski
How does it compare to fxCop?

~~~
nahname
Night and day. FxCop requires days of customization to include something into
your build step. Even then, the recommendations are weak at best. Furthermore,
think back to Carmack's comment on how static analysis has to _at least_ be in
your build step.

ReSharper has much better recommendations that require almost no tweaking
(variable names you might disagree with). The best part, it tells you these
recommendations at the time of writing. This is one of ReSharpers best
features (IMO). Not only do programmers write better code with, they often
write code that works the first time and compiling becomes just a
technicality. Code improvement is noticeably better. So much so, that when my
company is hiring and I am reviewing the candidates submission; I can
immediately tell whether or not they were using ReSharper.

Cannot recommend this tool enough. I feel crippled without it in VS.

~~~
rbellio
Another ReSharper user here that has used FxCop as well. I think both tools
are awesome, but ReSharper's ability to do the code analysis on the fly is
huge for me.

Just be aware of your default shortcuts if you look into ReSharper. I remember
that some of them change on you.

------
FredBrach
_There have been plenty of hugely successful and highly regarded titles that
were filled with bugs and crashed a lot_

I think it's false, and a huge mistake. There is rare cases, but not plenty.
Video games (I mean video games for core gamers) are products that demand
_first_ and _above all_ quality to be successful. There is rather plenty of
common games with a huge quality which become hits (Kingdom Rush for example
or Starcraft which was finally kind of common in its time). One of the rules
in the delovepment process at Blizzard is that they ship a game when it has
less than 100 known bugs. Also, I would add that, it seems that, ID software
did not make a successful games since Doom. Quake wasn't a commercial success,
Quake2, Quake3, Doom3 and Rage neither (that's why ID software has been bought
for _only_ $100M). After all, ID Software lost one of its core value co-
founder a long time ago (John Romero) who was responsible of the gameplay of
ID's games...

Quality in video games are everything, that's really my opinion. It's also
really an edge for every indy developper which want to start a company in this
sector, cases are countless.

~~~
Impossible
"Quake wasn't a commercial success, Quake2, Quake3, Doom3 and Rage neither
(that's why ID software has been bought)"

Do you have actual sales numbers to back that up or is this based on anecdotal
evidence, or is it just because you didn't like the games as much :)? Publicly
available internet numbers show that Doom 3 sold 3.5 million, and even Rage
sold 2.31 million, making Doom 3 extremely profitable and probably putting
Rage a little higher than break even given the long development time and much
larger team size.

Two of the Quake games are listed on wikipedia's top selling PC games and sold
1.7 and 1 million. ([http://en.wikipedia.org/wiki/List_of_best-
selling_PC_video_g...](http://en.wikipedia.org/wiki/List_of_best-
selling_PC_video_games))

Although none of these numbers are 100% accurate unless they are wildly off I
wouldn't call any of id's games except Rage unsuccessful.

~~~
klodolph
Don't forget licensing. On the top-selling games are other games which are
based on the Quake/id Tech source code: 3 Half-Life games, 2 Counter-Strike
games, and American McGee's Alice. I'm sure only a small percentage of Half-
Life 2 code can still be traced to its Quake roots, but it's still some
impressive technology. ( _Alice_ is a little more obviously derived from Q3A.)

~~~
FredBrach
You forget the Call of Duty engine which is, if I'm not wrong, still made of
Quake engine code... Sure, others made trully successful games with their
technology. Others, like Valve, actually did the video games history since the
2000's. ID not. I'm sorry for that, but it seems to be true.

~~~
klodolph
You seem to be arguing that id Software's games since around 2000 haven't been
very good... but I'm not sure why, since nobody here is really disagreeing
with you on that point.

The argument here is that lots of successful games are buggy, and id
Software's games aren't buggy. Trying to run BioShock on my computer is a
nightmare (you have to mess with compatibility settings, and even then you get
no sound at all). When I played Fallout 3 it crashed about once every few
hours of play time, and that was late 2012 with no add-ons -- 4 years after
the game's release you'd expect patches to address things. Bethesda has a kind
of reputation for releasing buggy games, it seems.

