
Rules for Writing Safety Critical Code (2006) - mpweiher
http://spinroot.com/p10/
======
michaelt
Rules like avoiding recursion and loop upper bounds are partly justified by
helping code analyzers to prove executions are bounded.

Does anyone have experience performing formal static analysis on code? And if
so, are there any open-source examples of the state-of-the-art tools being
used on nontrivial problems?

I've been looking at this recently, but I've never seen a proof of reasonably
standard function like, say, printf. Most of the attempts at formal proof I've
seen seem quite complex for very simple functions [1] and the next step up
seems too complicated for me to follow [2]. I'm familiar with heuristic-based
tools like clang analyser, coverity and flexelint - but none of those claim to
produce any sort of proof. I'm also aware of MISRA but again, that doesn't
claim to provide any sort of proof.

Those of you who are using formal methods: Is what I've seen really the state
of the art? Or are there powerful tools I've overlooked?

[1]
[https://github.com/Beatgodes/klibc_framac_wp/blob/master/src...](https://github.com/Beatgodes/klibc_framac_wp/blob/master/src/klibc/string/strnlen.c)
[2]
[https://github.com/seL4/l4v/blob/master/lib/Monad_WP/Strengt...](https://github.com/seL4/l4v/blob/master/lib/Monad_WP/Strengthen.thy)

~~~
wizeman
Take a look at dafny [1] and F* [2]. Both are open-source and it appears that
both of them are able to prove code to be correct with considerably less proof
code than traditional interactive theorem provers (such as Coq or Isabelle),
mostly thanks to the use of an SMT solver (Z3, in both cases) to aid in
proving.

Both languages are being used in Project Everest [3], which is building a
formally verified HTTPS stack, including a formally verified TLS
implementation [4].

Unfortunately, there isn't a lot of documentation / beginner guides about
these languages out there, but with the little there is, I was still able to
start proving non-trivial (but still relatively easy-to-prove) code with them,
even though I have no formal verification background whatsoever.

Just as a tip, I found the Dafny tutorial to be a lot easier to follow than
F-Star's, but from what I could gather, F* appears to be slightly more
general/sophisticated in what it can prove.

[1] [https://github.com/Microsoft/dafny](https://github.com/Microsoft/dafny)
[2] [https://www.fstar-lang.org](https://www.fstar-lang.org) [3]
[https://project-everest.github.io](https://project-everest.github.io) [4]
[https://mitls.org](https://mitls.org)

~~~
Animats
_" F star is an ML-like functional programming language aimed at program
verification. Its type system includes polymorphism, dependent types, monadic
effects, refinement types, and a weakest precondition calculus. Together,
these features allow expressing precise and compact specifications for
programs, including functional correctness and security properties. The F star
type-checker aims to prove that programs meet their specifications using a
combination of SMT solving and manual proofs. Programs written in F star can
be translated to OCaml or F# for execution."_

And you wonder why you're not getting any traction.

Dafny, though, seems to have the right idea.

------
ti32x
I work on software that goes to space. I believe we and NASA use the MISRA
coding standard, which is effectively this but taken to extremes. If you
actually want to create safety critical code I would recommend that you
familiarize yourself with the MISRA ruleset and DO-178B.

~~~
nstj
Space software? Nice! Going to check out MISRA now.

FWIW the initial page just reflects the JPL publication "The Power of Ten –
Rules for Developing Safety Critical Code" by Gerard Holzmann [0]

[0]:
[http://spinroot.com/gerard/pdf/P10.pdf](http://spinroot.com/gerard/pdf/P10.pdf)

~~~
mos_basik
Gerard Holzmann is the main author of the formal software verification tool
"spin" as well as the owner of spinroot.com. I took a class on formal
verification under him at Caltech.

~~~
4ad
As an anecdote, spin was used in Plan 9 to validate the kernel scheduler and
the IL network protocol. It was also used to validate a very early version of
the Go scheduler.

------
4ad
Full 31 rules of NASA JPL C coding standard: [http://lars-
lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf](http://lars-
lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf)

Previous discussion:
[https://news.ycombinator.com/item?id=4339999](https://news.ycombinator.com/item?id=4339999)

------
mrlyc
When I was writing safety-critical embedded programs for medical equipment and
an air traffic control system, I found MISRA
([https://en.wikipedia.org/wiki/MISRA_C](https://en.wikipedia.org/wiki/MISRA_C))
and Lutz's "Targeting Safety-Related Errors During Software Requirements
Analysis" ([http://trs-
new.jpl.nasa.gov/dspace/bitstream/2014/35179/1/93...](http://trs-
new.jpl.nasa.gov/dspace/bitstream/2014/35179/1/93-0749.pdf)) useful.

------
zwieback
I would add "use static type checker from day 1". It's amazing how much stuff
Coverity discovers and it can help with the clutter of rechecking invariants
of nested functions.

------
achou
Rules like this are much more useful if they're enforced. That's why efforts
like the C Secure Coding Rules[1], which were designed with the capabilities
of reasonably state of the art analyzers in mind, are a good idea. Like any
standard designed by a committee, there are some unfortunate compromises.

Many commercial (and open source) static analyzers will have checks that are
hard to codify in rules that are digestible by humans. For example, most
analyzers that use sophisticated interprocedural analysis will not be easily
described as a guideline for humans, unless those humans are compiler
engineers. This is especially true for analyzers that have heuristics built in
to minimize false positives, which is most often a real-world requirement.

[1]
[https://www.securecoding.cert.org/confluence/pages/viewpage....](https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=140705863)

------
huhtenberg
An opinionated and needlessly specific list. For example, this

    
    
      Use minimally two assertions per function on average
    

should've been

    
    
      Assert your invariants
    

And this

    
    
      Do not use goto statements
    

is a controversial advice at best, because there are several well-established
goto-based patterns that produce a much cleaner code.

~~~
kccqzy
I would also say perhaps

    
    
        avoid recursion
    

should be revised to

    
    
        avoid non-tail-call recursion
    

Because the justification to avoid recursion is basically avoid running into
the limit of stack, and tail-call recursion uses no stack space at all. Of
course you still might end up with situations where the function does not
terminate, but simple loops can do that too. In relation to Rule No.2, it is
also possible to prove an upper bound for recursive depths.

~~~
hvidgaard
Recursion is recursion. Tail recursion is optimized away by the compiler
except when it isn't (I've seen bugs with this exact setup more than once),
but now you have to verify that the compiler actually does this. It's simpler
to just avoid it all together unless you want to formally verify the recursion
depth.

~~~
4ad
Apart from that, TCO destroys information that is valuable for debugging
(especially for non-recursive tail calls), exactly what you want to avoid in
safety-critical software.

~~~
pja
Manual conversion to a loop also destroys the same information, so it’s a wash
either way on that front.

~~~
4ad
You can't convert arbitrary tail calls to a loop. You can only convert
_recursive_ tail calls to a loop. The information loss associated with non-
recursive tail calls is great. It's true that recursive tail calls lose less
information, but it is not true that it's equivalent to the information lost
in a loop. In a loop you always have access to {head, current, tail} while in
a tail call you _might_ (subject to algorithm and compiler optimization) only
have access to {current, head|tail}, depending if the loop goes forwards or
backwards.

~~~
spc476
But I've found that non-recursive tail calls are great in handling state
machines. I wrote a TFTP client and server in Lua [1] and the code was both
much cleaner and more reusable between both client and server. Granted, Lua
does proper TCO (as long as you properly code the return statement) and the
lack of stack trace wasn't much of an issue there.

These rules are mostly geared towards C, which doesn't guarantee TCO at all.

[1] Partly as way to learn Lua and TCO, and as a way to manage Cisco router
config files in a version control system.

------
Kurtz79
Nice.

These are reasoned, common sense measures, most of which I have seen in one
form or another in the coding guidelines of the companies I have worked for as
an embedded SW engineer.

Some of them are easy to enforce (no dynamic memory allocation, no recursion),
other less so (using asserts, limit function size).

The worst is adding your rules-compliant code to messy and bloated legacy code
:)

------
Tepix
It's interesting that testing is not among the rules and not among the
comments so far.

(The details for rule 5 and 8 mention testing)

------
atsaloli
See also my write-up on Dr. Holzmann's presentation at USENIX Hot Topics in
System Dependability mini-conf 2012,
[http://www.verticalsysadmin.com/making_robust_software/](http://www.verticalsysadmin.com/making_robust_software/)
and the full presentation video at
[https://www.usenix.org/conference/hotdep12/workshop-
program/...](https://www.usenix.org/conference/hotdep12/workshop-
program/presentation/holzmann)

------
swah
What have you guys been using to track requirements? I had a bad experience
adapting Redmine for a project, and wouldn't want to repeat that in a next
project.

~~~
ti32x
I have heard that Polarion is an _ok_ requirements management tool for
software, but it is expensive.

DOORS is good for tracking and tracing requirements, but from what I can see
doesn't do code parsing. This may have changed. However, my opinion of IBM
software is irreparably broken so I don't have a huge amount of trust in it.

We use Reqtify to glue requirement documents to code. It also sucks, but I
think it's still powerful. You just need to set up some regexes which can
recognize your requirement identifier style and you're good to go.

Whoever comes up with a solution to glue DOORS requirements to Jira issues to
code seamlessly is going to be fairly rich.

~~~
rubidium
I've done a bit a research on this. Company uses DOORS and hates it. We've
evaluated JAMA and Polarion (both expensive) and both are better than DOORS.
JAMA has a nice JIRA integration. Polarion can be integrated it seems but
untested.

------
svec
Funny timing: I just suggested those guidelines, and several related ones, in
a post over here: [http://embedded.fm/blog/2016/11/3/resources-for-software-
bes...](http://embedded.fm/blog/2016/11/3/resources-for-software-best-
practices)

(Check out "Rules for Defensive C Programming" by Dinu Madau for starters.)

------
aetherspawn
I wrote my capstone project a week ago on rolling out Simulink for
automatically generating SC code en masse and minimising hand written code and
manual translation.

I referenced the paper mentioned in my literature review

I would not link it here to the world to see as it is just an undergraduate
project and my research did not yield a highly quantitive result, but I am
applying the technique on a project right now with good success

------
PaulKeeble
10 Rules isn't going to cut it really, its a lot more detailed than that. Here
is the Spark reference manual (a standard for Ada that has been the industrial
standard for decades):
[http://docs.adacore.com/spark2014-docs/html/lrm/](http://docs.adacore.com/spark2014-docs/html/lrm/)

~~~
4ad
Yeah, the point of these rules (and the 31 JPL rules) is concision. Your
programmers will read a 10 page coding standard, and might remember it. Nobody
is going to remember hundreds of rules and hundreds of pages. They won't even
read it.

In his talks Holzmann always makes the point that a coding standard is a hard
sell for most people, and JPL hadn't had any success until they could come up
with something simple and concise.

------
TickleSteve
These are fairly typical modern embedded s/w guidelines, safety critical code
is much more rigorous specifically on the design & requirements side.

------
vbtemp
Why not just call it "Rules for Writing Code"?

Who would ever, for example, consider checking the return value of some
function but then think "ah, this isn't safety critical, I'm just skip writing
a quick _if_ statement and then leave this as a place that mysterious bugs can
arise"? Same thing with assertions.

~~~
pjc50
Have you met programmers?

Less snarkily, there are billions of lines of production code that don't check
return values for malloc(), printf(), and so on because those failures are
extremely rare and difficult to handle sensibly.

Banning dynamic allocation would ban large areas of programming language -
Lisp, Python, Javascript, and so on.

~~~
kazinator
Even if you check the return value of malloc, that doesn't help when virtual
memory is overcommitted. Malloc gets a valid pointer from mmap, but accessing
the memory later crashes.

A system OOM may actually show a soft behavior, whereby the whole system
thrashes more and more until it grinds to a halt. Whatever your program does
is futile and irrelevant.

To QA the program for how it handles null out of malloc, you need to tweak the
setup: disable overcommit and simulate low memory conditions. (Or even just
switch to an alternative allocator that simulates failures.)

Checking the result of malloc for null is of course good for maximal
portability: your code can be reused where the check actually means something.

Checking for null and not _testing_ that logic is almost no better than not
testing at all (in keeping with the general hypothesis that untested code is
junk).

------
BerislavLopac
Ask HN mods: How is this URL different from the one in
[https://news.ycombinator.com/item?id=12864032](https://news.ycombinator.com/item?id=12864032)?
I was under impression that submissions are unique...

~~~
detaro
They are not, it's even explicitly allowed to repost links that didn't see
activity as long as it is not overdone. I think currently the system only
rejects perfectly identical URLs for a few hours.

