
Resources about programming practices for writing safety-critical software - AlexDenisov
https://github.com/stanislaw/awesome-safety-critical
======
ctz
The obsession with C/C++ here is really weird. Like, take the MCO failure.
That's a classic, textbook problem that can be structurally guaranteed not to
happen with use of even a basic type system. It should be literally impossible
to confuse values of different types/units/dimensions like this in something
described as "safety-critical".

It seems like all the resources here are concerned with trying to whittle
C/C++ into an appropriate choice of tool, rather than choosing a different
tool. It seems like a 1980s-1990s mindset.

~~~
endorphone
How would a basic type system protect against incorrectly interpreting an
imperial floating point value as a metric floating point value? That seems
like an especially weak example, and fundamentally falls under the realm of
logical fault endemic of every possible programming language.

There are legitimate gripes about C/C++, especially in a space with hostile
actors an unknown inputs, but that example was particularly weak.

~~~
dualogy
> How would a basic type system protect against incorrectly interpreting an
> imperial floating point value as a metric floating point value

"Better" / richer / more refined ( _aiming /aspiring_ at least!) type systems
than C/++/Java/C#/Go/etc aim to make it ever-more "powerfully convenient,
effortless, and free-of-cost to denote eg. such different units as uniquely
distinct (yet 'compatible' when explicit conversion is finally expressly (and
visibly (and searchably)) called for) types" that all source code cannot
possibly mismatch accidentally without the compiler catching it --- however
the issue remains that we don't, and possibly _can 't_ have type systems that
also _enforce_ such styles (rather than just relying on a developer's/team's
own discipline & resolve to adhere to such convention even in the face of
deadlines & budget/schedule pressures) unless we actually totally forbid
primitives like lone (semantics ambiguous) ints, lone (likewise) floats etc.
Similar to "bool blindness", there's the general issue of "primitive/scalar-
type blindness" always lingering. Probably some PhD candidate will write a
Haskell extension for such a scenario some day soon.

Though of course the next thing the _deadline-driven developer_ will do is
construct a single "semantic" int type used for all different semantic sorts
kinds and types of "ints"......

~~~
gpderetta
For what is worth, C++ type system (which is more powerful than most,
actually) make it simple to encode units (including composite units, exponents
and ratios) and computing the exact units of complex expressions. See
Boost.Units or std::chrono design. And with zero cost abstraction and little
syntactic overhead of course.

~~~
pjmlp
The problem is getting embedded devs to use it, or when they do use it, not to
write "C with C++ compiler" code.

~~~
gpderetta
of course, but it would be even harder to get them to use any other language
other than C or C++.

------
Jtsummers
I don't have all my resources on hand right now, but off the top of my head
this book should be added:

[https://mitpress.mit.edu/books/engineering-safer-
world](https://mitpress.mit.edu/books/engineering-safer-world)

This list is barely scratching the surface of safety-critical system
engineering, but it's a start.

~~~
kqr2
There is a draft version that is freely available:

[http://sunnyday.mit.edu/safer-world.pdf](http://sunnyday.mit.edu/safer-
world.pdf)

~~~
kevinr
Parent's MIT Press link has a link to the final PDF, down the page on the
left.

------
swah
Other than the latest MISRA, I really enjoyed "Better Embedded System
Software" by Phil Koopman.

Ideally you should read it before starting your project, since it deal with
the product specification/gathering requirements phase, which is your starting
point in safety critical systems.

[1] [https://betterembsw.blogspot.com.br/2010/05/test-
post.html](https://betterembsw.blogspot.com.br/2010/05/test-post.html)

------
phelmig
Does anyone know how software quality is handled in complex supply chains,
e.g. automotive? From my point of view software is a 2nd grade citizen in
areas dominated by manufacturing and classical engineering.

I guess testing an over-the-update for a car that was build by ann OEM and
thousands of suppliers must be quite a task.

~~~
Jtsummers
It's getting better, but hardware companies tend to view software as second-
class. They think it's "easy", though they're finally accepting that it's not.
It's taken decades of fatalities, cost overruns, and missed deadlines for them
to realize this, but they're realizing it.

~~~
nerdponx
> fatalities

If someone dies because of a preventable bug in your software, shouldn't that
be considered manslaughter?

Obviously you formed a corporation in order to shield yourself from legal
action (among other things). Fine, so you personally don't get charged with
manslaughter. But in that case the corporation should be charged, and if
convicted should be sentenced to the corporate equivalent of 25 years in jail.
That would be a strong enough incentive to care about software. Of course, it
never works like that in real life.

Is this as ridiculous as it sounds to me or is my outrage misplaced somehow?

~~~
pjmlp
You are completely right.

As Hoare so elegantly described at his Turing award speech, regarding Algol
compilers, back in 1981.

"Many years later we asked our customers whether they wished us to provide an
option to switch off these checks in the interests of efficiency on production
runs. Unanimously, they urged us not to--they already knew how frequently
subscript errors occur on production runs where failure to detect them could
be disastrous. I note with fear and horror that even in 1980, language
designers and users have not learned this lesson. In any respectable branch of
engineering, _failure to observe such elementary precautions would have long
been against the law._ "

Full speech here:

[http://www.labouseur.com/projects/codeReckon/papers/The-
Empe...](http://www.labouseur.com/projects/codeReckon/papers/The-Emperors-Old-
Clothes.pdf)

------
yeslibertarian
hopefully in a future not so far away, most safety-critical code will be
formally verified, like [http://sel4.systems/](http://sel4.systems/) for
example

~~~
kevinr
Code like the Boeing 787's avionics package gets one better: the spec
specifies what the register values should be after each step of execution, and
there's a company which takes the code, puts the processor in single-step
mode, and checks.

------
danaliv
DO-178B has been replaced by DO-178C.

------
mrlyc
In addition to MISRA, I've found the safety checklist in Lutz's "Targeting
Safety-Related Errors During Software Requirements Analysis" at
[https://trs.jpl.nasa.gov/bitstream/handle/2014/35179/93-0749...](https://trs.jpl.nasa.gov/bitstream/handle/2014/35179/93-0749.pdf)
to be very useful.

------
RaiO
Is there anything like this that specifically addresses reliability in a
critical (but not "safety-critical") system?

~~~
jacquesm
Yes, Armstrong's thesis is a very good starting point:

[http://erlang.org/download/armstrong_thesis_2003.pdf](http://erlang.org/download/armstrong_thesis_2003.pdf)

~~~
amk_
The techniques in the conclusions and appendix starting about page 200 are
useful in any language

~~~
jacquesm
Yes. This goes far beyond Erlang.

------
partycoder
I have read the JSF standard. I learned a lot from reading it.

However, the JSF project has been reported to have lots of software defects.

~~~
hackuser
> the JSF project has been reported to have lots of software defects

I haven't read anything that differentiates between these two possible
scenarios:

1) Poor engineering, execution, etc.

2) The bugs expected in this software project. When I think of it this way,
I'm amazed it ever was completed (but maybe I'm thinking about it the wrong
way):

* Meet the specifications of not only three U.S. military services but also militaries and other entities in multiple national governments (with all the politics, compromise and complexity that involves).

* Invent and implement technologies to provide capabilities so bleeding edge that few people will imagine some of them for years, if not decades. There are no prior designs; nothing like it has ever been done. Part of the point is to exceed competitors' engineering capabilities by as much as possible.

* Integrate these technologies into a massive system of systems, arguably the most complex system in the history of humankind.

* The system is human-rated.

* Performance is the highest priority; there is no making easy compromises of performance for safety: Human lives, the outcomes of battles, the fates of nations, and the course of history may depend on performance.

* Accomplish this in secret, greatly restricting your access to outside resources. Will this work? You can't publish a paper and get feedback, or make a presentation at a conference.

* Accomplish this in coordination with thousands of suppliers in many countries.

* Because it's hardware and very expensive, your ability to iterate is limited. My completely amateur guess based on the above is that it's a massive, decades-long waterfall-style project.

~~~
nickpsecurity
"Integrate these technologies into a massive system of systems, arguably the
most complex system in the history of humankind."

You went a bit overboard there. There's plenty of systems probably more
complex that work fine on a daily basis. They were usually designed centrally,
though.

~~~
hackuser
Maybe, but plenty? What are you thinking of?

Also, can you distinguish between those two scenarios (which was my main
point)?

~~~
nickpsecurity
Huge systems with crazy amounts of code are what power most large companies.
The budgets are usually way smaller than for defense contracts such as we're
discussing where the profits of the contractors are assured through
corruption. The former have to get more done with less. There's thousands of
those teams, too, in just the Global 2000. Maybe Fortune 500, too.

Far as amazing examples, I'd go for the centralized, five-9's systems like
NonStop or the decentralized ones such as OpenBSD before the failure we're
discussing.

~~~
hackuser
> OpenBSD

I think there is a miscommunication. OpenBSD is as complex as the F-35? I
think the OpenBSD team would be very disappointed if that were true! OTOH, it
would make the ~10 minutes it took to install it on the laptop to my right
very impressive.

EDIT: What I meant to ask in the GP was, do you see a way to distinguish
between whether the F-35 is poorly executed, or whether we're seeing/saw the
normal bugs for a project as described above (even admitting hyperbole about
complexity, which I'm not sure of, it's still quite a project).

> where the profits of the contractors are assured through corruption

Not a major point, but these issues are too important to let pass these days,
IMHO: How much of whose profits are assured through what corruption? I'm sure
some of that goes on, as in all large institutions (including the large
companies mentioned above), but I'm not ready to assign corruption to all or
most defense industry profits with a broad brush.

Also, I rarely hear much attention given to reducing it. For example, the GOP
in Congress was working to take procurement authority out of a central DoD
office, where it was put to prevent corruption and to put real procurement
experts in charge, and into the hands of generals and admirals, people without
procurement expertise (commanding fleets and fighting wars is a much different
skill set) and with a bad track record regarding corruption in recent history
(look up the Fat Leonard scandal, as one example off the top of my head). What
I struggle to remember at this moment is whether that change passed; it was a
pet project of McCain's.

------
watwut
That is awesome, thank you.

------
throwme_1980
c++ is not considered safe for any RTOS system, in fact you won't find it used
in Aviation embedded devices (referring to the big 3 ) Tools yes, you can
higher level languages to your heart's content.

~~~
vonmoltke
Huh? The F-22A, F-35, P-8, and P-3 are all flying C++ code. Those are just the
programs I have personally touched (not necessarily the code, though). Where
did you get the idea that it "is not considered safe for any [real-time]
system"?

~~~
jordanb
The F-22 avionics are mostly Ada. F-35 is mostly C++ though. If you want a
good face-palm go read up on the "decision-makers" advocating making the
switch. I saw a quote by a general blaming the F-22's cost overruns on the
"Ada Operating System".

But don't worry, C++ is a "COTS Industry Standard" so you can bet there were
no overruns on the F-35. /s

~~~
vonmoltke
> The F-22 avionics are mostly Ada.

Yes, including the piece I directly touched. It was deemed to minor to be
worth the cost of rewriting.. Hell, when I left the program we still had an
arthritic VAX to build on, should we ever need to rebuild the code.

As for your last line, see my comment elsewhere about blaming the carpenters
instead of the tools. :-)

