
Safety-Critical Software: things every developer should know - fagnerbrack
https://smallbusinessprogramming.com/safety-critical-software-15-things-every-developer-should-know/
======
raxxorrax
The reality is that technical solutions can always fail. In medicine there is
emphasis on giving human actors a kill switch (no, luckily not for the
patient, phew...).

Of course humans can make errors too but are still superior to do sanity
checks. From what I understand about this issue, pilots were not able to
overrule the system that had wrong sensory information.

But to defend certifiers in such cases, their primary role is to ensure
engineers sit down and think about all possible risks. They cannot ensure that
every risk is accounted for. That is beyond anyone's ability. The primary role
is to ensure this process isn't skipped completely and is executed with due
diligence.

Of course a pilot can have a heart attack while the copilot is more interested
in the stewardess at that point, but a kill switch doesn't have to be a panic
button, but there can be a conscious procedure to activate a state for
handling exceptions.

I think with today's technology it is still important to have such a mechanism
to give back control.

