
Ironies of Automation (1983) - lcuff
http://www.bainbrdg.demon.co.uk/Papers/Ironies.html
======
greggyb
> For very fast failures, within a few seconds (e.g. pressurized water nuclear
> reactor rather than an aircraft), when there is no warning from prior
> changes so that on-line working storage would also be useless, then reliable
> automatic response is necessary, whatever the investment needed, and if this
> is not possible then the process should not be built if the costs of failure
> are unacceptable.

This excerpt from the article's section 2.2 really struck me. I have seen a
lot of software projects that were built, became mission critical, and only
then experienced their first significant downtime. I have rarely seen
organizations tackle the threat of failure first and design the system from
that point of view.

------
Jtsummers
This is a pretty good article. I like it, covers a lot of potential problems
and offers good questions to ask about your automation activities and their
impacts to overall capabilities.

> Perhaps the final irony is that it is the most successful automated systems,
> with rare need for manual intervention, which may need the greatest
> investment in human operator training.

This has played out, a number of aircraft mishaps are a consequence of this.
[0] is a recent article discussing this.

[0] [https://hbr.org/2017/09/the-tragic-crash-of-flight-
af447-sho...](https://hbr.org/2017/09/the-tragic-crash-of-flight-af447-shows-
the-unlikely-but-catastrophic-consequences-of-automation)

~~~
lcuff
Yes, I've read quite a bit about AF 447. As someone who has taken some flying
lessons, I know that in early aircraft, the controls were mechanically linked
together. There is no way one pilot could push the pitch control forward and
the other could pull it back. But they could on the Airbus with a 'glass
cockpit' being flown. The result: Software averaged the result, and the
elevator stayed neutral, with no indication to either pilot that this was
occurring. Tragic poor design.

