

Metrorail Crash May Exemplify Automation Paradox - bootload
http://www.washingtonpost.com/wp-dyn/content/article/2009/06/28/AR2009062802481.html

======
devicenull
I'm not convinced that the ship that crashed was a good design. When GPS
fails, it seems logical to switch to dead reckoning only if there is no human
available to take over. If you've got a human crew available, it seems like a
better choice to just stop the ship and let them handle it.

~~~
lnguyen
The design issue in this case is not making it blatantly and unavoidably
obvious that the GPS had failed. The dead reckoning system should probably
also give an error range instead of keeping any appearance of the same
"pinpoint" accuracy.

~~~
ars
The system was self steering (i.e. driving the ship) - there was no one to
give an error range to. No one was watching it because it normally works so
well that there is no reason to double check it.

An error indicator is no help if no one goes to look at it.

------
ZeroGravitas
Most of these just seem like standard bad design or human stupidity. I think
there may be a "paradox" but I'm not sure it's well sketched out here.

~~~
ars
The paradox is that the better the automation, the more humans rely on the
machine.

But when it fails, humans trust it too much.

Or you can make it less reliable, and cause the human to trust it less, but
that has issues too.

It's hard to find a balance point, neither direction is good.

A solution from outside the box is needed here - make the automation good, but
in a way that the human input is encouraged.

~~~
randallsquared
Make failure immediately visible.

~~~
skolor
Make failure immediately visible _in a non-destructive way_. A nuclear reactor
that fails and explodes immediately is not a good thing. A nuclear reactor
that fails and notifies someone qualified to fix the problem immediately is a
very good thing.

~~~
randallsquared
Well, the details are application-specific, yeah. The metrorail problem could
have been "solved" by making the system refuse to allow a train to travel on
track that didn't have a currently working train sensor, but such failures are
so common in the system that the whole Metro would grind to a halt, from my
reading this summer about this problem (mostly while on the Metro, fittingly).

