
Max Crashes Strengthen Resolve of Boeing to Automate Flight - Bostonian
https://www.wsj.com/articles/max-crashes-strengthen-resolve-of-boeing-to-automate-flight-11577816304
======
wutbrodo
My knee jerk reaction to this is "more automation after an automation failure?
Huh?". But whenever I think someone is being that stupid, it's always worth a
second look, and it turns out the reality is more complex.

This article[1] linked off of the OP goes into detail about their crashes. It
describes a design philosophy that pushes robustness fully onto the pilot,
expecting him to flawlessly stitch together independent automated systems and
unreasonably rapidly interpret and react to their error modes. In the case
described in the article, separate automated systems had contrasting views of
the world, and shouted their contradictory errors at the pilots, relying on
them to be able to instantly reconcile these models and figure out what the
root problem was. (In this case, a specific sensor failure made a single
system think the plane was pointed up, and this system made the isolated
decision to push the nose downwards while warning that the nose was too high.
Naturally, other systems simultaneously complained about the nosedive).

One solution, of course, is to remove automated systems from planes. But
assuming they provide any value (I don't think this is in dispute?): another,
more reasonable strategy is to commit more fully to an automated system design
that is intentionally fault-tolerant, robust to system and sensor errors, and
has a more complete view of the plane's world. Instead of requiring the
overall state of the meta-system to be lossily communicated to humans in a
time-critical emergency, the newer designs are intended to use all the
information available to the plane's systems to reconcile subcompoments'
contradictory view of the world. Note that this doesn't preclude human control
at all: if anything, it makes pilot control _safer_, since the system can
provide a coherent view to the pilot in a way that's currently impossible.

I don't know enough about large aircraft operation to fully comment on the
wisdom of this approach, but it's substantially more complex than "auto system
failed, why would they expand use".

[1] [https://www.wsj.com/articles/the-four-second-catastrophe-
how...](https://www.wsj.com/articles/the-four-second-catastrophe-how-boeing-
doomed-the-737-max-11565966629)

~~~
Traster
I think there's a different way to explore this idea. I don't think automation
had anything to do with the crashes. The problem was that the pilots weren't
trained to know what tools the planes were equiped with and how to use them.
Those crashes are like when I accidentally open a file with vim. There's
nothing wrong with me, there's nothing wrong with vim but how the hell am I
meant to know to hit esc :q? The lesson is orthogonal to automation. The
lesson is that the pilots need to know what each system does and how to
operate it. If the resolve of Boeing is to push forward with automation
without taking the role of the pilot seriously, then this is doomed to
failure, because automated systems will fail, and pilots will have to step in.
It'll be rare, but that's what pilots are there for. Indeed, the more you
automate things, the more liability you're inviting on yourself, which should
be worrying for Boeing.

~~~
paranoidrobot
> Those crashes are like when I accidentally open a file with vim. There's
> nothing wrong with me, there's nothing wrong with vim

I disagree, strongly. The root cause of the issue isn't pilot training.

The root fault is a design issue.

Without the existence of MCAS, these crashes would not have occurred.

Why? Because they would never have been built.

Why? Because the 737 MAX 8 design would not have received approval to be
incorporated into the 737 type certificate.

Why? Because the MAX 8, without MCAS, has different handling characteristics
during certain phases of flight than the original 737.

Why does it handle differently? Because it has larger and more powerful
engines, located differently on the wing, and these changes cause the aircraft
to want to pull up more when climbing.

I don't know that the MCAS's programmed behaviors have a good analogy, I can't
think of one right now.

~~~
bkdbkd
Those crashes are like: You tell your machine to open the file with Emacs, but
this new distro has unbeknownst to you (and you find out later, undocumented)
symlinked the emacs command to a version of vim with a emacs-like overlay,
because of some issue they had. The file crashes vim, but in a crazy memory
overflow before he dies manages to take out the system, causing spurious disk
writes which corrupts your boot drive.

\- The MCAS system. Got bad information and did a very bad thing. It was the
entire time operating as it was designed. The fact that the astronauts were
not given the information on how to turn HAL off in the event he goes berserk
isn't really their fault.

Just to be clear - The MCAS system was actively trying to fly the plane into
the ground. It thought it was trying to save the plane from a stall. It was
wrong. The pilots attempted to override the system. They could not. They
attempted to outfly the system. They could not. They could have killed the
system. Boeing did not teach them how.

------
turndown
For the life of me, I have absolutely no idea how you can look at what has
happened with the Max and believe that more entrenched automation is the
answer.

~~~
Retric
The argument goes something like this:

It’s the disconnect between pilots mental models of what’s going on and what’s
actually going on that caused the Max‘s and several other related crashes on
different aircraft. As such software designed based on how the system actually
works may actually be both simpler and safer than increasingly abstracted fly
by wire systems trying to safely intuit what the pilots want.

An example from 10 years ago: [https://www.fastcompany.com/1669720/how-lousy-
cockpit-design...](https://www.fastcompany.com/1669720/how-lousy-cockpit-
design-crashed-an-airbus-killing-228-people)

~~~
bkdbkd
While true. That disconnect is not related to the automation system
consistently doing the wrong thing. The 'thing the pilots didn't know' enough
about was how to turn off the automation. Which was what saved the Lion Air
flight the day before was that a jump seat pilot knew how to disable the
automation. Increasing the automation in this scenario, does the opposite of
what they hope.

~~~
jki275
The pilots did know to turn off the automation, and they did exactly that. The
problem was that they didn't decrease thrust, and then they turned the
automation back on in an attempt to regain control because they couldn't move
the control surfaces because they were going too fast.

~~~
bkdbkd
The turn off the MCAS system button on the 737 Max the pilots were using, does
_not_ in fact turn off the MCAS system. It is a "Pause the MCAS system button"
Even worse, After a few moments, the MCAS system reactivates and then adds
even more down trim. So pressing the button, not only does not turn off the
system, actually triggers the plane to make the situation worse by adding more
down trim.

When the FAA certified the 737 Max the MCAS Software was able to move the
horizontal tail a max of 0.6 degrees. (out of a total of 5 degrees) The
software flying on those planes was in fact able to move the horizontal tail
4.17 times that amount: 2.5 degrees. They never told anyone this changed. The
system went from being able to control 12% of the tail range to 50% of its
range.

Those pilots were pulling up on the nose of the aircraft with their feet
pressed against the controls. Directing the aircraft with clear intent and
skill. The 737's automation had unbeknownst to them and with their knowledge
of the aircraft - un-overridable, tilted the horizontal tail into a position
that would, under any operating condition direct the nose of the craft into
the ground.

~~~
jki275
They pulled the breakers for MCAS. That was the correct action. The problem
was that then they turned them back on because they flew the plane into a
situation where they could not trim it anymore by hand. When they turned the
breakers for MCAS back on, it flew the plane into the ground.

------
clon
Seems like a passive-aggressive PR scheme to still try to indirectly lay blame
on the dead pilots. Despicable.

~~~
redis_mlc
FYI: The Indonesian accident investigation report found the pilots, airline
mechanics, and Boeing roughly equally at fault for the Lionair crash.

But it's reasonable to hold Boeing to a higher standard of excellence as an
experienced aircraft mfg.

I'd like to learn more about the 31 pages of missing maintenance logs. That's
a very big deal.

[https://www.npr.org/2019/10/25/773291951/pilots-ground-
crew-...](https://www.npr.org/2019/10/25/773291951/pilots-ground-crew-share-
blame-for-lion-air-737-max-crash-indonesian-report-says)

------
logfromblammo
"Our automated system crashed several planes, when the pilots were not able to
disable it before it could erroneously trim the vehicle into terrain.

"Clearly, we need to remove the human pilots and double down on the
automation."

[This is where the people using shallow analysis go nuts. But wait.]

Because the flawed system was only installed for the purpose of avoiding the
expense of retraining human pilots to a new model of aircraft! In a fully-
integrated automated piloting system, there is no separate trim-compensating
subsystem. The trim parameters are plugged directly into the aircraft model,
and the automatic pilot has access to all the sensors in the craft to make its
trim decisions, instead of just the angle-of-attack sensor.

Instead of re-certifying hundreds of pilots, you just validate one new
aircraft data model.

~~~
bkdbkd
And with the design that both Boeing and the FAA certified to fly, that system
would simply fly the vehicle into terrain. In this case it is exactly the
automation that is at fault, not a pilot misunderstanding or mistraining. The
automation system was designed to prevent a pilot from doing something she
might have been familiar with doing while flying the old design on the new
design. It was designed to keep the pilot from putting the plane in a bad
state for flying. That entire system made an error that cost 347 people their
lives.

What is painful, is that the pilots recognized that the system was in error,
and attempted to correct it, but were unaware that it had the capability to
override them. They were unaware because the designers intentionally hid the
mechanism. That is made the mechanism hard to see, understand, and needing
special knowledge to disable. The designers had not considered all failure
modes, but acted as if their implementation was failure-proof and never to be
tampered with.

The question in this case is how can one verify something like an automated
aircraft system? And more importantly, if there is a technique or practice to
assure the system is valid, is the company trying to build it mature enough in
its engineering practices to follow it properly?

~~~
logfromblammo
Every aircraft-building company is mature enough in its engineering practices.
Not every company is ethical enough in its management practices.

------
mattrp
I don’t think it’s a mistake that the headline is intended to make Boeing
sound uncaring and wrong-headed. The headline is intended to provoke
discussion. However, if you pick almost any problem in any industry, the
solution invariably is less human / more automated. It’s not hard to see how
we got here - I can imagine when Boeing asks pilots what they need the answer
is something like reduce distractions so we can fly. How do you do that? More
automation. Take almost any operational role - everyone is asking to reduce
distractions and highlight things that need focus. I’m not arguing to absolve
Boeing from fielding the Max. But I can also see a logical argument here.

------
sansnomme
There are tons of math majors produced every year. Would be a viable business
model to offer factory-style program-correctness-proving-as-a-service. Current
proof services are mostly niche static analysis for stuff like Ethereum
contracts and occasionally a wannabe legal-tech company. I think aerospace
could use more proofs. You can't use Ada for everything, eventually you will
have to code in C/C++.

~~~
jandrese
Algorithmic correctness won't necessarily help you if the root problem is bad
sensor data.

~~~
Gibbon1
Everyone knows what bad sensor data is until they have to write software to
identify it.

~~~
jandrese
That said, the way Boeing designed the MCAS is inexcusable. They actually had
two sensors available, plus a full suite of other instruments and pilot inputs
to cross reference and their solution is to only pay attention to a single
sensor.

It was engineered like it was version 0.1 beta. When you realize that the -MAX
crashes were very close to single fault accidents it's even more crazy.
Normally to crash an airplane you need to thread the needle through multiple
failure states thanks to redundancy and safe design. Simple single issue
crashes are almost unheard of in the modern era thanks to decades of safety
engineering. Aggressive cost engineering is returning us to the perilous air
travel of the 40s and 50s.

------
ng7j5d9
Does anyone else miss the days when a single person could grok the design of a
plane? Or a piece of software, or a car, or whatever?

Now every new car needs to have adaptive cruise control and anti-lock brakes
and stability control and Apple CarPlay and over-the-air updates and who knows
what else, so there are a ton of things that are hopefully well-behaved black
boxes that people hopefully understand mostly correctly, but who knows what
could go wrong in a rogue OTA update or something.

I'm not all that familiar with the aerospace industry, but this feels the
same. Of course all of these components will be tested, but if no one
understands the whole package, who knows what fearsome edge case awaits when
Flight Safety Program 32 gets in a fight with Flap Threshold Control Program 3
or whatever.

~~~
mrtksn
I think in those days they needed to have an Engineer in addition to the
pilots to fly it and the cars were very unsafe and unreliable.

~~~
ng7j5d9
That's a great point. Air travel has gotten measurably safer and people
survive car accidents at much greater rates.

I guess there's just something more "understandable" about a plane crashing
due to an engine failure rather than software fighting the pilot. Or it's more
understandable when someone died in a car crash because they couldn't control
a skid rather than Autopilot driving into a concrete barrier it avoided
previously because something inscrutable in the programming changed.

Perhaps we're heading towards a world with on average better, but less well-
understood, outcomes.

~~~
mrtksn
I agree but we can still reason with the current systems in use. MCAS crashing
planes is not that different from metal fatigue IMO. In both cases, we can
look back and say "Okay, that's how it happened and we screwed up because we
made a design mistake" which is very different from Autopilot driving into
concrete barriers because that's a very high-level mistake.

We wouldn't have a problem with the software not recognising barriers or
mistake road lines correctly as long as this software doesn't claim to be able
to handle high-level decisions and fail on something extremely obvious to
humans leading to great damages.

------
pstrateman
Their automated systems caused two crashes and their solution is to automate
more systems?

Have they literally gone insane?

~~~
wutbrodo
This article[1] linked off of the OP goes into detail about their crashes. It
describes a design philosophy that pushes robustness fully onto the pilot,
expecting him to flawlessly stitch together independent automated systems and
unreasonably rapidly interpret and react to their error modes. In the case
described in the article, separate automated systems had contrasting views of
the world, and shouted their contradictory errors at the pilots, relying on
them to be able to instantly reconcile these models and figure out what the
root problem was. (In this case, a specific sensor failure made a single
system think the plane was pointed up, and this system made the isolated
decision to push the nose downwards while warning that the nose was too high.
Naturally, other systems simultaneously complained about the nosedive).

One solution, of course, is to remove automated systems from planes. But
assuming they provide any value (I don't think this is in dispute?): another,
more reasonable strategy is to commit more fully to an automated system design
that is intentionally fault-tolerant, robust to system and sensor errors, and
has a more complete view of the plane's world. Instead of requiring the
overall state of the meta-system to be lossily communicated to humans in a
time-critical emergency, the newer designs are intended to use all the
information available to the plane's systems to reconcile subcompoments'
contradictory view of the world. Note that this doesn't preclude human control
at all: if anything, it makes pilot control _safer_, since the system can
provide a coherent view to the pilot in a way that's currently impossible.

I don't know enough about large aircraft operation to fully comment on the
wisdom of this approach, but it's substantially more complex than "auto system
failed, why would they expand use".

[1] [https://www.wsj.com/articles/the-four-second-catastrophe-
how...](https://www.wsj.com/articles/the-four-second-catastrophe-how-boeing-
doomed-the-737-max-11565966629)

~~~
tehjoker
The auto system failed because they a) were band-aiding a flawed mechanical
design b) under-designed the compensating software (only provided a single AOA
sensor by default) to obscure the fact the plane was significantly different
to avoid training costs. These are management problems, not technical
problems. More automation would be subject to the same issues.

~~~
wutbrodo
Crap I meant to make the above comment as a top level comment, not a
response.. Somehow it came through twice.

Anyway, in response to your comment: perhaps I'm conflating these, but a
commitment to robustness seems like it's included in the newer philosophy? It
just extends that notion beyond the sensor-level to the system level.

Sure, you can frame an inadequate emphasis on robustness as a "management
problem" instead of a technical one, but that seems equally applicable to the
"management solution" being put forth, no?

~~~
tehjoker
I would say that framing a business problem in engineering terms will lead to
an unclear understanding of the problem. The executives were attempting to
thread the needle of a potentially business killing competition with Airbus
and were willing to make compromises to safety to save their skins. No
engineering methodology can save you from motivated reasoning and systems
level problems like that because management will force exceptions to whatever
process is in place.

The only thing that would be an appropriate counterweight is if the
engineering staff refuses to put a defective design into production. A robust
engineering process can make violations and exceptions clear, but if the staff
still does what management says, it won't help.

------
philwelch
To be fair, I don't think more automation is a bad idea here. The bad idea is
Boeing continuing to design and manufacture aircraft.

------
norto_flight
Way to introduce a false dichtonomy into the public conversation to save the
company from scrutinity. More or less automation is irrelevant to the
conversation and even with all goodwill, was never the motivating factor for
Boeing designing the MAX system as it did. Don't be blended by this PR spin!

------
bhouston
Fully automated flight is the far future. The question is how near future is
it?

I think Boeing and Airbus have such long delays between project initiation and
delivery that planning now to deliver a fully automated flight system in 10
years is likely in the realm of reasonable?

------
ilaksh
I think more or better automation makes sense. However, in this case, they are
just trying a new tactic to deflect blame.

The fundamental problem was that they had new requirements and instead of
allowing the engineers to create a new design, management wanted to save money
so insisted the engineers shoehorn the new requirements into the old design
with hacks resulting in a plane that was fundamentally unstable and ending
with hundreds of people dying.

The solution to that problem is to have several executives from Boeing and the
FAA in prison. Then break Boeing down and rebuild it as an engineering
organization.

After that, yes, continue increasing the automation.

------
newnewpdro
Somebody please take away Boeing's shovel already.

------
senordevnyc
My initial thought was it’s crazy to reach for more automation in the wake of
automation-involved fatalities, but if I step back and think about it, they’re
not wrong. There’s no way that piloting aircraft is going to become a more
manual process. More automation is a given in the future. And since so much of
the dangers of automation seem to be in the gray zone where it’s an opaque mix
between human and computer, it makes sense to cut humans out of the loop as
soon as possible.

~~~
_bxg1
I would've agreed with this stance _before_ the Boeing disaster. Granted there
are many things that could be (could have been) done to address the fragility
of the 737 Max's automatic system, but the point is _they weren 't_. And I
don't know who in the world is going to trust Boeing to take over _more_ parts
of the piloting process, because they'll probably be executed just as badly.

