
Boeing delays 737 MAX software fix delivery - lisper
https://arstechnica.com/information-technology/2019/04/boeing-delays-737-max-software-fix-delivery/
======
astockwell
Man oh man. Having (loosely) followed this saga, it sounds horrifically
similar to every other "feature rush" I've ever seen in software development.
I know this a fiction, but I can just imagine the scene - boss comes into SWE
meeting, says "some other team implemented this thing that the higher-ups
wanted, we have to change all of our stuff to make it work, and we only have X
weeks/months" (common sin: deadlines before effort estimates). Then more
issues are discovered in dev, SWE's try to push back but are ignored (common
sin: concerns are not heard). "Scope is reduced" (common sin: this ALWAYS
happens with hard deadlines, but SHOULD NEVER happen with human safety
involvement, although it still does all the time e.g. healthcare).

It's not until some poor, senior engineer, who is probably one of their best
staff, has a stellar reputation, and genuinely cares about their job, their
team, the product, and the company ... they have to fall on the sword and TELL
management what's going happen: we're not shipping this yet, IT IS NOT SAFE.
Then that goes all the way up the chain, management at every level wrings
their hands, "how oh how did this happen?" (although they specifically are
talking about the missed deadline, not the safety concerns). It gets high
enough that feeling is overcome by senior leadership wrath, which flows back
down, and ultimately that engineer and their entire team is punished.

What a dumpster fire.

~~~
cal5k
That's not really what happened here.

If you know what's going on, MCAS is quite safe - there are two highly visible
switches that kill electric trim completely (thus disengaging MCAS) and allow
the pilots to take manual control. Activating these switches is a memory item
(i.e. pilots have to memorize the checklist) for runaway trim and would be
universal across 737 variants.

Unfortunately, it appears that both the Lion Air pilots and the Ethiopian
Airlines pilots did not recognize that this was a runaway trim issue at all
(or until it was too late). If the Ethiopian Airlines disaster turns out to
have an identical cause to the Lion Air crash, one has to question why the
pilots did not take this simple corrective action.

~~~
msbarnett
> If you know what's going on, MCAS is quite safe - there are two highly
> visible switches that kill electric trim completely (thus disengaging MCAS)
> and allow the pilots to take manual control. Activating these switches is a
> memory item (i.e. pilots have to memorize the checklist) for runaway trim
> and would be universal across 737 variants.

> Unfortunately, it appears that both the Lion Air pilots and the Ethiopian
> Airlines pilots did not recognize that this was a runaway trim issue at all
> (or until it was too late). If the Ethiopian Airlines disaster turns out to
> have an identical cause to the Lion Air crash, one has to question why the
> pilots did not take this simple corrective action.

Not quite:

While the runaway trim checklist is a memory item, the symptoms that pilots
train on in simulator presents runaway trim as a continuous event (the
"classic" runaway trim being a relay becomes stuck and the trim just extends
continuously).

So while they've trained on the correct _response_ to save the plane, they've
never actually been exposed to the symptoms that an "MCAS runaway" presents
(stick shaker on the AOA too steep side when the plane is patently not in
stall combined with speed warnings and an uncommanded nose-down that goes away
for several seconds when the stick is pulled back).

The "backs off" part is critical when combined with the speed warning alerts
and lack of continuous trim, because this makes the nose down easy to
interpret as coming from the 737's speed trim system, which has existed for
decades, and the net result is that the combination of symptoms seems to
overload/confuse the pilots.

We have, now, at least 3 documented examples of "MCAS runaway" \-- the two
downed flights, and the prior Lion Air flight in which a deadheading pilot in
the jump seat _did_ suggest to cut the automatic trim, but ONLY AFTER several
minutes of confusion in the cockpit. And even after disabling it, they _re-
enabled_ the auto trim initially and only re-cut it out when symptoms re-
occured.

Which all boils down to: The MCAS system is in no way "quite safe". Safety is
not a single line of defence, and the fact that at least 3 sets of pilots have
struggled to identify the correct response to the MCAS issue indicates that
the design of the system _is flawed_ , because no system is safe when its sole
and only fallback relies on humans _100% providing the correct response to a
never-before-seen set of symptoms_.

~~~
nas
Some comments based on my understanding of the situation. I'm an engineer and
I'm interested in the study of engineering failures. So, I have been following
this. Initially I was thinking the blame was being unfairly placed on the MCAS
system. However, now I understand more, I suspect it is a serious safety
problem.

The reason is as you say, MCAS runaway (e.g. due to a failed AoA sensor) does
not behave like a normal runaway trim (e.g. continuous trim movement). That is
key. Watching Youtube videos was helpful in understanding what the pilot
experiences. I initially thought the pilot should easily notice the trim
moving because of the large trim wheel beside them. However, in normal
takeoff, the trim is being adjusted automatically and so pilots learn to
ignore it changing. Second, I think with the 737 MAX, the sound made by the
trim wheel has been changed (removed)?

Using the stab trim cutout switches is a drastic measure (occurs as nearly
last item on memory checklist) and, as I understand it, the only way to stop a
misbehaving MCAS from doing its bad work. If the pilots are even a little slow
in figuring out what's happening, I can easily imagine that leading to
disaster. Also, if they to realize the problem and flick the switches off,
manually cranking the trim wheel the other way takes time and it takes some
serious physical strength to do it. I can imagine that if the MCAS moves the
trim fully the wrong way, the pilots might not be able to fix it quickly
enough.

~~~
msbarnett
> The reason is as you say, MCAS runaway (e.g. due to a failed AoA sensor)
> does not behave like a normal runaway trim (e.g. continuous trim movement).
> That is key. Watching Youtube videos was helpful in understanding what the
> pilot experiences. I initially thought the pilot should easily notice the
> trim moving because of the large trim wheel beside them. However, in normal
> takeoff, the trim is being adjusted automatically and so pilots learn to
> ignore it changing.

Yeah, that's the 737's Speed Trim system. Even before the MCAS, automatic trim
adjustments to adjust for the '37s behaviour at different speeds were a
constant thing in the cockpit. It seems like one possibility here is that
pilots may be mistaking or unable to distinguish (since there's no real
differentiation) the MCAS' trim adjustments from normal Speed Trim adjustments
-- and keep in mind a speed warning is going off the entire time they're
fighting the MCAS, due to the malfunctioning AOA, so they may be primed to
think "Speed Warning + Automated Trim periodically occurring == Speed Trim"
and over-focus on the stall & speed warnings and ignore the trim component.

> Also, if they to realize the problem and flick the switches off, manually
> cranking the trim wheel the other way takes time and it takes some serious
> physical strength to do it. I can imagine that if the MCAS moves the trim
> fully the wrong way, the pilots might not be able to fix it quickly enough.

It's actually worse than that. If the AOA sensor is misbehaving, then it will
never show a recovery after the MCAS applies an automatic trim adjustment,
leading to the MCAS continuing to apply trim adjustments until the trim is at
maximum.

At this point, the aircraft is basically in a nose dive, and if they hit the
trim cutout, it _can be physically impossible to manually trim the aircraft_

An Avionics engineer discusses the issue here:
[https://www.satcom.guru/2019/03/aoa-vane-must-have-failed-
bo...](https://www.satcom.guru/2019/03/aoa-vane-must-have-failed-boeing-
fix.html)

> The standard response to just hit the stabilizer cutout switches and
> manually trim is actually flawed. If the nose has been pushed down by
> significant mistrim (nose down stabilizer, nose up elevator), and as
> airspeed increases, it may not be possible to trim the stabilizer manually
> nose up without letting the elevator go to a neutral position. The reality,
> under the MCAS runaway scenario, trimming nose up immediately stops MCAS as
> well as trims the stabilizer back towards an in-trim position. At that
> point, you would be best off to cutout the stabilizer.

> Many flight crews may not know that you have to relax the elevator to
> manually trim the stabilizer if the loads is too high.

It also mentions that the techniques needed to relieve the forces enough to
make trimming manually possible are not necessarily covered in Boeing's
current manuals -- because they completely ignored the possibility of the MCAS
trimming to max downwards.

------
btmiller
In terms of project management, aviation software is one industry where the
"three-legged stool" should always, always favor quality, so I'm more than
happy to see Boeing take the extra time to get this right. Though the nose-
pusher system's entire existence is due to the aerodynamic design of the
entire MAX...which seems to be driven by the other 2 legs of the stool (i.e.
budget and time).

~~~
sgc
Airplanes are not gliders. There is no reason to not correct aerodynamics via
powered systems since that is how virtually any airplane even flies, with the
caveat being - as long as it is done well. In other words, I have no problem
with the principle of their design choice, just its execution which lacked
typical redundancy, and transparency to the industry.

~~~
pron
Is there another unstable airliner in operation? Even if it's OK, at the very
least flying one (with or without software controls) would seem to me to
require a certification by the pilots, which, as I understand, is precisely
what Boeing tried to avoid.

~~~
sandworm101
All high performance aircraft are, by definition, unstable. An airframe
optimized for one flight regime will not be as suitable in another. An
airliner like the 737 is optimized for cruise flight (30k feet, almost
supersonic) and so isn't ideal during takeoff/landing. Making it good at both
isn't an option. So we have systems to modify and adapt the cruise airframe
for low-level flying such as this antistall system.

The only really stable commercial aircraft are, perhaps, the bushplanes. They
are not fast and not fuel efficient over any real distance.

~~~
pedrocr
> All high performance aircraft are, by definition, unstable.

This doesn't follow. Not even fighter jets were inherently unstable before
modern control electronics. Using the electronics to make the flight
characteristics better is not the same as the airframe being inherently
unstable.

~~~
sandworm101
There were stable at altitude, but at low speeds those "fighter jets" became
very twitchy. Delta or swept wings are a problem when low and slow. Their
landing speeds were sometimes extreme and/or their angle of attack so high
that visibility was an issue.

Fighter jets are also, in comparison to airliners, not very aerodynamically
efficient. They can be fast but burn lots of fuel while doing so. They are
like a motorcycle compared to a bus.

------
_1tan
Does anyone here know how updates like this are delivered? Sadly the article
doesn't answer this question. I suppose it's not as simple as downloading it
over 4G or something like that.

~~~
_trampeltier
A few years ago, we had to wait one and a half hour before takeoff in a 787.
After that time they said "Sorry, the we needed a bit more time for the
software update on the plane" over the speaker ...

~~~
joncrane
I wonder if it was the uploading of mapping/route information and not an
actual update to the systems running the plane.

~~~
_trampeltier
I think not just mapping information, the 787 had some really really bad bugs
when he was new, like shutdown the whole plane (even in the middle of a
flight) if you didn't reboot for a month and such.

~~~
crocal
Usually the safety requirement will mention a mandatory reboot period that
will be somehow related to a power of 2... Damn integer counters...

------
tus87
Software can't fix a broken air-frame.

~~~
ncallaway
I don't think the air-frame for the 737 MAX is broken, though.

I definitely agree that the MCAS system should have required two AOA sensors,
and been automatically disabled when there was a disagree. I agree that Boeing
should have briefed airlines and pilots about the MCAS system, the AOA
disagree indicator, and potentially required new training.

------
dmix
Anyone know what the text is on the controls? [https://cdn.arstechnica.net/wp-
content/uploads/2019/04/The_7...](https://cdn.arstechnica.net/wp-
content/uploads/2019/04/The_737-800_Flight_Deck_3852492599-800x600.jpg)

I find it interesting how airplanes jam as much information as possible in
wherever they can.

~~~
james_pm
[http://imgproc.airliners.net/photos/airliners/9/9/2/1495299....](http://imgproc.airliners.net/photos/airliners/9/9/2/1495299.jpg?v=v40)

Google search for "737 yoke checklist" reveals a variety.

------
wibblywobbly
Computer controlled instability in a military aircraft is one thing, but
instability in a civilian aircraft another thing entirely. Someone here said
built-in instability was an accepted design feature in modern civilian
aircraft. I do not remember being asked. I wonder why? John, PPL(IR),
passenger.

------
leowoo91
As a passenger do I have choice for not riding a particular aircraft model?

~~~
alkonaut
If you can choose the airline you know what fleet the plane comes from. If you
search on the flight number on e.g Flightradar24 you can see what model they
used on the flight for several recent flights.

But in the end I suppose the reality is that even an un-fixed MAX-8 is orders
of magnitude safer than your car ride to the airport. So before picking
aircraft models, ensure you ride the train to the airport and not a taxi...

~~~
stronglikedan
> an un-fixed MAX-8 is orders of magnitude safer than your car ride to the
> airport.

Of the 346 people that died in the last two plane crashes, not a single one of
them died in a car on the way to the airport.

~~~
empath75
Impeccable logic

~~~
stronglikedan
Just as logical as the statement I was poking fun at.

~~~
alkonaut
I was assuming the person was trying to avoid dying, not trying to avoid plane
crashes in particular. Someone who wants to avoid a plane crash but would ride
a motorcycle without a helmet to the airport obviously needs to use a
different strategy.

------
village-idiot
Of all the software to not deliver in a hurry, this has to be in the top 3.

------
Havoc
No luck moving those engines back with software yet huh?

------
sys_64738
Will we get to see the changeset?

~~~
SilasX
Do we (as in, arbitrary member of the public, or American, or passenger) ever
get to see the code that goes on an FAA-approved passenger aircraft?

~~~
Cacti
No, why would you? It’s typically highly, highly proprietary.

~~~
SilasX
That's what prompted my response to someone thinking we'd see the before or
the after.

