
Boeing software under scrutiny as Ethiopia prepares crash report - amaccuish
https://www.reuters.com/article/us-ethiopia-airplane-software/exclusive-boeing-software-engaged-repeatedly-before-crash-sources-idUSKCN1RF0YU
======
Someone1234
I strongly suggest people skip this re-hosted article, and instead read the
Reuters one here:

Title: Boeing software under scrutiny as Ethiopia prepares crash report

[https://www.reuters.com/article/us-ethiopia-airplane-
softwar...](https://www.reuters.com/article/us-ethiopia-airplane-
software/exclusive-boeing-software-engaged-repeatedly-before-crash-sources-
idUSKCN1RF0YU)

I have numerous problems with Arstechnica's re-write, but suffice to say that
nuance is lost and questionable conclusory statements added.

I will say in the case of BOTH articles, it sounds like we need to wait and
see. Neither article has an answer for how auto-trim was re-actived, be it the
pilots or something else. The answer to that question may have wide reaching
implications.

~~~
acqq
> Neither article has an answer for how auto-trim was re-actived, be it the
> pilots or something else. The answer to that question may have wide reaching
> implications.

I can imagine that being irrelevant in this case: _Even if_ the pilots
activated it, _it doesn 't mean_ they must have made any error, see my other
comment here for the details.

In short, if the pilots turned the switch off, turning off much more than just
the MCAS (as there's no switch "just turn MCAS off, keep the rest on"), but
then the plane was already not possible to save (it was only 1000 ft above the
ground!) and they turned the switch back on, hoping for obtaining more
power/possibility to trim, and the faulty MCAS kicked in again (despite being
the reason for the switch cutoff before) and even sped up the crash, then the
pilots in fact did absolutely all that was possible to do.

And I can even imagine that the code which included the promised "fix" by
Boeing until just a few days ago still had the built-in "obvious to implement
by default" logic: "oh, I'm booting, now I can point the plane to the ground
again because the sensors say so and I'm completely ignoring the fact that the
pilots already turned me off shortly before" and that that was the reason the
fix release is now delayed. But also note that that plane, the way it is
constructed, certified and sold simply needs the MCAS and can't reliably fly
without it, otherwise Boeing's fix would have been turning the feature off
completely.

~~~
SolaceQuantum
_"...if the pilots turned the switch off, turning off much more than just the
MCAS (as there's no switch "just turn MCAS off, keep the rest on"), but then
the plane was already not possible to save (it was only 1000 ft above the
ground!) and they turned the switch back on, hoping for obtaining more
power/possibility to trim, and the faulty MCAS kicked in again (despite being
the reason for the switch cutoff before) and even sped up the crash, then the
pilots in fact did absolutely all that was possible to do..."_

If this is indeed what is announced, I sincerely hope the responsible parties
for making the decision to go through with MCAS in this way are paid well in
jail time.

~~~
KMag
> If this is indeed what is announced, I sincerely hope the responsible
> parties for making the decision to go through with MCAS in this way are paid
> well in jail time.

I'm sorry, but your wording sounds like you're advocating taking revenge on
engineers / managers for poor decisions made in good faith. That's barbaric,
and I hope we as a society move beyond this sort of communal revenge. We'll
all be psychologically better off if we encourage more healthy ways of seeking
relief from tragedy.

~~~
WalterBright
Such a proposal would mean no more new airplane designs. Note that modern
airliners are much safer than their predecessors.

An awful lot of mistakes are only "obvious" in hindsight; engineering is full
of such.

~~~
acqq
> Such a proposal would mean

But who actually proposed what here?

~~~
WalterBright
Read the parent of my post.

------
CodeWriter23
Note to self: eventual consistency good for web/mobile apps, bad for avionics.

~~~
ummonk
Depends on the app. There are a lot of web / mobile app use cases where
stronger consistency is needed, e.g. banking.

~~~
cjbprime
You'd think, but banking (e.g. the ATM network) is _all about_ choosing
availability over consistency. If the ATM network can't reach your bank to
discover your balance, it'll just let you withdraw and figure your bank will
sort things out later after it comes back online.

------
mindslight
Pure handwaving speculation here, but this incident is feeling like a result
of the limitations of fly-by-cable - specifically taking the tried and true
design of fly-by-cable, but augmenting it with motors _in parallel_ to human
control.

System integration is a known common source of errors, and what is less
formally specified than the effective limits of a human being? If the human(s)
are _fully_ in the loop, then the buck stops with them and we know how to
socially characterize and educate about failure modes. A pure-computer portion
of the loop is characterizable by straight engineering.

But rather, we've got human pilots physically fighting against electrical
motors. Either the pilot is right and the computer is wrong (in which case any
physical effort expended is a further detriment), or the computer is right and
the pilot is wrong (in which case letting the pilot effectively sabotage the
computer is wrong). There is no middle ground between these two - weighing the
input of a human versus the input of a computer based on them physically
fighting is not a sound methodology.

Either powered drive of the trim is necessary or it is not. As it seems to be
necessary due to the forces involved, the sensible design is to declare the
actuator a critical part of the plane, and make the human input go through the
actuator rather than around it.

------
ceejayoz
That answers the "why didn't they remember the airworthiness notice" question
- they did, and tried to disable it, and it re-enabled itself.

Glad I didn't pick up Boeing shares on the cheap just yet.

~~~
everdev
> Glad I didn't pick up Boeing shares on the cheap just yet.

Unfortunately, it's a focus on profits over people that likely led to these
disasters. I don't know how you can stop or dissuade from people making a buck
off of a tragedy, but it might be worth putting our attention on the human
element of this mess. Something about merging this safety issue with an
opportunity to "buy low" and make some cash just doesn't sit right.

~~~
dboreham
Traditionally this is done by having a separate safety, QA or compliance
organization (depending on industry) that has the power to overrule the bean
counters and guys with white teeth.

~~~
bilbo0s
That's how it works in the FDA regulated medical device and medical software
industry. We had a woman who could shut us down on a whim. (And she would,
from time to time.)

That said, in the FDA regulated industry, every change was signed by not only
the engineer, but by the QA person, by the engineering manager, and by her. So
in fairness to her, _SHE_ would have been one of the people going to jail if
the software told a doctor to take out the wrong kidney because of a bug in
the calculation of an ortho normal basis or something. So I didn't blame her
one bit.

------
pupppet
Boeing deserves to get nailed for this. You can't tell me they didn't know it
was the MCAS at fault after the first crash 6 months ago, and decided to
do...nothing.

~~~
cameldrv
They weren't doing nothing. They've been working on a software fix since the
Lion Air crash, but it wasn't ready by the time of the Ethiopian crash. The
plan was to update some checklists as a stopgap and get new software ASAP.
Obviously in hindsight they should have grounded the fleet until the new
software was ready. At the time that would have been a very difficult decision
to make internally at Boeing, given that the 737MAX has $250 Billion worth of
orders, and arguably at the time, the updated checklists made a crash
unlikely. This is the situation where an independent regulatory body is most
important. The commercial pressure to keep flying was almost irresistable, and
you need someone without those commercial pressures to put the brakes on.

~~~
empath75
Or, you know, someone who values human lives more than money.

~~~
theredbox
Statistical value of human life. Learn it , breathe it.

~~~
joosters
Ok mr expert, please tell me exactly how much these two crashes have & will
end up costing Boeing.

------
tzs
I just had a 737 MAX fly low almost over my house [1]. That descending left
turn at around 2300 ft was centered on roughly my neighborhood. (The flight
track shown on Flightware is displaced to the NW a bit).

[1]
[https://flightaware.com/live/flight/BOE1/history/20190403/17...](https://flightaware.com/live/flight/BOE1/history/20190403/1700ZZ/KBFI/KBFI)

~~~
yarosv
I thought they were grounded.

Edit: Checked flightaware, it was Boing itself flying it.

~~~
rconti
Only from commercial service

~~~
danaliv
Sort of. All flights in U.S. territory, and by U.S. airlines anywhere in the
world, are banned. Non-passenger flights can be authorized but they require
either a Special Flight Permit (a specific process for authorizing ferry
flights of any non-airworthy aircraft) or an Experimental Airworthiness
Certificate (which you may know of from homebuilts, but in this case it's for
testing design changes).[1]

1\.
[https://www.faa.gov/news/updates/media/Emergency_Order.pdf](https://www.faa.gov/news/updates/media/Emergency_Order.pdf)

------
keyme
So the physical cutoff switches are still just inputs to software? WTF.

~~~
ams6110
No, the working theory I've heard is that the hand-crank trim wheels may have
been overwhelmed by the air load on the stabilizer given its "nose down"
configuration, probable full "nose up" pull on the elevators, and the very
high speed they were flying. Speculation is they may have turned the electric
trim back on to use the manual trim switches to trim up. By that time possibly
too late or even the electric trim was overwhelmed by the load, and of course
this would have brought MCAS back into play also.

~~~
keyme
This makes more sense, thanks.

I still don't understand why the trim is "stronger" on these planes than a
fully pulled back column. Is this necessary on large planes?

~~~
ncallaway
Mentour Pilot (as others have mentioned in various other comments) does a
great job explaining this.

On the 737 the trim adjusts the entire rear stabilizer (the entire horizontal
structure of the tail), while the elevator is a sub-component of that
stabilizer.

So, when you adjust the trim, it's a much larger surface area that is moving.
Ultimately, the stabilizer just has more aerodynamic control that the
elevator, due to the larger surface area.

I have no idea why the design is that way, though.

~~~
inferiorhuman
_I have no idea why the design is that way, though._

Cost, maintenance, weight, etc., probably. The only jet airliner I can think
of that uses an "all-flying tail" is the Lockheed L-1011. Nearly every other
plane you'd be riding on will have a big stabilizer and a smaller elevator.

~~~
phire
Redundancy too.

If both redundant hydraulic systems to the elevator fail, or the elevator gets
stuck for other reasons, the emergency procedure is to control pitch with just
the stablizer trim alone.

~~~
inferiorhuman
Presumably you could have a redundant drive system for the combined
stabilizer/elevator. The L-1011, for instance, had a safety record Boeing
should be envious of.

------
mdekkers
Avionics software that has direct impact on the handling of a passenger flight
should be globally mandated to be guaranteed faultless. There should be direct
liability to the software engineering team. I know Boeing's software wasn't
written in the USA, and was outsourced to the lowest bidder that met some on-
paper requirements.

If you think this is outlandish, read "They Write The Right Stuff"
[https://www.fastcompany.com/28121/they-write-right-
stuff](https://www.fastcompany.com/28121/they-write-right-stuff)

It can be done, but it is simply expensive. Boeing should hang for this.

