
Latest 737 Max Fault That Alarmed Test Pilots Rooted in Software - 1gor
https://www.bloomberg.com/news/articles/2019-07-27/latest-737-max-fault-that-alarmed-test-pilots-rooted-in-software
======
pcl
> _Because the fault was triggered by specific streams of erroneous flight
> data, a new software patch can be devised that monitors the computer for
> that highly unusual condition and prevents movement of the stabilizer when
> it occurs, one of the people said._

Oh man. That sounds like a mess.

~~~
teawrecks
Bandaid fix if I've ever heard of one.

~~~
burfog
It's proper. This is exactly how you do things with life-critical embedded
software. You detect when things are crazy, which of course shouldn't happen
but nevertheless does, and you react in a way that is hopefully not
disastrous.

The alternative is to ignore the problem.

Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing
goes bad. Even the RAM sometimes goes bad.

When the hardware is failing ("specific streams of erroneous flight data")
there isn't going to be a reliable solution. You can't even fully enumerate
all the possible failures. How could you possibly guess that a message routing
chip now flips bit 7 in every angle-of-attack measurement?

Boeing's answer is standard: when the values look crazy, stop doing stuff that
might make the situation worse.

~~~
tus88
The way I read it they are not detecting malfunctioning sensors, but
essentially a computer fault caused by it's inability to deal with erroneous
input. Rather than fixing the computer, they are essentially wrapping a great
big try/catch around it....good God.

~~~
comex
That interpretation seems like a stretch to me.

~~~
selestify
The way it's worded, that's exactly the interpretation that a lot of us in
this thread get. If this is not actually the case, they should really word it
differently.

~~~
ethbro
This is the same Bloomberg that reported on the SuperMicro hardware
"compromise."

------
Someone1234
I'm surprised nobody is discussing this paragraph from the article:

> The failure scenario was known previously and had been assessed in a safety
> analysis when the plane was certified before entering service in 2017. At
> that time, Boeing concluded that pilots could overcome the nose-down
> movement by performing a procedure to shut off the motor driving the
> stabilizer movement.

This isn't a new fault. Boeing certified this as safe along with MCAS back
when the aircraft first flew in 2017, using the same justifications ("pilots
can overcome it").

They weren't going to fix it this time either, except test pilots ran across
it in late-stage simulations monitored by the FAA and found it wasn't as easy
to overcome as Boeing had been asserting in their cost-safety analysis (just
like MCAS) and now the FAA are requiring a fix.

Has Boeing learned anything from MCAS? The company has cultural problems vis-
à-vis safety. I'm just glad the FAA are doing their job this time around.

~~~
ricardobeat
I’ve only recently started following the air industry more closely, but isn’t
it ironic that Boeing, the one on the “direct control” side vs computerized
systems from AirBus, is the one now suffering from software and automation
failures?

I wonder if it might be a consequence of being a late adopter, and rushing the
development of systems they never had before.

~~~
ulfw
The only reason Boeing is in the “direct control” business is because their
best seller is from the 1960s and they don’t dare touch it. Their newer wide
bodies 777 and 787 are fly-by-wire and in the latter case even fully electric
to a fault.

------
nocturnial
I really can't help to think that the design and production of this plane was
rushed.

They used a one sensor input to MCAS because the FAA wouldn't certify a two
sensor input without requiring a level D certification which meant sim
training for pilots.

This latest revelation in the article only strengthens my belief it was rushed
and possibly to the point that it was irresponsible.

~~~
onli
That's seriously overdiplomatic. The plane was rushed and it was constructed
very badly. It was absolutely irresponsible to bring it into service, it's not
a question anymore. It's a plane that crashes itself automatically into the
ground and that killed over 300 people.

~~~
marcyb5st
Agreed. However, the worst part is the FAA. Specifically, what baffles me is
the massive hand-wave they did on this considering it is one of their mandates
to check flight-worthiness of Airplanes/Airlines.

I hope all the other agencies worldwide will double check all the prior
certifications issued by the FAA, because all their credibility is gone.
Especially considering the US was the one the last countries to ground the 737
MAX.

I wonder what happened there. Bribes? Favors? Lobbying and ties? All of the
above?

~~~
jeremyjh
Regulatory capture. After their careers at the FAA they will go work at
Boeing.

~~~
linuxftw
Do you think regulatory capture actually exists? If so, why is the
pharmaceutical industry immune?

~~~
RockyMcNuts
why do you think pharma is immune, after fiascos like Vioxx, Accutane,
opioids, the fact that much regulatory machinery is in place to create drug
monopolies?

------
anticristi
Before upgrading to the latest Ubuntu, I like to wait for a few months until
my friends confirm that it's for the better. I guess I would do the same for
the MAX.

I can't imagine what it must be like to be one of the first pilots to fly the
MAX once the grounding is lifted. Can pilots refuse to fly a specific plane
"for a few months, 'cause I have kids waiting for me home"?

~~~
ninth_ant
There were almost 400 delivered by the time it was grounded, some of them in
service since 2017. Yes there were two high profile failures and this failure
rate is extremely out of the norm by the very stringent standards of aviation
— but these are not sudden-unavoidable-death machines that gets
sensationalized.

~~~
linuxftw
That's exactly what they are. How many planes have malfunctioned in the same
way and haven't nose-dived into the ground? So far, 0 planes that have had
this issue haven't been lost (first plane had the issue, was recovered using
non-standard procedures, then same plane killed everyone, even after sensor
was replaced!)

------
breatheoften
I was just reading an article on New York Times that describes the process
since 2005 whereby safety oversight responsibilities over Boeing were
gradually moved in house — until by 2018 Boeing was self-certifying for 96% of
safety testing.

If they are doing 96% of the safety validation work — then they should bear
96% of the responsibility for safety process failures ... in this case, given
the break down of safety validation capacity and what I’m increasingly viewing
as essentially a coverup after the first and second crashes — 96% of the
responsibility is and probably should be - enough to bring Boeing down ...

I’m beginning to think that might be the only way to ensure responsibility for
this failure gets allocated in a sufficiently accurate way to ensure this
scenario isn’t likely to happen again ... allowing the faa to have the
responsibility to bear the weight of this process breakdown would basically
just allow Boeing to shift responsibility for this outcome off themselves and
onto an organization which it seems has been deliberately engineered (by
Boeing) to not have the capacity to perform effective oversight ...

------
Aloha
Imagine if your bugfix process was broadcast to the world, every new IN added
to spirateam et al, every new crash condition found by test/QA, every time the
software failed in test, it got printed by news organizations the world over.

This is world the 737MAX software people are living in, for this alone, they
have my utmost respect.

~~~
toyg
Imagine if your software had already killed a few hundred people.

Boeing developers have been put in an untenable position and should consider
walking out until management does the right thing and kills the plane.

~~~
Aloha
The plane has no fundamental mechanical flaw, it has a software bug.

The only flaw the plane had was being treated as 'just another 737'

~~~
ulfw
It has plenty of uncertifiable components grandfathered in for 50 years,
topped with ‘still within range of grandfathering’ pieces of bandaid software
on top.

It’s fundamental because the whole fundament this rests one isn’t stable and
all the adding instability add-ons do not make sense.

~~~
Aloha
Since you seem to be an expert in airplanes and complex systems, which
components are not certifiable?

the 737MAX could have been certified as a new airplane, with a new type
certificate, and a new pilot type rating to go with it - but that would have
removed much (or all) of its economic justification to exist.

~~~
nraynaud
It would be a not very good airplane. It has only 2 selling points: its type
certificate, and its 737 footprint at the terminal.

Any new airplane would be a bit higher, it brings some downsides (ground
operations people really like the easy access), but having bigger engines
under the wings is such an advantage that it overrides all other
considerations.

~~~
Aloha
Thats kinda my point, the shared type certificate is most of the justification
for the airplane to exist, Boeing would rather build a whole new small
aircraft, people who fly on them would rather fly on a whole new aircraft -
airlines however strongly prefer something that costs as little as possible to
put into service - and has commonality with existing fleets, which means _not_
a whole new aircraft.

I bet if you offered Delta some brand new MD-95's they'd take up on them, as
they have basically every MD-95 produced.

~~~
ethbro
And by "airlines", let's be specific: American Airlines was the carrier that
specifically asked Boeing to build an 737 with new engines.

~~~
CydeWeys
Although they didn't explicitly ask for such insane engineering compromises to
have been made.

~~~
ethbro
Didn't they? They asked for it fast and cheap...

------
laythea
Can anyone in the know explain why, if ground clearance due to the new engines
is the issue, why Boeing cannot just add a few inches to the wheel stems?

This would raise the whole aircraft off the ground. Surely that wouldn't
effect aerodynamics too much as the wheels are up most of the time.

~~~
Glawen
Because you then need longer and heavier landing gear, which you need to fit
somewhere in the plane as they retract. This is a ripple effect where you
probably need to redesign the complete airframe, from fuselage to the wings.

They actually cleverly modified the landing gear to gain a bit of heigth, but
it was not enough:
[https://youtu.be/F4IGl4OizM4](https://youtu.be/F4IGl4OizM4)

------
s_T_e_v_o
One thing I learned many years ago, you can solve infrastructure problems with
software. If prior jets have multiple sensors and the new jet has one sensor,
then software isn't going to help in this situation. Gyros worked back in
1911. why not use one to back up the failed sensor? seems like a simple fix
that could be mounted anywhere on the plane.

~~~
cesarb
> Gyros worked back in 1911. why not use one to back up the failed sensor?

An angle of attack sensor measures something a gyroscope cannot measure: the
angle of the plane _relative to the wind_. The amount of lift produced by the
wings depends on this angle.

------
kjar
It appears the 737 Max is a total failure. Pushed through by Boing to outpace
Airbus, from physical design, attitude sensors, re-certification bypass
imperative, and flight control software, finally killing nearly 600 people
before being ground. In short I’m never flying in one!

------
kitchenkarma
Move fast break things kind of doesn't work well with planes.

Also too big to fail (fall) doesn't apply here.

~~~
nimish
Au contraire, Boeing is most definitely too big too fail and will be bailed
out somehow.

------
SkooterIn228
Laying off senior Boeing software engineers and outsourcing their work to
India for $9/hour is really paying off for Boeing.

------
kyberias
Are there any public databases that would show plane model failure statistics
so that I could choose how to fly?

------
hammerbrostime
Sounds to some degree like a usability issue. Echoes of Three Mile Island.

