
737 Max: 1960s Design, 1990s Computing Power and Paper Manuals - bushido
https://www.nytimes.com/2019/04/08/business/boeing-737-max-.html
======
CivBase
Why exactly does this author think an airplane needs a lot of computing power?
Airplane software does not deal with especially large amounts of data or
intense computations. Faster hardware does not provide an inherent safety
benefit.

There is a large degree of coupling between avionics hardware and software amd
updating that software is prohibitively expensive. There's just no point in
doing it unless you get a tangible benefit.

Furthermore, avionics systems take a long time to develop are not often
rebuilt from scratch, so the hardware naturally lags far behind the computing
power we're used to.

The recycling of avionics systems is definitely fueled by a desire to reduce
costs. However, it's also important to note that older, mature systems with a
history of service inherently offer a degree of safety compared to something
which is untested.

The issue this article complains about can easily be seen as a safety feature.

To be fair, the author does spend most of the article talking about missing
features and design issues rather than harping on "processing power". It is a
poor title.

~~~
dmix
Wait until they finds out what software powers the nuclear weapons of 90%+ of
the world, top-tier fighter jets, "critical" infrastructure like power
(including nuclear), and countless financial/banking systems.

To be fair the article doesn't critique using 1990s (or earlier)
hardware/software. It's merely a clickbait headline. The only reference is
dismissed by the very next sentence as insignificant [Boeing could be replaced
with any ~~expert~~ somewhat-knowledgable person on the subject]:

> The flight-control computers have roughly the processing power of 1990s home
> computers. A Boeing spokesman said the aircraft was designed with an
> appropriate level of technology to ensure safety.

This only adds to my lowered standards when approaching NYT articles over the
years. They've long been abandoning being correct for being entertaining at a
seemingly accelerating rate. Which is too bad considering plenty of people
continue to hold them to a higher credibility over the typical clickbaiter
sites. Considering they are a subscription service I'm not convinced this is
justified.

~~~
huffmsa
The NYT fired basically their entire editorial staff in 2017. The accelerated
degradation has been noticable.

~~~
et2o
This is a good and comprehensive article. You and the grandparent comment are
focusing on one minor component and using it to raise FUD about the rest of
the article and indeed the entire NYT now. It's a lazy form of debate. HN used
to be better than this.

~~~
huffmsa
I'm just providing facts. NYT fired a ton of editors, because they're
expensive.[1]

They've also hired younger, cheaper writers.

As a result, the overall quality of the product has fallen, noticeable.

[1] [https://www.google.com/amp/s/deadline.com/2017/06/new-
york-t...](https://www.google.com/amp/s/deadline.com/2017/06/new-york-times-
editors-decry-humiliating-layoffs-1202121627/amp/)

------
pinewurst
The "1990s Computing Power" comment is ignorantly harsh. The MAX (to the best
of my knowledge) uses a similar AMD29050-based architecture as the 777 (where
it was pioneered) and the 737-800/900\. It's amply fast and very very
reliable, hardware and software (setting MCAS aside which is new to MAX).

Any sane person would prefer the solid well-proven choice over the bleeding
edge. People used to make similar comments about the AGC and the Space Shuttle
GPCs e.g. that such-and-such-a-PC-was-faster. Yeah, so what?

~~~
londons_explore
I'm not sure I agree here.

There exist possibilities for far more safety with more compute.

For example, when a mechanical failure occurs (for example an engine explodes
and partially falls off) the flight characteristics of the plane change
dramatically. Current systems fall back to human control, and hope that human
will be able to figure out how to control a now very "unique" machine.

Future systems could dynamically create new flight models based on collected
data to be able to fly the machine in entirely new ways.

For an example of this, did you know it's possible to fully control a
quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? [1].
That sort of 'fly it how it has never been done before' is out of reach of
human control, and might save lives.

[1]:. [https://www.ethz.ch/en/news-and-events/eth-
news/news/2013/12...](https://www.ethz.ch/en/news-and-events/eth-
news/news/2013/12/new-algorithm-makes-quadrocopters-safer.html)

~~~
exelius
This tech is _just now_ making it into military aircraft in the F-35 (it’s
designed to compensate for failures or combat damage while keeping the plane’s
handling relatively consistent), so I would guess it’ll be a while before we
see a civilian version.

~~~
AnimalMuppet
"Figure out a way to fly the thing in the new, crippled configuration" is
hard. Figure out a way that's _better_ than what the human pilot came up with
may be even harder.

~~~
londons_explore
It's hard, but it's a very measurable problem.

You randomly deform the airframe in a physics simulator, then get a real pilot
to try to fly the simulated damaged airframe to the nearest airport.

You then get an AI to try doing the same.

As soon as the AI manages to get to the airport more often than the human, the
tech will save lives, on average.

My guess is a PhD student could come up with an AI passing the above test
inside a year. Yet aviation standards are stringent enough AI will be decades
away from production use, if ever.

~~~
notahacker
Your link to ZTE trumpeting the success of an algorithm helping an actual
physical model of a quadcopter with a disabled rotor operating in lab
conditions to crashland a bit more softly might be interpreted as a hint that
the general case of _safely landing a stricken aircraft_ is the sort of
problem that probably isn't solved by a single PhD in a year...

Keeping those with enough hubris to say "yeah, shouldn't take long if we've
got some sort of sort of simulator to devise scenarios for a neural network to
overfit to" about a wide range of failure modes that an entire field has spent
decades studying well away from safety critical systems might be one of the
more underappreciated aspects of tight regulation.

------
whoisjuan
Kind of unfair. Do they really think that an aircraft should be powered by the
same software or hardware that powers a modern computer or smartphone? An
aircraft needs incredible reliability. You don’t get that with modern
software/hardware architectures.

~~~
da_chicken
Yeah, the B-52, adopted in 1952, is planned to be is service through the
2050s. An airframe design expected to last for over 100 years. Keep in mind
that flight itself is barely 100 years old.

And paper manuals? Are they expecting to use iPads for in flight documents or
something? Shall we compare failure states of paper vs tablets?

~~~
tedivm
That article says that other plans will display the error and the recommended
checklist automatically, which certainly seems faster than flipping through a
bunch of books trying to find a specific page based off of some lights.

~~~
ReptileMan
A lecturer of mine once said - knowing what kind of people we teach here for
programmers, I am scared of going to the doctor.

The less software is in something(and less cloudy/iot) the better. And the
more I program the more I prefer stupid and mechanical things when reliability
is on the line.

A paper manual is always there for you.

~~~
JauntyHatAngle
>A paper manual is always there for you.

Then have the paper manual as a choice and/or backup.

If the programmers working on aircraft are so incompetent that they can't get
a the digital manual correct, than they have no business programming any of
the other multitude of systems that are critical to fly-by-wire systems.

~~~
huffmsa
And the time it takes to realize your iPad has run out of battery or you
accidentally open an advertisement before grabbing your paper manual is too
long.

The manuals have never been an issue. In fact, having a physical, rigorous
checklist has been shown to improve the likely hood of a successful outcome.

~~~
tyingq
iPad (and others) based electronic flight bags are already in place on
passenger airlines. Your points are valid, but the debate is pretty much over.

------
myth2018
Mass media in general is stressing too much the fact that 737 MAX is allegedly
based on dated designs.

I think it's fair to say that MAX versions present some compromise solutions
(like the now-infamous MCAS, which is there to compensate for the "unnatural"
bigger engines). But I think that is not the main point. They would be good
solutions it they worked as intended.

There are some other more fundamental and more daunting, afaik unanswered
questions.

Like, why does such a critical system like MCAS take only a single AoA sensor
as input, when there are two sensors available? Specially considering that the
inputs from both are hardware-available to MCAS (the new software version is
going to take data from both).

Boeing affirmed in its manuals that the elevators would be able to compensate
for the trimmed vertical stabilizers. Now the preliminary report in the
Ethiopian's crash shows that the pilots wheren't able to perform such
compensation, even by pulling the control columns all the way back.

Those and some other issues are much more critical.

~~~
kacamak
>Like, why does such a critical system like MCAS take only a single AoA sensor
as input

The classic approach is to have three sensors, so in case one fails you can
know which one. Having two only indicates something is wrong but is not useful
on the fly.

~~~
flyinglizard
Three with a disagree algorithm is definitely what I’d expect out of such a
critical system, but two with signal averaging would still be much better than
just one.

~~~
cm2187
Or alternatively with two, and disabling MCAS if they disagree, seems a better
solution than having one and having no way to tell if it is working (keeping
in mind both can still fail simultaneously). Not an ideal solution but better.

~~~
myth2018
This is one of the features of the MCAS software update.

------
lqet
Hm. I hesitate to defend Boeing here, but I think the outset of this article
is a bit unfair.

> Pilots start some new Boeing planes by turning a knob and flipping two
> switches.

> The Boeing 737 Max, the newest passenger jet on the market, works
> differently. Pilots follow roughly the same seven steps used on the first
> 737 nearly 52 years ago: Shut off the cabin’s air-conditioning, redirect the
> air flow, switch on the engine, start the flow of fuel, revert the air flow,
> turn back on the air conditioning, and turn on a generator.

So? What does this have to do with anything? Is the goal to produce an
airplane where pilots press a button "fly to destination", and the plane does
it?

> The strategy, to keep updating the plane rather than starting from scratch,
> offered competitive advantages. Pilots were comfortable flying it, while
> airlines didn’t have to invest in costly new training for their pilots and
> mechanics. For Boeing, it was also faster and cheaper to redesign and
> recertify than starting anew.

> But the strategy has now left the company in crisis, following two deadly
> crashes in less than five months.

How was it the strategy to keep updating the plane that left to this crisis?
The strategy itself is not to blame here, and I very much like the idea to
gradually improve a proven model. It was a _bad execution_ of this strategy
that left the company in crisis.

In Germany, the national train agency Deutsche Bahn (and its predecessors)
basically had a policy for nearly a century to order rolling stock that was
designed to be produced for around 40 years. During this production run, the
model was gradually improved. Some of the rolling stock designed in the 50ies
is still in use, and quite reliable at that [0]. During the 90ies, agency and
industry switched to a policy where basically every train generation was newly
developed from scratch (for example, the ICE high speed trains). Guess what -
you can channel _a lot_ of public money into private hands that way, but
bleeding edge technology is not what you want or need when you are trying to
build a reliable transportation system.

[0]
[https://de.wikipedia.org/wiki/N-Wagen](https://de.wikipedia.org/wiki/N-Wagen)

~~~
throw0101a
> So? What does this have to do with anything? Is the goal to produce an
> airplane where pilots press a button "fly to destination", and the plane
> does it?

When you're nose down heading into the ground, taking your hand off the stick
to adjust the trim can get panic-y. Especially since the MCAS may be working
against your trim adjustments.

The design of the Max line seems to be _just_ enough compete with the A320neo,
and _just_ enough to not "need" re-certification. But these two _just_ s,
together, may have cost several hundred people their lives.

------
iliketosleep
When it comes to flight computers it makes sense to be conservative when it
comes to computing power - the goal should be simple, proven, and functional.
But when it comes to diagnostics why not use more computing power? As the
article states _A second electronic system found on other Boeing jets also
alerts pilots to unusual or hazardous situations during flight and lays out
recommended steps to resolve them._ Seems better than flipping through a paper
manual when you've only a minute or two to save the plane.

~~~
theclaw
Because new diagnostics mean new pilot training, meaning the new plane is not
economical to buy when compared to the competition, meaning no plane at all.

Those other Boeing jets required costly new training. Boeing were trying to
build an update to the 737 that did not require this.

------
kbos87
The thing that baffles me - why the hell has there never been any innovation
in the form of making it easier to diagnose a problem and move through the
necessary steps in a checklist beyond fumbling through a paper manual? In a
situation where seconds count, it seems illogical that this has never been
improved upon. Didn’t at least one of these planes crash because the pilots
are believed to have only gotten X number of steps through a list of possible
reasons?

~~~
jonas21
As mentioned in the article, other Boeing jets do automatically present a
checklist on the display when they detect a problem:

> _A second electronic system found on other Boeing jets also alerts pilots to
> unusual or hazardous situations during flight and lays out recommended steps
> to resolve them._

The 737 is stuck in the past because given the choice, nobody wants to have to
retrain their pilots on more modern systems.

~~~
EricE
>The 737 is stuck in the past because given the choice, nobody wants to have
to retrain their pilots on more modern systems.

Ding ding ding!

------
andy_ppp
Does anyone else here feel aeroplane software should be mandated as being open
source? Boeing would have never even dreamed of releasing the 737-Max software
in the state it was in and the more eyes you have on things like this the
better.

~~~
StreamBright
Do you think there are a massive amount of developers out there who are at the
same time avionics experts? I just don't see how being opensource would help.
You wouldn't be able to even test these software projects because it requires
and emulator or actual hardware to run on that you do not have. What would it
chage if it was opensource?

~~~
andy_ppp
I think the argument you're making is completely false.

First Premise is that there re not many avionics expers - I agree so having
_more_ of those experts able to look at review and learn from different
implementations would be a good thing.

Second premise is that you need to be an avionics expert to review and improve
avionics software, I'm not convinced, there are many clever people out there.

Third premise you can't test or run the software becuase there is no emulator;
maybe if the software was open source someone would start writing an emulator
- maybe an emulator could be open sourced?

Fourth point; nothing would change if it was open source - I think the quality
of code that would be released would go up and that's the main thing I'm
talking about, embarrassment would be the optimal solution to something like
this - I'd be very suprised if Boeing would release software that used a
single sensor as input to a critical system like this.

~~~
StreamBright
I see so basically it would mean no benefits at all. Thanks for confirming.

Now the other problem that I have this in this thread. The Boeing issue is not
a software issue at all. Even if we chose the best case scenario for the
software it would be still a physical design & lack of pilot education
problem.

~~~
drinfinity
Airbus agrees Boeing should make its software opensource.

------
chmaynard
The entire management chain that dictated these design compromises should be
surgically removed from Boeing and prosecuted for negligence.

Of course, that will never happen.

~~~
PhantomGremlin
This doesn't get said enough.

Within days of the Lion Air crash, MCAS was strongly suspected to be the
problem. And yet Boeing management allowed 157 more people to die in the
Ethiopian Airlines crash. And then the Boeing CEO still tried to tell everyone
that it was a safe airplane.

The entire top management of Boeing, and the entire board of directors, should
be perp-walked out the door.

Of course, that will never happen.

------
foldr
>In the recent crashes, investigators believe the MCAS malfunctioned and moved
a tail flap called the stabilizer,

Fact checking at the NYT seems to be dead. The MCAS moves the entire
stabilizer -- which is not a "flap" in any sense. Can they not find a pilot or
some other knowledgeable person to read through an article like this before
publishing it?

------
buserror100
I'd rather take a well tested i386 than a new processor that crashes Linux
with microcode errors.

Paper manuals actually instill confidence. Should pilots contact Stackoverflow
in an emergency?

------
filereaper
I guess pilot retraining will always be time consuming, but can anything be
done about recertification so that new designs don't face this issue?

~~~
ajross
Good grief. No!

Because as we are literally discovering, not knowing the details of how and
why the automatic stabilization software is driving your trim wheel _can kill
you_. The biggest single part of the failure chain here was the fact that this
_should_ have been an invasive change requiring recertification as a different
aircraft and retraining for pilots instead of pretending that it was just like
a 737-600.

~~~
Gibbon1
You are on my page. Type certification is the process to discover issues like
this.

I'm not familiar with pilot training and certification but it seems to me that
even if the 737 MAX was type certified certifying a pilot with previous 737
experience would have been just supplemental training. Ultimately a nothing
burger.

I've seen a lot of cases where people get bent thinking of all the bad things
that will happen because they think customers won't like some change. Always
in the end if it's not a show stopper the customers just lump it.

------
throw0101a
I'm sure this conversation will be looked at during discovery in any future
lawsuits:

> _Boeing also designed the system to rely on a single sensor — a rarity in
> aviation, where redundancy is common. Several former Boeing engineers who
> were not directly involved in the system’s design said their colleagues most
> likely opted for such an approach since relying on two sensors could still
> create issues. If one of two sensors malfunctioned, the system could
> struggle to know which was right._

> _Airbus addressed this potential problem on some of its planes by installing
> three or more such sensors. Former Max engineers, including one who worked
> on the sensors, said adding a third sensor to the Max was a nonstarter.
> Previous 737s, they said, had used two and managers wanted to limit
> changes._

------
tzakrajs
Three angle of attack sensors should have never been something that could be
overridden by cost savings measures. The engineers wanted three sensors and
the management said no without ensuring all planes sold had sufficient
mitigating controls. We need justice.

------
spacedog
>> “They wanted to A, save money and B, to minimize the certification and
flight-test costs"

This seems to be the root cause of the problem. Competition is not always
good.

~~~
reacweb
I think Boeing was in a wrong mindset: certification was seen as a burden
instead of being seen as an help to build a better aircraft. But maybe it was
not only the fault of Boeing. The certification process of FAA is perhaps not
friendly enough toward big air-frame changes (like putting the wings slightly
higher to make room for the bigger reactor). Here, the certification process
has not identified that a failure of one AOA sensor was able to cause a
difficult to avoid crash. Maybe we should allow the use of less reliable
source of data (like GPS, ...) each time a sensor has a problem.

------
stunt
Except rumors from media there is no clear explanation about what is going on
inside Boeing. I just hope Boeing management is not the old Microsoft
management.

------
adontz
While i386 may be enough to run a plane, what other software or hardware of
the same age is still in use?

------
tibbydudeza
The Windows XP of passenger jets.

~~~
bcatanzaro
Windows XP was actually pretty modern for its time. Maybe you mean Windows ME.
;)

~~~
Gibbon1
Thank you for reminding me of something I'd rather not remember.

One thing I remember is Microsoft tried repeatedly with Win95, 98 and ME to
get USB support to work correctly. Failing each time and having to start over.
Much to the horror of peripheral manufacturers. With Windows XP that finally
mostly worked.

