
Software Won’t Fix Boeing’s ‘Faulty’ Airframe - farseer
https://www.eetimes.com/document.asp?doc_id=1334482
======
mhandley
Boeing's software fix, announced today, is to compare readings from both
angle-of-attack sensors and disable MCAS if they disagree significantly. The
obvious question is why they didn't do this in the first place?

One possibility is incompetence. But Boeing engineers are smart people, so I'm
not convinced by this. The elephant in the room is the requirement to maintain
a common type rating with older 737 models.

Suppose they did originally do what the fixed software does now, and disable
MCAS if the AoA sensors disagree. The problem Boeing face is that with MCAS
disabled when this occurs, the plane no longer flies like an older 737. They'd
need to announce to the pilots an AoA disagree, and announce that MCAS was
disabled. Now what? A pilot certified and trained on the older 737 would not
know how the Max now differs from what they trained on. If they'd done this,
they'd have needed to provide additional training, and this must have
concerned Boeing management that it might jeopardize the common type rating.
Hence it seems likely they didn't add the AoA sensor comparison for this
reason, reasoning that it was unlikely to be a problem anyway. We now know
that reasoning was flawed.

What does this mean going forwards? Will EASA and other CAAs refuse to certify
the modified 737 Max under the same type rating as the older 737? This
certainly seems possible. If they did require a separate type rating, this
would likely kill 737 sales, regardless of whether the plane is now safe.

~~~
GuB-42
> One possibility is incompetence. But Boeing engineers are smart people, so
> I'm not convinced by this.

That's still a possibility. Stupid decisions can emerge out of smart people.

Boeing is huge, and what they develop is incredibly complex. There are a lot
of people with differing level of competence, ethics, and goals.

For example (I am not saying that happened), the engineers designing MCAS
didn't expect incorrect AoA data, thinking the checks were done elsewhere. At
the same time, the "sensors" team thought that raw, unchecked data was
expected. The integration guy didn't read the specs correctly (sometimes, it
comes down to a single word), didn't catch that, and checked the OK box. His
manager, focused on a more pressing issue took that as granted and it went to
production.

It is possible that the engineers did an excellent work, but didn't question
the specs they had. The integration guy is normally super reliable but he just
had a bad day. And his manager handled the other problem beautifully and
overlooked the MCAS/AoA because, normally, the integration guy is reliable. A
series of small mistakes that ended up in a catastrophe.

There are a lot of safeguards but the complexity is so high that sometimes,
something goes through. Especially if the company is under pressure.

~~~
vonmoltke
> For example (I am not saying that happened), the engineers designing MCAS
> didn't expect incorrect AoA data, thinking the checks were done elsewhere.
> At the same time, the "sensors" team thought that raw, unchecked data was
> expected. The integration guy didn't read the specs correctly (sometimes, it
> comes down to a single word), didn't catch that, and checked the OK box. His
> manager, focused on a more pressing issue took that as granted and it went
> to production.

> It is possible that the engineers did an excellent work, but didn't question
> the specs they had. The integration guy is normally super reliable but he
> just had a bad day. And his manager handled the other problem beautifully
> and overlooked the MCAS/AoA because, normally, the integration guy is
> reliable. A series of small mistakes that ended up in a catastrophe.

What you describe here would be a major failure of systems engineering on the
project.

The systems engineers are responsible for flowing top level requirements down
to the individual systems. They are responsible for ensuring the specs the
engineering teams receive for their systems are correct, and for handling
requests to change said specs. If the spec of the output of the AoA sensors
does not match the spec flowed down to other teams on the input from those
sensors the systems engineers responsible did not do their jobs.

Systems engineers exist to manage complexity like this, to ensure that the
various engineering teams across the various disciplines are technically
coordinated by providing clear, consistent specs and interfaces for them to
work to. If that didn't happen, then I would not say those engineers are
competent, let alone smart. It would be especially disappointing to me (as a
former systems engineer) if this were the case, as systems engineering is the
_last_ place I would expect such incompetence from a company like Boeing. It's
at the core of everything they do.

I want to address one specific point in a different context.

> the engineers designing MCAS didn't expect incorrect AoA data, thinking the
> checks were done elsewhere

You _always_ expect out-of-spec conditions to be a possibility and have
something in place to handle those conditions appropriately. To not do so is
incompetence bordering on negligence.

~~~
mtw
2 planes crashes in less than 5 months. So yes, there was either a system
engineering failure, or a completely inept decision in choosing an oversized
engine for an antic airframe. Your choice.

~~~
ethbro
Or a training requirement failure. Or a UX failure. Or a documentation
failure. Or an unrelated failure, given that the RCAs haven't been completed
on the crashes.

It's amazing the hubris software engineers have in assuming everyone else in
an idiot.

I am not an expert on aircraft-scale hardware/software codesign projects,
aerospace engineering, high-reliability engineering (to aerospace standards),
or complex systems analysis. I strongly suspect you, and most of HN, aren't
either.

Boeing has people with all those skillsets. As does Airbus. As do NTSB and the
variety of national certification agencies.

So before we toss rocks of certainty around in a comment thread, maybe wait
and listen to the experts?

~~~
bobthepanda
Given that at least some of the expert companies involved have attempted
coverups of previous failures before, a little bit of skepticism is healthy.
[https://en.m.wikipedia.org/wiki/Boeing_737_rudder_issues](https://en.m.wikipedia.org/wiki/Boeing_737_rudder_issues)

Authority should not be free from skepticism, even by people who are not
authorities.

~~~
ethbro
We can agree that not all skepticism is valuable skepticism though?

I came across this Rand report on the NTSB [1, 1999?]. The conclusions were
not good about the funding & staffing levels vs modern accident load. Partly
due to more incidents, but moreso due to increased systems complexity per
incident.

So the NTSB, with a budget of around USD$100M, takes multiple years to deliver
a report. And they're a professional worldwide standard on accident
investigation. Suffice to say, we're not going to crack this case open on HN.

Which isn't to say it isn't productive to debate the relative merits of
different regulatory approaches, takeaways for other disciplines, lessons
learned, and all manner of things. But let's just have some humility in
pretending we're all experts on everything [2].

[1]
[https://www.rand.org/content/dam/rand/pubs/monograph_reports...](https://www.rand.org/content/dam/rand/pubs/monograph_reports/MR1122z1/MR1122.1.sum.pdf)

[2] Though in fairness, I wouldn't be surprised if there were at least one
expert on any given topic here. You folks are awesome.

------
csours
This reminds me of The Slow Winter by James Mickens [0]

> "John was terrified by the collapse of the parallelism bubble, and he
> quickly discarded his plans for a 743-core processor that was dubbed The
> Hydra of Destiny and whose abstract Platonic ideal was briefly the third-
> best chess player in Gary, Indiana. Clutching a bottle of whiskey in one
> hand and a shotgun in the other, John scoured the research literature for
> ideas that might save his dreams of infinite scaling. He discovered several
> papers that described software-assisted hardware recovery. The basic idea
> was simple: if hardware suffers more transient failures as it gets smaller,
> why not allow software to detect erroneous computations and re-execute them?
> This idea seemed promising until John realized THAT IT WAS THE WORST IDEA
> EVER. Modern software barely works when the hardware is correct, so relying
> on software to correct hardware errors is like asking Godzilla to prevent
> Mega-Godzilla from terrorizing Japan. THIS DOES NOT LEAD TO RISING PROPERTY
> VALUES IN TOKYO. It’s better to stop scaling your transistors and avoid
> playing with monsters in the first place, instead of devising an elaborate
> series of monster checks-and-balances and then hoping that the monsters
> don’t do what monsters are always going to do because if they didn’t do
> those things, they’d be called dandelions or puppy hugs."

0:
[http://scholar.harvard.edu/files/mickens/files/theslowwinter...](http://scholar.harvard.edu/files/mickens/files/theslowwinter.pdf)

~~~
songeater
this is awesome and probably needs a thread of its own

~~~
csours
There have been a few, but it looks like none of them really sparked a lot of
discussion:

[https://hn.algolia.com/?query=Slow%20Winter&sort=byPopularit...](https://hn.algolia.com/?query=Slow%20Winter&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

\---

I've submitted it again:
[https://news.ycombinator.com/item?id=19514259](https://news.ycombinator.com/item?id=19514259)

------
SeaSeaRider
There is a debate to be had, but this is a naked propeganda piece. The crux of
the article is based on:

“Among Boeing’s critics is Gregory Travis, a veteran software engineer and
experienced, instrument-rated pilot who has flown aircraft simulators as large
as the Boeing 757.”

... someone who uses flight simulators. This is not credible journalism.

~~~
clawoo
The article reeked of anything but credible journalism as soon as it opened
with "The saga of Boeing’s 737 MAX serves as a case study in engineering
incompetence, and in engineering ethics – or the lack thereof."

By this point's it's obvious to everyone that the engineering of the plane is
pretty far down the line of causes which lead to this.

There was a Twitter thread[1] a few weeks ago which explained it very clearly:

Some people are calling the 737MAX tragedies a #software failure. Here's my
response: It's not a software problem. It was an

* Economic problem that the 737 engines used too much fuel, so they decided to install more efficient engines with bigger fans and make the 737MAX.

This led to an

* Airframe problem. They wanted to use the 737 airframe for economic reasons, but needed more ground clearance with bigger engines.The 737 design can't be practically modified to have taller main landing gear. The solution was to mount them higher & more forward.

This led to an

* Aerodynamic problem. The airframe with the engines mounted differently did not have adequately stable handling at high AoA to be certifiable. Boeing decided to create the MCAS system to electronically correct for the aircraft's handling deficiencies. During the course of developing the MCAS, there was a

* Systems engineering problem. Boeing wanted the simplest possible fix that fit their existing systems architecture, so that it required minimal engineering rework, and minimal new training for pilots and maintenance crews.

The easiest way to do this was to add some features to the existing Elevator
Feel Shift system. Like the #EFS system, the #MCAS relies on non-redundant
sensors to decide how much trim to add. Unlike the EFS system, MCAS can make
huge nose down trim changes.

On both ill-fated flights, there was a:

* Sensor problem. The AoA vane on the 737MAX appears to not be very reliable and gave wildly wrong readings. On #LionAir, this was compounded by a

* Maintenance practices problem. The previous crew had experienced the same problem and didn't record the problem in the maintenance logbook. This was compounded by a:

* Pilot training problem. On LionAir, pilots were never even told about the MCAS, and by the time of the Ethiopian flight, there was an emergency AD issued, but no one had done sim training on this failure. This was compounded by an:

* Economic problem. Boeing sells an option package that includes an extra AoA vane, and an AoA disagree light, which lets pilots know that this problem was happening. Both 737MAXes that crashed were delivered without this option. No 737MAX with this option has ever crashed.

All of this was compounded by a:

* Pilot expertise problem. If the pilots had correctly and quickly identified the problem and run the stab trim runaway checklist, they would not have crashed.

Nowhere in here is there a software problem. The computers & software
performed their jobs according to spec without error. The specification was
just shitty. Now the quickest way for Boeing to solve this mess is to call up
the software guys to come up with another band-aid.

I'm a software engineer, and we're sometimes called on to fix the deficiencies
of mechanical or aero or electrical engineering, because the metal has already
been cut or the molds have already been made or the chip has already been
fabed, and so that problem can't be solved.

But the software can always be pushed to the update server or reflashed. When
the software band-aid comes off in a 500mph wind, it's tempting to just blame
the band-aid.

[1]
[https://threadreaderapp.com/thread/1106934362531155974.html](https://threadreaderapp.com/thread/1106934362531155974.html)

~~~
rootusrootus
This is an excellent analysis, thanks! Wish I could upvote more than once.

------
manfredo
I've read a variety of articles on this and they often said somewhat different
things. What I've been able to gather about the timeline of events is:

1\. The new engines on the MAX shifted the center of gravity forward (and I
assume center of lift stayed the same).

2\. Boeing was worried that #1 would cause the plane to nose up during high
angles of attack (so, take off and landing?), and added software, MCAS, to
pitch up to counteract this.

3\. There's some confusion over when this software kicks in and how to cancel
it (something about the trim controls not cancelling MCAS?)

4\. Regardless of #3, this software seems to have confused pilots and the
current belief is that MCAS was active when pilots didn't want it active.

5\. ????

6\. Planes crash.

Also, I've read about some concerns about the fact that the handling behavior
changed so much but the plane wasn't reclassified as a different type. I'm
still unclear about how classifications plays into this story.

My core point of confusion is, if MCAS is the culprit why isn't the solution
to remove MCAS? Is tendency to pitch during high angles of attack unusual, and
something pilots cannot be expected to counteract manually? I've only played
sims like DCS and X-Plane (and not very much at that) but "nose goes up when I
don't want it to, so I push stick forward" doesn't seem too complicated to me.
Of course, I'm no pilot so I'm probably drastically oversimplifying the
situation.

~~~
mhandley
Your point #1 is incorrect. The problem with the larger engines is that their
larger nacelles placed further forward produce extra lift at high angles of
attack. This lift is further forward than the centre of mass. The
certification requirement is that to produce steadily increasing angles of
attack, you need to steadily increase back pressure on the yoke. The problem
with the Max is that this is no longer true. Past a certain angle of attack,
the back pressure needed to further increase angle of attack reduces. The
plane is not actually unstable, but it's closer to being so than the
certification requirements allow. And it's certainly behaviour that Boeing
couldn't claim was similar enough to older 737s to allow a common type rating.
Hence MCAS, which was supposed to detect this condition and make the aircraft
fly like an older 737. This allowed a common type rating, and allowed the
aircraft to be certified. But fundamentally, the airframe has an undesirable
property, and you'd never have designed it this way unless the desire for a
common type rating dominated other design decisions.

~~~
reacweb
It is incredible: the airframe has been designed to reuse the certification of
737. A flaw that could be fixed in a proper way, instead has been worked
around using unreliable subterfuges.

The aim of the certification process is to ensure the safety (and reliability)
of aircraft. The required mindset shall be that the certification process
helps to highlight defects in order to build a better aircraft. Here, the
mindset was that the certification process is a burden with arbitrary
constraints that have to be fulfilled even if this means a worse aircraft.

IMHO, the people (managers) with the wrong mindset shall be replaced and the
faulty airframe of 737 shall be killed.

~~~
greeneggs
This is unfair. If done appropriately, making a plane behave the way pilots
expect should also make it safer. A "worse aircraft" with a better UI can
actually be a safer aircraft.

~~~
rtkwe
Only so far as we can assume the UI never breaks and the sensors are always
correct. The issue is pilots need to be trained to understand where the UI
hides the underlying performance of the plane because when it breaks it can go
wrong incredibly quickly. MCAS altered the way to aircraft reacted and also
would restart it's changes unless completely cut out. Pilots didn't receive
enough training to recognize the issue as being the MCAS system adjusting the
trim fast enough to prevent the crashes during takeoff and ultimately the
whole point of the MCAS was to avoid having to retrain and maintain the common
type rating.

------
Luc
Well eetimes got some clicks, so job well done for the journalist who wrote
this article about a blog post by some guy with experience flying large planes
in a flight sim.

~~~
dustindiamond
Well, we at HN got some good discourse out of it, no?

~~~
rootusrootus
Not sure I would call it 'good' but we did sure get a lot more chatter. I
wonder if collectively this is the most comments a particular issue has ever
received on HN. So many articles posted, each gets hundreds of comments.

------
rwhitman
The paper that this article cites is a far more interesting read than the
summary: [https://drive.google.com/file/d/1249KS8xtIDKb5SxgpeFI6AD-
PSC...](https://drive.google.com/file/d/1249KS8xtIDKb5SxgpeFI6AD-
PSC6nFA5/view)

~~~
SeaSeaRider
... it’s written by a guy who uses flight simulators, this is crank stuff.

~~~
GregTravis
The 757 flight simulator that I've flown was a full-motion flight simulator at
UPS's Louisville training facility. I was helping conduct 6-month
certification flights as co-pilot. I would say that if those simulators are
good enough for UPS's 757 pilots' recurring training, they're good enough for
this Cessna driver to understand that difference between his plane and a 757.
Answer: Not that much.

~~~
jordan_clark
Greg - Serious question - How is that air frame on the Max8 faulty due to
engine placement when the A320NEO has almost the same design?

~~~
salawat
Because the A320neo can fit the engines under instead of in front of the
wings, thereby negating the nacelles contribution to leading edge flow
separation.

~~~
GregTravis
Exactly. Except it’s not flow separation that is the problem. It’s the lift
that the nacelles generate combined with their mass and moment far ahead of
the center of gravity

~~~
salawat
Yes, my mistake. Stated the outcome, not the action that gets you there.

------
GregTravis
Hello,

This is Gregory Travis, who wrote the original article on which the EE Times
article is based. If any of you have a specific question regarding my
conclusions or how I got them or want to discuss any statements of fact, I'm
more than happy to engage.

~~~
mhb
Why is this: “once this thing pitches up, it wants to keep pitching up”? And
why is it more of an issue with the engines in their new position? Thanks.

~~~
GregTravis
Couple of things here to keep in mind. First, when the A320 came out I wrote
extensively about its fly by wire system, which was highly controversial at
the time (early 1990s). It’s been nearly thirty years since then and long
story short, Airbus has vastly more experience with implementing cockpit
automation than Boeing. Boeing simply got in far over their expertise with
deadly results

Second, Airbus’ 320 airframe does not impose the same issues with larger
engines that the 737 does. For starters, the 330 airframe started life in the
era of large high-bypass turbofans — its initisl engines were much larger than
the 737’s initisl engines

~~~
GregTravis
It wants to keep pitching up because the engine cowlings are now far ahead of
both the center of gravity and the center of lift. And the cowlings generate
significant lift themselves. Aerodynamically they act as levers that pull the
nose up and the higher the angle of attack, the more they pull. That is
dynamic instability and as I point out you want to have ejection seats in any
dynamically unstable aircraft (ir fighter jets)

------
isquared23
This article appears to be fairly thinly sourced. The one named source I can
find appears to be a blog post by a fellow software developer who is an
instrumented-rated pilot, however has flown airliners in simulation only. The
article does not claim the source is a professional pilot or that they have
ever flown the 737Max.

With due respect, I am not sure whether that counts as enough expertise to
qualify someone’s opinion as news worthy?

------
kerng
Wondering if Boeing will be able to recover from this in regards to keeping
the MAX flying at all. I mean, I will always pick a different plane from now
on - just not a risk I'm willing to take in the foreseeable future. Not sure
if my stance is common though.

~~~
Brakenshire
How easy is it for a consumer to make that choice? Is the information surfaced
to consumers?

~~~
codesforhugs
You can see it in most search engines, and after the Ethiopian crash Kayak
announced they'll add an aircraft type filter. Other flight search engines
will probably follow suit.

------
weyman
I may be irrational but I’ll avoid the 737-MAX no mater what software they
push.

~~~
kurthr
I guess I'll keep looking for my keys under the lamp post.

~~~
TheSpiceIsLife
I vaguely recall this being a reference to something.

Would you mind elaborating.

~~~
function_seven
[https://quoteinvestigator.com/2013/04/11/better-
light/](https://quoteinvestigator.com/2013/04/11/better-light/)

------
CoolGuySteve
1) Like most people, I’m far from qualified to give aeronautical engineering
advice, but as fly-by-wire technology gets more advanced, won’t this be the
norm? ie: Airframes that are difficult to fly might always be more efficient,
so have a computer do the hard part.

2) This part seems like the real damning misdesign:

Boeing offered the single angle-of-attack sensor as standard equipment, and
charged extra for a second along with a “disagree” indicator that would allow
737 MAX pilots to “cross-check” a faulty sensor. Citing those decisions,
another observer noted: “Who would design a system with a single point of
failure?”

~~~
kalleboo
> “Who would design a system with a single point of failure?”

According to the original design, the MCAS was only supposed to adjust trim to
a level that was easily overridden by the pilot essentially just pushing on
the stick/adjusting the trim. If it had been implemented this way, sensor
failure would not have been catastrophic and hence doesn't require redundancy.
At some point this was either changed or implemented incorrectly so that MCAS
had much more authority.

~~~
kuzehanka
It was changed. Flight testing showed that much larger trim was required for
MCAS to function, and that was implemented. The failure was not reassessing
the risks after that.

~~~
russdill
Echos of the Hyatt Regency walkway collapse.

------
trhway
today i first time saw the 737 MAX frontal view. Initially i thought that it
was that typical funny plane-themed photoshop. I kid you not, it is the real
thing -
[https://i.stack.imgur.com/GFzcj.jpg](https://i.stack.imgur.com/GFzcj.jpg)

Just look at those nacelles. Deep breath. Look again. Take them in. Besides
clearly visually screaming that this Frankenstein thing was quickly&cheaply
slapped together and wasn't properly engineered and thus should just have
never seen the light of the day, these nacelles obviously add more lift than
normal symmetrical ones. So:

1\. the engines placed more forward than pre-MAX 737 - that results in
additional pitching up moment as the engines are below the centers of
pressure, gravity, etc.

2\. the engines are 2x higher-by-pass than pre-MAX 737 and thus the center of
thrust is shifted even more forward and lower - as a result it adds even more
of the pitching up

3\. these asymmetrical nacelles generate more lift just due to the shape - and
again due to the position of the engines that lift results in the additional
pitching up moment.

Basically that thing just can't really fly steady straight, and looking at all
this some people at Boeing decided that a bandaid software patch would just
fix it. Sounds like it were the same people who did the "curl" fix in today
Cisco story
[https://news.ycombinator.com/item?id=19508472](https://news.ycombinator.com/item?id=19508472)
:)

~~~
ihuk
I will not comment on (1) and (2) but (3) is wrong. I don't think that's even
a 737 MAX on the picture.

737 was originally designed to be low to the ground to make it easier for
ground crew to "bulk" load i.e. just throw stuff into the cargo area. The
reason why 737 could be so low to the ground was that they were using
turbojet[1] engines. Turbojet engines are very slim compared to new turbofan
engines used on later 737 generations. When they moved to turbofan engines
which have bigger diameter they needed to move them higher up from the ground.
So they moved the engines in front of the wings. To gain even more ground
clearance they moved accessory gearbox and fuel pump from underneath the
engine to the side. That's why engine appears flat on the bottom. Obviously
the engine is still round because the fan is round but the lip is flattened.
All this allowed Boing to fit more efficient and quieter engines to 737
without extending landing gear. The shape itself does not generate much
aerodynamical lift if any.

On the 737 MAX they actually extended the landing gear and it does not feature
"flattened" engine shape any more.

[1]
[https://en.wikipedia.org/wiki/Pratt_%26_Whitney_JT8D](https://en.wikipedia.org/wiki/Pratt_%26_Whitney_JT8D)

------
kchoudhu
This doesn't sound right: aren't unstable airframes controlled by computers
the norm now? If that is the case, a software fix _is_ the answer.

If unstable airframes are _not_ the norm, then the question we need to be
asking is how the regulatory regime let an unstable airframe into service.

~~~
neuronic
If you want to fly from Germany to the Canarian Islands for 50 EUR round trip
then you're going to end up with unstable airframes.

The 737 MAX is in part unstable due to repositioned engines. They are larger
and thus significantly more fuel efficient, however they would not keep enough
ground clearance if they were positioned the same way as on the 737-NG. MCAS
corrects for the repositioning, otherwise the plane may start to climb which
in turn could lead to a stall.

~~~
jlangenauer
> If you want to fly from Germany to the Canarian Islands for 50 EUR round
> trip then you're going to end up with unstable airframes.

How does B follow from A? That's quite the non-sequitur.

~~~
neuronic
B follows from A because in part A makes B economically possible.

Reductions in fuel consumption factor greatly into ticket prices. It's easy to
reduce services (luggage, refreshments, leg space) but reducing fuel
consumption is entirely up to the aircraft manufacturers.

~~~
usrusr
But you don't need instability to get fuel economy, all you need is an
airframe that actually matches the current engines instead of a relic from the
1960ies that was designed for tiny inefficient low-bypass engines. A plane
designed for the new engines could be perfectly benign without sacrificing
efficiency. Even a more properly adapted 737 that had the changes required to
solve the problems aerodynamically instead could be fuel efficient.

~~~
neuronic
Oh that's true, except a new airframe leads to new everything, including
certifications and training for pilots. Those do not come free.

------
isquared23
This appears to be a thinly sourced article, based on a blog post opinion by
someone who is a software developer and instrumented-rated (hobby?) pilot who
has flown airliners in simulation only.

With due respect, not sure whether that constitutes enough expertise to to
qualify and opinion as news-worthy?

------
qwertox
It would be interesting if these kind of companies (aviation, car companies)
were forced to publicly disclose the patch they are applying in order to fix a
broken piece software.

Maybe then they'd be more careful because of the extra scrutiny and the
potential leaking of secrets.

On the other hand, maybe then they'd patch as little as possible, although in
this case, if a second patch would be required, a very hefty fine could be
forced onto the company, or possibly force a full disclosure of all the
relevant source code.

Maybe the Blockchain could be used for some accountability here, where hashes
of the blobs of all the software in the system, including the secret one,
could be used as a means to prove that only a specific section of a codebase
has been altered.

------
novaRom
Everything made in US has become extremely expensive. As a product
manufacturer you have to pay a lot more than your Asian competitors. You can
open an R&D subsidiary in Asia to reduce costs, but in very short time you
will see your technology has diffused and now you have even more pressure from
competitors.

More bugs and design faults. Growth, innovation, effectiveness, all in a
shorter time. All with increasing costs.

And even more complexity, and more pressure. iCloud leaks, empty root
password, reboot by WhatsApp message, Meltdown, Spectre, 737 MAX, etc.

------
mosselman
I find the concept of fixing an airplane's hardware issues with a software fix
incredibly scary.

------
cmurf
As an example of the seriousness of approach to stall, buffeting and proper
recovery (you don't fall out of the sky like a rock, but recoveries take
hundreds and possibly a couple thousand feet), this two year old serious
incident in a 747 involving inadvertent stall while entering a hold, just had
its final report issued.
[http://avherald.com/h?article=4a787699&opt=0](http://avherald.com/h?article=4a787699&opt=0)

------
jacquesm
You can't really fix any hardware fault in software. The best you can do is a
workaround, but the whole will never be as solid as a properly designed system
would be.

~~~
jillesvangurp
My bet is that Boeing will spend some time banging on this and then the 737
goes back to being a money maker for them for years to come. I'm guessing that
at this point they know what is the problem and are probably pretty far done
with coming up with many solutions and picking one that gets the job done. The
way this works is that they have to do something because they are grounded.

This may or may not involve installing some extra hardware but it will most
certainly involve a software update.

------
chx
The big question is ... who will trust the Boeing - FAA duo after this? The
777X is coming, there surely will be rather pointed questions from airlines,
the EASA and more.

~~~
rjf72
I think many do not understand typical practices of regulatory agencies. As a
related example, what do you think the FDA requires in terms of genetically
engineered foodstuffs? Many seem to think there's extensive oversight and
safety testing. There isn't. They treat genetically engineered products and
natural products identically. If a company has all their regulatory issues in
order to market e.g. corn, they can cook up a new genetically engineered corn
in the lab and bring it to market with literally 0 additional oversight
necessary. All the FDA offers here is a completely voluntary consultation, and
that in turn basically is little more than the company signing off on some
checkboxes.

This leads to a bemusing and disconcerting run around.

Monsanto: "The Food and Drug Administration (FDA) is responsible for the
safety and appropriate labeling of food and feed products grown from GM
crops." [1]

FDA: "It is the manufacturer's responsibility to ensure that the food products
it offers for sale are safe and otherwise comply with applicable
requirements." [2]

Sound similar? It'll be the exact same story if/when a company inadvertently
releases a harmful genetically engineered product. The assurance of safety
provided by regulatory agencies is often illusory. As an aside, this is all
clearly described on the FDA's page as well. [3] But the phrasing is designed
to mislead consumers. They state repeatedly that it is unlawful to ship unsafe
food to consumers without ever directly clarifying that they themselves never
actually test the foods. Inventions go straight from Monsanto's lab to your
plate. Obviously they have a major incentive to ensure their products are
safe, but they have a long history of failing in that obligation yet remain a
multi billion dollar company.

[1] -
[https://monsanto.com/company/commitments/safety/statements/a...](https://monsanto.com/company/commitments/safety/statements/are-
gmos-safe/)

[2] - [https://health.usnews.com/health-news/health-
wellness/articl...](https://health.usnews.com/health-news/health-
wellness/articles/2013/04/25/gmos-a-breakthrough-or-breakdown-in-us-
agriculture)

[3] -
[https://www.fda.gov/Food/IngredientsPackagingLabeling/GEPlan...](https://www.fda.gov/Food/IngredientsPackagingLabeling/GEPlants/ucm461831.htm)

~~~
avar
Food safety isn't a function of relatively small changes in the genome of
plants you eat, this is pseudo-scientific nonsense. The "natural" corn or
animals you eat also experience genetic drift, and the FDA isn't tasked with
sequencing them and certifying each "change".

If the purpose of aircraft was to feed them to giants who'd digest them for
their raw materials Boeing wouldn't need to certify the 737 MAX either. But
aircraft are flown, so minute changes to their construction can make a lot of
difference. This comparison of yours makes no sense.

~~~
rjf72
Boeing obviously felt there was a basically 0% chance of their decision being
in anyway unsafe. And they are, arguably, the most qualified people on this
Earth to decide this. Of course they probably got blinded by profit a bit, but
it's not like this was a Ford Memo moment. A single plane going down is a
catastrophe. Two planes going down is something much worse. They obviously
felt everything was perfectly safe; they were wrong. Lots of people died. Even
though the most likely outcome is they'll get a slap on the wrist, I think
there's no way they would have gambled on this.

The reason I mention this is because I don't believe you believe it's
impossible to create an unsafe product as you are implying in your statement.
Genetic engineering technology enables us to hybridize anything. As a not
entirely random example you could combine an orange with genomic data from an
arbitrary virus or perhaps certain aspects of various plants in the nightshade
family, if you so wished. You can theoretically do great things with genetic
engineering, and you can certainly also do awful things. And there is no doubt
that you can also _accidentally_ do awful things. And I don't think short term
safety is the real concern. You're not going to drop dead after drinking a
cola because of some genetically engineered corn syrup in it. My concern would
be longterm unforeseen consequences.

For instance weight gain, fertility, and even cognitive and psychological
factors are all connected to what we consume in various ways that remain
poorly understood. And we're currently running a compulsory experiment in that
nearly all foodstuffs in the US now contain substantial components of
genetically engineered products. The rest of the world works as a control, to
varying degrees, due to radically less consumption of engineered products.
What will be the longterm consequences of this? Perhaps we're already seeing
them. Or perhaps the issues plaguing the US are caused by something altogether
different. The point I was making is that it's ultimately up to the individual
to come to their own decisions here. If you're happy to consume any
genetically engineered product in full faith then I fully respect your view,
even if I might disagree with the soundness of it [1]. I'd ask for nothing but
comparable treatment.

[1] -
[https://link.springer.com/article/10.1186/s12302-014-0034-1](https://link.springer.com/article/10.1186/s12302-014-0034-1)

~~~
avar
Instead of a single study you should look at systematic reviews.

Here's an article discussing a wide-ranging review the National Academy of
Sciences conducted, which is the sort of thing that informs the current FDA
policy: [https://sciencebasedmedicine.org/national-academy-of-
science...](https://sciencebasedmedicine.org/national-academy-of-sciences-
report-on-gmos/)

> I fully respect your view, even if I might disagree with the soundness of
> it. I'd ask for nothing but comparable treatment.

I'm not claiming you have to eat GMO food, or food that's been exposed to cell
phone tower waves or whatever.

But you weren't expressing a personal preference. You were suggesting that a
government organization like the FDA should be regulating something based on a
hypothesis that the current scientific consensus shows is baseless.

At that point you aren't asking for your view to be respected, you're
suggesting that government policy should be changed to enforce it on the rest
of us.

~~~
rjf72
I did link to an overview of much of the current state of the science. You
linked to a pop science article written with the impartiality and
professionalism of a Breitbart article, though it does in turn reference
something meaningful. Here [1] is the actual report from the NAS that that
page references. They comment directly on our little discussion. Page 513:
"FINDING: Not having government regulation of GE crops would be problematic
for safety, trade, and other reasons and would erode public trust."

It also goes into detail on the problems with "weak" regulatory regimes. I put
"weak" in quotes as any genetic engineering specific regulatory regime would
be stronger than the US' reliance on self regulation. For instance in one
study referenced (page 194) scientists ran a typical regulatory test (90 day
whole food study) with rice that was genetically engineered specifically to be
toxic. And indeed it was toxic. But over the standard 90 day test, no ill
effects were found. This is a quite a serious problem.

And the one final thing I'd hit on is that much of the research on genetic
engineering is driven by the companies that stand to profit from proving
everything is safe and beneficial. Similar to how at one time nearly all
science on e.g. leaded fuel was driven by interests that had a motivation to
prove that everything was safe and beneficial, and so that's exactly what they
did.

The NAS paper when discussing rat studies mentions, "Some found no
statistically significant differences [from consumption of genetically
engineered feed], but quite a few found statistically significant differences
that the authors generally did not consider biologically relevant, typically
without providing data on what was the normal range." later emphasizing again
after discussing various dismissed abnormalities detected in rodent studies
that "There was no presentation of standards used for judging what would be a
biologically relevant difference or for what the normal range was in the
measurements." In other words statistically significant differences were
simply completely dismissed as "biologically irrelevant", without ever
defining what would actually be considered biologically relevant. That's not
good science, to say the least - but it's the typical pattern in much of the
research for GE products, which tends to rely heavily on direct or indirect
industry funding.

And, I think you'll find your view that negligible regulation is acceptable to
be something very few outside of those directly connected to the genetic
engineering industry would find satisfactory. The only reason more people do
not voice concern is because they're generally completely unaware of the lack
of safety inspections for these products. This state of 'regulatory
subterfuge' is itself reason for a significant degree of cynical skepticism.
You want to regulate? Ok. You don't want to regulate? Ok. You don't want to
regulate, but strongly imply that you are? That's not ok.

[1] -
[https://www.nap.edu/read/23395/chapter/1](https://www.nap.edu/read/23395/chapter/1)

------
ncmncm
The key failure of the MCAS system I have not seen discussed is that if it is
overriden, and triggers again, it cranks the trim another notch. Trigger it
five times, and each time it makes the plane less flyable. When Lion Air
crashed, it had been triggered many times.

Making MCAS pay attention to two sensors might help a bit, but the disaster is
still latent. Once it trims, it should never trim again without a full reset
back to baseline. There are standards relating to this sort of thing in flight
assist, about how much "authority" an automated system may assert, in total,
and they were ignored, apparently because they did not treat it as part of the
autopilot system.

If the standard had been observed, the bad sensor could not have had much
effect on the flyability of the plane. The pilots would have needed to apply
some force to keep the nose up, but would have succeeded, long enough to
discover a fix or to turn around and land.

------
not4noob
Somewhat off-topic but this has been bothering me: Several years ago I was in
a program review and ended up in an argument with the lead Boeing software QA
person for a particular group. The disagreement was because the person made a
blanket statement that their QA process ensures there are no defects in their
flight software. My response was that such a statement is absurd and that all
software beyond some minimal complexity has defects. (A statement that I still
agree with even though it is hyperbolic.) None of this has any direct
relationship to the 737 Max issues as this wasn't even an airplane program but
I think it points to what might be a cultural flaw if this attitude is
widespread.

~~~
alkonaut
> ensures there are no defects in their flight software.

"Testing only proves the presence of defects, not the absence"

However, I think you are talking past each other here. The person you taled to
was likely _defining_ "no defects" to mean "all 1000 boxes ticked on the
spec/testing protocol". They should call it "known defects".

------
alkonaut
What if it doesn't? I can see two ways in which they risk ending up with
"major" problems

\- They need to make changes that means it's no longer a 737 for
certification.

\- They need to make changes that delay production and even mean recalls of
built plane, say moving engines rearward which in turn would mean big
airframe/landing gear changes to manage ground clearance. These would be very
expensive, cause tons of cancelled orders, and possibly also cause the same
issue with certification as the above.

Any of those chnages would maybe kill the whole MAX program. And that would
leave Boeing without a competitive plane in the most common class? Is this a
possibility?

------
sagebird
Why does an active MCAS system need to exist in order to tamp down on pitch
up? I wonder if modifying the surface of the wing and or tail could achieve
this. Eg, make a computer model of the plane, verify that it captures the real
pitch up behavior, and “evolve” a wing that counteracts it. I suppose the hard
part is that we are searching for a perhaps complicated non-linear response,
that needs to behave differently across speed, atmospheric conditions, pitch
and turning. But maybe there are enough degrees of freedom when evolving a
surface that it can capture it all?

------
ggm
The source article is attracting some heat. But, this aside, does anyone else
think the meta-questions about the FAA and their relationship to Boeing, and
the 'same type' certification process begs questions?

I "get" that people wanted this. But a regulator has to ask a subtly different
question: _is this actually in the wider public interest_ which is not
neccessarily what Boeing wants.

------
mcguire
" _Among Boeing’s critics is Gregory Travis, a veteran software engineer and
experienced, instrument-rated pilot who has flown aircraft simulators as large
as the Boeing 757._ "

Ok, what? I'm a veteran software engineer and I've flown (MS) flight
simulators such as the 747 (badly), and even I know that none of that gives me
any grounds to weigh in on this situation.

~~~
txsoftwaredev
"instrument-rated pilot" \- That's the key part that makes this individual
able to weigh in on the situation.

~~~
mcguire
There are a lot of instrument-rated pilots; it's not a high bar
([https://www.aopa.org/training-and-safety/active-
pilots/ratin...](https://www.aopa.org/training-and-safety/active-
pilots/ratings-and-endorsements/instrument-rating)).

There are far too many people making pronouncements on this issue who have no
business doing so.

------
novaRom
>> “By laziness, I mean that less and less thought is being given to getting a
design correct, and simple – up-front,” he wrote.

Competitors have more less expensive engineering resources. You simply cannot
produce high-quality, safe, stable complex systems in shorter time if you have
limited human-engineering very expensive resources. This is not only related
to aircraft-manufacturing.

------
z5h
What's this about "engineering incompetence", "engineering ethics" and
"“cultural laziness” within the software development community"?

Do we have _any_ reason not to believe that management is at fault? That
management forced engineering to do things quicker/cheaper and cut corners and
hack out a solution in those interests?

~~~
slavik81
Management pressure is not an excuse for faulty engineering. As an engineer,
your first duty is to the public, then your client, then your employer. That's
straight out of the engineering ethics handbook. Your boss comes third.

When you sign-off on a design, it is _your_ approval. _Your_ name goes on the
document. There's no passing that off as someone else's fault. If you are
unable or unwilling to say "no" to your employer, then you do not meet the
criteria to be a professional engineer.

Your employer may find another engineer to sign-off, but perhaps that person
will think on it carefully, knowing that someone else refused. If something
later went wrong, it could not be passed off as a simple oversight.

------
OJFord
What causes such inaccurate AoA readings? 'Vanes' freezing?

From what I understand MCAS is about pointing up/down, so although AoA can of
course be a more sophisticated angle in 3D space, couldn't the measurement for
MCAS purposes be accomplished with a sensor based on gravity that could be
entirely internal to the aircraft and so perhaps more reliable?

~~~
zamfi
There’s no real way to measure “gravity” in a plane, unfortunately.

AoA is about airflow directions anyway, so wind angle plays a role too, beyond
just plane angle to the ground.

~~~
OJFord
Suspended mass inside container fixed to plane structure, measure angle of
mass relative to container?

> AoA is about airflow directions anyway, so wind angle plays a role too,
> beyond just plane angle to the ground.

That doesn't sound desirable though, is it?

~~~
zamfi
> Suspended mass inside container fixed to plane structure, measure angle of
> mass relative to container?

Unfortunately not! This measures acceleration of the airframe, not gravity.
Was a huge challenge in the early days of flight, and there’s a great history
online that I can’t find right now. Anyway, this might be useful reading:
[https://en.wikipedia.org/wiki/Attitude_indicator](https://en.wikipedia.org/wiki/Attitude_indicator)

> That doesn't sound desirable though, is it?

Given that wing-angle-to-airflow is what matters when it comes to stalls, I’d
say it’s eminently desirable?

------
l31g
This article is clearly clickbait. I don't think they've done a good
investigative job. Even though I don't have any pilot license, I have flown
"full" 747, 767, 777 and 787 simulators, but this doesn't give me the
authority to make broad statements about an airframe being faulty or not.

------
ycombonator
The author mentions this engine in the article and there is a neat video of
the behemoth engine. Pure mechanical engineering excellence !
[https://www.youtube.com/watch?v=5CytG5M5Jcs](https://www.youtube.com/watch?v=5CytG5M5Jcs)

------
fixermark
It's odd to call the airframe "faulty" when (a) it flies and (b) the FAA
requirements for positive static stability in the airframe itself don't forbid
the 737 MAX from being flown.

Does the author mean the FAA airframe acceptance criteria are faulty?

------
RenRav
you can only make so many systems redundant before the weight increase and
lower capacity outweighs any potential risk of systems failing, or this is how
it seemed to me when i compared cargo carriers to other aircraft. you need
constant dedicated maintenance and regular troubleshooting, i don't understand
how more ptoblems like this haven't already occured. there is only so much you
can do without a reliable pilot to disable anything causing problems. and
having seen firsthand the problem solving methods and goto solutions for
maintainers, everytime a plane lands successfully it's a miracle

------
sudoaza
If the cost of the recertification is more than the expected profits, we don't
do one

------
systemBuilder
I can't understand how a switch this simple could fail, frankly. Wouldn't a
Mercury switch with an arc'ed path and several electrical contacts work? It
seems blazing simple and idiot-proof. We are not being told the entire story!
STAY TUNED!

~~~
kawfey
A mercury switch is probably not as reliable as a wind vane in high
turbulence.

------
jcims
Seems more like a faulty plan than a faulty plane.

------
w-ll
how `faulty` is the airframe compared to the F-35? I hear that needs massive
amounts of sw to keep it balance.

~~~
TheSpiceIsLife
If I recall correctly the F-16 is also an unstable airframe dependant on
software, yet arguably _higly successful_.

Although the F-16 is from a different era. Perhaps we’re no longer capable of
building complex systems?

~~~
Deestan
The F-16 requires specifically trained pilots to fly safely.

The Boeing in question was flown by pilots who were not trained for or even
made aware of significant modifications to the plane's behavior.

A training wheel on a motorbike is not very hard to make well or drive with
safely, but if you suddenly discover it during a sharp turn at speed, it's not
going to go well.

