
Boeing identifies new software problem on grounded 737 Max - rafaelm
https://www.bloomberg.com/news/articles/2020-02-06/boeing-identifies-new-software-problem-on-grounded-737-max-jet
======
MDWolinski
All software is buggy. The problem with the MCAS system is that pilots were
not informed that it was there, nor were they given a way to override it and
take full control of the airplane. Also, while the MCAS system relied on two
sensors, if either failed, the MCAS system itself failed, so there was no
built in back-up for it.

Bugs in software happen because situations where they arise are sometimes hard
to predict. You can test your software all you want but it's not until it's in
the field that you start discovering new issues because people tend to do
things in ways developers didn't consider.

Tesla's software has over a billion miles of data on it and it still has
issues in some basic functionality. And let's not talk about Iowa which in
itself was a major failure in software release management.

~~~
mnm1
Not all software is buggy to the point of killing almost four hundred people.
Comparing some shit app some interns built for Iowa with avionics software is
frankly insulting to the people who work hard to make avionics software. The
same goes for Tesla. The avionics industry, including Boeing, used to have a
great record in this area. Even if the mcas bugs were unavoidable, the fact
still is that the design was fatally flawed due to either sensor being a
single point of failure. And of course, the main problem that the whole,
entire airplane is unstable in the air. How can you still make excuses for
Boeing at this point in time? The only reason this bug should be irrelevant is
because this plane should never carry another commercial passenger. But I'm
sure profits will prevail over lives once again starting this summer or
whenever the FAA gives their go ahead.

~~~
catalogia
> _And of course, the main problem that the whole, entire airplane is unstable
> in the air._

That's not really true. The airframe is fine, except it doesn't handle like a
737. MCAS was meant to make the MAX handle like a 737.

Mentour Pilot, a 737 instructor with a youtube channel, has covered this
fairly extensively:
[https://youtu.be/TlinocVHpzk?t=951](https://youtu.be/TlinocVHpzk?t=951)

~~~
mnm1
Great video. But he says the exact same thing I did. The MCAS is necessary
because of the different engine placement. So the airplane cannot recover from
a stall without it. That, to me, makes the entire airplane unstable and
improper for commercial flights as this is an expected condition at times that
the plane should be able to recover from. The airplane cannot function without
a deeply flawed software system no one understands and no one knows how to
operate. Changing the software doesn't change any of these things.

~~~
catalogia
> _That, to me, makes the entire airplane unstable_

That's fine, but know that the way you're choosing to use the word "unstable"
doesn't square with how it's actually used in the aviation industry.

~~~
mnm1
Ok. I should have said completely unsafe for commercial passenger use instead.

------
kayfox
I think the main thing we are seeing here is hundreds of smaller fixes that
usually form the steady stream of Airworthiness Directives that an aircraft
currently supported by the manufacturer sees turning into a news event every
single time one comes out.

So far only one "aircraft" has had perfect software, and that was the Space
Shuttle, every single other aircraft out there has had software issues that
are worked out over the life of the aircraft, just like every piece of
software, even that which has very strict testing regimes, has had defects in
it.

~~~
blattimwind
> So far only one "aircraft" has had perfect software, and that was the Space
> Shuttle

Actually it had 3+ known bugs.

~~~
kayfox
Drat, we have no known examples of perfect software.

~~~
radiorental
10 goto 10

~~~
glitchc
Great for those cold winter nights!

Will probably be optimized out by a modern compiler though. Sad.

------
swiley
I really wonder about these large engineering corporations, Toyota seems to
have similar problems with software.

Part of me feels like many of these companies don’t keep code secret to
protect IP, instead they do it because they know it’s a burning train wreck
and don’t want people to find out.

~~~
salawat
That's interesting. Toyota would be one of the last companies I'd expect to
hear that about. They're notorious in Quality circles for taking Quality
seriously; at least as far as their production line is concerned. Do they not
apply that same philosophy to in house software?

~~~
mark-r
The investigations carried out for unintended acceleration in Toyotas didn't
paint a good picture.

[https://www.safetyresearch.net/blog/articles/toyota-
unintend...](https://www.safetyresearch.net/blog/articles/toyota-unintended-
acceleration-and-big-bowl-%E2%80%9Cspaghetti%E2%80%9D-code)
[https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_...](https://users.ece.cmu.edu/~koopman/pubs/koopman14_toyota_ua_slides.pdf)

~~~
s5ma6n
Damn...

\- "No configuration management"

\- "No bug tracking system"

\- "No formal specifications"

\- "9,273 – 11,528 global variables"

\- "Uses recursion, no mitigation for stack overflow. Memory just past stack
is OSEK RTOS area"

I thought of Toyota as a much better company in terms of safety and
reliability. I can't imagine other manufacturers and their code.

------
vikramkr
So not only are they trying to fix a fundamental hardware issue with a
software patch, their inability to do software properly extends beyond just
their MCAS system? This is a good reminder that air travel's extraordinary
safety record isn't just a given, it's something that takes real work to
achieve and when the people responsible for putting in that work (Boeing,
regulators) begin taking safety for granted, that's when people die.

~~~
totalZero
I agree. But we should keep in mind that there's no such thing as a bad apple;
we can't blame individual executives or regulators.

It's a bad barrel: a company that has, on a cultural level, put its business
motive above its responsibility to deliver a safe and high-quality product. We
have seen documented evidence that employees knew there were dangers and
problems, and discussed these issues, but nobody cared enough to slow things
down and get the product right.

~~~
vikramkr
Absolutely. That's why I said Boeing and the FAA failed their responsibility
as opposed to a Boeing exec or a particular legislator - there are
organizational, structural problems. Sure, some individuals made the decision
to ignore reports or set a new culture, but the fact that they succeeded is
concerning - why did everyone else enable them? Is there anything we could
have done to encourage those engineers to whistleblow their concerns before
the planes crashed? Would they have been taken seriously by the FAA or the
media or investors who were pushing for growth at all costs? These are deep,
structural problems.

~~~
salawat
Interesting you're getting downvotes. Not sure if the general audience or not.
Regardless, the primary motivation to get a worker in a highly consolidated
industry to blow the whistle is for them not to feel like they only have one
choice. I can see where Boeing having consolidated as much as it has can give
the culture extra resilience against employee disruption simply from the fact
there isn't anywhere else to go.

~~~
vikramkr
I think if you get one or two downvotes for whatever reason, as hn starts to
grey out the text, a bit of bandwagon effect happens, not sure why.

Good point about whistleblowing. Perhaps the faas reliance on self regulation
alsobplayedbibto that consolidation, so even the one other place they might
have gone was just something that looped right back to the monolith.

~~~
stallmanite
Perhaps the threshold for graying out text should be much higher as the
current setting is very path-dependent in that the first few people to vote on
a comment have a disproportional effect.

------
clSTophEjUdRanu
As a former software defense worker, I wish there were 3rd party audits of
code and dev ops. If you saw the code that's flying in missiles, aircraft, etc
and how they got there youd want to go live in a cave.

~~~
jacquesm
Some whistleblower should one day post an archive of Airbus or Boeing's
software archives. That would make for interesting reading.

~~~
Glawen
It's usually worthless without knowing what is attached to the input/output of
the microcontroller. A lot of things are done ecternally on the wiring.

------
platz
> designed to warn of a malfunction by a system that helps raise and lower the
> plane’s nose

So, they can't even name the mcas system anymore?

~~~
slumdev
I thought that's what the MCAS was. Unless there's more than one system that
overrides the pilot to pitch the nose down?

~~~
kayfox
Speed trim and the trim system in general.

------
onychomys
In their defense, probably every piece of software of any complexity at all
has bugs waiting to be found, and it's not super surprising that they found
some new ones while doing a rigorous testing regimen.

~~~
coldpie
In their defense, the software industry is a complete joke in terms of quality
control. I hope that this and the spate of ransomware will wake the industry
up to realize we need new processes, languages and tooling to make software
provably correct. It's clear we can't use our existing languages and tooling
to make high quality software. Realistically though, that's not going to
happen. The smart move is to eliminate as much software from your life as
possible. It's only going to get worse as decades of laziness catches up to
us.

~~~
m_fayer
Watching Android descend from a place of dorky stability to sleek sealed
glitch-city, kinda makes me think it's the institutions, as opposed to the
tooling, that's behind the plunge in the quality of mainstream software.

~~~
peteradio
Feed me features I shit backlogs.

~~~
salawat
I couldn't help but laugh, because it is so damn true.

It's taken me a while to come to the realization that all project management
is is a backlog manufacturing and prioritization layer that operates on top of
actual software engineers.

Most just want to implement things, and don't care what it is they are
implementing as long as they are getting paid.

------
benwerd
There is a less than zero chance I'll be boarding one of these planes again,
ever. Trust is an important idea in any product, but particularly in areas
like aviation, and I don't see how they can possibly build it back.

I _do_ see an opportunity for software that ensures you are only booking
journeys on the aircraft you feel are safe.

~~~
apexalpha
on the other hand, the software for this plane might be the most scrutinized
ever when all this is over.

~~~
oska
The hardware remains fundamentally flawed.

------
frandroid
> Asked about a likely date for a return to service for the Max, Dickson said
> it isn’t helpful to talk about timelines. Boeing needs to concentrate on
> making complete, quality submissions on its fixes for the plane, he said.

Ahh, "we'll ship it when it's ready, not on some arbitrary deadline." Music to
any engineer/builder's ears.

------
stagas
Can someone explain how a hugely complex machine with mostly parallel working
analog parts fits into the digital computing paradigm? Isn't it predetermined
to fail under extreme conditions, like those that are found while flying
inbetween clouds and thunderstorms with all that pressure and fluctuations?
How does sampling not fail, like, all the time? What kind of tooling is being
used to mitigate for all these? Does anyone know?

~~~
garbage_88564
I am very naive to commercial aviation but this is my experience with building
and crashing model aircraft repeatedly. I fly mostly FPV which puts me in the
first person view from the cockpit.

Yes, electronics fail in the most weirdest ways due to connector failures, RF
interference, software error, sensor failure.

When my systems start failing or acting up due to improper stabilization PID
gains, etc. I have a big switch for MANUAL mode. I am able to fly this thing
as long as the servos, radio, and camera get power. All sensors could be
sheared off. I have no idea what my airspeed is ever because I don't use pitot
tubes so I use a known engine throttle % whose stall characteristic I
understand for level flight in various wind conditions and I don't make sudden
maneuvers at throttle below this point.

Fixed wing planes have remarkable aerodynamic stability and I don't understand
why 737 MAX cannot be piloted in a fly by wire manner with all computer aids
disabled, giving the pilots direct control of the servos with a big red switch
that mechanically disconnects the flight computers. This requires almost no
code to implement.

~~~
jaywalk
On Boeing aircraft, the pilots essentially _do_ have "direct control of the
servos" at all times. MCAS was implemented to make the MAX fly just like the
NG despite the difference in engine size and placement. What MCAS actually did
was not modifying the pilots inputs, but adjusting the stabilizer trim in
certain scenarios.

The pilots _do_ have direct control over the stabilizer trim, and have always
had the ability to disable the electronic system in case of stabilizer trim
runaway. This was not new to the MAX, and would have effectively disabled
MCAS.

~~~
garbage_937648
Pilots do not have direct servo control of the aircraft if there is any
possibility of any computer system adjusting the servos aside from the throw
commanded by the sticks held by the pilot.

Reading comments such as:

> The problem was that an indicator light, designed to warn of a malfunction
> by a system that helps raise and lower the plane’s nose, was turning on when
> it wasn’t supposed to, the company said.

Implies that there is intrinsically some computer system that continually
parses the commanded stick deflection and applies an overlay.

What I am suggesting is a single toggle to make everything shut up and reset
all servos to their midpoint all at once in one shot and let the pilot just
fly the plane.

I have not seen any evidence that such a system exists. It is the elephant in
the room. Airplanes do not need complex electronics to just fly if they are
aerodynamically stable, and this plane is more or less stable except that
under some conditions it will make the pilot soil their pants at higher AoA,
which is where the promises of MCAS come in. Big deal. They can mentally
compensate against that manually better than fighting a computer system
working actively against your commanded inputs.

I have experienced the joy of a badly tuned PID controller turning my
stabilization system into involuntary high speed descent. The fix is always to
tell the computer to shut up and just fly the plane 100% manually.

~~~
pdonis
_> Implies that there is intrinsically some computer system that continually
parses the commanded stick deflection and applies an overlay._

That's not how the 737 works. The 737 is not a fly by wire aircraft. The
pilots control the rudder, ailerons, and elevators electrohydraulically; there
is no computer filtering. The electric stability trim system, which is what
MCAS feeds its input into, controls the trim tabs on the elevators. This does
not change anything about the pilots' inputs to the elevators, but it does
change the aerodynamics of the elevators in a way that can limit the pilots'
ability to control pitch.

 _> this plane is more or less stable except that under some conditions it
will make the pilot soil their pants at higher AoA, which is where the
promises of MCAS come in. Big deal._

If the 737 MAX had been a new aircraft type, it would not have been a problem.
There might still have been some adjustment needed to meet FAA certification
requirements for stick force (basically, the stick force is supposed to
increase with increasing angle of attack, so the pilot has to pull harder to
keep the nose going up as you get closer to a stall). But there would not have
been a need to cobble together anything like MCAS.

The problem was that Boeing wanted the 737 MAX to be certified under the
existing 737 type certificate (because otherwise the potential customers
wouldn't want it, since they didn't want to have to re-train and re-certify
all their pilots), which meant that the stick force as a function of angle of
attack had to be the same as for previous 737s. But the new engines on the 737
MAX made the plane aerodynamically different, so the "natural" stick force was
different. MCAS was a software kludge to try to change the stick force.

~~~
kayfox
>If the 737 MAX had been a new aircraft type, it would not have been a
problem.

If the aircraft was a new type this still would have been an issue that would
have to be corrected, see FARS 25.173.

~~~
pdonis
Yes, which is why I said: "There might still have been some adjustment needed
to meet FAA certification requirements for stick force"

------
notadoc
Throw in the towel on the 737 Max and go back to the drawing board.

------
bsimpson
I'm shocked they keep publicly working on this plane.

I get that planes cost more money than I can fathom, and that making a whole
fleet of impossible amounts of money costs a gazillion dollars. Still, this
one seems spent. Nobody is going to knowingly fly on a 737 Max.

They ought to have retired the plane last year. They can design a new plane
(that because of economics, will probably be very similar to this plane),
release it when it's been properly vetted, swap out Maxes for it, retrofit
those Maxes, etc.

I realize this is naive armchair quarterbacking from someone who has never
worked in aviation, but there's a reason that Philip Morris is called Altria
now and that Weinstein Co was merged into Spyglass. If the public doesn't
trust your brand, no amount of "but we fixed it with this patch we rushed out
the door" is going to change that.

~~~
yellow_lead
> Nobody is going to knowingly fly on a 737 Max.

I will respectfully disagree with you. Most airlines won't tell you your
aircraft when you book a ticket and even if they do, they seem allowed to
change it at the last minute. If you have spent $500,$1000, $XXXX on a ticket,
will you not board if you discover the plane has been switched? Will you avoid
airlines that don't guarantee you a certain plane?

There are not many great options to travel long distances quickly. If this
plane becomes commonplace, I'm afraid consumers will be forced to use it. If
there are other ideas on how this could play out, I'm happy to consider them,
but I fear this is nearly a given.

~~~
blt
> Most airlines won't tell you your aircraft when you book a ticket

I don't think this is remotely true. I just checked Delta and British Airways;
both show the aircraft type in "details". (Pick the 747 flights from BA while
you still can!)

> even if they do, they seem allowed to change it at the last minute

Airlines shuffle aircraft occasionally, but usually the aircraft type for a
particular flight is predictable. If airline A flies the 737 Max and airline B
flies the A320 on the same route, people will flock to airline B.

~~~
gpm
Last time I was on a delta flight, their terms of service (or whatever they
call them) explicitly reserved the right to swap out the type of aircraft at
any point in time. That was less than a year ago, I assume it is still the
case.

~~~
oska
They won't swap you to a 737-MAX if they don't own any of that aircraft.
Delta, your example, does not have any in service and has not ordered any.
It's not hard to find out which airlines use the plane and which don't.

------
sheeshkebab
There are two kinds of software:

\- buggy

\- the one where bugs were not yet found

~~~
gibbonsrcool
\- formally verified software

I only remember hearing about "mathematically proven software" as an undergrad
and just googled to find this name. I've always been interested in learning
more of what it's about but never jumped in.

~~~
aidenn0
Formally verified software still has its limitations. Knuth's famous "Beware
of bugs in the above code; I have only proved it correct, not tried it." is
funny but true.

Formal verification is a good and useful tool, but it provably cannot cover
the entire system, and practical limitations will limit it even further.

Formal verification of source code is still subject to compiler bugs. Formally
proven compilers are subject to bugs in the larger system (IIRC Csmith was
able to find an incorrectness in code generated by CompCert because of a bug
in a system header file).

~~~
Gibbon1
Also formally verified code may behave very badly when the underlying hardware
fails.

~~~
gpm
If the hardware is behaving out of spec, it's not the software failing.

If the hardware is behaving in spec (e.g. 1 out of 3 computers fails) and you
properly formally verified the software to that spec, the software will not
behave badly.

------
djsumdog
I hope these planes get scrapped and never see flight again. Tear them down
and recycle the parts, and build something that's modern from the group up
instead of recycled, deprecated bullshit.

~~~
V_Terranova_Jr
This just isn't going to happen.

