Hacker News new | past | comments | ask | show | jobs | submit login
Boeing Believed a 737 Max Warning Light Was Standard (nytimes.com)
503 points by aaronbrethorst 6 months ago | hide | past | web | favorite | 324 comments

> But months after the planes were flying, company engineers realized that the warning light worked only on planes whose customers had bought a different, optional indicator.

If I am to read this correctly, the angle-of-attack disagree light is installed on all Boeing 737-MAX aircraft. For customers who did not buy on optional additional indicator, the angle-of-attack disagree light was always off, even if there is a disagreement between to sensors of the angle of attack of the aircraft.

Doesn't this seem dangerous in-and-of itself? If a pilot assumes that the light is off because there is not a disagreement, instead of because there is no information available, she is acting on incorrect information, and might try to address problems unrelated to the actual problem.

Also, pilot may switch aircraft and expect the same functionality of safety features and may make incorrect assumptions about the functioning of this light.

The "light" is just a warning text that appears on the flight display, so the "no light installed" and "light installed but off" states are not distinguishable (as far as I have understood it).


The "light" is just a warning text that appears on the flight display, so the "no light installed" and "light installed but off" states are not distinguishable (as far as I have understood it).

They're absolutely different. If you're a pilot and you expect to see an error message when the two alpha vanes disagree, you'll be lulled into a false sense of security when that text never appears even in the case of a failure.

A pilot needs to be able to trust all of the instruments in the plane.

I'm a pilot, and absolutely this is a crucial difference.

It also reminds me of a time years ago when a team I was on had to extend a Memcache client library, as it couldn't differentiate between the following two conditions:

- "I tried to retrieve a key from cache but was unsuccessful (due to connection error, etc)"

- "I tried to retrieve a key from cache but determined conclusively that key does not exist in the cache"

If it was important for a caching layer on our silly social game, the same thing is clearly important in aviation.

Most people experience the same when using DB, NULL in a database is hard to distinguish from, data never stored or data don't exist.

Do you mind explaining what you mean? To my knowledge, inspecting the number of rows returned eliminates this problem.

I think he's referring to something like a row with First, Middle, and Last name fields. If the Middle name field is null, does it mean "we don't known this value" or "this value is <nothing> (the person doesn't have a middle name)". For most database schemas, there is no way to tell the difference between the two states.

While this is overkill for most applications, the HL7 V3 data types used in healthcare even have multiple different "flavors" of null. So you can distinguish between "no information", "unknown", "not asked", etc.


The latter case, no middle name, should be stored as an empty string. In practice, not enough care is devoted to the difference between Null and empty string for text fields.

"First not null, Middle null, Last not null" represents an a priori assumption that everyone has a First, Middle, and Last - but sometimes we don't know what the Middle is. Allowing a zero length string in any of those attributes implies that some rows in a table have First, Middle, or Last represented by a zero length string. What a zero-length string actually means is subject to interpretation and thus beyond the data model.

in the real world it turns out not to be useful to require that people have names just in order to keep data about them...nor does a name particularly distinguish one person from another.

Reminds me of that Red Dwarf episode where Rimmer wants to go to red alert and Kryten asks "are you SURE sir? it does mean changing the bulb!".

Reminds me of a different Red Dwarf moment. Any damage? I don't know, the damage-report machine's been damaged. Apparently they played the same gag twice:



Good one!

I think you’re misinterpreting. The parent means that by design the plane makes the states indistinguishable.

If you see two AoA sensors that aren't in agreement then you can safely assume either one sensor has failed or physics itself has.

The 737 has AoA sensors by default, the autopilot uses them and the flight computer corrects the flight speed based on the AoA and issues stalling warnings based on the values.

But the display which displays raw AoA readings to the pilots is an optional extra, only 20% of 737 maxes ship with it enabled. Normally pilots (or at least commercial pilots) are trained to fly without referencing the AoA by staying well within the safety margins.

So, if the AoA isn't shown anywhere in the default configuration, there is no way for the pilots to crosscheck them and see the fault. Which is why you need the AoA disagree warning message.

Unfortunately, Boeing messed up and the AoA disagree warning was only enabled on the planes with the AoA display option enabled.

> if the AoA isn't shown anywhere in the default configuration, there is no way for the pilots to crosscheck them and see the fault. Which is why you need the AoA disagree warning message.

But the article buried the lede!

There is much bigger problem reported now in this article:

The warning wouldn't have helped even when flying with those companies that paid extra for it!


"“We were told that if the A.O.A. vane, like on Lion Air, was in a massive difference, we would receive an alert on the ground and therefore not even take off,” said Dennis Tajer, a spokesman for the union representing American Airlines pilots. “That gave us additional confidence in continuing to fly that aircraft.”

But in the last several weeks, Boeing has been saying something different. Mr. Tajer said the company recently told American pilots that the system would not alert pilots about any sensor disagreement until the aircraft is 400 feet above the ground."

Note what happens then: at the moment you are above the ground, and the MCAS activates, we already know that if you "just follow the procedure" the forces against pilot corrections could be too big to do anything. That's what can be concluded from the analysis of the Ethiopian crash.

That means the plane from the "20%" with "paid option" would still eventually crash as soon as AoA vanes fail, just there would be one more of many alerts in the cockpit for some short time before everybody dies.

The sense of "security" Boeing gave the American pilots flying for the companies who paid for that additional feature was completely false. They would have been doomed too as soon as a single AoA vane that the MCAS depended on failed: The warning worked only at the moment it was completely useless.

The U.S. MAX planes were grounded the last, yet they were in the same danger as the other planes.

That's certainly not how the information is presented. But if you read closely you can see how they set it up carefully in the first half.

This is about adding to the narrative of Boeing's "general mismanagement" of the aircraft. Not about adding to the particular 737-800 crash investigation story (directly).

On both the Lion Air and Ethiopian Air crashes, the AoA sensor had failed before takeoff.

So the warning message could have potentially prevented both crashes and anything like it (assuming it activated during the takeoff roll, before V1 and the pilots were trained to abort take off when it happened)

Though you are right, if the AoA sensor had failed halfway though a flight, they still would have had problems.

> On both the Lion Air and Ethiopian Air crashes, the AoA sensor had failed before takeoff.

Are you sure? The AvHerald narrative for the Ethiopian Airlines flight:

"The takeoff run was entirely normal, both AoA sensors were in agreement. Shortly after becoming airborne the left AoA sensor however began to deviate and reached a position of about 85 degrees nose up, which obviously triggered the left hand stick shaker motor (and provides a very noisy and distracting cockpit environment), which in turn, different to the 737 NG behaviour (where the AoA does NOT correct the pitot data), also caused the IAS and ALT data to differ from the right hand system, in particular the left IAS became 12 knots higher than the right hand IAS (the preliminary report does not mention that the IAS DISAGREE warning activated however, which would invoke the unreliable airspeed procedures). The captain was pilot flying and focussed to keep the aircraft on track (via the flight director) due to terrain around while at the same time trying to understand why the stick shaker activated. Several required callouts by the first officer did not occur (e.g. speed, ...), as result the crew lost speed as well as thrust control completely out of sight."

As I said somewhere below, I don't think the AoA disagree indication would have made much of a difference here.


The new information in the article (buried in the last part of it!) is that apparently the warning message just didn't work on the ground at all, even with the already non-functioning sensors, and even if that option was paid for; read it again:

"We were told that if the A.O.A. vane, like on Lion Air, was in a massive difference, we would receive an alert on the ground" ...

"But in the last several weeks, Boeing has been saying something different. Mr. Tajer said the company recently told American pilots that the system would not alert pilots about any sensor disagreement until the aircraft is 400 feet above the ground."

Knowing nothing about these things, are the AoA sensors accurate on the ground? I assume they depend on predictable airflow, of which there's initially none and Boeing have to draw the line somewhere as to when they become reliable?

Even if they're accurate, due to the ground effect[0] when an airplane is close to the ground it can have significantly higher AoAs without stalling because the stalling speed is so low at those heights.

[0] https://en.wikipedia.org/wiki/Ground_effect_(aerodynamics)

> when an airplane is close to the ground it can have significantly higher AoAs without stalling

No. A wing will stall at a consistent critical angle of attack. Ground effect works by increasing the lift available for a given angle of attack, not by allowing a larger angle of attack to develop.

Parent comment was asking about the need for minimum airspeed to set the AoA sensors into a reliable state such that they could be checked at the gate, though, which is unrelated to ground effect.

Not necessarily at the gate, but possibly during taxi and definitely during the takeoff roll (ideally before V_1).

Granted, btw, that AoA readings on the ground may be unreliable - the question remains whether AoA disagree could be detected reliably before takeoff.

If the issue you're worried about is only MCAS related, it seems like the takeoff roll might be a poor place to want to detect it.

An abort just prior to V1 is a no-joke exercise from a safety standpoint. MCAS operation is suppressed with any flaps deployed, so taking an airplane with a failed AoA sensor into the air and troubleshooting/returning to the departure airport if needed or simply flying with an INOP MCAS may very well be safer than a high-speed abort.

Good point. Instead of aborting take-off, either do a circuit back WITHOUT retracting flaps, or set Stab Trim switch to CUTOUT and fly to destination trimming manually with the trim wheel? Wonder what pilots are briefed to do currently (well, if MAXes were flying).

Sure. Problem is, if you can see the two disagreeing AoA indications, then you will get the "AoA disagreement" warning light/message, even though you don't need it.

Conversely, if your airline has not ordered the optional AoA indicators, and you'd really want the "AoA disagreement" warning light/message, you don't get it.

Serious tangent, but the idea of physics failing amuses me. One night we had an Earthquake in an area that basically almost never has them. Gentle shaking. It woke me and my wife up. I remember looking out my window for some sign of nuclear weapons going off or something. Nothing. I blurted out “Earthquake” all in about 10 seconds of processing. Rolled over. Went back to sleep. My wife was anxious the rest of the night and could not sleep from “what if” scenarios. One of her concerns was “what if physics was broken”?!

What if the moon’s too bright? https://en.wikipedia.org/wiki/Inconstant_Moon

What if it’s only there when someone looks?


Damn...I read that link as "Incontinent Moon"

> One of her concerns was “what if physics was broken”?!

I think this is the wish of very a many physicist as they drink their morning tea and contemplate the day.

No place on the planet is immune to earthquakes, it's just that they're much more common in some places than others (i.e., near fault lines). The continental plates are always under stress and can move at any time.

That's kind of the plot of 'Ravages' (https://en.m.wikipedia.org/wiki/Ashes,_Ashes) a French SF novel by Barjavel.

Electricity suddenly stops working.

Also Fade Out by Patrick Tilley


So neurons stop working?

I've seen stories that address that by positing that it turns out that our intuitive understanding of ourselves as fundamentally "simple" entities is in fact correct, and the electricity our neurons used was just an epiphenomenon based on our local conditions. If you physically move your body into another location, suddenly those rules will work on you, and lo, you really will be running on the 4 humors, or whatever other theory is in play.

If they're on either side of the fuselage and you're slipping, it seems like they'd give different readings. Pitots can give misleading readings if you slip away from the side they're on. I've never flown a plane with an angle of attack indicator.

If they're on either side of the fuselage and you're slipping, it seems like they'd give different readings.

Which is why there's a threshold that much be crossed (IIRC ten degrees) before the disagree warning appears.

Most of the time when you're slipping close to the ground you're going to also have flaps deployed, in which case the MCAS is inoperative. Your AOA sensors will definitely disagree, but at least the MCAS won't act on them because it doesn't act when flaps are deployed.

If you're slipping without flaps I guess it could still be a problem if the disagreement is outside the epsilon spec. I'm not sure if [normal] slip could ever cause that much disagreement.

I'm now imagining a 737MAX8 in a flat spin...

(I've had the controls of a 172 and a Tiger Moth briefly, but have never _really_ flown a "real plane". I have flat spinned many many rc planes, mostly recovered fine, but also to their occasional doom...)

Or the sensors are on different sides of the plane and the plane is slipping or banking. Which happens routinely during normal flight.

While that's (partly) true, it's also pretty irrelevant to whether one can reasonably infer "AoA sensors agree" from the lack of an error notification indicating they don't, when the system that should be providing that notification is, in fact, disabled, because the airline one works for opted not to purchase some premium feature.

Looking at the mechanism of a vaned AoA sensor, I am guessing a fast enough roll would induce disagreement. Probably insufficient in degree to be material, though.

Thank you for sharing that image. Now the UX question is: would that small bit of yellow text in a cockpit with hundreds of visible UI elements make any difference?

It would imho. If the plane is in a failure mode, the user would search the UI specifically for error messages.

As a UXer I would _assume_ error messages would be in a consistent section of UI real estate. However, even if this is not the case I would expect both experience and training to mitigate the "can't find the error message among all das blinkenlights"

Yes absolutely, all failures are announced in that same place on the PFD.

The pilot flying has eyes on the PFD continuously, only looking away for a few seconds at a time if handling other systems. Essentially when flying instruments you "scan" the 4 primary pieces of information in the middle of that screen every few seconds.

On top of that there is an audible reminder when a new failure annunciator appears, so even if they were on autopilot and not paying attention (rare so early after takeoff) they would hear it and look.

I see. It is still the same problem where the pilot does not have an indication as to whether the safety feature is telling them that there is not a problem or if the safety feature is not telling them anything about the existence of a problem.

Does a pilot know whether the particular plane has it installed or not?

No, that's the point. All 737 MAX planes were supposed to have an angle of attack disagree annunciator. Boeing thought all 737 MAX planes were sold with a working angle of attack annunciator. Turns out only some of them had working warnings. Boeing discovered that the annunciator would only work under certain configurations but decided that this wasn't serious enough to notify airlines or the FAA. And then Lion Air crashed one of these managerial marvels.

The more that comes out of this, the more my opinion solidifies that they just have no effing clue about what the heck they are doing.

I'd expect building safe, efficient, high quality planes to be the answer. Whereas in their minds it's probably generate record profits by maximising ROI on past investments, protecting profit margin at all times having priority over things that might improve quality. Oh and with a complete spaghetti monster of responsibility layers.

I just keep getting more and more disgusted they are getting away with manslaughter.

Indeed, the 737 MAX is a dangerous bastardized old design but a managerial marvel.

- "We thought the 'AoA disagree' warning was functional in all jets we delivered, but turns out it works only if this other paid upgrade has been installed."

- "We planned for, and certified, an MCAS operating within certain boundaries, but then delivered a plane that exceeds these limitations multiple times."

- "To save money, we built a plane whose handling characteristics are so crappy that we could not certify it, unless we put in an extra system interfering with primary flight controls, but let's not tell anyone about it."

- "We've added this powerful system interfering with primary flight controls, and it depends crucially on AoA data, which we receive from two sensors. Well, let's just pick one of them and ignore the other one. It'll be fine."

Managerial marvel? I assume you're not sarcastic, so what do you mean?

You assumed wrong, or at least half-wrong.

It's a managerial "marvel" because they managed to certify and deliver that mutant flying turd adding billions to their bottom-line, earning their bonus while spending Boeing's good name and capitalize their decades-long "investment" on FAA.

That's top-notch management right there...

That's just typical management for American corporations these days. If you want a company you can really trust with human lives, you buy from a company that isn't run by Americans. Of course, that's no guarantee (see: VW Dieselgate), but American management these days is all about next-quarter profits over all else. Japanese-run companies are probably the most trustworthy ones; people there care more about the long term and personal honor. Too bad they don't make planes, but that's changing, as Mitsubishi is now getting back into the airplane business.

> You assumed wrong

Glad to hear :-)

It is the point, because if the pilot doesn't know whether or not it's installed, then absence of warning does not imply absence of problem.

If a pilot knows or believes it's installed, then they might discount a theory because the instrument appears to disagree.

> Doesn't this seem dangerous in-and-of itself?

To me, more dangerous, is that some safety systems are considered optional. Surely the plane either needs a safety feature or not, and it shouldn't be down to someone at an airline to make a decision on?

There is usually no such thing as "safe" or "not safe". It's more like "X% chance to happen". X might be really, really small, but not zero. Usually you can make X arbitrarily small, but the amount of money needed grows non-linearly. Do you pay 10x the price of an aircraft to reduce the chances a problem so that it goes from happening once every 100 years to once every 1000 years?

Who should make the decision whether an airplane is "safe enough"? Should it be the buyer who wants the safest plane? If you did that then rich airlines could keep out new competition by specifying safety features that make airplanes more expensive than they need to be. Should it be the country where the airline is based? But different airlines are based in different countries. Should there be options so that one country can specify higher standards than others?

Basically, that's how you get where we are I think. I'm not sure how you avoid that.

Who should make the decision whether an airplane is "safe enough"?

How about these guys? https://en.wikipedia.org/wiki/International_Aviation_Safety_...

it's worth pointing out, however, that it was a software-only feature that they turned on or off depending on whether the customer had paid. If I understand right?

I think that changes the calculus of "Do you pay 10x the price of an aircraft to reduce the chances a problem so that it goes from happening once every 100 years to once every 1000 years"

They had already invested the money to develop the feature. Delivering a plane with or without it to a customer had the same cost. If the customer paid more, they got the safety feature turned on, the end.

> Delivering a plane with or without it to a customer had the same cost

Not exactly, surely. The ongoing/recurring costs to Boeing would be increased with the feature enabled - it's another thing that can go wrong and must be monitored, it changes support requirements and the overall software runtime complexity. I assume that would get factored in, somehow...

Different countries have different standards of safety, and different budgets. Boeing got more $$$ from airlines in the US by making planes in Ethiopia and Indonesia less safe. Yay capitalism.

Maybe should have run adds in the cockpit

That’s weird to say. Lion Air is a sloppy, cheapskate airline — look at their horrible maintenance and crash history for evidence of that. That they didn’t want to pay slightly extra for additional options on the plane was their choice. And Ethiopian is a flag carrier: they could have easily gotten the extra option, but likely some accountant thought it wasn’t necessary. Who spends over $100 million on an airplane and then declines a $1 million safety option?

Yeah, blaming the airlines and not the obvious flaws in the plane and the way it got its security certifications is the way to go...

People should have paid for the "warning when auto-crash enabled" option and I'm sure Boeing advertised their plane as absolutly unsafe without it...

> Who spends over $100 million on an airplane and then declines a $1 million safety option?

Someone being pitched 30 different $1 million safety options?

Someone who wants to buy 100 of those airplanes, but knows that the safety option reduces the risk of incurring a $100 million total hull loss by less than one percent, thus saving less than the $100 million extra cost for the option? Obviously the numbers are different, but see the oft-quoted 'Fight Club' line about product recalls for similar logic.

Southwest didn't buy the option either until after the Lion Air crash:

In manuals that Boeing gave to Southwest Airlines — the biggest operator of both the MAX and 737s in general — the warning light was depicted as a standard feature just as it is on older 737s, according to Southwest spokeswoman Brandy King.

After the Lion Air crash, Ms King said, Boeing notified Southwest that it had discovered the lights did not work without the optional angle-of-attack indicators, so Southwest began adding the optional feature too.

(quotes from https://www.abc.net.au/news/2019-05-06/boeing-knew-737-probl... )

Who ships safety-critical features as an optional extra?

There's levels to safety-criticality so these aren't binary decisions. Often they are based on severity/probability/detectability measures.

For example, airbags on your car may be mandatory but automatic emergency braking not so much.

I agree, but was responding to a post that was heavily criticising Lion Air for not paying for the option and implicitly laying the blame for the crash at their door. Either the option was safety critical, in which case Boeing had no business making it optional, or it wasn’t, in which case blaming Lion Air is unreasonable.

Given that thanks to Boeing, the Lion Air crew didn’t even know the MCAS subsystem existed, and that MCAS would continue to rely on the single faulty sensor however many extra redundancies were installed, the decision not to buy seems kind of irrelevant.

> Either the option was safety critical, in which case Boeing had no business making it optional

This is the part I was trying to clarify. I don't know Boeing's internal processes, but there certainly could be options that are safety-critical but still optional because they are ranked low enough on the severity/probability/detectability scale.

I agree that it's wrong to blame Lion Air as the consumer. The fault seems to be in Boeing misapplying the risk categorization.

You appear to assume that the 20% of aircraft with the optional feature are in the US. That's possible, but... Citation needed.

It's not a random assumption. I read in some recent article that the airlines in the US have the feature--sorry I don't remember which, but you can look it up as easily as I.

How can I look that up?

You're the one who made the factual statement. I'm not the one that's obliged to back that up.

Wikipedia lists the customers, and the number of aircraft delivered to US airlines is a good match for the 20% of aircraft that are said to have that feature. However, that assumption implies that 0% of leasing companies ordered it, that 0% of Canadian customers did, that the American airlines bought all their aircraft and didn't lease any, etc. Might all be true but you can see why I asked for a citation.


Not less safe. Perhaps not as safe at others. Such as of those where the front doesn't fell off.

Well, it's not built of cardboard and rubber...

Does everyone know this delicious skit? "Clarke and Dawe - The Front Fell Off"


I mean less safe relative to shipping the same hardware with the warning feature enabled, which they could have done at no cost except the loss of loss of price discrimination (which I'm sure was substantial, but probably not enough that they don't now regret it)

Since the 737 MAX also has military customers, it may be more likely that Boeing's design of the modular components was compromised by the need to accommodate military specs, which often leave off 'unnecessary consumer safety options' ..

It has to do with the classification during the system safety analysis (SSA). IIRC, the MCAS was classified as 'hazardous' if failed but not 'catastrophic'. If it were classified as a higher hazard level, it presumably would levy additional requirements. It's a risk-based approach, which sounds like they failed to appropriately classify the risk.


Did you buy all the optional safety features of your car model ?

Last time I bought a new car, this was actually a pretty important factor for me -- one car was nominally cheaper, but had a number of optional safety features (like rear passenger pillar airbags, IIRC), and they brought the price to parity with the other car I was considering. I figured I'd hate myself if I declined to buy some optional safety features and ended up with an injured passenger as a result, so ended up including all of them in the purchase price, and eventually went with the higher-base-price car.

The really sociopathic part is the upselling of it as a separate feature when the actual marginal costs are nill. It is one thing if they have to retrofit it but the 737 is newer - just doing it the most expensive option seems like it should give the most logistical savings and be the best for long term profit. I mean people look at the safety record and a crash makes them look bad even if it was pilot error.

Yeah, that's jarring - it's not even that Boeing has to put in extra parts or even extra software. It's just flipping a switch in the software.

It's a tool. Without the warning you can add up the approximate weight of passengers and luggage, and check it's below the maximum. I'd like to think this is done as a matter of course.

I'm not sure what you're responding to, but AoA indicators have very little to do with weight and balance.

Fuel loads, critical airspeeds, and maneuvering characteristics are indeed governed by weight and balance. That has nothing to do with AoA reading mismatch indicators.

Was there a separate question about automatic weight and balance measurement as a feature?

> Surely the plane either needs a safety feature or not, and it shouldn't be down to someone at an airline to make a decision on?

To you own the safest car that is available?

Actually, when I bought my last car, I did get the trim level that had all the extra safety features.

However, I'm not transporting passengers for money and accepting liability for passengers' lives; airlines do, so their financial calculation should be very different from Joe Blow private car buyer.

Finally, if I understand correctly, these extra safety features were software only, meaning they don't cost the manufacturer anything to add on, so there's no excuse to not make them standard. On cars, the extra safety features require expensive sensors, so they really do add to the BOM cost of the car, and putting them on an economy car really could push the price too high for buyers to afford, so carmakers put them in higher-cost models to help make them more commonplace and then move them into lower-cost models as the volumes increase and economies of scale make costs lower.

> Doesn't this seem dangerous in-and-of itself?

I would describe it as extreme criminal negligence with wanton disregard for human life.

I'll see that, and raise with "I would describe it as extreme criminal negligence with wanton disregard for human life that has resulted in more than 300 deaths."

My question is if this ON-OFF switch was really a safety feature (and I believe it was and is) then why FAA turned the blind eye during the aircraft certification process and did not ask Boeing to make it Standard on every MAX plane they ship? Shouldn't the blame for those 2 air disasters be equally then shared by FAA along with Boeing because every air regulation agency in the world looks at FAA as a leading org among them all.

Edit (as seeing downvote due to the possible tone of the comment): What I am saying is basically this should have been flagged by the FAA during the certification process. Maybe that was not part of the process but making sure to check which optional features offered are safety related could be absolutely critical for air safety.

Edit 2: I meant ON-OFF Light here

I have no idea what you mean. Which ON-OFF switch? No-one was talking about an ON-OFF switch. There is no optional ON-OFF switch. The article is about an optional warning. Not an optional switch.

I meant the ON-OFF Light -- the AoA disagree light that turns on when both sensors have different readings (during a possible malfunction of the sensor with bad data) and keeps off when both sensors have same reading on the angle of attack.

Oh. Then the answer appears to be that Boeing misled the FAA into thinking that MCAS didn't have the capability to cause a catastrophic failure, by understating the degree of control authority it was capable of, and therefore AoA sensor failure (and the disagree light) wasn't tied to any airworthiness issue.

I don't think a light (rather than annunciator) can be a very serious safety feature in any case, simply because in an emergency pilots aren't going to be looking at random places throughout the cockpit to see which small lights are on. They're going to be looking at the central EICAS display, which is what they're trained to do.

I hadn't realized planes were sold with software-only features that were only turned on if you paid more. Which is what sounds like happened, right?

I guess I shouldn't be surprised, this makes sense as a money-maker, and is how software is (necessarily) sold. When you have to pay extra for certain software features, whether or not they were actually included in the binary you have as a lesser-payer, it could have been, software is intangible and has no extra per-unit cost to include the "mechanism" once developed. This makes sense as a business model for selling software, in that that's how software works.

But yeah, I hadn't realized planes were sold that way. When we're talking about "features" that are pilot dashboard indicators... it really doesn't seem wise. Are there any pilot dashboard indicators that would be sold as add-ons that wouldn't increase safety were they there? I mean, what's a dashboard indicator for if not making it easier for the pilot to fly the plane? If you have to pay extra for some actual physical device, that's one thing, the physical device has to be installed. But when they're just holding back safety features with no extra per-unit cost that could simply be switched on if you paid... does not make me feel great about flying.

The bigger issue is that there should be no price discrimination on safety gear unless it's complex and expensive.

Set the rules that way and you're likely to get a lot of safety gear that meets the "is complex and expensive" test.

This disagree doesn't help anything if you don't know that MCAS exists and is coupled to the sensors. Afaik only Brazil regulators insisted of telling pilots of MCAS existence (though I don't know whether the implications where clear to pilots).

I am surprised these mechanisms come in optional packages. What else do I have to order separately? Wings, engines, ruder?

Is this some kind of airplane DLC and can I get the season pass for less money?

This sounds so much like my enterprise experience, where you can bring up a huge security vulnerability, have everyone senior to you explain to you that it isn’t a problem. Leaving you utterly confused.

Then when someone else finds out a few months later, suddenly alarm bells start ringing, and everyone up to the CTO is called into an emergency meeting because there is a huge security vulnerability.

Yeah, no shit.


Sometimes I think this is a victim of the "invented here syndrome". I don't know why it is, but many, many people seem to have absolutely no respect for their coworkers. If you bring up an issue (of any kind), it's often treated with suspicion before people even understand what it is. But if the issue is generated by "an authority", suddenly it's a massive problem -- again even before people understand what it is.

I've found this also happens for good things. "I've found this amazing thing that will save us millions of $". "It'll never work and as soon as I understand what you are talking about, I'll tell you why".

What's absolutely horrible about this is that I've caught myself having the same attitude! I've been trying to stop myself from doing it, but man is it difficult. The default, "My coworkers are brilliant" point of view is just so contrary to my cultural upbringing ;-)

>The default, "My coworkers are brilliant" point of view is just so contrary to my cultural upbringing ;-)

It's not just cultural upbringing. It's difficult to really realize that the intelligence/competency distribution at work is vastly different than in the rest of your life and all those years before work. Especially if you work for a highly selective employer in a high-skill job market. We learn from experience afterall and encounter a lot of incompetency throughout life. We also tend to weight negate experiences much higher than expected ones.

I still catch myself being happy and surprised, if I encounter someone competent and usually expect incompetency. It's rude and on average probably wrong, but trained from experience.

> It'll never work and as soon as I understand what you are talking about, I'll tell you why

This hits so close to home.

And yes, it requires a lot of mental energy to avoid falling for the same trap.

This right here is why management consultants exist.

The disturbing thing is that this "ordinary corporate crap" level of quality and decision making seems to have crepped into the production processes of a company that making aerospace products - ie planes that fly through the air. 'Cause there's a reason the standards for aerospace are different. Your plane crashing is a different matter than your insurance advertising campaign tanking.

That isn't surprising. A security bug that is known only to the vendor does not cause any damage in reputation or finances. In fact, fixing it costs money for no perceived economic benefit.

I guess the "responsible disclosure" culture is partly to blame here. If you get 90+ days to fix any vulnerability, then software that ships with (internally) known security issues is much more viable.

How did "someone else" report the bug differently from "you"? Or why was the response different at all?

Different organizational standing.

It's all about costs of impact / costs to fix. If you hope for anything to be fixed, better be a catastrophic issue that is trivial to understand and fix.

When other people or audit start noticing too, the potential for fines and PR disaster will go up orders of magnitude. Resolutions that were previously impossible can be evaluated again.

Before somebody says that it is a trivial fix to turn on a light. It is not. That would have to be retrofitted in the few existing planes and future planes, starting by notifying clients and regulators. A long and costly fix, creating a long paper trail of something being wrong.

This is a sad truth about corporate culture, but it's human nature.

Seniority in a position (imo) is the soft skills, and communication chops to either present a better argument, make sure there are written email statements of it not being a problem, or not take no for an answer.

It's easy to throw up your arms and say, well I said it in some meeting.

> After discovering the lapse in 2017, Boeing performed an internal review and determined that the lack of a working warning light “did not adversely impact airplane safety or operation,” it said in its statement.

> As a result, Boeing said it did not inform airlines or the Federal Aviation Administration about the mistake for a year.

There was a warning light installed in the cockpits which was completely non-functional in 80% of the 737 max fleet, and Boeing did not notify the airlines for an entire year? Did they honestly think that "lack of a warning light" is equal to "lack of working warning light"?

The "warning light" is an "AOA DISAGREE" text that appears on the main flight display:


So there was no non-functional physical light on the cockpit.

I think that makes it worse on Boeing's part, that it was a software-only feature that was disabled from those who didn't pay. They didn't even have to "install" any hardware.

It sounds like the software accidentally coupled functionality of this light to another feature. It is in fact a software bug. To fix the bug would have required a software redeploy -- which is definitely non-trivial thing in safety-critical software, and may have required some kind of recertification, or at any rate significant expense.

Instead, they could have done what they in fact _have done_ only _after_ people died:

> In the months after the Lion Air crash, Boeing quietly worked to appease some customers, according to a person briefed on the matter. In several instances, it activated the angle of attack indicator for free, which then turned on the disagree alert.

Well, quietly, for some customers, who know to insist on it and have the purchasing power to get what they insist on.

All Boeing had to do was flip a switch to turn the software feature on, without charging for it.

Ah, I was not aware of that. But surely the pilots were under the impression that such a warning message existed?

So there a weird thing about angle of attack: the autopilot uses it, but civilian pilots don’t use it (they use airspeed and descent angle). It’s used in the military, and airlines who recruit a lot of military pilots tend to buy the angle of attack option to get them comfortable. Ignoring the little MCAS issue, it was a fine reasoning, the autopilot can detect a disagree (I guess they just drop out, but I don’t know about cat 3 landings), ex-military pilots need a disagree because they read it, and other pilots don’t need to know.

I found it amazing that civilian pilots don't use angle of attack so went looking and found this article about the debate on installing angle of attack indicators on commercial aircraft, which cleared up for me why this would be the case. But not in the most reassuring manner.


>"From day one, pilots should be—and in most cases are—taught that stall occurs when the critical angle of attack is exceeded. They are then told to memorize the stall speeds in various configurations. By the time they have become proficient, they relate stall to a certain airspeed, and don’t really relate it to an AOA anymore."

Air France 447 immediately comes to mind, where the crew (or maybe specifically the copilot who kept pitching up for minutes) lost their airspeed indication and thought the stall warning was frivolous despite falling vertically at terminal velocity for minutes.

Critically in that case, the stall warning stopped when the nose was pulled up even further, because the computers rejected the indication as too large to be correct. So every time the nose was lowered, the stall warning activated.

Specifically, the airspeed became too low for the AoA reading to be deemed reliable, silencing the stall warning. Definitely a terrible UX.

Ah, thanks for the clarification. From an engineering perspective that makes total sense - and is disastrous from the pilot's perspective!

Fucking hell. That is a fucker of a dark UX pattern.

AF447 is really a terrible accident, just hard to analyze. It's not even clear would breaking one link in the chain save the plane, since so many thing happened.

True, UX was inadequate. But there was one more detail, the indications were incorrect and mismatched for a short period, while the pilot inputs were continual. It's not hard to interpret that the pilot actually knows something you don't (speaking from the computer's POV) and disengage ("Wait, we're 40 degrees pitch-up, and the man keeps pulling up, alternate flight law engaged, sensors might be incorrect, listen to the man").

Indeed. And it goes to show that rejecting “impossible” values is a dangerous strategy.

I think you can reject impossible derivatives, though only if being very fucking careful.

I seem to remember reading that the ability to cause a power-on stall is both part of standard pilot training and a surprise for most pilots when demonstrated.

Military or not you should know about the AoA disagree and the presence of MCAS as the combination can be quite deadly. Knowing the actual angle of attack could be useful, but knowing what sort of cascading failures to expect is critically important.

I am trying to explain Boeing’s point of view before the accident, you’re busting an open door.

But how can you even diagnose a failing AoA sensor, if there is no warning message but you expect it to be indicated by one? Is there any testing done on ground by the engineers after each flight?

That's the rub.

Let's plan a theoretical test case. At a minimum, you'd need two maintenance guys, maybe one and a pair of tools. One guy to tweak the sensor outside (as several hundred mile per hour airstreams aren't exactly in plentiful supply, or easy and trivial to reconfigure in a hangar near you), and a guy in the cockpit, reading off the measurement on the computer against some sort of calibration chart.

The "tweaking" part would probably be done with some sort of precision test mount (that I doubt they even build into the plane.)

Also, odds are, you'd only detect very specific types of disagrees on the ground. Bad springs should be trivially deducible from the forces required to create measurable deflections.

Bad potentiometer connections should be deduceable from checking output voltage against the component spec sheet while deflecting.

Unfortunately, this dive into Electrical Engineering won't stop until you've taken into account every length of wire and eventually, every circuit board and line of code between that sensor and the avionics suite. If some FOD for instance, was slowly wearing through the wiring and insulation leading from the sensor to the computer, or an inductive cross-talk event occurred, or something shorted. Etc, etc.

And all of this work and diagnostics would need to be done by someone whose documentation told them the AoA sensor was a non-critical component so the manufacturer could cut corners.

Oh yes. <double facepalm> Been here before. Way. Too. Many. Times.

Not in safety critical systems , mind, but oh the web's I've seen woven.

The more I learn about the story and the more I realize the bureaucracy that is airplane flying. The boeing line is that they had to use the runaway trim checklist and that’s it. They missed the auto throttle item on the list so it’s their fault.

I feel like it’s all about reading documents and never trying to use their brain.

A lot of my bubble about diagnosing things and connecting information together has been bursted.

In “the complete private pilot” they definitely talk about AoA in the context of stalls. I’m pretty surprised to hear that civilian pilots don’t use it.

it's a very important concept, but it's a far cry from having a (redundant) gage and introducing it into the scanning pattern. Air speed is generally a very good indicator of how far you're from stall, unless you're in the army doing crazy attitudes very close to the ground.

I'm pretty sure that small aerobatic planes don't have an angle of attack indicator, while the pilot would know everything there is to know about attitude control, having a number is not useful.

I would add an analogy: wing loading is an extremely important concept, and there is absolutely zero plane with a direct force reading on the wing spars.

Civilian planes (Cessnas etc) don't even have AoA indicators, so civilian pilots couldn't use them even if they wanted to. They have started to add them just in the last few years. But of course that means they're still not in the planes most private pilots are flying.


An in the aftermath of the Lion Air crash, Boeing were telling pilots that this warning message would alert them to that kind of problem before takeoff.

Boeing were telling pilots that this warning message would alert them to that kind of problem before takeoff.

I doubt that, the alpha vanes aren't particularly useful until the plane is actually moving at a fairly decent clip.

Yes, it does not work. According to the OP article, Boeing said to American Airlines pilots in November that the warning would work on the ground, but later said it in fact does not.

By before takeoff, I assume that means "during the takeoff roll"

And I hope that also means "before V1".

I know the DC 8 couldn't do it's elevator check until it was moving quite fast, as there was no direct connection between the pilots controls and the elevator and it couldn't respond without airflow.

What speed is that? I don’t know about 737s in particular but some AoA sensors work at speeds as low as 20 knots—far less than V1 on any runway a 73 is going to use.

So if you were to get a value that doesn't make sense on the ground when you aren't moving, that would be a assuring sign that you have correctly detected a fault.

The angle of attack of a wing that has zero airspeed is undefined. You could take two traditional weathervanes and put them inside your house and they wouldn't tell you the direction of the wind outside.

No, you need airflow to measure its angle.

If that annunciator isn’t in the checklist, you wouldn’t think about it being there or not. It wouldn’t be part of your scan.

So is that text warning always available and just software enabled if the packaged is purchased, or is there extra hardware involved?

100% software

if (crashing && paid_enough) show_warning();

That's kind of a distinction without a difference. A standard function was non-functional.

Time to call their BS and fine heftily to avoid such behavior in future. Imagine Boeing responsible for manned space flights.

Boeing is in fact in the certification pipeline for crewed spaceflight at NASA, and has received more NASA funding for this than every other applicant together, including SpaceX.


> In the months after the Lion Air crash, Boeing quietly worked to appease some customers, according to a person briefed on the matter. In several instances, it activated the angle of attack indicator for free, which then turned on the disagree alert.

Instead of activating the indicator for all customers, Boeing only activated it for customers that explicitly asked. Greedy until the end.


Boeing believed they shipped it as a standard feature, and included it in the plane's the manual. They found out in 2017 that it wasn't working. After their shitty planes start falling out of the sky they turn it on for the loudest customers, keeping what was intended to be standard now a premium.

Which part of this shitty behavior exactly are you trying to defend?

At least Southwest thought that they had the AoA disagree light enabled, and they released a shockingly brutal press release calling Boeing liars over the issue. [0]

Beyond that, critical safety features are not supposed to be charged add ons. If 10,000 extra people died because they didn’t pay for Ford’s airbags, who would be at fault? Now imagine that all of those people thought they had airbags because the steering wheel said “airbag” and the brochure talked about airbags, even though they were turned off by software. Who would you blame?

0: https://www.cnbc.com/2019/04/28/boeing-didnt-tell-southwest-...

If Boeing had told the airlines what the consequences of MCAS and not having the indicator could be they probably wouldn't have bought the plane. This is a clear case of shipping a different product than what was sold.

Their scheme to generate some extra revenue resulted in hundreds of deaths.

Why was a safety feature optional to begin with?

Had boeing mentioned that not having that feature frequently causes planes to crash and kill everyone onboard, I'm sure they would've bought it.

Boeing statement: https://boeing.mediaroom.com/news-releases-statements?item=1...

This PR is helpful in that it reminds me that there are actually people paid to take a simple set of information and mix in with a ton of PR bullshit so that you have to read it 4 times before you actually understand that yes, they screw up.

Here is the ass saving statement from the press release: "Senior company leadership was not involved in the review and first became aware of this issue in the aftermath of the Lion Air accident."

Courious they try the same strategy as VW. We all know how that is most likely to end...

With increased sales and better revenue?


And got a the former CEO and other exects indicted in two countries and cost the conptany billions. The story is still ongoing. And might have contributed to the issue with the new Golf 8. Did I mention that some former managers are in jail already?

VW is a conglomerate comprising tens of manufacturers from Audi to Porsche. The perception of VW as a brand is of minor importance when it comes to global sales and revenues.

It is unlike Boeing, that only sells planes under the Boeing name. When airlines buy an Airbus instead, the money ain't going to Boeing.

What worries me even more is the monkeypatching they are now trying to do in order to get the permission to fly the plane again.

In the original design they had designed the MCAS to be able to make drastic changes which then caused the crash.

Now they are trying to fix that by making the system less intrusive, but didn't the engineers design the system to be so strong for a reason (because of the changed position of the engines)?

This seems to me like going through a codebase from someone else because there were errors and just deleting stuff that made problems. Even if the initial version had problems, it was designed that way for a reason?

> In several instances, [Boeing] activated the angle of attack indicator for free, which then turned on the disagree alert.

This sounds like the "indicator" and "warning light" are not physical devices that must be installed at extra cost, but rather are software features that can be "activated" by changing some configuration file.

If so, then Boeing has blood on their hands. The active control system is already a cost-cutting hack. Charging extra for zero-cost safety features related to the new, unproven control system is an act of appalling greed.

So, the AoA indication is a feature that can be activated for a fee (which most likely exists to cover costs of modifying the configuration from the base configuration, so imagine as an analogy a $2 fee for some really minor change to a car you ordered). The reason this is an option is that it requires additional training for the pilots, some airlines already have this training (American Airlines is one example) and theres no enabling it and then not training the pilots on it.

As many have noted, AoA is not commonly used in non-military piloting in the United States, so its no surprise its not commonly ordered.

I am also under the impression that the AoA Disagree indication was new on the 737 MAX, I haven't checked the 737 NG manuals for it.

Of course they're just software. "On-disc DLC" is how the world works now. Pay for the efficiency package and the computer starts mixing fuel/air better. Pay for the "enhanced maintenance" package and two dozen extra sensors turn on.

This story is really starting to annoy me. Now there's a new person named "Boeing" who somehow lacks self awareness to know what they're doing or selling even though this Boeing character designed, built, and sold two variations on their product. I feel like Boeing will soon become the main character, and try to plead insanity.

Corporations are only people when it comes to benefits, not responsibility. This may sound facetious, but please prove me wrong.

Hey now, if we started holding individuals accountable for the criminal negligence of the corporation they run, surely bedlam would errupt.

It's Limited Liability in it's purest, most distilled form.

Interesting linguistic phenomenon: Commonwealth grammar uses plural verbs for collective nouns -- i.e. English newspaper articles will say "Boeing have decided.." instead of "Boeing has decided..".

Strikes me as subversive in this current environment: language reminding you that a corporation's decisions are made by actual people.

As an Indian, I was surprised at "Collective are..." and believe it arose after the colonial days.

That's interesting! I wonder when this strong preference for plural verbs appeared in British English.

>> It's Limited Liability in it's purest, most distilled form.

I didn't say what should happen. The company is just publicly waffling about it's own behavior. I find that annoying is all even if it's kind of expected.

We had the same conversation about VW last a couple years back. Who knew? When? Can they blame a "rogue" engineer or was it the C level? Were shareholders and customers, both deceived, entitled to something? At least nobody got hurt in the VW thing.

People absolutely got hurt in the VW thing, and will continue to be harmed.

One study claims "1200 premature deaths" in Europe alone: http://news.mit.edu/2017/volkswagen-emissions-premature-deat...

Actually, the extra pollution is estimated to cause 60 excess deaths in the US.

Boeing had a history of being CEOd by people who started as engineers and worked many years in the industry, then for ten years until 2015, it had a Business major & MBA in charge. The 737 MAX development was mostly under the MBA CEO James McNerney who started a Proctor and Gamble brand manager then as a McKinsey management consultant.

The current CEO is an engineer did come up from inside the company, and was in charge as CEO from June 2015. The first commercial flight according to wikipedia of the MAX 8 was in May 2017.

It makes one wonder.

This is a phenomenal failure of management. The failure of the team is always the fault of the leader.

I've seen this so many times in industry. Some decision on high comes through that costs need to be cut, which means cutting corners. It always means cutting corners, because the only way they have to measure costs is Salary × Butts-In-Seats. So they hire lower cost people (i.e. less experienced, often off-shore where there is no oversight on process) and cut the number of people (often several key roles that would put the brakes on bad plans).

They make a short-staffed team of people who don't know any better, then look around confused when things go wrong. "But the PowerPoints said we'd be successful. The management consultants said everything was on track".

When it's a website for a government bureau, people just throw up their hands and act like this is normal. The customer eats the cost of the useless product and they start over, usually with the same contractor on an extension, often not learning any lessons in the process (how could they? They can't imagine a world in which any of this is their fault).

It should never happen for any project. It should be a damn crime, upwards of fraud, with massive fines. But stuff that can kill people, this should be sending people to jail.

I can hear the meetings in my head, I've been through them so many times. "Who knew this was a problem?" "Uhhh, I brought it up at the outset, you said it wasn't my place to bring it up." "You're on report for your negative attitude."

> Boeing performed an internal review and determined that the lack of a working warning light “did not adversely impact airplane safety or operation,”

Why is this review not done by a third party auditor? The cynical view is that corporations have a conflict of interest. They are only incentivized to act ethically up to the point that the cost of ethical behavior exceeds the damage done by unethical behavior... These costs do include future fines/penalties, but these are often woefully disproportionate to the damage done as evidenced by the fraudulent NASA metal supplier incident.

Because the third party auditor (the FAA) has had it's operating budget slashed so viciously, it can't compete in attracting the talent required for independent certification.

Remember, to regulate something, you need to be 10% smarter than what you are regulating. Which means you have to be able to make sure you can attract the best talent possible.

As a regulator, you have to be on your game all the time. The regulated just need to get lucky once to get away with what they shouldn't.

“Boeing Believed a 737 Max Warning Light Was Standard. It Wasn’t.”

This is retrospective ass-covering, to distract us from the fact that the the pilots were never informed as to the existence of MCAS (Maneuvering Characteristics Augmentation System) and therefore could not even know how to disable it. The presence of or lack thereof of a warning light being rendered specious. And MCAS read only the one AoA sensor, from the captains side.

So to summarize, the pilots don't know about MCAS, MCAS is still engaged even when the auto-pilot is disengaged. A single AoA sensor failed, triggering the MCAS into a nose-down mode, the pilots begin ignorant of this, repeatedly tried to regain level flight using the trim wheel. This they failed at.

“The warning light notifies pilots of a disagreement in the sensors that measure which direction the plane is pointed”

This wouldn't even work in the way described, in a two sensor system, the computer can't tell which sensor is giving false readings. That's why they hooked up MCAS to only the one sensor. And Boeing didn't tell the FFC or the pilots as they would of had to get the plane re-certified and the pilots retrained.

It's (almost?) wilful. If ever there was a suitable time for using a military acronym such as FUBAR, this is it.

In my own view, leadership should be held criminally responsible for the loss of lives. Of course it will never happen, but that's what ought to happen.

Hopefully they'll at least be responsible for liability in a lawsuit by the victims' families. They deserve being taken to the cleaners, well and proper.

At this point I feel like it's fair to call Boeing's actions malicious. You charge for safety features that cost you nothing to deploy? Yeah, that's malicious. They knew people could die, but they wanted more money, so they let people die. Now, the company needs to die. Buyers of aircrafts would be clever not to trust Boeing ever again, and buyers of tickets would be clever to avoid flights that use Boeing planes. If everybody acted cleverly, Boeing would die and other aircraft manufacturers would take note. I doubt everyone will act cleverly, but repeatedly informing people of their malicious actions could help facilitate it. Not flying on Boeing planes would help too, but unlike a traditional boycott you have an actual selfish incentive to do so (not dying). Just like a traditional boycott, your individual participation is statistically meaningless unless you're actually in a position to buy a large number of aircrafts. I honestly don't think I'll avoid flying on Boeing planes in the future, but I'm also probably more willing to gamble with my life than most people are; I never said I was clever.

Agree on malicious, especially given that the Max change was a big hack of an existing airframe. It needed all the extra safety features it could get; to charge through the nose for them was shocking and callous.

However, given the amount of Boeing planes out there, and the number of employees, the company dying would be harsh. What needs to happen is a fundamental shift of culture; this is why I think that leadership should be held responsible for what's happened, and that this (with leaders in jail, new ones would have to take their place) would help force the changes that need to happen for safety and engineering quality again to be the number one drivers - not just profits and bean counting.

It's an opportunity for them to fully reset the company. Fingers crossed, it will happen.

>However, given the amount of Boeing planes out there, and the number of employees, the company dying would be harsh.

346 dead from wilful incompetence, followed by utter bullshittery, half-truths, outright lies and finally, when it has to admit what it has been doing, a desperate insistance that executives could not possibly have known.

And all the while fighting against them being grounded.

Break the damn thing up already.

> Break the damn thing up already.

I dont think any US politician has the guts to take down Boeing, if it goes down there wont be any major US aircraft manufacturer.

That's another reason, given the evident issues with the total fuckwits currently in management, it should be broken up. If it isn't, then the US may end up without a major US aircraft manufacturer.

So far, Boeing stocks went up since the time before the first crash...

>However, given the amount of Boeing planes out there, and the number of employees, the company dying would be harsh.

It would be, but it needs to happen. And it wouldn't be as bad as people think.

The same thing happened when GM was on the verge of going under 10+ years ago. Why does everyone somehow think a giant manufacturer going out of business somehow means that thousands of employees will suddenly be destitute and unable to find another job? Do people really think that enormous manufacturing facilities are somehow going to go unused forevermore?

No, what happens is the company's competitors swoop and and buy up all the assets for cheap, and put them to work, because now there's suddenly not enough manufacturing capacity to meet global demand, but there's all these assets and a highly-skilled workforce suddenly available to be put to use.

For the same reason bailing out Chrysler and GM was wrong, Boeing should not be allowed to survive. The company can be broken up and all its assets sold to (hopefully foreign) competitors, who can then build better planes with them.

> You charge for safety features that cost you nothing to deploy?

If you read the article, that's not what happened.

The warning message was meant to be supplied by default. Boeing though it was supplied by default. The airlines thought they were receiving it.

But they later discovered they had a bug which disabled the warning message unless another feature (an optional luxury feature) was enabled.

This situation is still bad. Boeing should have done something back when they discovered this bug: released a software update, given everyone the optional feature for free, or at the very least notified the pilots.

But it's not "charging for safety features" bad.

I get the distinction you're making, but I still feel like the actions taken by Boeing resulted in charging for a safety feature. I also never bought into Hanlon's razor; I don't accept anything Boeing claims as being true just because they claim it. I don't accept that upper management didn't know, I don't accept that they intended to add this feature in the next software update before they were forced to talk to the FAA about the first crash, and I don't accept that they didn't know about this "bug" until after the plane shipped. Like, really, they didn't test all of the indicators on an airplane before shipping it? Really? If that isn't malice, it's an unbelievable amount of incompetence. I find it more likely that the airlines conspired with Boeing to act like this was a bug rather than admit that the didn't pay for a safety feature, so Boeing gets to act like they intended to ship the software-only safety feature all along. Maybe I'm crazy, the crazy rarely know it.

> This situation is still bad. Boeing should have done something back when they discovered this bug: released a software update, given everyone the optional feature for free, or at the very least notified the pilots.

Do you feel like the failure to do this qualifies as malicious behavior?

Edit: tense.

> I still feel like the actions taken by Boeing resulted in charging for a safety feature

I still feel like a distinction needs to be made. According to this article, at no point before the grounding were airlines aware that they needed to pay extra to get this safety feature. Infact, they thought they already had it.

Boeing got no benefit, it was clearly a mistake.

Maybe you could argue the coverup was malicious behaviour, but such coverups can also result from incompetence.

The reason I think a distinction is necessary, is I think incompetence is much more dangerous than maliciousness.

A Malicious company evaluated each corner it cuts. It decides "Can I get away with this particular thing" and it will only cut the corners it thinks it can gets away with, leaving the more dangerous corners un-cut. A Malicious company will have notes on each corner they cut, or at least people who remember, so we can go in later and re-evaluate everything.

On the other-hand, an Incompetent company has no idea where it went wrong. It made random mistakes without realising. The mistakes could be anywhere along the risk-spectrum. They aren't documented. Nobody knows where they are. The only way to be sure you caught all these mistakes it to completely re-examine the whole design.

In the case of disagreement the only sensible option is to disable the MCAS and alert the pilots. Simply ignoring the secondary sensor is not a valid solution.

As I understand it, they wouldn't be trained for a 737 Max without MCAS. Without MCAS it's a different plane.

This is true. Which means per FAA certification standards MCAS should have been a "hazardous failure" system. Which means it should have had at least two levels of redundancy. Which means it should have never been certified with no redundancy at all for a failed sensor.

The handling characteristics change, but from what I understand it just tends to pull up. Any pilot should be able to deal with that as long as the artificial horizon is working and they are paying attention.

Plus, even if their attention does wander there are other safety systems (like stall alerts) that will kick in to help them.

It's not like the MCAS disengagement makes the plane uncontrollable.

What's worse: Last I read, they are planning a software patch that makes the system read from both systems. If the sensors disagree, MCAS is disabled.

Hopefully it doesn't disable itself when it's needed.

I don't think the quoted text refers to the computer being able to resolve conflicts in the sensor readings. The computer cant resolve conflicting signals from two sensors, but can certainly detect if the signals are conflicting, and if there was an indicator to warn about such cases, can turn it on giving pilots indication of such conflict.

* Manslauthering Coordinated Augmentation System

One question to ponder: Would the two crashes still have happened without this particular mistake, that is, if the planes had had a working AoA disagree indicator?

It depends, I think:

1. Did the AoA sensor malfunction occur before the takeoff roll?

2. Would the AoA disagree indicator come on early enough to abandon takeoff?

3. Would the crew reject takeoff once the AoA disagree indication comes on?

If it's a "yes" on all three questions, then presumably the accidents would not have happened.

Once you're in the air, and all hell breaks loose (stick shaker etc.) while MCAS tries to dive you in the ground, I doubt that the indication would have made any difference: before the Lion Air crash, crews didn't even know about MCAS. After, they did know, and the Ethiopian Airlines crew, from what I gather, diagnosed the problem, though maybe too late - but again, I doubt whether the AoA disagree indication would have helped.

That leaves above questions.

1. For the Lion Air crash, the AoA malfunctioning was there before takeoff. For Ethiopian, I'm not sure.

2. I'd think yes.

3. Don't know. Anyone an idea? (Today, presumably you'd abort with a MAX and AoA issues. Back then?)

I think you're too quick to dismiss the importance of the disagree indicator, and doing so on a shaky basis. A significant amount of time was spent trying to figure out what the hell was happening. I think it's safe to assume that if the pilots saw an error warning pop up warning that there was a disagreement between two sensors, that would focus their attention on that system—which quite conceivably could have caused them to shut it off manually sooner.

> that would focus their attention on that system—which quite conceivably could have caused them to shut it off manually sooner.

But you can't shut off the AoA measurement "system", AFAIK. And the system causing the trouble (under wrong AoA information), the MCAS, can be shut off by setting the Stab Trim switch to Cut Out, but how should the crew know the connection between AoA and MCAS when they weren't even briefed about MCAS in the first place?

iirc the Malaysian flight had in the cockpit a pilot that was hitching a ride, who figured out what was going on but too late. He knew about the MCAS. Had they vocally stated something about the AoA sensor early on, or had he caught a glimpse of the message, perhaps he could have intervened sooner.

For the Ethiopian flight, you're right. They likely wouldn't have known what to do with that information. But it might have prompted them to get on the radio and ask someone, or consult a certain part of the manual (assuming there was any relevant information there.. I don't know).

It certainly seems like crucial information. And even in the small chance it could have made a difference, the absence of it is critical

For Lion Air, he remedied the symptom; he realized there was some sort of periodic trim runaway, and guided them to remedy it with Stab Trim cutout without having any intrinsic understanding of what the cause or underlying principle of the runaway was.

Big difference. Still didn't do anything for the fellows on the next flight.

Re:2. The article says the alert will only come once the aircraft is above 400ft. Originally Boeing told pilots it would activate on the ground.

True, it does say that... Why would that be the case, though?

At any rate, having had that AoA disagree warning (limited to above 400 ft) would not have saved these planes, it seems to me. Having had an AoA disagree warning on the ground might have.

It's because you have to have moving air for an AoA sensor to work. Unmatched sensors when there isn't enough air movement for proper operation doesn't mean anything.

But the air certainly moves fast enough at the moment of taking off, and if the sensors are sensibly sensitive, they should register AoA well before V_1.

EDIT to add:

Hm, Boeing states:

"During rotation, pitch angle is the critical parameter that ensures tail clearance. Once the airplane is airborne and at a sufficient altitude where ground effect and crosswinds do not affect the sensor reading, AOA will provide valid information."


Looking at the FDR for Ethiopia [1], it looks like the AOA sensors disagreed 2 seconds before they left the runway. The sensor disagreement happens after 05:38.43, they started climbing at 05:38:45 and were at 1000ft at 05:39:06 ish if I am reading the PDF image correctly.


The warning light might matter for squawk reporting (pilot to maintenance bug report, if you will).

But even if the pilot notices the warning, if they don't know about MCAS, then they don't know about "MCAS upset" which can result when MCAS acts on bogus AoA information. And this particular variety of "MCAS upset" quickly induces a mistrim, and high airspeed. So I'm unconvinced the indicator is a "safety" feature in-flight that will result in any different behavior on the part of pilots in an "MCAS upset" situation.

It is a mistake to think that trim is only about attitude (nose up or down relative to the horizon), it also implies an airspeed. Nose down trim effectively translates into a higher airspeed, all other things being equal. If you reduce power, the plane will nose down (this is a function of positive static stability, a FAR 25 certification requirement for transport category airplanes), and the plane will stabilize to maintain the airspeed implied by that trim. Add power and it will climb, to maintain the airspeed implied by the trim.

The central problem is still that the pilots weren't trained about MCAS in either it's intended working condition, or "MCAS upset", or how the airplane behaves with autotrim disabled and therefore without the benefit of MCAS. The pilot not being aware of these three of these behaviors is just flat out wrong. It's bad enough they weren't also required to demonstrate competency (knowledge and practical test) for all three.

So I draw two conclusions from this clusterF of astronomical proportions:

1. The A320neo really caught Boeing off guard. The 737 MAX was a panicked reaction to something that they thought would threaten their business to a significant degree. The A320neo and A320 have a common type rating too. Googling seems to suggest the differences are fairly minor however. I guess Airbus benefits in the A320 being a newer airframe than the 737 (which is ~50 years old now)?

2. Somewhere along the line this is a management failure of gigantic proportions. That management would risk Boeing's reputation on something that, in hindsight, is so incredibly foolish is mind-blowing. And the buck stops with the CEO. If this ends with anything less than the CEO deciding to spend more time with his family it's a joke. And whoever replaces him should clean house with anyone remotely responsible for this.

This isn't unprecedented either. Through this fiasco I came to learn about the 1990s 737 rudder issue [1] and it's scary how similar the reaction is. "The planes are fine". "There is no issue". You know, until there is.

I expect the only thing stopping a lot of airlines from cancelling their 737 MAX orders and buying A320neos instead is at this point they'd probably have to get in the back of the line and wait years. They really have no choice but to stick with the 737 MAX now.

Actually I guess airlines like Southwest (who are 100% 737 IIRC) are kind of stuck anyway. It's the likes of American Airlines who has such a large and diverse fleet that really bet on the wrong horse.

I really hope this ends in a court case or even better, a criminal case. I really want to know who signed off on this plan and at what point warnings were given and ignored.

[1] https://en.wikipedia.org/wiki/Boeing_737_rudder_issues

> The A320neo and A320 have a common type rating too.

> Googling seems to suggest the differences are fairly minor however.

> I guess Airbus benefits in the A320 being a newer airframe than the 737 (which is ~50 years old now)?

The 737 is uniquely low to the ground among modern aircraft. (This was an advantage at airports without jetways high off the ground a long time ago.) Which means for every other model there’s plenty of space for new larger style engines under almost all of the existing planes. This makes airbus’s neo programs comparatively simple and reasonable for keeping the same type rating. The 737 doesn’t have the space, so they moved the engine location and came up with the MCAS as a workaround. It was clever, but unfortunately, too clever.

The truth of the matter is that the 737classic update from 1984 was already too low on the ground for the CFM56 engines to fit. So they had to reshape the nacelles into that odd shark mouth looking oval shape unique to 737s for that little extra bit of ground clearance. The problem isn’t new, was well known and yet greed made them update that same old airframe for 35 more years until it all came crashing down (literally, sadly)

Greed? The 737 is an excellent airframe. Suggesting that they kept the old airframe around because of greed is just silly.

Question: At what point does maintaining the 737 type rating become too much "technical debt"? Why not create a new (modern) type that fills the same commercial requirements as the 737?

I wish the FAA would grow some balls and come right out and declare the old 737 type rating as deprecated. No new designs (including the 737 MAX) are allowed to fall under it. The type is many decades outdated and increasingly poor decisions are being made to conform to it.

I'm wondering how other certification agencies will react.

Specifically after more and more of the incestuous relationship between the FAA and Boeing comes to light.

Including claims that experienced engineers, which were in charge for certain certifications were demoted or fired if they were too critical or outright refused to certify certain sub systems as was reported in the Seattle Times[1]

It seems that since 2004 engineers in charge of certification were stopped to communicate directly with techies at the FAA, but reported to their managers, which in turn communicated with the FAA.

He who pays the piper calls the tune.

[1] https://www.seattletimes.com/business/boeing-aerospace/engin...

Call your members of Congress and demand this.

Boeing was considering that. But were caught completely off-guard by the announcement of Airbus' 320NEO.

American Airlines (a life long Boeing customer) placing a massive order of 320NEO put Boeing into full throttle panic mode and triggered the unfortunate decision to apply yet a number of hacks to an airframe, which was not suitable for the purpose.

It seems we unfortunately found the answer to that question empirically.

> Question: At what point does maintaining the 737 type rating become too much "technical debt"?

I was about to snark, "Sometime prior to people dying as a result of the debt."

But then I considered that (potentially) the $X dollars saved by allowing the technical debt to live on, could probably be invested in some other way that saves far more lives. I.e., obesity reduction, smoking cessation programs, etc. So perhaps the issue isn't that simple.

> I really hope this ends in a court case or even better, a criminal case. I really want to know who signed off on this plan...

There are multiple lawsuits against Boeing going already, but don’t forget the FAA signed off on the plan, and some time ago formally adopted a policy of trust whatever the airlines say without checking carefully. Why even have a federal regulatory agency for air safety if they’re not going to independently scrutinize what the airlines do?

No 737s crashed in the US, so why would the FAA be on the hook for anything? Shouldn’t the civil aviation authorities in Ethiopia and Indonesia be the ones “in trouble?”

The FAA is supposed to catch problems before the crashes, not after them. It's pure luck that a 737 Max didn't crash in the US, that doesn't in any way absolve the FAA of it's charter and responsibility to check Boeing's activity and it's failure to assess the risk of the 737 Max. Our FAA did put US citizens at risk when it approved the 737 Max, so the FAA is on the hook absolutely.

Your point about Ethiopia's and Indonesia's agencies (and airlines) is reasonable, they ought to do their own due diligence on the aircraft that they use & allow on their turf. Smaller countries and agencies with fewer resources than us have tended to trust our FAA and the aircraft manufacturers to assess these risks. In that sense, the FAA's failure to regulate the Max and it's policy of outsourcing regulations to the manufacturers themselves is (rightly so) truly embarrassing for the FAA, is resulting in loss of trust of the US FAA globally, and maybe on the bright side is going to cause all other countries to start doing more of their own due diligence.

Free Trade is the answer. Most national civil aviation organisations defer to the aviation authority of the country where the manufacturer is based. So FAA is trusted/expected to maintain the standards for Boeing.

The last thing Boeing or Airbus wants is to have to prove their aircraft for every country in the world.

> It's the likes of American Airlines who has such a large and diverse fleet that really bet on the wrong horse.

American Airlines was going to place a major order with Airbus for the A320neo, Boeing promised AA that they will deliver a new aircraft quickly which will be better than A320neo if they cancel the A320neo order.


If the airlines switch from 737 to A320neo then all the 737 pilots would have to be trained on the new plane which would cost the airlines big $$$

Southwest has ~10k pilots. At $20k type rating per pilot, that’s $200mil. An A320neo is $110mil. So, a big cost but not the biggest.

There are on-going costs of maintaining each type rating, so it isn't a one time cost. Also, you'd want most or perhaps all the pilots maintaining both type ratings, so they can still be matched with any 737.

Apparently, pilots can only be rated for one type at a time.

So during the cross-over period, you will need two completely independent fleets of pilots, one to operate the old 737s and one to operate the new A320-NEOs.

Each fleet needs to have different spare pilots to cover absences or overtime emergencies. And the pilots who currently are training for the new type rating can't be in either fleet.

The airline will have to temporally increase the number of pilots until the whole switch-over finishes.

> Apparently, pilots can only be rated for one type at a time.

You can have endorsements for multiple ratings but you will need to renew each independently, which costs. It would cost twice as much if you want to maintain two type ratings.

Interesting, if this is the case it would also have meant needing a second set of pilots for the MAX if it received a different rating.

It gets more complex.

Airbus have designed their A320, A330/A340/A350 planes with almost identical cockpits and handling characteristics to allow cross-crew qualification.

A pilot with an A320 type rating can take a 7 day course to become qualified on the A330 (instead of the 40 day course normally required). And they can maintain the type rating on both types simultaneously.

Since the A330 shares a type rating with the A340 and A350, this allows a pilot to fly quite a few airbus planes.

Given the Max is so close to older 737s, I see no reason why Boeing and the FAA couldn't come to a similar arrangement, were pilots can be cross-trained with a 1 week course and maintain simultaneous type ratings on both the Max and Older 737s.

Boeing does have a few common type ratings: The 757 and 767 share one; The 777 and 787 share another. But as far as I'm aware, pilots can only be qualified on one Boeing type rating at a time.

Given the Max is so close to older 737s, I see no reason why Boeing and the FAA couldn't come to a similar arrangement, were pilots can be cross-trained with a 1 week course and maintain simultaneous type ratings on both the Max and Older 737s.

Greed and hubris, would be my best guess.

Is it really possible for Boeing to fix this problem with out making changes that will require pilots to recertify anyway? If not then it seems like the training costs are a wash.

Add a third AoA sensor.

My understanding from what I’ve read elsewhere is that changing the AoA sensor from non-safety critical to safety critical is enough for it to be considered a new type of aircraft. This makes sense as you would expect pilots and crew to be trained on how to interpret all safety critical instruments.

If Boeing were to hire me to consult on this project, I'd suggest that they just go directly to five blades and/or make changes to instruments so they can be dialed to 11.

Sure, but the same goes for the 737 MAX-2 or whatever they call the eventual result of this, probably. I really doubt that they'll be able to continue to claim it's just another 737.

A320neo is also having issues. [1] Though the issues mostly have been related to a type of Pratt & Whitney engine, rather than the airframe.

[1] https://qz.com/india/1527728/india-places-checks-on-airbus-a...

> Boeing performed an internal review and determined that the lack of a working warning light “did not adversely impact airplane safety or operation,”

> after the crash of Lion Air Flight 610 […] conducted another review and again found the missing alert did not pose a safety threat

How was it possible to come to this conclusion? Twice!? Even after it caused a crash!?

"Well, we didn't tell pilots that MCAS exists, which means we didn't tell them that AoA is hooked up to it, which means that AoA failure isn't relevant to their decision making, so there's no safety threat from not communicating it."

(In case there's doubt: I am not joking, I think this is seriously the argument.)

>“You better start knowing things about the airplane you’re building and selling because my life and the passengers that I carry safely across the globe depends on it,” Mr. Tajer said.

haha wow. If that's representative of how pilots feel then Boeing is in trouble.

Pilots don't buy the planes.

May I ask how this gets Boeing into trouble? The only thing pilots could do is strike which I find improbable.

SAS had a seven-day pilot strike just last week, over (mainly) working conditions, so how would a pilot strike over this be so improbable?

So even now Boeing will not include all safety features for all customers. Not that Airbus is doing better, but this indicates that Boeing is not concerned that it could face serious charges for continuing to sell planes with some safety features missing. That's interesting! Why are they so assured?

The USA have a long history of being very supportive of their companies ("regulation is evil!").

Corruption. After the Ethiopia crash, the CEO/Chairman was on the phone to Trump, to ask him not to ground the rest of the 737 MAXs. (He changed positions on that once it was clear Boeing wasn't going to be able to dodge culpability.) The board member responsible for the audits which should have demanded a fix for these issues has been re-elected. Even the dual CEO/Chairman role is a significant governance failure.


Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact