
How the pilots of Lion Air Flight 610 lost control - skilled
https://www.nytimes.com/interactive/2018/12/26/world/asia/lion-air-crash-12-minutes.html
======
Animats
There have been a surprising number of air disasters in recent years caused
primarily by air data sensors returning false data.

\- Birgenair Flight 301 - B757. Pitot tube clogged possibly by insect nest,
false overspeed indication, autopilot commanded pitch-up, alarms, stall
warning, crew confused about speed, loss of control. 189 dead.

\- Air France Flight 447 - Airbus A330. Well known. Pitot tube clogged by ice,
confusion about airspeed, loss of control. 228 dead.

\- Saratov Flight 703 - An-198. Pitot tube frozen. Three airspeed indicators
all disagreed. Loss of control. 71 dead.

\- Lion Air - Angle of attack vane failure, from parent article.

That information could be checked against GPS, and at least one aircraft does
this. But that has its own problems.[1] Checking against an inertial system is
another possibility. Those are complicated, though. The classic airspeed,
altimeter, and angle of attack vane are so simple.

[1] [https://www.gpsworld.com/gps-disruption-a-full-fledged-
aviat...](https://www.gpsworld.com/gps-disruption-a-full-fledged-aviation-
problem/)

~~~
gugagore
Do aircraft inertial measurement systems ever provide linear velocity
information that isn't trash? I had assumed not (except for special cases
where you can do e.g. a zero-velocity update)

An inertial measurement system cannot measure velocity. It's insensitive to
which inertial frame it is in. (to move from a non moving frame to a moving
frame, you must accelerate, which it could detect. but then you have to
integrate these accelerations over time)

~~~
SomeHacker44
Your assertion is simply demonstrated to be untrue.

Inertial systems integrate all measured accelerations over time to compute
present velocity and position. They are periodically corrected with external
references. The accuracy is limited to the accuracy of the accelerometers, the
sampling frequency and the accuracy of the digital math, among other things.

This IMU idea could be used, but I am not aware of any current commercial IMUs
in airliners - but I am not an airline pilot so that does not count for much.

If I were given conflicting information I would see how the plane is acting. I
have flown planes with various primary instrument failures over my decades,
some in actual instrument conditions, and always had sufficient secondary
sources of data to determine the situation. As an example in a piston plane:
Tachometer fails? Cross check fuel flow, manifold pressure and airspeed -
subject to mixture control which can be reasonably set using "lean to
roughness and enrich" method. Been there, done that. I have had an altimeter
fail insidiously slowly, when water got into the static lines (despite
redundant ports and properly routing them upward), while in actual near
minimums in a non precision approach. (I used GPS altitude cross reference and
odd seeming behavior of altimeter needle to diagnose and switch to alternate
static.)

Importantly, if the plane is flying straight and level and sufficiently fast,
over time this becomes a "feel" and you can tell if something is
extraordinary. Maybe not on airliners, but definitely the piston planes I fly
a lot and probably the small jets I fly on occasion.

This is why some of these things are mystifying to me. If your engines are set
to a certain power and seem to be operating properly, and your attitude is
just so, in the absence of unusual meteorological conditions your plane should
be performing in a manner consistent to your last experience. If your stall
warning is going off, well, maybe the instrument broke. Air France is really
scary mystifying.

Look, I am a software engineer so I get it, people can get locked into a
misconception about a fault and not step away long/far enough to consider
other alternatives. It happens a lot in debugging. But for whatever reason my
brain works much faster and more dynamically when shit happens in the left
seat of an airplane having problems.

~~~
lolc
> But for whatever reason my brain works much faster and more dynamically when
> shit happens in the left seat of an airplane having problems.

Maybe you didn't intend it but this sounds a lot like "couldn't happen to me!"

~~~
SomeHacker44
LOL. Oops.

I meant to mean "It has most definitely happened to me" although not yet in
the context of piloting - but in other contexts.

------
WalterBright
What the pilots experienced was indistinguishable from runaway stab trim, and
shutting off the stab trim from the switches on the console is the correct
response. There's a loud distinctive sound when the trim runs, and the wheels
that bracket the console turn, so it's pretty obvious. The previous flight's
pilots had indeed done this. The pilots are trained for this. They has 12
minutes to shut off the repeated action of the runaway stab trim.

Additionally, the airplane should have been grounded after the previous
flight, as runaway stab trim is a serious problem, until the fault was found
and corrected.

Equally badly, the flight crew was probably not informed of what had happened
on the previous flight.

~~~
cameldrv
Well sort of but not quite. When the pilots clicked the trim up switch on the
yoke, it disabled the MCAS system for 5 seconds -- i.e. it made the problem go
away temporarily. Then MCAS comes right back with more nose down trim. This is
not a typical runaway trim situation where it's just continuously rolling in
more trim. The pilots had no idea that this system existed or that a "runaway
trim" failure could have these characteristics.

Sure, it's easy to say from the ground with what we know now that all they had
to do was flip a couple of switches, and that a previous crew managed to land
safely. However, the job of an airplane is not to be safe only with quick
thinking, above average pilots. If a single sensor failure can present a
situation that 99% of pilots will successfully diagnose and recover from,
you're looking at multiple crashes per year.

~~~
WalterBright
The pilots did not need to diagnose the system. It would have been obvious
that the trim system was running, and was causing the nose down. The trim
cutoff systems are right there on the center console. They dealt with the
issue for 12 minutes, lightning reflexes were not necessary.

The NTSB will of course look into the CVR, the pilots' training, background
and track record to try to figure out why they did not use the cutoff
switches. I'm very curious about that.

Similar types of accidents have occurred in the past and turned out to be CRM
(Cockpit Resource Management) issues, where the copilot recognized what was
wrong but was intimidated by the pilot into doing nothing.

------
privateSFacct
What is never mentioned is that since forever runaway stabilizer is one of the
FEW memory items.

Here it is. To be qualified as a pilot you have to have this memorized.

I. Runaway Stabilizer

CONTROL COLUMN - HOLD FIRMLY

AUTOPILOT (if engaged) - DISENGAGE

Do not re-engage the autopilot.

If the Runaway Continues

STAB TRIM CUTOUT SWITCHES (both) - CUTOUT

All this drama around "fighting the controls" and "fighting the plane" is
weird. This is not some procedure you need to lookup, this is one of a few
memory items.

~~~
dingaling
That process isn't sufficient on the 737Max. That is the point; to make it fly
like a 737NG Boeing added an additional system, MCAS, that requires the AoA
sensors to be manually disconnected during a malfunction or it will continue
commanding elevator pitch. You could run through your memory drill as many
times as you like, it wouldn't have helped in this case.

~~~
mannykannot
The FAA Airworthiness Directive (2018-23-51) can be found at the address below
(pdf). There is no mention of manually disconnecting the AofA sensors. The
addition to the Airplane Flight Manual required by the AD explicitly says "do
the existing AFM Runaway Stabilizer procedure above, ensuring that the STAB
TRIM CUTOUT are set to CUTOUT and stay in the CUTOUT position for the
remainder of the flight."

While I don't think pilot training is the only issue, we can't assume a crash
was the most probable outcome following from the failure of the AofA indicator
in question.

[https://rgl.faa.gov/Regulatory_and_Guidance_Library/rgad.nsf...](https://rgl.faa.gov/Regulatory_and_Guidance_Library/rgad.nsf/0/fe8237743be9b8968625835b004fc051/$FILE/2018-23-51.pdf)

------
AceyMan
The page is graphics loaded page with floating paragraphs over a single "sheet
of wallpaper" showing interior and exterior diagrams of the aircraft. I can
see it not working worth a damn in any kind of Reader View.

FWIW, nothing new here, but it's a good 3rd grade overview for the layperson.

And I am a Boeing guy, through and through, but they screwed the pooch on this
one. RIP, Lion Air 610.

/Acey

(me: FAA licensed dispatcher)

~~~
sargun
Can you point us to a better article?

~~~
trevyn
Here’s the actual safety board report, including flight data recorder graphs:
[http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/20...](http://knkt.dephub.go.id/knkt/ntsc_aviation/baru/pre/2018/2018%20-%20035%20-%20PK-
LQP%20Preliminary%20Report.pdf)

Looks like there was a problem with the autopilot trim, likely due to a faulty
sensor, and the pilot just needed to switch off the autopilot instead of
fighting it — the previous flight of that aircraft experienced a similar
issue, and they just turned the autopilot trim off and carried on normally. :(

The graphs on PDF page 23 are particularly morbid, and you’ll note that two of
the final auto nose-down commands are of longer duration than all the rest.

~~~
trevyn
Ah, this is quite good as well: [https://leehamnews.com/2018/11/14/boeings-
automatic-trim-for...](https://leehamnews.com/2018/11/14/boeings-automatic-
trim-for-the-737-max-was-not-disclosed-to-the-pilots/)

Apparently it wasn’t the standard autopilot trim, but a special safety program
specific to the 737 MAX. Which still is just a switch to disable, if you know
that switch is there, or that there is a checklist for dealing with runaway
trim, or that you’re aware that auto-trim commands are the reason your plane
keeps nosing down...

~~~
caf
I think the most important point described in that article is that the MCAS
trim, unlike the other kinds of elevator trim, _isn 't_ defeated by limit
switches on the yoke - and that this critical information was not in the
training or flight manual.

~~~
dotancohen
If true, then this right here is the problem root. Why isn't pilot limit
respected with regard for the MCAS specifically? Are any other systems so
independent?

I remember years ago the Boeing folks claiming that the automated Airbuses
would be dangerous because the pilot always knows best. So how did it come to
this on a Boeing?

~~~
caf
The answer appears to be something along the lines of _" because the whole
point of MCAS is to stop the pilot flying the aircraft beyond the limits of
stability"_.

------
iamhamm
These human-machine interface failures always fascinate me. The NYT makes the
quip about not being able to look down and note the trim, but speed doesn’t
need to be high to have these failures. It seems to be more related to system
complexity misunderstandings or compounding interpretation issues. Look at the
grounding of the Royal Majesty - not exactly some high speed object; it all
amounted to a misunderstood icon. See:
[https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20t...](https://ti.arc.nasa.gov/m/profile/adegani/Grounding%20of%20the%20Royal%20Majesty.pdf)

------
WatchDog
You would think it would make sense to automatically disable the MCAS system
and sound a warning if the aircraft detects conflicting readings from it's
sensors.

~~~
akira2501
Due to the way the engines are mounted on the frame, which enhances their
efficiency, they also cause more of a "upward thrust vector." This makes it
very easy for this plane to reach dangerously high AOA in certain scenarios,
particularly during turns.

The MCAS has a specific and important function and just turning it off is
probably not going to increase safety. The real problem was the Boeing did not
disclose the existence of this device and it's functions in aircraft training,
according to one source, because they did not want to inundate new pilots with
too much information about the plane and it's attendant safety systems.

Perhaps, had the pilots known, they would have seen the stick shaker/stall
warning system activating on _one side only_ as a serious indication of an
Airspeed/AOA system fault and the potential for incorrect MCAS outputs being
generated.

They might have known to disable the electronic trim control, bypassing the
MCAS, and then to manually fly and trim the plane with the aforementioned
thrust vectoring taken into consideration. They could have trained for this.
That would have all given them the best safety margin for survival here.

~~~
cameldrv
In the case of a disagreement between the AoA sensors, the obviously correct
thing to do is for the computer to disable MCAS and put up a warning light.
The conditions MCAS addresses only happen when the AoA gets very high, such as
in a slow speed over banked turn. The plane is stable in normal operation.

Think of how a human would react: One sensor says everything is normal. The
other says a very rare emergency situation is occurring. Since the sensors
disagree, you know one of them is defective and wrong. Applying a very rarely
needed emergency correction when you know that you have a sensor fault is not
reasonable.

Even applying the stick shaker is confusing to the pilots and dangerous. Much
more appropriate is a warning light of a sensor malfunction/MCAS disabled.
Then the pilots must simply be extra careful to not make overly banked turns
for the remainder of the flight, and replace the sensor on landing.

~~~
akira2501
> the obviously correct thing to do is for the computer to disable MCAS and
> put up a warning light.

The only problem with that is that problems rarely happen isolation, and you
have to consider whether the pilots are going to notice the warning in the
midst of several others and if they are going to give it the appropriate
priority and consideration while flying.

> Think of how a human would react

That's exactly it, though.. look at Air France 447. The system automatically
disabled itself and put the plane into an "alternative law." All
automatically. The pilots did _not_ notice this, and still flew the plane into
the ocean even though they had several minutes to work the problem.

It's not that simple.

> Even applying the stick shaker is confusing to the pilots and dangerous.

All evidence to the contrary. The stick shaker is an amazing safety device
because it demands priority of consideration. It's not going to get lost in
the noise of a degrading cockpit. Seriously, go listen to some cockpit voice
recorders of a disaster.. it's never what you would expect.

~~~
cameldrv
> problems rarely happen in isolation

Problems usually happen in isolation. Your dataset is skewed because you've
read a bunch of accident reports. When there are multiple failures together,
this is much more likely to lead to an accident. When there's just a single
problem and it's handled with no loss of life, they don't write a report about
it.

I agree, the stick shaker gets a pilot's attention. What the stick shaker
tells the pilot is that he's about to stall, but that's not what was
happening. Shaking the stick is loudly yelling false information at the pilot!

Let's look at the epistemology here. If we're only looking at the two AoA
indicators, and one reads 5 degrees and the other reads 25 degrees, we know
that there has been a sensor failure. All you can say that the airplane does
not know what its angle of attack is. That's fine though, we've been flying
planes for a hundred years without AoA indicators, even ones that had way
worse pitch instabilities than the 737MAX. If the airplane doesn't know the
angle of attack, there is no reason for it to activate the stick shaker, put
in nose down trim, or do anything else except to calmly notify the pilots that
AoA is unavailable and therefore MCAS is disabled. All the pilots need to do
then is fly the plane normally and not do any crazy banked turns or extremely
abrupt pullups at low speed. It's definitely wrong for the plane to start
dialing in nose down trim "just in case", because the "just in case" can kill
you if it's not necessary!

On AF447, there were as usual, a lot of mistakes made. One problem clearly
though was that the plane was giving the pilots a lot of conflicting
information that confused them. If the plane was seeing three different
airspeeds, the best thing for it to do would have been to put a big red X over
the airspeed tape and let them fly by pitch and power. This is exactly why a
lot of instrument pilots in older smaller planes carry a little instrument
cover. If say your AI fails in IMC, you don't want to see the wrong indication
at all, so you cover it up and use your other instruments. Seeing a wrong
indication, even if you know it's wrong is very confusing and can lead people
to make errors in reasoning, especially in a stressful situation.

------
reactor
From the article, "Outside the plane, one of the plane’s angle of attack
sensors falsely indicated that the plane’s nose was pointed too high, and the
aircraft could stall."

I'm not in any way eligible to comment on aviation systems, but why that
equipment needs to measure the angle of airplane be outside? Couldn't
something like gyroscope mounted inside do the job?

~~~
JshWright
It's measuring the angle of the plane relative to the air moving past it (i.e.
the angle the plane is "attacking" the air). Knowing the angle of the plane in
an absolute sense isn't as useful when it comes to detecting stalls, etc.

~~~
kwhitefoot
It's not as useful but it is far from useless. The ordinary artificial horizon
indicator should have been able to show that the external sensor was far out.
I don't know anything about the aircraft or Boeing but as a software designer
and implementor I would have wanted to use the output of the artificial
horizon as well as the angle of attack sensor in an attempt to gauge the
quality of the readings. I'm guessing here but it seems likely the aircraft
was equipped with a normal gyroscopic instrument in addition to the external
angle of attack sensor.

Of course the Boeing engineers who worked on this could easily tell us exactly
how it works and why. I don't suppose that will ever happen though.

~~~
7952
I think the problem is that moving air (such as an updraft) can alter the
angle of attack independently of the pitch angle. That is important during
landing where low thrust levels are set and the air is turbulent. But in other
phases of flight it should be possible to keep the aeroplane stable using know
thrust and pitch settings and ignore air speed and AOA sensors completely. Of
course that requires fully functional control surfaces and engines. Maybe it
would be better to remove the erroneous sensor type completely and have
procedures that can cope with that.

------
jumelles
I'm amazed that Boeing isn't the focus of more - much, much more - criticism.
It seems clear to me they are vastly more culpable than the pilots or airline.

~~~
pcurve
I agree that Boeing's share of responsibility is big. I think media coverage
has largely reflected it too. I don't recall reading any article where it put
the blame on the pilots. Based on what I've read, it felt more like, 70%
Boeing, 29% airline, 1% pilot.

~~~
privateSFacct
If this is the case we seriously need to consider getting rid of the pilots.
Stab trim issues are a memory checklist item under existing training with
clear cutoffs easily available.

1% pilot fault is ridiculous.

~~~
SomeHacker44
"Getting rid of the pilots" has already mostly happened. You know this.
Autopilot on at 400-1000' and off at MDA/200'/DH to rollout. In between they
are system administrators. Heck, even I can and often do fly my much smaller
planes this way. And, as you also probably know, autopilot is required to be
flying the plane above 28,000' (Google RVSM for interested readers).

I am sometimes glad I have access to Cubs, Decathlons, and other simple planes
to keep me an honest pilot. :)

~~~
privateSFacct
So true, and overseas actually a bigger issue I think because they don't
always have the recreational / GA flying world going on. Some pilots in
airlines trying to develop local skills are going basically straight onto an
Airbus flight deck. Not saying it's bad, but they don't seem as comfortable
with manual flying. And frankly manual flying is more complicated on these
planes - the number of autopilot and throttle modes etc etc - a cessna -
pitch, power... I do wonder if adding a basic safe flight pitch power button
(ignores everything and just goes to a safe pitch power setting smack in
middle of power curve) might help. Literally ignore AoA, ignore speed, ignore
everything. Just trim to middle and set power. Air france? This flight? These
planes are flying fine otherwise. You don't 100% need all the automation
(though the MAX did introduce some weird instability near stall which is
annoying in a big way). My own view - make a naturally stable plane (MAX moved
away from this), and have a way to fall back to a basic setting that takes
advantage of that.

------
cameldrv
What the plane did here is not technically reasonable. We had optimal
estimation and sensor fusion in the sixties. With two AoA sensors, three
pitot/static systems, GPS, and a full IMU, the plane had more than enough
information to determine that there was no high AoA situation requiring its
intervention.calling for the computer's intervention.

Since this dangerous high AoA situation is so rare, even a simple rule
requiring both AoA sensors to agree that AoA was dangerously high to override
the pilot completely solves the problem. Even with just one AoA sensor and a
little memory, the simple fact that the system had rolled in so much nose down
trim, which should have lowered AoA, with no apparent effect should have clued
it into the fact that it’s model of the system was wrong and caused it to
stop.

This is not a particularly sophisticated insight. The engineers all knew this
when they designined the system. They had a reason for doing it the way they
did, though, and that was to slip the new system in a bit under the radar of
the FAA. Since the MCAS is required to maintain the stability of the plane at
high angle of attack, it should have been certified as a Stability
Augmentation System. That would have subjected it to more redundancy
requirements and eliminated this failure. The problem was that Boeing wanted
the MAX to not require significant training or certification beyond the
737/737NG. A big new Stability Augmentation System would have required extra
certification and probably pilot training. Instead, they chose to sort of
launder this system through an existing one, the Elevator Feel Shift system.
The EFS adds some nose down trim at certain speeds and altitudes to make the
737NG feel like a classic 737.

Since the FAA already determined that the EFS wasn't a stability augmentation
system, but it could control the elevator trim, Boeing figured they could
piggyback the new MCAS onto it, adding just one new input, the AoA sensor, and
no new outputs. Since it only controlled the trim and not the main flight
controls, Boeing could keep to its manual control philosophy, and they could
slip it through certification. They couldn’t give the computer both AoA
inputs, because the air data system is supposed to have two totally
independent and manually selectable sets of sensors. If you give a computer
both AoA inputs, you lose that redundancy concept.

So why did the EFS system never cause any problems despite having the same
lack of redundancy? My guess is that the EFS is essentially an open-loop
controller. It applies a fixed amount of forward trim for a given
speed/altitude combination. If the pitot/static system goes haywire, the worst
that happens is fixed, moderate amount of nose down trim, easily and naturally
compensated for by the pilot or autopilot. The MCAS appears to be closed loop
in that it will just keep adding more and more nose down trim until the AoA
sensor says things are OK again, the pilot pulls its circuit breaker, or the
plane smashes into the ocean.

People in the industry blaming the pilots one bit are making a mistake the
industry collectively stopped making seventy years ago. We don’t blame the
pilot anymore, we blame the system. Given enough time, humans will make any
mistake that can be made. If the plane cannot be flown completely safely by
significantly below average pilots, it's an unsafe plane. Demanding that the
system be safe even with imperfect pilots is why commerical aviation is so
amazingly safe today. It’s also ironically why the MCAS is in there in the
first place. Boeing could have just put in the manual to never exceed 14
degrees of AoA, and even average pilots would never have a problem. The FAA
would never certify such a plane to carry paying passengers with such a
limitation though. It would be too dangerous. Eventually someone would screw
up and the plane would go out of control, so there had to be a computer to
prevent this. As it stands, the plane does not IMO meet certification
requirements for the transport category, and the only reason it isn't grounded
is because there are literally half a trillion dollars worth of these things
in service or on order. I'm racking my brain to think of another single
product produced anywhere that is so valuable.

~~~
unionemployee
I agree that that more focus should be placed on the system, however, with so
many NTSB reports naming "pilot error" as the cause of accidents/incidents, I
think we're still very much focused on the the pilots.

~~~
AceyMan
GP says it all right here,

>>> People in the industry blaming the pilots one bit are making a mistake the
industry collectively stopped making seventy years ago. We don’t blame the
pilot anymore, we blame the system. _Given enough time, humans will make any
mistake that can be made._

Excellent rundown by cameldrv all-around. Framing this a bit of certificatory
sleight-of-hand is spot on, far as I can tell.

------
NelsonMinar
All these years of listening to Boeing bigots talk about how bad Airbus planes
are because they are too automated, flown by machines, unsafe.. And then a
Boeing automated system seems to a significant cause of a fatal accident. Not
sure what if any conclusion to draw from that, just the context.

------
1stranger
Would it be possible to have a single switch that disables all automatic
systems and puts the plane in complete manual control as much as possible? Can
these planes even operate in a complete manual mode?

~~~
drpixie
Certainly possible but not required. In the B737 family, most of the main
controls are essentially power assisted direct controls.

Boeing consistently has preferred more-or-less manual controls, while Airbus
took the other direction (in which the pilot guides the computer, which
operates the controls).

All airlines can fly happily in completely "manual" mode. Boeings are doing
this most of the time, Airbus's only do this when the situation is outside the
computer's envelope.

------
BXLE_1-1-BitIs1
There was an erroneous stick shaker that captured the crew's attention and
diverted attention from the trim stuffing the nose down. The NYT graphics show
a miniscule part of the problem. Most likely the FDR data has been played
through a full motion simulator - it would be a hairy ride, but the crews have
to keep it confidential until the Indonesian authorities release the data.

------
heyjudy
[https://outline.com/pELX5F](https://outline.com/pELX5F)

------
gumby
It doesn't feel like the NYT is trying to communicate something to their
reader. Or if they are I can't figure out what the message is.

Apart from the whizzbang combined with tiny paragraphs (making it hard to read
or comprehend), not all the text is available to Safari's reader mode.

------
Havoc
Both inconsistent readings and a plane actively working against the pilots
will.

That seems like a pretty massive failing on the manufacturers part.

~~~
privateSFacct
Except that runaway trim is a MEMORY checklist item.

Literally - they should have the procedure here memorized, and there are LOTS
of cutout options, from temp right at fingertips to cutoff switches.

~~~
Havoc
The fact that the FAA issued an emergency order with instructions after this
crash suggests to me this wasn't sufficiently covered by existing procedure.
In fact:

> The FAA's directive orders airlines within three days to update flight
> manuals to include specific steps pilots should take to recover.

