
Navy’s flawed technology set the USS John McCain up for disaster - thomasjudge
https://features.propublica.org/navy-uss-mccain-crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/
======
crmrc114
In a panic people get tunnel vision. When you design HMI's for a living you
have to consider. If my brother, sister, grandmother was using this system
during an emergency could we guide them to make the right decisions.

The fact that the prop controls could be split but the UI would not gray out
the sliders on the screens that did not have control is a poor user interface
design choice. If the controls only appeared "active" on the screen they were
transferred too this could have made things much more simple.

The fact that the mushroom button for emergency steering yanked control to
wherever it was pressed- this should have displayed a red on black box.
Emergency Override called from X Station.

The issues with the navigation system are a whole other bag of worms- who
knows what the mess was there.

Disclosure, I have done HMI design and install in production systems for
manufacturing, chemical and warehousing operations. I have never built
anything for a ship or vehicle. So I may be speaking total stupid to someone
in that industry. I would love to hear someone from that market sector drop in
and talk about how or why the HMI was setup like this.

Sounds like this system was sub-contracted to the lowest bidder and pushed out
without enough testing.

~~~
rwmurrayVT
Integrated bridges are being installed all across USN ships, including Mt.
Whitney with MSC right now. They're filled with a hodgepodge of HMIs that are
all linked to a control system filled with gyros, ECDIS, RADAR, propulsion,
etc. etc. System integration is complex and the design work is done by
contracted companies. You don't have the first party vendor doing the
integration work. It's a recipe for disaster.

~~~
magduf
This is pretty typical of military contracts. They like to spread the work
around, so different vendors provide different pieces of the system, and it
ends up being a hodgepodge that doesn't integrate well and doesn't work well.

~~~
lonelappde
The Navy doesn't like to spread the wolk around. Congressmembers like to
spread it around to get a piece of the pork.

------
allovernow
>Immediate responsibility, the Navy ruled, rested with Sanchez, his officers
and senior sailors. They had been lax, even complacent, in their training of
the sailors steering the ship. Sanchez had made a critical error in not adding
more sailors to stand watch as the McCain navigated the treacherous strait

From what I read consistently about the Navy, and hear from veteran friends,
any sailors they could have asked to stand watch would have already been
overworked and under rested. Everything I read about the U.S. military, and
especially the NAVY, seems to indicate a broken culture of bloat and
incompetence from the top down. I think we've been too comfortable as the
premier unchallenged world superpower and the services have started to rot
from within.

~~~
Balgair
I can't find the source [0], but there was another expose'/interview on the
_USS John McCain_ where they interviewed darn near the entire crew. My details
are fuzzy, but from what I remember the crew has been _severely_ short staffed
for years. To the point where 'normal' days were like 18 hours long and sleep
for the captain was 2-4 hours a day for years on end. The internal email
system for the ship was so broken that they resorted to just using free Gmail
addresses, they had to tape over control knobs to keep them from being
disturbed, training was completely abandoned, etc. It paints a hell of a bad
picture of the Pacific fleet and it's command.

[0] I think it was propublica as well, but I can't find the article there.

~~~
mandevil
Your google-fu failed you because it wasn't about the USS John McCain, it was
Propublica's article on the _other_ fatal 7th Fleet destroyer collision in
2017, the USS Fitzgerald. The collisions were two months apart.

The article you are thinking of is here:
[https://features.propublica.org/navy-accidents/uss-
fitzgeral...](https://features.propublica.org/navy-accidents/uss-fitzgerald-
destroyer-crash-crystal/)

~~~
Balgair
Bingo! Thank you!

------
waterside81
> In 2014, Navy officials discovered a flaw in the IBNS. One component could
> not keep track of more than 150 ships at a time without malfunctioning,
> according to Navy investigators. The Navy’s solution? Sailors were told to
> delete tracked ships before the total hit the magic number.

This blows my mind. Can anyone here guess what's going on under the hood? Is
this a magic number that a developer came up during testing to avoid running
of out memory/swapping?

~~~
klodolph
Just a guess—probably designed using high-reliability / realtime principles.
This might involve e.g. allocating memory only at startup, which would explain
things. The limit has to be _some_ number when programming this way.

~~~
magduf
In a real-time safety-critical OS, you absolutely do allocate memory only at
startup, but this kind of OS usually isn't used for any kind of GUI interface,
it's used for much simpler systems. Another trait of these systems is that the
CPU caches are all disabled, because those prevent determinism. But again,
this isn't something you'd do on a GUI system; it's something you'd do on an
ABS brake controller or an engine ECU, where the amount of memory ever needed
is easy to determine in the design.

~~~
vonmoltke
This is also done in the signal processors that power these systems. I don't
know this system in particular, but the USN has been moving towards commodity
computing hardware for these systems. Essentially, a rack of commercial
servers (usually IBM) running stripped-down RHEL with a real-time kernel. The
software for these systems is firm real-time, and generally uses the same
principals as other real-time software (including, but not limited to, up-
front allocation).

~~~
jki275
These are just PCs running Windows, and I don't believe there has ever been a
thought of real time processing on them.

------
schwanksta
Not to be overlooked: ProPublica actually constructed a 3D version of the
navigation system's control board: [https://www.propublica.org/article/how-we-
reconstructed-the-...](https://www.propublica.org/article/how-we-
reconstructed-the-flawed-navigation-controls-behind-the-navy-worst-maritime-
accident-in-40-years)

~~~
mikeyouse
Couldn't find it in your article, but it looks like the 3D models are
integrated into the story here: [https://features.propublica.org/navy-uss-
mccain-crash/navy-i...](https://features.propublica.org/navy-uss-mccain-
crash/navy-installed-touch-screen-steering-ten-sailors-paid-with-their-lives/)

------
unlinked_dll
Something unclear from the article, but jumped out at me.

It seems that the Achilles heel of the IBNS system was that too much
information tended to bork it, but what’s unclear to me (as implied by the
article) is if dropping incoming data was actually the sensible thing to do
(as is often the case) and if incoming data was prioritized over existing data
and that was dropped instead.

The other thing I would point out is that the technical failure is not that a
system is complex, but that a complex system has a user interface that allows
a sequence of actions to leave it in an invalid or unpredictable state. The
transfer of throttle control being incomplete while other actions were taken,
for example.

I think the thing lacking here was consistent and sound changes to the state,
lack of fuzzing user input to observe this possibility, and the reliance on
training to avoid the issues at all. These are problems that can be solved
with design.

~~~
iudqnolq
I haven't worked on any project anywhere approaching significant complexity,
but the magic number of 150 supported tracked ships sounds insanely low to me.
Is this at all reasonable, or does it indicate the design and implementation
were shit?

Does anyone know how much processing is needed here? Is it just updating an
internal model of locations and warning about impending collisions? Presuming
constant speed and heading that sounds easy, not presuming it sounds not
feasible even for just a handful of ships.

Also, if only 150 ships are supported, wouldn't it makes sense for the
computer to automatically untrack them? Maybe allow manual prioritization of
what to drop first, but surely within a few days of a competent vendor
learning adding too many items borked their system they would put out a patch
that does the absolute minimum by automatically preventing adding too many
items?

> In 2014, Navy officials discovered a flaw in the IBNS. One component could
> not keep track of more than 150 ships at a time without malfunctioning,
> according to Navy investigators. The Navy’s solution? Sailors were told to
> delete tracked ships before the total hit the magic number.

> The navigation system could also become overloaded if too much information
> streamed in from a ship tracking database used worldwide to prevent maritime
> collisions. The Navy’s second solution was similar to the first: Drop the
> feed when it became too much.

> They were patches on top of patches that left the Navy’s destroyers without
> a full picture of the seas around them. But none of the problems was serious
> enough to attract high-level attention. A Navy system designed to track
> problems in major ship systems did not contain any reports that mentioned
> the IBNS until last year, according to a Navy official.

~~~
touisteur
150 tracks without knowing the range of their navigation radar systems doesn't
say much. It's hard to put more than 150 objects in a 5-10NM radius. Since
it's a navy ship the nav radar is probably more powerful, but still, it's not
like a 200NM range air-defence radar. I'm guessing they spec-d the system with
incoherent requirements, and either the contractor that won went for the
cheapest working hardware without asking too many questions, or maybe they
took an old system's spec and asked for an incomplete upgrade... or nobody
wanted to go to their boss saying the system didn't work because of some badly
written spec or badly managed contractor and we need more money, please ? That
would track with leaving the system badly working like this without even
having some face-saving 'we're trying to work it out with the contractor right
now'.

Or, most likely, normalisation of deviance. The system came at first with so
many bugs and problems that when they got it in some stable state, they'd got
so used to work around the bugs that they kept on doing that. Look we already
put so much effort in this, what's a bit more? Yeah, we're quite happy with
the system, at least it doesn't crash every hour like when it was delivered...

~~~
aidenn0
It sounded like it was "150 tracks since last reboot" which is absurd, since
the workaround was to manually delete out-of-range tracked objects.

------
remarkEon
Absolutely phenomenal reporting. Take the time and read the entire thing in
detail. It’s worth it.

Can any Navy vets comment or speak to these systems? I really worry about the
proliferation of these digital interfaces. I’m curious to know how folks feel
about them.

~~~
linuxftw
I can't speak to that specific system, but I can give you plenty of other
anecdotes about the systemic unreadiness of the US Navy.

I was forced to be the on-duty electrician from time to time, despite not
being an ship's electrician. One of the responsibilities for the on-duty
electrician was to cut the right circuits in case of a compartment fire.
Despite my repeated protests that I didn't know what I was doing and needed
additional training, I was forced to be the on-duty electrician. Mind you,
there were no other duties I had to perform, just that one, and just in case
of a fire, so it's not like my every-day workload was increased. Determined
not to get people killed in the event of an emergency, I asked the actual
electricians what I should be doing. Turns out, they didn't know either. Their
plan was, in case of fire, kill power to the whole ship. I don't think I need
to tell you, this is exactly the wrong move to make.

There's an ever-growing number of electronic systems onboard ships, and an
ever-shrinking number of sailors to run them. Qualifications are a joke in
pretty much every department, at least in the surface fleet. Ships are in a
constant state of disrepair, it's not uncommon for 30-50% of the
communications equipment to be broken at any given time, parts taking months
to receive (mostly due to budget restraints).

Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up
by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with
no sleep, followed by maybe 2 hours of sleep before having to work again the
next day, and possibly have another watch from 20:00-12:00. Rinse repeat.

~~~
jandrese
That last paragraph screams of a massive failure in Crew Resource Management.
There are acres of case studies about this subject throughout industry, and
when people take them to heart lots of accidents are avoided.

Airline pilots are a prime example. How many airliners crash these days? CRM
is one of the factors in making them so safe. Truck Drivers now have similar
requirements and a much improved safety record as well. This is literally life
or death, industries that ignore it are being unnecessarily reckless and
unprofessional.

~~~
Noumenon72
It would be cool if anyone has a source for the improved trucker safety
record. It is just common sense, though. I don't know how this no-sleep system
ever got traction.

------
jschwartzi
It strikes me that at least 16 seconds of screwing around could have been
saved by changing the caption of the "Emergency Override - To Manual" button
to say "Emergency Override - Take Control." The button's caption would then
contain enough information for anyone pressing the button to know that they
were responsible for all controls at the station after pressing.

------
FlyMoreRockets
Very sobering article. Makes me reconsider the proliferation of glass cockpits
and fly by wire in aviation. Granted, there is no putting the genie back in
the bottle, but failsafe and redundancy need to be a central design tenet from
the start.

Back in the 80's, I was throttleman aboard a similar sized USN ship. There was
no direct link from the bridge to the throttle. Throttle commands were
communicated via an indicator and the throttle operator manually turned a big
wheel to meet the command. I remember thinking at the time there should be
direct throttle control from the bridge. Hindsight is 20:20.

~~~
magduf
Glass cockpits and FBW are great, in theory. In practice, I don't know because
I haven't had to use it. However, don't make the mistake of assuming that just
because the US Navy can't get a well-engineered system put together means that
Boeing and Airbus are guilty of the same. (Their glass-cockpit systems come
from other vendors, who also supply to smaller aircraft makers BTW.)

Also, how many foreign militaries have had problems like this with basic ship
steering?

From what I can tell, the US Navy is uniquely broken here. UK, Australia,
Japan, France etc. do not have have their ships plowing into slow cargo ships.
Commercial aircraft are not flying into the ground left and right. (The 737MAX
is pretty bad, but that's not even related to glass cockpits; MCAS was a
Boeing-engineered component, not a glass cockpit sourced from some vendor like
Thales.)

~~~
nwallin
> Also, how many foreign militaries have had problems like this with basic
> ship steering?

> From what I can tell, the US Navy is uniquely broken here. UK, Australia,
> Japan, France etc. do not have have their ships plowing into slow cargo
> ships.

Norway lost a destroyer last year after colliding with a tanker. Keep in mind
the US Navy is larger than the next 13 largest navies combined. 2 of 3 of
these events happening to the US Navy isn't statistically significant enough
to say it's a uniquely US Navy problem.

Not to say they don't need to get their shit together.

------
_Nat_
This story's worrisome in that it makes the Navy sound very vulnerable to
electronic warfare.

I mean, if overall operations are so hazard-prone that they're not reliable in
the absence of hostilities, how could they possibly be resilient against an
advanced actor?

Is the situation as bad as it sounds from the story?

------
paranoidrobot
I find it a little ironic that for an article about unintuitive and broken
electronic navigation system, the UX on mobile of this article breaks
navigation in weird and unintuitive ways.

They value their narrative and slow animated exposition over my ability to
scroll and read the article, meaning at multiple times I have to stop and wait
for their painfully slow animations before I can continue.

~~~
pbourke
I thought the visuals were well done, but I agree that they should have just
scrolled normally on mobile instead of capturing the scroll to advance the
animation.

------
fmakunbound
> could not keep track of more than 150 ships at a time without
> malfunctioning, according to Navy investigators. The Navy’s solution?
> Sailors were told to delete tracked ships before the total hit the magic
> number

> “Usually when we have a fault with that system,” Sanchez said, “their
> resolution is to reboot the system.”

Of course it is.

Software is going to kill us all.

------
userbinator
_He ordered Bordeaux to take over steering the warship while another sailor
controlled its speed. The idea was to avoid distractions by having each man
focus on a single task in the heavy maritime traffic._

I've not controlled anything that floats bigger than a small fishing boat, and
I realise the timescales of reaction here are much larger, but that just
sounds wrong to me --- changes in speed can affect direction, and vice-versa.
Splitting that to two separate brains that are communicating only via a low-
bandwidth voice channel seems like a recipe for disaster.

~~~
at-fates-hands
>> changes in speed can affect direction, and vice-versa. Splitting that to
two separate brains that are communicating only via a low-bandwidth voice
channel seems like a recipe for disaster.

It's interesting you mention this. My uncle and cousins have been sailing for
years on Lake Superior. They navigated some of the largest races on the lake
for decades, including the Trans Superior in every imaginable weather
conditions. They are, IMHO incredibly experienced sailors.

I just sent this very article to them and my uncle responded within the hour
about how having two people doing these in tandem was a recipe for disaster
because of the very issue you pointed out. He said you should have one capable
person who should be able to do both since it can be a delicate balancing act.
Too much speed and over correction with steering can have disastrous results.

~~~
nwallin
The people with the controls in their hand aren't the people deciding what to
put the controls at. They're told where to set them, and they do it or report
that the mechanisms have failed.

The reason for this is that the people making the decisions have more
important shit to worry about. Things are different when your primary concerns
are getting shot at and shooting back. In war time it's not atypical to get
your engine or rudder shot out, and if you have someone who has one job - to
set the rudder - he'll notice and diagnose it a lot faster than someone who's
tasked with avoiding incoming fire, bearing weapons on the enemy, etc.
Distribution of responsibilities is incredibly important in war. Combat tends
to break down careful, considered decision making.

Also keep in mind most of these people aren't seasoned. This dude was just out
of high school. Information overload is a thing.

------
mannykannot
The only thing here that is worse than this shitstorm of a system design is
the vindictive attitude of the Navy towards the sailors who have to cope with
it. Flogging and hanging from the yardarm may be gone, but the spirit lives
on.

~~~
mannykannot
Update: I have this suspicion that the Captain was charged with homicide in
order to get him to take a plea deal, so that the problems with this system,
and the Navy's failure to properly train crews in its use, would not become an
issue in a court-martial.

------
AareyBaba
Those look like Motif gui widgets from the 1990's on the touchscreen!
[https://features.propublica.org/navy-uss-mccain-
crash/assets...](https://features.propublica.org/navy-uss-mccain-
crash/assets/images/ui-overview-9x7.jpg)

------
slics
These days we are so hyped on the cool shiny objects, but failing at the most
basic concept, HSI = Human System Integration:

“ Systems are comprised of hardware, software, and human personnel all of
which operate within a surrounding environment. Too often, acquisition systems
programs fail to consider the human capacity or requirements as part of the
system. This leads to poor task allocation between hardware, software, and
human users or supporters. To promote ideal task allocation, it is critical
that the human element be considered early in system development.”

------
kbos87
Any entry level UX designer would tell you that you can’t pack more and more
control into an opaque and confusing system and still lay all the
responsibility on the user when something breaks down.

I really wonder, what does UX testing and iteration look like at Northrup-
Grumman?

~~~
CapricornNoble
>>I really wonder, what does UX testing and iteration look like at Northrup-
Grumman?

As someone who worked daily with two different NG-produced command and control
systems, it's probably between "non-existent" and "flaming dumpster fire".

[https://www.northropgrumman.com/Capabilities/C2PC/Pages/defa...](https://www.northropgrumman.com/Capabilities/C2PC/Pages/default.aspx)

[https://dzone.com/articles/war-fighter-netbeans-
platform](https://dzone.com/articles/war-fighter-netbeans-platform)

Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old
version of Google Earth. Newer versions of C2PC are a polished turd, advancing
the app's interface from 1995 to 2005.....despite hitting the operating forces
around 2017-2018. We had bug submissions on some key mission-critical
visualization features in the new version with patch turn-around times looking
like 9-12 months at best.

------
throw7
The Navy has a serious, serious leadership problem. Their seals have also been
fucking up left and right.

This would all be comical if it wasn't so tragic.

------
SQL2219
touch screens are a horrible idea for this type of system. I also think
they're a bad idea in cars. there is no muscle-memory with a touch screen.

------
shitgoose
Tried scrolling article up and down. Animated images hijacking the mouse
control randomly, showing some motion then freezing. A good illustration.

~~~
stefan_
This is the latest innovation in "interactive story telling".

It is just the ultimate in irony: here is an article crying murder because a
touch interface allowed a sailor to un-gang engine power, then slide a big fat
green power slider for one engine to half of what the other big green slider
for the other engine directly next to it showed.

Meanwhile here we are furiously scrolling to hit invisible "scrolled distance"
breakpoints to advance a slideshow.

~~~
ceejayoz
It's even worse if you're used to using spacebar to page-down.

~~~
aidenn0
Meh, I use spacebar to page-down, but am ready to fall-back at a moments
notice because so many sites draw their header _over_ the body, so a spacebar
will end up by scrolling by more than a page...

------
partiallypro
This reminds me of "Pentagon Wars."

------
iPhone1
DO NOT LET YOUR FRIENDS/FAMILY JOIN THE NAVY.

-ACTIVE DUTY SAILOR

~~~
anticensor
Openly defaming your employer might have serious consequences.

------
galaxyLogic
I wonder was the system coded in Ada?

