The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.
The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station.
The issues with the navigation system are a whole other bag of worms- who knows what the mess was there.
Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this.
Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.
This reminds me of the Airbus control sticks, where the systemd averages the two inputs if e.g. one pilot pushes forward while the other pulls back. There is a warning but it can be easily ignored, as happened on https://en.wikipedia.org/wiki/Air_France_Flight_447 .
Also, I suspect there may have been at least one person who raised the idea of disabling/graying out the controls if they're inactive, but it was not listened to or even actively opposed because "it's not in the spec" and the one who brought it up didn't have the power to do anything about it. I've worked in environments like that (thankfully not for long) enough to know that these big and highly bureaucratic projects will tend to have lots of "lack of common sense" problems, either because no one bothers to actually think about what they're trying to do, or the division of labour was so fine that no one could. Lots of process, lots of specification meetings, lots of cargo-culting dogma, but no real thought put into anything. The fact that such developer positions are treated as disposable and easily replaceable probably adds to the problem too.
Now I’m starting to understand the hostility towards systemd.
As for the topic at hand, at a 30k ft view, having seen the inside of DoD stuff from a ground troop level, and later from DC itself, I mostly blame the good-old-boy network from being too established and greedy. I figure it's always been this way and technology complicating things has just pulled the curtain back, but essentially they do DC/Virginia back-room dealing to get bids and then under-perform in the delivery but don't really care because DoD's budget is massive and they keep throwing money at them. This matters much less while we engage countries that can't engage back like Iraq and Afghanistan, but as we shift more towards the return of the tri-polar world and DoD has to start planning for more peer level agitation, these problems need to be fixed ASAP or people are going to die.
There is a reason a handful of counties in Virginia had the highest growth of millionares in America post-9/11. I'm tired of contractors wasting money while the real military gets shite, and while maybe some of that is bias from my Marine Corps days (The Corps was notorious for "hand-me" down equipment and small budgets compared to Army, etc), I still say it's about time we financially body checked the good old boys and gave our boots on the ground better treatment.
Hell, how about just not having boots on the ground in the first place in unconstitutional bullshit wars, but I digress.
In this case, I see that the ship's left and right power controls were transferred from one station to another one at a time. Regardless of whether, in the system's byzantine command structure, there was a way to transfer them together, the fact that they could be transferred individually suggests that someone thought there was a scenario where it made sense to have two different people, possibly in widely separated places, handling the two sides of this function. perhaps someone could enighten me as to the scenarios in which that is useful?
That's actually the normal mode of operation (split responsibility between Lee Helm on the right and Helm on the left) on a CVN (which uses a very similar system, although with a larger console). It has the ability to control both functions from one side of the console, but that's not the normal mode of operation.
If the bridge ceases to exist, then steering could be controlled from aft steering (directly above the rudders) and throttles from engineering central control (engineering owns the engines). That's a canonical thing, it's the way it's always been -- you have redundancy in those places for many obvious reasons for a warship.
The article says: "As the years passed, Northrop Grumman continued to make improvements to the system. That meant that controls on one ship might not look exactly like controls on another. It was like hopping from the driver’s seat of a Ford truck into a Mercedes-Benz roadster. Sailors could adjust. But it took a while to get a feel for the different controls — and on a fully armed warship, no less."
>I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry.
You don't need to have experience building ship control systems to know this stuff is stupid. You probably drive a car, right? Can you imagine a car with UIs anything like this? If military contractors were doing car UIs, the roads would be a bloodbath.
Unfortunately, car UIs have also declined in usability thanks to the touchscreen fad, and very poor software behind them too. At least there's still some choice, but I suspect the majority of new buyers were taken in by a flashly salesman presentation and only later discovered all the problems.
You're correct about inflexible working arrangements though. WFH isn't very common, and work locations tend to not be in desirable geographic locations.
The pay is generally good, certainly much better than working for the government (though without the job security), though it's probably not going to beat a top salary at a FAANG company by a long shot. But how many other jobs do? There seems to be a trend here on HN to assume that the very top-paid positions at FAANG companies are utterly average for the tech industry, and they're just not.
Overall, defense contracting is fairly lucrative to people who can obtain and keep a security clearance and aren't completely incompetent, and want a job that doesn't have crazy overtime and lets them have a normal family life. But it definitely won't attract the very top-most talented people, because those people can go to work at FAANG companies for a lot more money.
The only reason this should be true is if the Navy is directly providing this service instead. If nobody is coordinating integration, then that's absolute insanity. Integration is the most important job in a distributed system like this, also probably the hardest.
Knowing the military they probably thought they could avoid needing an integrator by writing out tens of thousands of pages of specs, never stopping to consider that the specs themselves may be incorrect or unworkable or that they would be so complex that no vendor could ever deliver a product that exactly meets the specs.
Tim Harford covers this in his new podcast series, "Cautionary Tales", episod "DANGER: Rocks Ahead!":
As described in the episode, there were a number of contributing factors: tight schedule, miscommunications and assumptions between captain and helm, changes of watch, sleep, limited visibility, and the unexpected presence of other traffic.
But at a critical moment, rudder inputs failed to change the ship's course because authority had been transferred to the other set of helm controls.
I was one of the very early maintenance techs on the system in the 90s. No one with even a foggy understanding of HMI has ever been allowed to touch its design. The issues called out in the article barely scratch the surface.
But the system was designed so that control of steering or thrust could be taken over from any "inactive" side. If an interface appeared inactive, how would you easily take control from that side?
The idea that this thing behaved that way and made it through a bunch of people is mind boggling to me.
From what I read consistently about the Navy, and hear from veteran friends, any sailors they could have asked to stand watch would have already been overworked and under rested. Everything I read about the U.S. military, and especially the NAVY, seems to indicate a broken culture of bloat and incompetence from the top down. I think we've been too comfortable as the premier unchallenged world superpower and the services have started to rot from within.
 I think it was propublica as well, but I can't find the article there.
The article you are thinking of is here: https://features.propublica.org/navy-accidents/uss-fitzgeral...
The Air Force is a huge can of failure with the F35. At least they're standing by the goals of having a reasonable number of combat-worthy planes ready to fly, but the refusal to play ground-support roles and use light-combat airframes sets them up to be replaced by an army-run drone program within a few decades.
The Navy, though, hasn't had to do anything more useful than stop Somali pirates for two decades. It shows.
Lockheed gets something like 90% of its revenue from the federal government. They're effectively an arm of the Air Force.
Besides, it's not like the Air Force did no design at all, they came up with a list of wants and needs, had prototypes made by two contractors, refined their requirements after selecting one of the two competing prototypes, etc...
This blows my mind. Can anyone here guess what's going on under the hood? Is this a magic number that a developer came up during testing to avoid running of out memory/swapping?
It seems that the Achilles heel of the IBNS system was that too much information tended to bork it, but what’s unclear to me (as implied by the article) is if dropping incoming data was actually the sensible thing to do (as is often the case) and if incoming data was prioritized over existing data and that was dropped instead.
The other thing I would point out is that the technical failure is not that a system is complex, but that a complex system has a user interface that allows a sequence of actions to leave it in an invalid or unpredictable state. The transfer of throttle control being incomplete while other actions were taken, for example.
I think the thing lacking here was consistent and sound changes to the state, lack of fuzzing user input to observe this possibility, and the reliance on training to avoid the issues at all. These are problems that can be solved with design.
Does anyone know how much processing is needed here? Is it just updating an internal model of locations and warning about impending collisions? Presuming constant speed and heading that sounds easy, not presuming it sounds not feasible even for just a handful of ships.
Also, if only 150 ships are supported, wouldn't it makes sense for the computer to automatically untrack them? Maybe allow manual prioritization of what to drop first, but surely within a few days of a competent vendor learning adding too many items borked their system they would put out a patch that does the absolute minimum by automatically preventing adding too many items?
> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number.
> The navigation system could also become overloaded if too much information streamed in from a ship tracking database used worldwide to prevent maritime collisions. The Navy’s second solution was similar to the first: Drop the feed when it became too much.
> They were patches on top of patches that left the Navy’s destroyers without a full picture of the seas around them. But none of the problems was serious enough to attract high-level attention. A Navy system designed to track problems in major ship systems did not contain any reports that mentioned the IBNS until last year, according to a Navy official.
Or, most likely, normalisation of deviance. The system came at first with so many bugs and problems that when they got it in some stable state, they'd got so used to work around the bugs that they kept on doing that. Look we already put so much effort in this, what's a bit more? Yeah, we're quite happy with the system, at least it doesn't crash every hour like when it was delivered...
The number of 150 seems odd as well, but I have no idea what that 150 limit corresponds to in terms of actual data, methods to poll it, update it, connections to leave open, packets to receive, etc. It could be totally reasonable. It could also be insanely small.
My point is, it’s difficult to pass judgement on tech specs out of context.
Combine this with the sort of logic used in real time displays (or video games) and as the number of ships increases, the processing time required for the analysis goes up rapidly until you miss a critical time deadline and get your analysis ignored. At which point the system is classified as broken.
Can any Navy vets comment or speak to these systems? I really worry about the proliferation of these digital interfaces. I’m curious to know how folks feel about them.
I was forced to be the on-duty electrician from time to time, despite not being an ship's electrician. One of the responsibilities for the on-duty electrician was to cut the right circuits in case of a compartment fire. Despite my repeated protests that I didn't know what I was doing and needed additional training, I was forced to be the on-duty electrician. Mind you, there were no other duties I had to perform, just that one, and just in case of a fire, so it's not like my every-day workload was increased. Determined not to get people killed in the event of an emergency, I asked the actual electricians what I should be doing. Turns out, they didn't know either. Their plan was, in case of fire, kill power to the whole ship. I don't think I need to tell you, this is exactly the wrong move to make.
There's an ever-growing number of electronic systems onboard ships, and an ever-shrinking number of sailors to run them. Qualifications are a joke in pretty much every department, at least in the surface fleet. Ships are in a constant state of disrepair, it's not uncommon for 30-50% of the communications equipment to be broken at any given time, parts taking months to receive (mostly due to budget restraints).
Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with no sleep, followed by maybe 2 hours of sleep before having to work again the next day, and possibly have another watch from 20:00-12:00. Rinse repeat.
Airline pilots are a prime example. How many airliners crash these days? CRM is one of the factors in making them so safe. Truck Drivers now have similar requirements and a much improved safety record as well. This is literally life or death, industries that ignore it are being unnecessarily reckless and unprofessional.
It always seems to me that modern staffing modeling/logic just skips emergency situations especially for critical systems. When I read this idea that generalization and reduced staffing was to replace specialization for exactly this reason. It's hard to moonlight as an electrician during the worst moments, even if you can somehow skimp by at the best moments. The whole point of a navy vessel is to be as self sufficient in all conditions not to have to page a vendor halfway around the world just because the system blew up again, and that's just orthogonal to a business where you can put in a few extra hours to fix a problem or can wait until Monday for a vendor to call back. I'd be interested to know if some people have seen better operations research and modeling to cover these sorts of organizational systems compared to the apparent style of cut a labour force until you see blood and reap the profits from minimizing that expense against long term stability.
The real problem is that they're reducing the crews on ships of all kinds, not just the newer ones. I inherited 2-3 people's worth of equipment responsibilities when I checked on board, and that's on top of all the extra nonsense the Navy has you do other than work on your equipment.
Back in the 80's, I was throttleman aboard a similar sized USN ship. There was no direct link from the bridge to the throttle. Throttle commands were communicated via an indicator and the throttle operator manually turned a big wheel to meet the command. I remember thinking at the time there should be direct throttle control from the bridge. Hindsight is 20:20.
The complexity of the IBNS system seems to be the ultimate culprit, in that the various members of the stressed and tired bridge crew were each operating under a different mental model of the state of the system. Because of IBNS's flexibility, there were too many possible states for a crew to reason about under pressure.
Software makes it very easy for states to proliferate, but there hasn't been enough research (nor, more soberly, enough spilled blood) yet around designing flexible systems which are easily comprehensible in urgent situations.
Of course this complexity does cause problems in aviation, even with that culture's extreme emphasis on rigorous training, procedure, and human factors. AF 447, for example.
Also, how many foreign militaries have had problems like this with basic ship steering?
From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Commercial aircraft are not flying into the ground left and right. (The 737MAX is pretty bad, but that's not even related to glass cockpits; MCAS was a Boeing-engineered component, not a glass cockpit sourced from some vendor like Thales.)
> From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships.
Norway lost a destroyer last year after colliding with a tanker. Keep in mind the US Navy is larger than the next 13 largest navies combined. 2 of 3 of these events happening to the US Navy isn't statistically significant enough to say it's a uniquely US Navy problem.
Not to say they don't need to get their shit together.
I mean, if overall operations are so hazard-prone that they're not reliable in the absence of hostilities, how could they possibly be resilient against an advanced actor?
Is the situation as bad as it sounds from the story?
They value their narrative and slow animated exposition over my ability to scroll and read the article, meaning at multiple times I have to stop and wait for their painfully slow animations before I can continue.
> “Usually when we have a fault with that system,” Sanchez said, “their resolution is to reboot the system.”
Of course it is.
Software is going to kill us all.
I've not controlled anything that floats bigger than a small fishing boat, and I realise the timescales of reaction here are much larger, but that just sounds wrong to me --- changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.
That was standard practice back in the day and for many decades before that: One sailor served as the "helmsman" doing the steering and another as the "lee helmsman" transmitting speed orders to the engine room, each as ordered by the conning officer. (Former aircraft-carrier bridge officer here.)
Example: Conning officer: "Right standard rudder, steady on course 090." Helmsman: "Right standard rudder aye, sir [or ma'am]. Sir, my rudder is right standard." Conning officer: "Very well." Helmsman: "Sir, steady on course 090." Conning officer: "Very well." Conning officer: "All engines ahead standard, make turns for 15 knots." Lee helmsman: [Repeats back the order in the same way; rings up "ahead standard" on the engine-order telegraph ; and verbally tells the engine room over a sound-powered telephone .] Lee helmsman: "Sir, engine room acknowledges all engines ahead standard, make turns for 15 knots." Conning officer: "Very well." The quartermaster on the bridge records the orders in the ship's log, and the watch officer in the engine room does the same for the engine room log.
Don't know how they do things now.
CVN CDCWO / TOPWO here.
So much so that the military even tried to do this with aircraft cockpits, until it was obviously unworkable. The "hotshot young maverick pilot" archetype was allowed to form because they physically couldn't help it.
At least, this is the story told in The Right Stuff.
It's interesting you mention this. My uncle and cousins have been sailing for years on Lake Superior. They navigated some of the largest races on the lake for decades, including the Trans Superior in every imaginable weather conditions. They are, IMHO incredibly experienced sailors.
I just sent this very article to them and my uncle responded within the hour about how having two people doing these in tandem was a recipe for disaster because of the very issue you pointed out. He said you should have one capable person who should be able to do both since it can be a delicate balancing act. Too much speed and over correction with steering can have disastrous results.
The reason for this is that the people making the decisions have more important shit to worry about. Things are different when your primary concerns are getting shot at and shooting back. In war time it's not atypical to get your engine or rudder shot out, and if you have someone who has one job - to set the rudder - he'll notice and diagnose it a lot faster than someone who's tasked with avoiding incoming fire, bearing weapons on the enemy, etc. Distribution of responsibilities is incredibly important in war. Combat tends to break down careful, considered decision making.
Also keep in mind most of these people aren't seasoned. This dude was just out of high school. Information overload is a thing.
The two driving are under the command of someone senior to them (the conning officer) who gives direction to both and is responsible for ensuring the ship is being driven correctly.
See this 1941 British Navy training film:
Especially on a ship that might have to spring from "situation normal" to active combat (maneuver/attack/defend all at once) without a pre-knowable trigger condition.
“ Systems are comprised of hardware, software, and human personnel all of which operate within a surrounding environment. Too often, acquisition systems programs fail to consider the human capacity or requirements as part of the system. This leads to poor task allocation between hardware, software, and human users or supporters. To promote ideal task allocation, it is critical that the human element be considered early in system development.”
I really wonder, what does UX testing and iteration look like at Northrup-Grumman?
As someone who worked daily with two different NG-produced command and control systems, it's probably between "non-existent" and "flaming dumpster fire".
Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old version of Google Earth. Newer versions of C2PC are a polished turd, advancing the app's interface from 1995 to 2005.....despite hitting the operating forces around 2017-2018. We had bug submissions on some key mission-critical visualization features in the new version with patch turn-around times looking like 9-12 months at best.
It's probably very limited, if it even exists at all. What I'm betting happens is that somebody says "The Navy needs a new widget for X purpose!". and the only input the designer gets to have is where it goes.
I'd love to hear how UX designers in research challenged areas such as the military do their testing. It's not like they can just go on UserTesting.com and get feedback from somebody for $10 a test.
This would all be comical if it wasn't so tragic.
It is just the ultimate in irony: here is an article crying murder because a touch interface allowed a sailor to un-gang engine power, then slide a big fat green power slider for one engine to half of what the other big green slider for the other engine directly next to it showed.
Meanwhile here we are furiously scrolling to hit invisible "scrolled distance" breakpoints to advance a slideshow.
This one just sticks and does nothing, which is exceptionally awkward and feels broken.
-ACTIVE DUTY SAILOR