In a panic people get tunnel vision. When you design HMI's for a living you have to consider. If my brother, sister, grandmother was using this system during an emergency could we guide them to make the right decisions.
The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.
The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station.
The issues with the navigation system are a whole other bag of worms- who knows what the mess was there.
Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this.
Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.
The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.
This reminds me of the Airbus control sticks, where the systemd averages the two inputs if e.g. one pilot pushes forward while the other pulls back. There is a warning but it can be easily ignored, as happened on https://en.wikipedia.org/wiki/Air_France_Flight_447 .
Also, I suspect there may have been at least one person who raised the idea of disabling/graying out the controls if they're inactive, but it was not listened to or even actively opposed because "it's not in the spec" and the one who brought it up didn't have the power to do anything about it. I've worked in environments like that (thankfully not for long) enough to know that these big and highly bureaucratic projects will tend to have lots of "lack of common sense" problems, either because no one bothers to actually think about what they're trying to do, or the division of labour was so fine that no one could. Lots of process, lots of specification meetings, lots of cargo-culting dogma, but no real thought put into anything. The fact that such developer positions are treated as disposable and easily replaceable probably adds to the problem too.
As someone who has been on both sides of the systemd debate, I heartily guffawed at this. Thanks.
As for the topic at hand, at a 30k ft view, having seen the inside of DoD stuff from a ground troop level, and later from DC itself, I mostly blame the good-old-boy network from being too established and greedy. I figure it's always been this way and technology complicating things has just pulled the curtain back, but essentially they do DC/Virginia back-room dealing to get bids and then under-perform in the delivery but don't really care because DoD's budget is massive and they keep throwing money at them. This matters much less while we engage countries that can't engage back like Iraq and Afghanistan, but as we shift more towards the return of the tri-polar world and DoD has to start planning for more peer level agitation, these problems need to be fixed ASAP or people are going to die.
There is a reason a handful of counties in Virginia had the highest growth of millionares in America post-9/11. I'm tired of contractors wasting money while the real military gets shite, and while maybe some of that is bias from my Marine Corps days (The Corps was notorious for "hand-me" down equipment and small budgets compared to Army, etc), I still say it's about time we financially body checked the good old boys and gave our boots on the ground better treatment.
Hell, how about just not having boots on the ground in the first place in unconstitutional bullshit wars, but I digress.
I have never understood why averaging the stick inputs makes any sense (who is controlling the aircraft in that case?)
In this case, I see that the ship's left and right power controls were transferred from one station to another one at a time. Regardless of whether, in the system's byzantine command structure, there was a way to transfer them together, the fact that they could be transferred individually suggests that someone thought there was a scenario where it made sense to have two different people, possibly in widely separated places, handling the two sides of this function. perhaps someone could enighten me as to the scenarios in which that is useful?
The general use case is Helm controlling the rudders on the left and Lee Helm controlling throttles on the right. They're two separate functions, and while they require coordination to run effectively they don't have to be the same person or the same station or even in the same room.
That's actually the normal mode of operation (split responsibility between Lee Helm on the right and Helm on the left) on a CVN (which uses a very similar system, although with a larger console). It has the ability to control both functions from one side of the console, but that's not the normal mode of operation.
If the bridge ceases to exist, then steering could be controlled from aft steering (directly above the rudders) and throttles from engineering central control (engineering owns the engines). That's a canonical thing, it's the way it's always been -- you have redundancy in those places for many obvious reasons for a warship.
Thanks for your informative reply, but if I am following what you are saying, the Lee Helm is normally controlling all the trottles, and responsibility for port and starboard throttles is not normally split between two people? - though I take your point that, in the case of damage, there might be no single console capable of controlling both the port and starboard power, so I can see why complete flexibility in the assignment of controls to consoles would be a requirement.
That correct - throttle control should not be split between consoles. I don’t believe that’s even possible - it was not in the system I worked on. I suspect the article is not exactly correct in this. The controls normally are ganged so they don’t have to be individually changed but for maneuvering individual screws can be commanded separately on the same console if they are not ganged together.
> I don’t believe that’s even possible - it was not in the system I worked on.
The article says: "As the years passed, Northrop Grumman continued to make improvements to the system. That meant that controls on one ship might not look exactly like controls on another. It was like hopping from the driver’s seat of a Ford truck into a Mercedes-Benz roadster. Sailors could adjust. But it took a while to get a feel for the different controls — and on a fully armed warship, no less."
The thing you're completely missing here is that these systems are built by military contractors, so it really shouldn't be too surprising that they have such awful UI. These companies do not make any commercial products (except Boeing). Their real strength in the marketplace is being really skilled at acquiring valuable government contracts, and being a big enough company (thanks to history and inertia) to actually win these contracts and deliver on them (even if it's late and the product is poor, though it meets all the specs). These companies would not survive if they had to sell their products to consumers. The negative reviews on Amazon would kill them quickly.
>I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry.
You don't need to have experience building ship control systems to know this stuff is stupid. You probably drive a car, right? Can you imagine a car with UIs anything like this? If military contractors were doing car UIs, the roads would be a bloodbath.
Can you imagine a car with UIs anything like this? If military contractors were doing car UIs, the roads would be a bloodbath.
Unfortunately, car UIs have also declined in usability thanks to the touchscreen fad, and very poor software behind them too. At least there's still some choice, but I suspect the majority of new buyers were taken in by a flashly salesman presentation and only later discovered all the problems.
It's also worth pointing out that there's a serious lack of talent in the defense contracting industry (not universal -- but certainly on average). The pay is typically not very impressive, the working conditions tend to be stressful (constant cost over-runs, impossible deadlines) and inflexible working arrangements (good luck with WFH). Talented people generally gravitate elsewhere with better pay and perks.
This isn't quite true. The working conditions tend to be less stressful usually, because by law you cannot be required to work overtime without being paid for it. Every hour you work as a defense contractor must be billable to a government contract, and you must get paid for it. You're also not allowed to work overtime unless the customer agrees to it, which they typically only do if there's a schedule crunch.
You're correct about inflexible working arrangements though. WFH isn't very common, and work locations tend to not be in desirable geographic locations.
The pay is generally good, certainly much better than working for the government (though without the job security), though it's probably not going to beat a top salary at a FAANG company by a long shot. But how many other jobs do? There seems to be a trend here on HN to assume that the very top-paid positions at FAANG companies are utterly average for the tech industry, and they're just not.
Overall, defense contracting is fairly lucrative to people who can obtain and keep a security clearance and aren't completely incompetent, and want a job that doesn't have crazy overtime and lets them have a normal family life. But it definitely won't attract the very top-most talented people, because those people can go to work at FAANG companies for a lot more money.
Integrated bridges are being installed all across USN ships, including Mt. Whitney with MSC right now. They're filled with a hodgepodge of HMIs that are all linked to a control system filled with gyros, ECDIS, RADAR, propulsion, etc. etc. System integration is complex and the design work is done by contracted companies. You don't have the first party vendor doing the integration work. It's a recipe for disaster.
> You don't have the first party vendor doing the integration work.
The only reason this should be true is if the Navy is directly providing this service instead. If nobody is coordinating integration, then that's absolute insanity. Integration is the most important job in a distributed system like this, also probably the hardest.
Knowing the military they probably thought they could avoid needing an integrator by writing out tens of thousands of pages of specs, never stopping to consider that the specs themselves may be incorrect or unworkable or that they would be so complex that no vendor could ever deliver a product that exactly meets the specs.
This is pretty typical of military contracts. They like to spread the work around, so different vendors provide different pieces of the system, and it ends up being a hodgepodge that doesn't integrate well and doesn't work well.
looking at these touch controls, I am wondering how one can possibly operate them with any precision in a heavy sea, and avoid clicking the wrong thing. We evolved hands to grasp, push, pull and twist, not to point and tap a glassy surface.
Split controls, in this case, of the rudder, played a major role in an earlier landmark shipping accident, the 1967 wreck of the Torrey Canyon, the first of the modern age of major supertanker oil spills.
Tim Harford covers this in his new podcast series, "Cautionary Tales", episod "DANGER: Rocks Ahead!":
As described in the episode, there were a number of contributing factors: tight schedule, miscommunications and assumptions between captain and helm, changes of watch, sleep, limited visibility, and the unexpected presence of other traffic.
But at a critical moment, rudder inputs failed to change the ship's course because authority had been transferred to the other set of helm controls.
You'd be amazed at who built this system. I suspect it's easy enough to find out, but I won't throw it out there in case.
I was one of the very early maintenance techs on the system in the 90s. No one with even a foggy understanding of HMI has ever been allowed to touch its design. The issues called out in the article barely scratch the surface.
> If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.
But the system was designed so that control of steering or thrust could be taken over from any "inactive" side. If an interface appeared inactive, how would you easily take control from that side?
All the controls would be inactive except for one "take control" button, which when pressed activates all the other controls and disables itself and the controls at the other stations.
>Immediate responsibility, the Navy ruled, rested with Sanchez, his officers and senior sailors. They had been lax, even complacent, in their training of the sailors steering the ship. Sanchez had made a critical error in not adding more sailors to stand watch as the McCain navigated the treacherous strait
From what I read consistently about the Navy, and hear from veteran friends, any sailors they could have asked to stand watch would have already been overworked and under rested. Everything I read about the U.S. military, and especially the NAVY, seems to indicate a broken culture of bloat and incompetence from the top down. I think we've been too comfortable as the premier unchallenged world superpower and the services have started to rot from within.
Norway recently lost one of their five frigates[1], and a significant factor was the officers on the bridge lacking experience and being overworked due to training new personnel. Nine of the fifteen[2] safety recommendations made by the accident investigation board were directed to the Norwegian Navy.
I can't find the source [0], but there was another expose'/interview on the USS John McCain where they interviewed darn near the entire crew. My details are fuzzy, but from what I remember the crew has been severely short staffed for years. To the point where 'normal' days were like 18 hours long and sleep for the captain was 2-4 hours a day for years on end. The internal email system for the ship was so broken that they resorted to just using free Gmail addresses, they had to tape over control knobs to keep them from being disturbed, training was completely abandoned, etc. It paints a hell of a bad picture of the Pacific fleet and it's command.
[0] I think it was propublica as well, but I can't find the article there.
Your google-fu failed you because it wasn't about the USS John McCain, it was Propublica's article on the other fatal 7th Fleet destroyer collision in 2017, the USS Fitzgerald. The collisions were two months apart.
I think the Army works significantly better. Obviously still wildly wasteful, but they've been under pressure to deliver results for a decade, and actually operate in a war theater, so they actually have processes to get things done, given sufficient money thrown into the hole.
The Air Force is a huge can of failure with the F35. At least they're standing by the goals of having a reasonable number of combat-worthy planes ready to fly, but the refusal to play ground-support roles and use light-combat airframes sets them up to be replaced by an army-run drone program within a few decades.
The Navy, though, hasn't had to do anything more useful than stop Somali pirates for two decades. It shows.
They didn't go the airplane store and pick one off the shelf either though...
Lockheed gets something like 90% of its revenue from the federal government. They're effectively an arm of the Air Force.
Besides, it's not like the Air Force did no design at all, they came up with a list of wants and needs, had prototypes made by two contractors, refined their requirements after selecting one of the two competing prototypes, etc...
I served in the Navy from 74 to 80, school for the first 18 months and at sea for the last 4.5 years. I worked on a nuclear powered cruiser in the engine room. It was not unusual to stand "port and starboard" watches which meant only 2 people were qualified to stand that particular watch and we alternated duty.
Yet gun related deaths in the US are still MUCH higher than other countries[1].
22,938 (76%) are by suicide which can't be prevented by gun laws.
I beg to differ. Gun laws that reduce the number of firearms in homes, and/or reduce the number of guns sold to those with mental health issues, would almost certainly reduce suicide rates. A gun is a dangerously quick, easy, and effective method that sits in the closets of many Americans.
>There are about 30,000 gun related deaths per year by firearms, this number is not disputed.
"There were 39,773 gun deaths in 2017, up by more than 1,000 from the year before".[1]
>Do the math: 0.00915% of the population dies from gun related actions each year.
'Deaths per capita' is the more common unit, isn't it?
>22,938 (76%) are by suicide which can't be prevented by gun laws.
As I understand it, suicide-by-gun are more 'successful' than suicide-by-other-means. Each failed attempt is a chance to save that person or for the person to save themselves from trying again in the future. How could gun control laws not potentially impact suicide rate?
>• 987 (3%) are by law enforcement, thus not relevant to Gun Control discussion.
For some narrow definition of gun control, maybe. There has been talk of the increasing militarization of the police over the last decade+, putting more police in schools in response to school shootings, use of force by police officers in situations that seemingly could have easily avoided it. etc. All seems pretty relevant to the discussion.
>Still too many?
Yes. Staggeringly so[2].
>That's over 30% of all gun crime in just 4 cities.
America has gang/drug/poverty problems that fuel violent crimes. No one is denying that. All part of the gun control discussion.
>We don't have a gun problem...
Can you please help me understand that in relation to [2]? Because it sure looks like we have a gun problem.
We have a gun problem in certain particular areas; if you remove those areas from the data the rest of the US looks more like Switzerland in terms of gun violence. And all of those areas where there is a gun problem already have gun control laws that are among the toughest in the country. That does not seem like evidence that gun control is the way to solve a gun problem.
> America has gang/drug/poverty problems that fuel violent crimes.
Which is further evidence that the way to solve the gun problem in the areas that have it is to fix the gang/drug/poverty problem, not gun control. Just ending the war on drugs, all by itself, would be a huge win since it would remove the major incentive for violence associated with drugs.
You're missing the point. The data that makes it look concerning comes from areas that have other problems that actually cause those data points. The fault is not the hardware, it's all the other issues those places have.
I occasionally see a therapist, drive a safe car outside rush hour, avoid drugs, get a flu shot, do not live in areas with high violence statistics, exercise and wear sun screen, but simply have no recourse against most gun violence. The magnitude of risk is commensurate with problems we take head on, but somehow it's wrong to consider gun violence preventable or a worthy issue.
What recourse do you have against a random stabbing? Or a hammer blow to the back of the head?
>but somehow it's wrong to consider gun violence preventable or a worthy issue
It's not about believing it's worthy, it's minimizing or completely ignoring the tangible and intangible benefits of minimally restricted gun ownership.
The cost/benefit analysis is treated only as a cost analysis.
I beleive I'm as prepared as I can be against random stabbings or hammers. Also, I often carry a knife.
I'm genuinely curious about the benefits of minimally restricted gun ownership. In a game theoretic sense, theres only two stable power dynamics. 1. Nobody has weapons 2. Everyone does. Otherwise, the power dynamic is so skewed as to be trivially exploitable.
Since 1 is not possible 2 must be the only option.
Firearms are trivially easy to build with any machining experience. Takes a little more chemistry to make gunpowder. That said we've been doing it in far more primitive situations for hundreds of years.
If people want to kill you. Simply put, you are dead. Our focus should be on building a world where we dont want to kill each other.
But thats not actually what I asked. What are the benefits of minimally restricted availability of weapons or how do I find a "canonical list" of such benefits?
(also, option 2 is not possible either. What you propose is trending towards 2 has some scaling of benefit, and I believe the opposite: that removing as much as possible the possibility of gun violence will have far reaching de-escalating effects on police, politics, health, etc etc.) I just dont have a clear picture of how proliferation of guns helps to prevent gun violence except perhaps in the limiting, unobtainable case where everyone fears immediate lethal reprisal for any violent act?
"benefits of minimally restricted availability of weapons"? Seriously? There are over 2000 laws on firearm ownership over the US. It like there is a new law every time there is a shooting. In Maryland, if I want to purchase pistol, I have to take a course (min 8 hrs), pass a test, prove marksmanship ability, get fingerprinted, and pass a background investigation. This, to qualify for an HQL - Handgun Qualifying License. This is usually $150 or more for the training, plus $45 for the license, plus $55 for the fingerprinting. Then I can select handguns approved by the state of MD, which are on a list. Then I can order or purchase one from a licensed dealer, or an individual. Then I fill out additional paperwork, and another background check. BTW, if I have been convicted of a misdemeanor which might have a possible sentence of 2 years I'd be disqualified. MD laws are written such that almost any misdemeanor COULD cost you 2 years. When the firearm arrives, there is a 7 business day waiting period before you can claim it. Of course you have to pay for the firearm, so figure $350-$500 for a reliable firearm which hopefully will not jam when used in self-defense. At this point, you've spent $600-$1000 dollars to purchase a handgun. And you can only use it at a gunrange (public or private) or in your home for self-defense. You cannot carry it in your pocket. You cannot carry it in your car except to and from a gun range. and you can't carry it loaded then. This is what legal firearms owers go thru.
Gang-bangers in downtown Baltimore or St Louis... not so much.
I'd like to keep this conversation going. Maybe email me or find me on keybase from my bio.
I understand that regulation is enormously inconvenient for owners. But making it easy to acquire firearms isnt a benefit of more people owning firearms, its a prerequisite. I want to understand why people should own firearms at the level of scenario 2 above.
Those suits will stop most commonly-used pistol rounds. If you desire protection against rifle rounds, you can purchase SAPI plates from China for fairly cheap, but them, and their plate carriers, are fairly bulky....worse than wearing an armored motorcycle jacket, for comparison.
A $3500 suit (sold only to feds) is not a recourse against random acts of violence. Body plating is not either. This is lunacy to recommend as a common man / everyday solution that takes the place of regulation and enforcement.
You are more likely to trip, fall into a ditch or puddle and drown than be shot. So you can lament your perceived need for body armor, or forgo walking in exceptionally high crime areas.
Given that the grandparent comment was snuffed, apparently the 2nd amendment now even trumps the 1st.
And, when the grand-parent says that "some people are in denial" about problems with medical, housing and other costs and problems. Apparently they're not just in conspiracy theory denial but even seek to prevent the merest mentioning of these issues.
This is subjective. Most Americans care more about roads than public transportation.
>housing costs
In all but the most expensive metros, US housing is way cheaper per sq ft than most Western European nations. This is partly due to urban sprawl due to the love affair with cars, but it is cheaper.
>gap between rich and poor
Richer middle class than most other countries. Just because the US is a good place for business and a bad place to be poor says very little about the average experience.
>gun-related deaths
Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons.
> Most Americans care more about roads than public transportation.
And the roads are garbage. I brought some friends over from Australia and driving on the US interstates they said "wow.. it's like everything is half finished".
> You'd want to compare murder/suicide rates with other countries for any meaningful comparisons.
And it's way, way higher than any other developed country.
For what it's worth, the The US is at or very near the worst among OECD countries in: infant mortality, child poverty, child health and safety, life expectancy at birth, healthy life expectancy, rate of obesity, disability-adjusted life years, doctors per 1000 people, deaths from treatable conditions, rate of mental health disorders, rate of drug abuse, rate of prescription drug use, incarceration rate, rate of assaults, rate of homicides, income inequality, wealth inequality, and economic mobility.
>And the roads are garbage. I brought some friends over from Australia and driving on the US interstates they said "wow.. it's like everything is half finished"
Uh, the U.S. is huge, with 50 semi independently managed states and thousands of semi independently managed cities. Having lived and traveled all over, most places have roads in good shape - which is an achievement because many of these roads are quite remote.
>For what it's worth, the The US is at or very near the worst among OECD countries in: infant mortality, child poverty, child health and safety, life expectancy at birth, healthy life expectancy, rate of obesity, disability-adjusted life years, doctors per 1000 people, deaths from treatable conditions, rate of mental health disorders, rate of drug abuse, rate of prescription drug use, incarceration rate, rate of assaults, rate of homicides, income inequality, wealth inequality, and economic mobility.
Though unpopular, it should really be said that almost every one of these problems is isolated to one or two demographics in this country. That therefore is not indicative of a failure at large. If you remove the same couple groups of people from the various statistics, stats for the U.S. in almost every category are at or near the top for the global average.
Who cares. Why do you want to compare the US to the Central African Republic. At least compare pro teams to other teams on the pro ladder if you want any kind of meaningful comparisons.
You misunderstood my comment. Perhaps global average was not the right term. I mean to say top of the list of countrywide averages. As in best on Earth.
America gets a ton of shit online, but the shitty areas that drive averages down are a tiny percentage of total population/land. They're outliers. For example, Houston is rated as one of the most dangerous cities in the country - but only a handful of neighborhoods/areas are actually bad.
Well unfortunately we don’t live in a spreadsheet where you can just drop the rows that pull your average down, but rather a real country. Somehow the other countries have done better despite facing their own challenges.
They consist of entirely different majority and minority cultures. And the point is that random city, USA is statistically likely to compare favorably to random city, Nordic country you're probably referring to.
> And the point is that random city, USA is statistically likely to compare favorably to random city, Nordic country you're probably referring to
I take it you've never been to "random Nordic city" then.
In stark contrast to "random town USA", not a single person is without healthcare without paying directly for it, everyone can have unemployment money if they need it, everyone has had (or can have) free university education, etc. etc.
The median commute time in the US is 27 minutes each way [0]. Lower in less dense and more car-oriented geographies. Our system of decentralized, sprawling, unremarkable towns and "cities" has many problems, but those things are fucking great for commute times. Residents of giant world cities with excellent public transit infrastructure spend much more of each day getting around.
This feels like a tautology - commute times are close to 30 minutes because on average people have lived about 30 minutes from where they work for thousands of years.
You say that yet both Portugal and Italy have an average one way commute of ~15 minutes. So, clearly 30 minutes is far from any real natural custom. At best you can say commute times are going to some limited fraction of the day as people need to sleep, commute, and work in the same day.
Edit: The metric they use for that may be slightly different as they put US commutes at 48.8 minutes per day. At which point 2 EU countries have worse commutes. Hungary and Romania. If the daily average for the US was 27x2 = 54 minutes that’s worse than very EU country.
> Edit: The metric they use for that may be slightly different as they put US commutes at 48.8 minutes per day. At which point 2 EU countries have worse commutes. Hungary and Romania. If the daily average for the US was 27x2 = 54 minutes that’s worse than very EU country.
It's the same method, the CNBC article is just using the more recent survey, so their number is 10 years newer than the one in that report. This, comparing to the non-US numbers in it is not fair.
> That’s terrible by EU standards or Canadian (24 minutes each way) standards
I don't agree with your definition of "terrible" in the relative sense. Daily, the US number in that report is only 2 minutes higher than the Netherlands, 6 minutes higher than Germany, and 7 minutes higher than Belgium. All three are over 40 minutes per day, so IMO if the US time is "terrible" so are theirs. Ditto the Canadian number, which is only 6 minutes per day less.
It takes a lot to move an average by even one percent across millions of people. Look at a chart of say life expectancy across these countries and it seems almost flat by comparison. The wide spread here represents significant differences that have huge impacts on people’s lives.
Also, 1 minute * 261 work days a year * 40+ working years = 174 hours extra commute time per person. Thus, are talking well past 3,000 hours longer than the best country on this list. Or every walking hour for more than six months for the average working American over the average Portugal worker. So, I am not saying every EU country is doing ok, just by comparison it’s clear the US huge issues in this area.
> Yep, that comes with gun ownership. Many people are knowingly making that trade-off. You'd want to compare murder/suicide rates with other countries for any meaningful comparisons.
And our murder rate is pretty bad, likely thanks to our extremely high rate of gun violence.
While over 2/3 of US Murders involve guns. You can exclude gun violence only from the US numbers and the US still has a higher murder rate than the UK.
So, it’s likely a major factor but hardly the root cause.
Keep in mind that private citizens in the United States possess roughly 35-40% of all small arms ON THE ENTIRE PLANET, with more than double the per capita ownership rate of the #2 country (active warzone Yemen).
> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number.
This blows my mind. Can anyone here guess what's going on under the hood? Is this a magic number that a developer came up during testing to avoid running of out memory/swapping?
I doubt it was a magic number, but just an emergent property of how quickly the system could track a certain number of targets. When they saw problems in testing, a workaround was devised, and the 150 number probably came from just counting the number of targets when things started to go haywire. Most likely a bug report was put in the queue, and it never got prioritized because new complaints stopped coming in.
Just a guess—probably designed using high-reliability / realtime principles. This might involve e.g. allocating memory only at startup, which would explain things. The limit has to be some number when programming this way.
In a real-time safety-critical OS, you absolutely do allocate memory only at startup, but this kind of OS usually isn't used for any kind of GUI interface, it's used for much simpler systems. Another trait of these systems is that the CPU caches are all disabled, because those prevent determinism. But again, this isn't something you'd do on a GUI system; it's something you'd do on an ABS brake controller or an engine ECU, where the amount of memory ever needed is easy to determine in the design.
This is also done in the signal processors that power these systems. I don't know this system in particular, but the USN has been moving towards commodity computing hardware for these systems. Essentially, a rack of commercial servers (usually IBM) running stripped-down RHEL with a real-time kernel. The software for these systems is firm real-time, and generally uses the same principals as other real-time software (including, but not limited to, up-front allocation).
Something unclear from the article, but jumped out at me.
It seems that the Achilles heel of the IBNS system was that too much information tended to bork it, but what’s unclear to me (as implied by the article) is if dropping incoming data was actually the sensible thing to do (as is often the case) and if incoming data was prioritized over existing data and that was dropped instead.
The other thing I would point out is that the technical failure is not that a system is complex, but that a complex system has a user interface that allows a sequence of actions to leave it in an invalid or unpredictable state. The transfer of throttle control being incomplete while other actions were taken, for example.
I think the thing lacking here was consistent and sound changes to the state, lack of fuzzing user input to observe this possibility, and the reliance on training to avoid the issues at all. These are problems that can be solved with design.
I haven't worked on any project anywhere approaching significant complexity, but the magic number of 150 supported tracked ships sounds insanely low to me. Is this at all reasonable, or does it indicate the design and implementation were shit?
Does anyone know how much processing is needed here? Is it just updating an internal model of locations and warning about impending collisions? Presuming constant speed and heading that sounds easy, not presuming it sounds not feasible even for just a handful of ships.
Also, if only 150 ships are supported, wouldn't it makes sense for the computer to automatically untrack them? Maybe allow manual prioritization of what to drop first, but surely within a few days of a competent vendor learning adding too many items borked their system they would put out a patch that does the absolute minimum by automatically preventing adding too many items?
> In 2014, Navy officials discovered a flaw in the IBNS. One component could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number.
> The navigation system could also become overloaded if too much information streamed in from a ship tracking database used worldwide to prevent maritime collisions. The Navy’s second solution was similar to the first: Drop the feed when it became too much.
> They were patches on top of patches that left the Navy’s destroyers without a full picture of the seas around them. But none of the problems was serious enough to attract high-level attention. A Navy system designed to track problems in major ship systems did not contain any reports that mentioned the IBNS until last year, according to a Navy official.
150 tracks without knowing the range of their navigation radar systems doesn't say much. It's hard to put more than 150 objects in a 5-10NM radius. Since it's a navy ship the nav radar is probably more powerful, but still, it's not like a 200NM range air-defence radar. I'm guessing they spec-d the system with incoherent requirements, and either the contractor that won went for the cheapest working hardware without asking too many questions, or maybe they took an old system's spec and asked for an incomplete upgrade... or nobody wanted to go to their boss saying the system didn't work because of some badly written spec or badly managed contractor and we need more money, please ? That would track with leaving the system badly working like this without even having some face-saving 'we're trying to work it out with the contractor right now'.
Or, most likely, normalisation of deviance. The system came at first with so many bugs and problems that when they got it in some stable state, they'd got so used to work around the bugs that they kept on doing that. Look we already put so much effort in this, what's a bit more? Yeah, we're quite happy with the system, at least it doesn't crash every hour like when it was delivered...
This is what I mean, I have no idea if it’s reasonable or not. Or why the issue exists (bandwidth, memory, etc etc). Or if the “dropped” data is actually dropped at all past 150 ships, and not dropped when it is lowered in priority compared to incoming data.
The number of 150 seems odd as well, but I have no idea what that 150 limit corresponds to in terms of actual data, methods to poll it, update it, connections to leave open, packets to receive, etc. It could be totally reasonable. It could also be insanely small.
My point is, it’s difficult to pass judgement on tech specs out of context.
No inside knowledge here, but in tracking moving objects it would be easy to build in an O(n^2) algorithm when predicting interactions or intersections between ships.
Combine this with the sort of logic used in real time displays (or video games) and as the number of ships increases, the processing time required for the analysis goes up rapidly until you miss a critical time deadline and get your analysis ignored. At which point the system is classified as broken.
Absolutely phenomenal reporting. Take the time and read the entire thing in detail. It’s worth it.
Can any Navy vets comment or speak to these systems? I really worry about the proliferation of these digital interfaces. I’m curious to know how folks feel about them.
I can't speak to that specific system, but I can give you plenty of other anecdotes about the systemic unreadiness of the US Navy.
I was forced to be the on-duty electrician from time to time, despite not being an ship's electrician. One of the responsibilities for the on-duty electrician was to cut the right circuits in case of a compartment fire. Despite my repeated protests that I didn't know what I was doing and needed additional training, I was forced to be the on-duty electrician. Mind you, there were no other duties I had to perform, just that one, and just in case of a fire, so it's not like my every-day workload was increased. Determined not to get people killed in the event of an emergency, I asked the actual electricians what I should be doing. Turns out, they didn't know either. Their plan was, in case of fire, kill power to the whole ship. I don't think I need to tell you, this is exactly the wrong move to make.
There's an ever-growing number of electronic systems onboard ships, and an ever-shrinking number of sailors to run them. Qualifications are a joke in pretty much every department, at least in the surface fleet. Ships are in a constant state of disrepair, it's not uncommon for 30-50% of the communications equipment to be broken at any given time, parts taking months to receive (mostly due to budget restraints).
Sailors are routinely sleep deprived. It's not uncommon for a sailor to be up by 6AM, work all day, and then be steering the ship from 00:00 to 04:00, with no sleep, followed by maybe 2 hours of sleep before having to work again the next day, and possibly have another watch from 20:00-12:00. Rinse repeat.
That last paragraph screams of a massive failure in Crew Resource Management. There are acres of case studies about this subject throughout industry, and when people take them to heart lots of accidents are avoided.
Airline pilots are a prime example. How many airliners crash these days? CRM is one of the factors in making them so safe. Truck Drivers now have similar requirements and a much improved safety record as well. This is literally life or death, industries that ignore it are being unnecessarily reckless and unprofessional.
It would be cool if anyone has a source for the improved trucker safety record. It is just common sense, though. I don't know how this no-sleep system ever got traction.
The real issue is that the Navy never wants to blame senior leadership and prefers to blame the easiest targets; they also never want to blame vendors who they might want to buy something from again. The vendors of course never want to take responsibility for problems or they won't ever get another project. Congress never wants to deal with things because it might anger some powerful/wealthy constituent like a defense contractor who holds a lot of jobs. In the end there is no one that will take responsibility to say "this is crap, fix it" because everyone is more concerned with ruining their promotion/contract/influence. So you wind up with ordinary sailors and ship's officers bearing the brunt of this as they are expendable.
It always seems to me that modern staffing modeling/logic just skips emergency situations especially for critical systems. When I read this idea that generalization and reduced staffing was to replace specialization for exactly this reason. It's hard to moonlight as an electrician during the worst moments, even if you can somehow skimp by at the best moments. The whole point of a navy vessel is to be as self sufficient in all conditions not to have to page a vendor halfway around the world just because the system blew up again, and that's just orthogonal to a business where you can put in a few extra hours to fix a problem or can wait until Monday for a vendor to call back. I'd be interested to know if some people have seen better operations research and modeling to cover these sorts of organizational systems compared to the apparent style of cut a labour force until you see blood and reap the profits from minimizing that expense against long term stability.
The ships referenced in the article are a newer class of ship. It was designed for reduced crew capacity, but as you stated it's simply impractical to try to perform specialized skills outside of your training area.
The real problem is that they're reducing the crews on ships of all kinds, not just the newer ones. I inherited 2-3 people's worth of equipment responsibilities when I checked on board, and that's on top of all the extra nonsense the Navy has you do other than work on your equipment.
It strikes me that at least 16 seconds of screwing around could have been saved by changing the caption of the "Emergency Override - To Manual" button to say "Emergency Override - Take Control." The button's caption would then contain enough information for anyone pressing the button to know that they were responsible for all controls at the station after pressing.
Very sobering article. Makes me reconsider the proliferation of glass cockpits and fly by wire in aviation. Granted, there is no putting the genie back in the bottle, but failsafe and redundancy need to be a central design tenet from the start.
Back in the 80's, I was throttleman aboard a similar sized USN ship. There was no direct link from the bridge to the throttle. Throttle commands were communicated via an indicator and the throttle operator manually turned a big wheel to meet the command. I remember thinking at the time there should be direct throttle control from the bridge. Hindsight is 20:20.
Looks like quite a lot of redundancy here - you can split up controls over various stations really in kind of any way you want. That being the problem. The article led me to think the crash was more a confluence of factors: staff shortages, high operational demands, culture of tolerating cursory training, and a very complex new system that was both substantially harder to understand than traditional fixed stations, and constantly changing.
The complexity of the IBNS system seems to be the ultimate culprit, in that the various members of the stressed and tired bridge crew were each operating under a different mental model of the state of the system. Because of IBNS's flexibility, there were too many possible states for a crew to reason about under pressure.
Software makes it very easy for states to proliferate, but there hasn't been enough research (nor, more soberly, enough spilled blood) yet around designing flexible systems which are easily comprehensible in urgent situations.
Of course this complexity does cause problems in aviation, even with that culture's extreme emphasis on rigorous training, procedure, and human factors. AF 447, for example.
Glass cockpits and FBW are great, in theory. In practice, I don't know because I haven't had to use it. However, don't make the mistake of assuming that just because the US Navy can't get a well-engineered system put together means that Boeing and Airbus are guilty of the same. (Their glass-cockpit systems come from other vendors, who also supply to smaller aircraft makers BTW.)
Also, how many foreign militaries have had problems like this with basic ship steering?
From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships. Commercial aircraft are not flying into the ground left and right. (The 737MAX is pretty bad, but that's not even related to glass cockpits; MCAS was a Boeing-engineered component, not a glass cockpit sourced from some vendor like Thales.)
> Also, how many foreign militaries have had problems like this with basic ship steering?
> From what I can tell, the US Navy is uniquely broken here. UK, Australia, Japan, France etc. do not have have their ships plowing into slow cargo ships.
Norway lost a destroyer last year after colliding with a tanker. Keep in mind the US Navy is larger than the next 13 largest navies combined. 2 of 3 of these events happening to the US Navy isn't statistically significant enough to say it's a uniquely US Navy problem.
Not to say they don't need to get their shit together.
This story's worrisome in that it makes the Navy sound very vulnerable to electronic warfare.
I mean, if overall operations are so hazard-prone that they're not reliable in the absence of hostilities, how could they possibly be resilient against an advanced actor?
Is the situation as bad as it sounds from the story?
I find it a little ironic that for an article about unintuitive and broken electronic navigation system, the UX on mobile of this article breaks navigation in weird and unintuitive ways.
They value their narrative and slow animated exposition over my ability to scroll and read the article, meaning at multiple times I have to stop and wait for their painfully slow animations before I can continue.
I thought the visuals were well done, but I agree that they should have just scrolled normally on mobile instead of capturing the scroll to advance the animation.
> could not keep track of more than 150 ships at a time without malfunctioning, according to Navy investigators. The Navy’s solution? Sailors were told to delete tracked ships before the total hit the magic number
> “Usually when we have a fault with that system,” Sanchez said, “their resolution is to reboot the system.”
He ordered Bordeaux to take over steering the warship while another sailor controlled its speed. The idea was to avoid distractions by having each man focus on a single task in the heavy maritime traffic.
I've not controlled anything that floats bigger than a small fishing boat, and I realise the timescales of reaction here are much larger, but that just sounds wrong to me --- changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.
> He ordered Bordeaux to take over steering the warship while another sailor controlled its speed.
That was standard practice back in the day and for many decades before that: One sailor served as the "helmsman" doing the steering and another as the "lee helmsman" transmitting speed orders to the engine room, each as ordered by the conning officer. (Former aircraft-carrier bridge officer here.)
Example: Conning officer: "Right standard rudder, steady on course 090." Helmsman: "Right standard rudder aye, sir [or ma'am]. Sir, my rudder is right standard." Conning officer: "Very well." Helmsman: "Sir, steady on course 090." Conning officer: "Very well." Conning officer: "All engines ahead standard, make turns for 15 knots." Lee helmsman: [Repeats back the order in the same way; rings up "ahead standard" on the engine-order telegraph [0]; and verbally tells the engine room over a sound-powered telephone [1].] Lee helmsman: "Sir, engine room acknowledges all engines ahead standard, make turns for 15 knots." Conning officer: "Very well." The quartermaster on the bridge records the orders in the ship's log, and the watch officer in the engine room does the same for the engine room log.
Back in the day I didn't get verbal engine order instructions after the telegraph changed. I matched the telegraph pointer and that signaled to the bridge when they saw it move to match their order.
Button pushing is beneath a gentleman-scholar commanding officer. On the other hand, a single worker-drone with all the controls in his hands might get his own ideas. Multiple stations with partial control, coordinated by the captain's voice, fit the organizational hierarchy best.
So much so that the military even tried to do this with aircraft cockpits, until it was obviously unworkable. The "hotshot young maverick pilot" archetype was allowed to form because they physically couldn't help it.
At least, this is the story told in The Right Stuff.
>> changes in speed can affect direction, and vice-versa. Splitting that to two separate brains that are communicating only via a low-bandwidth voice channel seems like a recipe for disaster.
It's interesting you mention this. My uncle and cousins have been sailing for years on Lake Superior. They navigated some of the largest races on the lake for decades, including the Trans Superior in every imaginable weather conditions. They are, IMHO incredibly experienced sailors.
I just sent this very article to them and my uncle responded within the hour about how having two people doing these in tandem was a recipe for disaster because of the very issue you pointed out. He said you should have one capable person who should be able to do both since it can be a delicate balancing act. Too much speed and over correction with steering can have disastrous results.
The people with the controls in their hand aren't the people deciding what to put the controls at. They're told where to set them, and they do it or report that the mechanisms have failed.
The reason for this is that the people making the decisions have more important shit to worry about. Things are different when your primary concerns are getting shot at and shooting back. In war time it's not atypical to get your engine or rudder shot out, and if you have someone who has one job - to set the rudder - he'll notice and diagnose it a lot faster than someone who's tasked with avoiding incoming fire, bearing weapons on the enemy, etc. Distribution of responsibilities is incredibly important in war. Combat tends to break down careful, considered decision making.
Also keep in mind most of these people aren't seasoned. This dude was just out of high school. Information overload is a thing.
The coordination between steering and speed is the job of the conning officer, not the helmsman or lee helmsman. Things happen quite a bit slower on a large warship than they do on a sailboat, so that method of control works fine.
Your relatives don't know what they're talking about. Offshore racing powerboats typically have separate crew members for helm and throttles. Sailboats are completely different.
That's the standard for driving Navy ships and has been for a very long time.
The two driving are under the command of someone senior to them (the conning officer) who gives direction to both and is responsible for ensuring the ship is being driven correctly.
I've not controlled anything that floats bigger than a pedal boat, but to the extent sci-fi shows and navy fictions show can teach me anything, I've always had the impression that the direction and speed of ships are controlled from two different stations on the bridge.
I'm curious why these things are manually controlled at all. Seems like you should give it a heading and it goes that direction, self-correcting along the way.
Because sometimes he "wrong" action is the right thing to do. No automated system can anticipate everything. Think the NASCAR "drive at the wreck in front of you" - by the time you get there it will have moved.
Especially on a ship that might have to spring from "situation normal" to active combat (maneuver/attack/defend all at once) without a pre-knowable trigger condition.
The only thing here that is worse than this shitstorm of a system design is the vindictive attitude of the Navy towards the sailors who have to cope with it. Flogging and hanging from the yardarm may be gone, but the spirit lives on.
Update: I have this suspicion that the Captain was charged with homicide in order to get him to take a plea deal, so that the problems with this system, and the Navy's failure to properly train crews in its use, would not become an issue in a court-martial.
These days we are so hyped on the cool shiny objects, but failing at the most basic concept, HSI = Human System Integration:
“ Systems are comprised of hardware, software, and human personnel all of which operate within a surrounding environment. Too often, acquisition systems programs fail to consider the human capacity or requirements as part of the system. This leads to poor task allocation between hardware, software, and human users or supporters. To promote ideal task allocation, it is critical that the human element be considered early in system development.”
Any entry level UX designer would tell you that you can’t pack more and more control into an opaque and confusing system and still lay all the responsibility on the user when something breaks down.
I really wonder, what does UX testing and iteration look like at Northrup-Grumman?
>>I really wonder, what does UX testing and iteration look like at Northrup-Grumman?
As someone who worked daily with two different NG-produced command and control systems, it's probably between "non-existent" and "flaming dumpster fire".
Agile Client is the better of the two apps UI/UX-wise, close to a clunky, old version of Google Earth. Newer versions of C2PC are a polished turd, advancing the app's interface from 1995 to 2005.....despite hitting the operating forces around 2017-2018. We had bug submissions on some key mission-critical visualization features in the new version with patch turn-around times looking like 9-12 months at best.
>I really wonder, what does UX testing and iteration look like at Northrup-Grumman?
It's probably very limited, if it even exists at all. What I'm betting happens is that somebody says "The Navy needs a new widget for X purpose!". and the only input the designer gets to have is where it goes.
I'd love to hear how UX designers in research challenged areas such as the military do their testing. It's not like they can just go on UserTesting.com and get feedback from somebody for $10 a test.
This is the latest innovation in "interactive story telling".
It is just the ultimate in irony: here is an article crying murder because a touch interface allowed a sailor to un-gang engine power, then slide a big fat green power slider for one engine to half of what the other big green slider for the other engine directly next to it showed.
Meanwhile here we are furiously scrolling to hit invisible "scrolled distance" breakpoints to advance a slideshow.
The websites that pull this off "well" do so by giving you feedback that your scrolling is still doing _something_. For example, Apple pulls this off reasonably in their marketing pages by moving things in the background: https://www.apple.com/iphone-11-pro/
This one just sticks and does nothing, which is exceptionally awkward and feels broken.
Meh, I use spacebar to page-down, but am ready to fall-back at a moments notice because so many sites draw their header over the body, so a spacebar will end up by scrolling by more than a page...
It's even worse on mobile, locking the page up entirely with Javascript. I assumed it was untested and broken there, and shifted to desktop, but it's broken here too.
The fact that the prop controls could be split but the UI would not gray out the sliders on the screens that did not have control is a poor user interface design choice. If the controls only appeared "active" on the screen they were transferred too this could have made things much more simple.
The fact that the mushroom button for emergency steering yanked control to wherever it was pressed- this should have displayed a red on black box. Emergency Override called from X Station.
The issues with the navigation system are a whole other bag of worms- who knows what the mess was there.
Disclosure, I have done HMI design and install in production systems for manufacturing, chemical and warehousing operations. I have never built anything for a ship or vehicle. So I may be speaking total stupid to someone in that industry. I would love to hear someone from that market sector drop in and talk about how or why the HMI was setup like this.
Sounds like this system was sub-contracted to the lowest bidder and pushed out without enough testing.