
F-35 Program Office Signs Off on Air Force 3i Software - Gravityloss
http://www.defensenews.com/story/defense/air-space/2016/05/09/f-35-program-office-signs-off-air-force-3i-software/84138390/
======
chris_overseas
> Pilots found that jets’ systems would shut down midflight about once every
> three or four hours and have to be rebooted.

> But the new, improved version of 3i is a significant improvement, according
> to the JPO. The new version showed approximately twice the level of
> stability as the previous load, Block 2B, and three times better stability
> than the original 3i software

Does that mean it "only" needs to reboot once every 6-8 hours? If that's
considered stable enough to sign off for production, does that mean they will
simply reboot before each flight and just hope for the best?

~~~
billforsternz
My intuition is that occasional software crashes like this probably follow a
Poisson distribution statistically speaking. Like occasional cars appearing on
a lonely street. One car appearing doesn't affect the expected arrival of the
next car. So rebooting before the start of a mission is not an effective
strategy. Just idle speculation based on a stats and probability class 36
years ago. Happy to be proven wrong.

~~~
woodman
Unless the crashes are caused by things cosmic rays, they're very likely
influenced by the internal state of the aircraft and therefor not independent
events. If that car showing up causes the world to reset - then you won't
likely see another car for a few million years.

------
jfoutz
As much as i want to hate on the F-35, _mutter 1.5 trillion mutter_ , good for
them. Shipping software is hard. By all accounts it's millions of lines of
code. They're slowly picking off the critical bugs and upgrading.

------
nateberkopec
> "The best analogy is you are starting up your computer and you want to use
> Word, Excel, PowerPoint and Outlook, and you are trying to get your work
> done for the day, and PowerPoint and Outlook came up but you are having an
> issue with Excel,” Brendan Rhatigan, director of engineering and test
> operations for the F-35 ITF, told Defense News last week at Edwards. “So you
> say, I don’t know what’s going on, so let me x out of that, let me restart
> it again.”

So what did they do with this software update? Just install "CTRL+ALT+DEL"?

~~~
digler999
they cite a design _defect_ in MS products - designed to forward memes and cat
pictures and add up a few columns of numbers - to justify design defects in
aerospace (aka your-life-depends-on-it) code..

------
natch
The analogy they used to MS Word crashing once in a while doesn't quite fit.
MS Word isn't loaded with jet fuel and flying over populated areas at high
speed. I wouldn't be disappointed at all if they would keep this thing away
from large cities for now.

~~~
engi_nerd
The portions of the software that need to be restarted have to do with
avionics, not with the vehicle management computers that actually control the
aircraft and keep it in the sky. The VMCs are triply redundant and are
extremely reliable. The stuff that's making the news and causing all the
problems has nothing to do with actually keeping the plane in the air.

~~~
natch
Citation needed.

And I'm not only concerned about the plane staying in the sky.

I also want the jet fuel and the weapons to stay in the plane, to name a
couple other items that are pluses.

Nor do I have any confidence that a plane that stays in the sky today will do
so tomorrow, given the record of this team.

All kinds of things can go wrong[1]. Focusing on one aspect and claiming
(without evidence) that this aspect is safe doesn't help a bit. And triply
redundant means very little (as with a famous bomb accident example, sometimes
all those redundancies are shown to still be cutting it very close)[2]. This
is especially the case when the entire system is a poster child for projects
going wrong[3].

1\. [http://catless.ncl.ac.uk/Risks/](http://catless.ncl.ac.uk/Risks/)

2\. [http://www.theguardian.com/world/2013/sep/20/usaf-atomic-
bom...](http://www.theguardian.com/world/2013/sep/20/usaf-atomic-bomb-north-
carolina-1961)

3\.
[https://www.google.com/#q=f-35+failed+project](https://www.google.com/#q=f-35+failed+project)

~~~
engi_nerd
The split between the F-35 vehicle systems and mission systems is described in
[0].

"This new design approach breaks avionics into two categories: mission systems
and vehicle systems. The former includes tools that help the aircraft do its
job, such as sensors, displays, and weapons. The latter are subsystems that
help the aircraft function correctly, such as power generation, cooling, and
flight control."

For more on the Vehicle Managment Computers, see [1], particularly:

""Each F-35 will have three boxes, making it a triple-redundant system," said
Tom Burbage, executive vice president and general manager of the Lockheed
Martin JSF program. "Each box 'votes' and compares its decision with that of
the others before executing a command -- a process that takes place in much
less than the blink of an eye. If one or even two boxes were to be damaged or
malfunction, the aircraft would continue to operate normally." The all-digital
VMCs, which save weight and space while improving precision, are at the heart
of the distributed F-35 Vehicle System."

When we say "Block 3i" we are talking about the mission systems hardware[2]
and software that are all about "sensors, displays, and weapons". All I
intended was to point out this difference, and to point out that the vehicle
systems are controlled by physically distinct components and simply saying
"The F-35 software" isn't completely accurate.

You'll note I made no claim about safety, nor do I intend to do so. Future
performance of the system does not necessarily depend on its past performance.
Allow me to point out,though, that through 50,000+ flight test hours, no F-35
has been lost through a flight control failure (again, no guarantee that there
isn't some latent failure case awaiting discovery!) And, as an engineer who is
struggling to tame some complexity on a rather bothersome flight test
instrumentation system, I'm well aware of risks (I love Risks Digest!). Heck,
check some of my post history, I'm often exhorting people to read John Gall's
Systemantics and heed its lessons.

Discuss technical failings of the program if you wish, but remarks about the
"record of this team" and a google search to "f35 failed project" are rather
offputting and are, in my humble opinion, needlessly negative.

[0]:
[http://www.militaryaerospace.com/articles/print/volume-14/is...](http://www.militaryaerospace.com/articles/print/volume-14/issue-5/features/special-
report/f-35-jet-fighters-to-take-integrated-avionics-to-a-whole-new-
level.html)

[1]:[http://www.prnewswire.com/news-releases/first-lockheed-
marti...](http://www.prnewswire.com/news-releases/first-lockheed-
martin-f-35-joint-strike-fighter-vehicle-management-computer-
delivered-55707452.html)

[2]:
[http://www2.l-3com.com/displays/pdfs/redesign/ICP(2011)_LR.p...](http://www2.l-3com.com/displays/pdfs/redesign/ICP\(2011\)_LR.pdf)

------
kunai
Why do government contractors refuse to use COTS hardware and software? Is it
just a bureaucracy thing, tradition, or is there any significant ACTUAL
benefit to not using something like RTLinux or QNX for systems instead of the
proprietary mess of legacy spaghetti code they usually use instead?

~~~
mavhc
"Much of the F-35's software is written in C and C++ due to programmer
availability, Ada83 code also is reused from the F-22. The Integrity DO-178B
real-time operating system (RTOS) from Green Hills Software runs on COTS
Freescale PowerPC processors" "24 million lines of code"

From
[https://en.wikipedia.org/wiki/Lockheed_Martin_F-35_Lightning...](https://en.wikipedia.org/wiki/Lockheed_Martin_F-35_Lightning_II)

Integrity-178B is the DO-178B–compliant version of Integrity. It is used in
several military jets such as the B-2, F-16, F-22 and F-35, as well as the
commercial airframes Airbus A380. Its kernel's design guarantees bounded
computation times by eliminating features such as dynamic memory allocation.

The auditing and security engineering capabilities have allowed it to obtain
the EAL6 rating by the NSA.

Integrity-178B has a unique feature: an EAL6 rating.

[https://en.wikipedia.org/wiki/Integrity_%28operating_system%...](https://en.wikipedia.org/wiki/Integrity_%28operating_system%29)

EAL6: Semiformally Verified Design and Tested

~~~
outworlder
> Much of the F-35's software is written in C and C++ due to programmer
> availability

This sounds ridiculous. Surely it would be very easy to get as many
programmers as they wanted for whatever tech they need with the 1 trillion USD
that they have.

It would be fine if C++ was the best tool for the job, but I suspect it isn't.
If it is not the best tool for the job, then whatever is gained in "programmer
availability" is lost in productivity, correctness, reliability or whatever
metric a fighter jet's software development is measured against.

Furthermore, this whole IT industry fetish with hiring for a particular
programming language is completely misguided. Domain knowledge is usually the
bottleneck. I'd expect that to be even more so in flight or sensor software.

~~~
albinofrenchy
C++ is likely the best bet for this kind of thing. Or rather, a strict subset
of C++.

You want real time, you want a minimal OS and you probably want some of the
nicer type capabilities over c.

~~~
ArkyBeagle
It depends muchly. _In general_ , I'd rather have a seasoned set of 'C'
programmers rather than people who are unconcerned with what happens at the
level of the metal.

But that's probably a personal bias, one that's been hard-earned. It's a way
of leveraging the decades of mistakes made by those folks - and aid for by
previous employers.

A "'C' with classes" approach is probably a good one - but I'd hate to find
some subtle template or Boost bug in mid flight...

This is all IMO, but an approach like Bruce Powell Douglass "Doing Hard Time'
is a pretty good way to satisfy the "systems engineer" customers for UML
charts and still have rigorous development.

UML is quite the anti-pattern and Rose is kind of awful but this should be a
compromise that can work. CASE tools have gone out of fashion.

~~~
rat87
> A "'C' with classes" approach is probably a good one - but I'd hate to find
> some subtle template or Boost bug in mid flight...

I'm not sure how much benefit that would bring, especially since dynamic
dispatch is probably not encouraged. As for templates I'm not much of a c++
guy but wouldn't most template errors exist at compile time?

~~~
albinofrenchy
Templates are mostly problems at compile time, but it's also possible to have
subtle issues with complicated template specializations where it doesn't call
the function you expect. So complicated usages are discouraged, BUT simple
ones that give code reuse with strong typing are great advantages over c in
most cases.

I think dynamic dispatch would be more permissible than you might think. You
end up doing it in c manually in complicated code bases, might as well let the
compiler dou the heavy lifting.

------
jokoon
The more I hear about the F35 softare, the more it sounds like it is something
you actually find in scifi movies. It really sounds like it is 20 years ahead,
if not more.

So of course it's going to fail at times. I really want to know the actual
capabilities of the things, but I know I'll have to wait some years before
russia or china are even able to copy those systems.

------
hackuser
Denmark's government likes it too:

[http://breakingdefense.com/2016/05/f-35-wins-denmark-
competi...](http://breakingdefense.com/2016/05/f-35-wins-denmark-competition-
trounces-super-hornet-eurofighter/)

~~~
ju-st
Probably they were "encouraged" by the US, too... (see
[https://en.wikipedia.org/wiki/Lockheed_Martin_F-35_Lightning...](https://en.wikipedia.org/wiki/Lockheed_Martin_F-35_Lightning_II_procurement#Norway)
)

------
zepearll
"The best analogy is you are starting up your computer and you want to use
Word, Excel, PowerPoint and Outlook, and you are trying to get your work done
for the day, and PowerPoint and Outlook came up but you are having an issue
with Excel,” What about "Word"? Did it start?

------
ArkyBeagle
I read these things, and I think "My disicipline - computer science - has
failed." Rent seeking - now with more death.

~~~
ilaksh
It's not computer sciences failure it's a management issue that stems from a
very primitive organizational design of the military.

For example, the engineers did not sign off on buggy systems. The executive
officers did because they had a contract with an archaic structure that is not
compatible with software development and required the milestone to be met or
there be penalities. But the reason the milestones weren't met in the first
place was poor management and contract structure created by those executives
in the first place.

Biggest issue is too much ambition and too much budget trying to be crammed
into one project. Other big issue is lack of closed feedback loops. It looks
like the devs do not have a production system to do integration testing on.
There are probably communication barriers between QA testers ie pilots and
devs, caused again by poor structure and culture of military. Also likely very
long lag between releases so few opportunities for iteration. Also likely
using outdated programming languages like C++ for application-level logic.

The nation-state's deadly enforcers eg military are directly descended from
historical organized crime, since nations generally started as simply the most
powerful families.

Look at the first premise of this whole thing: we are going to fly around and
missile/bomb the shit out of you if you don't go along with our global
domination. So from the beginning it is poorly planned.

~~~
ArkyBeagle
But medicine works in the military. We just don't do "first, do no harm." This
is just brought more sharply in focus by the nature of the F35.

I understand completely your point. The distinctions between militaries and
crime families is the chain of command and accountability - they're subject to
oversight by a legitimate government.

The hot job in DoD contractor is contracting itself - feed the beast. The
actual work is a minor annoyance to them. It's just careerism on stilts.

~~~
hackuser
> medicine works in the military

There have been many large scandals with military hospitals and veterans care
in the U.S.

~~~
tssva
There have been scandals involving veterans care; however, in the U.S. the
hospitals and care for veterans is provided by the Veterans Administration, a
cabinet level civilian run agency, and not the Department of Defense, so not
military hospitals.

~~~
hackuser
There were major problems at Walter Reed medical center, which is where
seriously wounded active-duty soldiers are treated IIRC. Just skim these
headlines:

[https://duckduckgo.com/?q=walter%20reed%20scandal%20site%3An...](https://duckduckgo.com/?q=walter%20reed%20scandal%20site%3Anytimes.com)

