
The F-35’s Software Is So Buggy It Might Ground the Whole Fleet - Wonnk13
http://motherboard.vice.com/read/the-f-35s-software-is-so-buggy-it-might-ground-the-whole-fleet
======
chishaku
Highlights from the GAO report:

* Pilots, maintainers, and administrators at three of the five sites we visited are concerned about ALIS’s ability to deploy and function in forward locations. For example, users are concerned about the large server size and connectivity requirements, and whether the system’s infrastructure can maintain power and withstand a high-temperature environment.

* ALIS users at three of the five sites we visited are concerned that a failure in the system’s current infrastructure could degrade the system and ground the fleet. Currently, ALIS information, including data from all U.S. F-35 sites, flows from the Standard Operating Units (SOU) to a single national Central Point of Entry, and then to the lone Autonomic Logistics Operating Unit (ALOU).26 This data flow process has no back up system for continuity of operations if either of these servers were to fail.

* Maintainers and pilots at three of the five sites we visited were concerned that ALIS does not have much interoperability with legacy aircraft systems... The ability to share information between ALIS and these legacy systems is vital due to the way the services operate.

* Maintainers at four of the five sites we visited told us that the current Action Request (AR) process does not allow for the effective reporting and resolution of F-35 aircraft and ALIS issues.

* ALIS users at all five sites we visited are concerned with data accuracy issues within the system, including missing or inaccurate data and inaccessibility of raw data within ALIS.

[http://www.gao.gov/assets/680/676576.pdf](http://www.gao.gov/assets/680/676576.pdf)

Description of ALIS risks starts on page 15.

~~~
Analemma_
One other highlight from the report: the total lifetime cost of the program is
now estimated at 1.3 trillion dollars, or $1300000000000 if you prefer
longhand (up from the 1 trillion dollar estimate of a few years ago). Other
things we could've done with that money besides spending it on this useless
garbage heap of a plane are left to the reader's imagination.

~~~
rodgerd
> the total lifetime cost of the program is now estimated at 1.3 trillion
> dollars

But the would-be US presidential candidate suggesting free higher education is
the fiscally irrespomsible fantasist, apparently.

~~~
kristopolous
I've seen numbers floating at 70 billion/year for that. So that would be 18.5
years.

I've seen estimations of about 1.7 trillion for the wars since 9/11\. That'd
be another 24 years.

Peter Dale Scott wrote a book about something he calls "The Deep State" which
at least, to me helps explain how seemingly regardless of who gets elected,
such unpopular expensive programs continue without elected officials people
having any discussion of legislation to stop it.

~~~
NoMoreNicksLeft
Have you ever noticed that whenever government gives a bunch of money to
higher education to operate, the cost to operate those universities goes up to
match whatever extra was given?

~~~
Loughla
That's just not true. Higher education costs have held steady for about 30
years.

What has increased is the share of that cost that is being fronted by the
student, instead of the government. You want to be mad, be mad. But at least
point it in the right direction.

For example - at the 'state funded' institution I was previously employed by,
only 11% of our annual funding came from the state. Compare that to the
1970's, when around 60% came from the same source.

The cost of tuition didn't increase (outside of annual inflation adjustments)
overall. The cost to the student increased because the state cut back so hard
on their payments. Meanwhile, students demand the same, or better services
every year. You can have cheap, you can have good, but you can't have both.

The fact that people going to college in the 1970's were able to pay for
college on part-time summer jobs wasn't because it was miraculously cheap. It
was heavily subsidized.

These aren't secrets.

------
PeCaN
Does anyone know what it's written in?

Boeing writes (nearly) everything in Ada and as far as I know has never had
any software problems of this scale. Most of it is in SPARK as well, a
restricted subset of Ada that can be partially formally verified[1]. Frankly,
_not_ formally verifying mission-critical software where a failure can
potentially threaten someone's life seems rather irresponsible.

1\. Contrary to what seems popular opinion, regular Ada is not particularly
strict (less so than Rust). SPARK is very, very strict about what you can
compile, and if it compiles it follows the spec (which is usually easier to
audit).

~~~
acomjean
Speaking as someone who left a company that did radar software development in
ADA (and some C). We were a subcontractor of Boeing and wrote the software for
them.

We didn't formally verify the code, its way too complex, but everything was
reviewed and heavily tested. And tested again. Then integrated and tested...

Ada is pretty strict and if it compiles you usually had some confidence it was
going to at least run ok.

The industry was moving to C/C++ because developers know it and promises of
higher productivity.

I did kinda grow to like ada. It had its warts, but it was good. It reminds me
a bit of GO with its packages.

~~~
PeCaN
> It reminds me a bit of GO with its packages.

Ada Tasks also bear some semblance to goroutines. (In the sense that they
share memory by communicating instead of communicating by sharing memory.)

It's definitely a nice and well-designed language IMO—with a share of
annoyances[1] and flaws—but overall deserving of more attention than it gets.
Lately I've gotten comfortable enough with it that I have been making a habit
of reaching for Ada for side-projects where I would formerly use C. (For
example, I'm writing a toy X window manager in Ada at the moment—using XCB
from Ada is a little ugly at times but definitely bearable.)

1\. For example, why can't a record discriminant be a range bound? Answer:
standards committee didn't think people would use it. Of course, this means if
you have an array of discriminated size, you can't conveniently have a type
for a position in it. e.g.

    
    
      type Thing_Stack (Max_Size : Positive) is record
        Elements : array (Positive range 1 .. Max_Size) of Thing;
        Top_Index : Positive range 0 .. Max_Size := 0;  -- 0 = Empty stack
      end record;
    

Fails to compile with "Discriminant cannot constrain scalar type". According
to [2], this is because

> This restriction is left over from Ada 83 (RM83 3.7.1(6)). We considered
> removing this restriction in Ada 95, but ultimately decided that the
> implementation effort to support this modest extension would outweigh the
> (more?) modest benefits. Perhaps a short-sighted decision, in retrospect...

> As for why the original Ada 83 restriction existed, noone knows for sure,
> but it probably relates to the historical purpose of discriminants as only
> for controlling the "shape" of the record, rather than being seen (as in Ada
> 95) as more general type "parameters."

And apparently it got left in Ada 2012 also.

(For C people, this is basically:

    
    
      typedef struct {
        size_t max_size, top_index;
        thing elements[];
      } thing_stack;
    

but bounds-checked and without the annoyance of allocating flexible array
members.)

2\. [http://computer-programming-
forum.com/44-ada/82b646ab38d529a...](http://computer-programming-
forum.com/44-ada/82b646ab38d529af.htm)

~~~
acomjean
Its been a while

I liked the records in ada. You could control the position and size of each
field (which was really usefully for dealing with in binary messages) We had a
ton of constrained types.

If I remember we has assert 'size on some of the records to make sure type
changes didn't cause some records to become unexpected sized in case the types
that made up the record changed.

variant records were fun, but when you start trying to do much (array of
variant records?) it fell down.

The string handling left a little to be desired though..

------
protomyth
"In one instance, maintainers even had to manually burn data onto CDs and
drive off base to send the massive files across a civilian WiFi network."

That is a level of scary, I'm not comfortable with in our military's data.
Logistics data flying across a civilian WiFi is really not good.

~~~
maxxxxx
Similar stuff happens in medical sometimes too. Everything is so restricted
that at some point if you want to get something done you have to circumvent
all security systems in a really bad hack way.

~~~
striking
So instead of gently decreasing security when it's not possible to be totally
secure... it's better to throw the baby out with the bathwater. Wonderful.

~~~
noxToken
The gist of a convo with IT when I needed to work around some parameters:

"Hey IT dept. I understand the reason for our network restrictions, but we
have a lot of data that we need to get to [some endpoint] using [some
method]."

"Sorry, we can't relax the restrictions. You have to work within the
parameters."

"But the parameters do not allow me to complete my job."

"Sorry."

I'd love to work within the parameters, but when IT won't budge, you're stuck
with the least preferrable method.

~~~
protomyth
and where exactly is your boss during this time?

~~~
noxToken
Right beside me infuriated, because there was nothing that he could do about
it either.

~~~
protomyth
That's not an IT issue, that's an organizational issue. If your boss cannot do
anything including appeal to his/her boss, then you shouldn't do anything.
That type of thing needs to be sorted out.

Clearly, in the case of the F-35 systems, federal law was violated doing the
offsite thing. We see developers (and frankly some sysadmins) violating HIPA
and this kind of stuff has to stop.

~~~
noxToken
> _That 's not an IT issue, that's an organizational issue._

I knew this at the time. I never blamed IT directly for the issues, because I
knew that they were just doing their job as well.

~~~
tremon
I know it's not the easiest thing to do, but the proper response is to walk to
HR (with your boss alongside) and explain that you can't do your job. The
improper response is to wardrive to the first open access point you can find
and use that.

~~~
maxxxxx
In theory you are right but that's not how it works in most places.

------
cpeterso
Here is an interesting talk about the F-35 (Joint Strike Fighter) C++ code
given at CppCon 2014 by an engineer working on the project:

[https://www.youtube.com/watch?v=sRe77Mdna0Y](https://www.youtube.com/watch?v=sRe77Mdna0Y)

    
    
      The Joint Strike Fighter (JSF) is the first major DOD aircraft program to use
      C++. Much of this software is either safety critical or mission critical and so
      must be written in such a way as to be clear, readable, unambiguous, testable,
      and maintainable. We discuss the driving requirements behind the standard and its
      evolution. We give a quick overview of our standard and discuss how it differs
      from later standards such as MISRA C++. We discuss lessons learned over our nine
      year history of applying the standard to a large embedded software program. We
      also address ambiguities in rules and difficulties with automated checking of
      conformance with the standard.

~~~
engi_nerd
As I said in another comment, these remarks apply to the aircraft only, and
not to ALIS. To be honest I have no idea what ALIS is written in.

------
tibbon
Ok dumb question; if it's so damn broken why are we paying Lockheed Martin for
it? Did someone not think to put a term in their contracts saying that if they
can't meet basic milestones (working software) that the US Government (and the
people) aren't going to pay for it?

It's just software...

~~~
jsprogrammer
Why would LM sign such a contract?

~~~
tormeh
Hopefully they wouldn't, when you look at the end result of their contribution
to the project.

------
alphapapa
Some context is necessary here, because as engi_nerd explained, it's not as if
the primary flight control system is crashing and having to be rebooted in
mid-air. It's more like a separate system that's used by maintenance personnel
on the ground. Of course, if the aircraft's maintenance procedures are
designed around it, then deploying a whole wing of them isn't going to work
very well, because high-performance aircraft are high-maintenance aircraft.

But I can't help but think of Battlestar Galactica. At what point does this
software become too complicated? At what point does it become a vulnerability
to be exploited by the enemy, or simply a liability reducing readiness below
what's needed in wartime? I don't recall hearing stories about the F-16's or
F/A-18's computers crashing and needing to be rebooted (or even the F-22's).

~~~
noir_lord
F-22 did have some howlers (the computers crapping out when they flew across
an international dateline rings a bell) but as someone who casually follows
high tech stuff including military equipment I don't remember anything on the
scale of the issues they are having with the F-35.

I think a lot of it comes back to scope creep, the F-22 was designed for one
branch by one branch to do one thing well, the F-35 was designed for multiple
branches by multiple branches to do a bunch of things somewhat well.

It's an absolute bear of a programme.

------
kogus
How does this reconcile with this article?
[https://www.f35.com/news/detail/f-35-chief-software-bugs-
no-...](https://www.f35.com/news/detail/f-35-chief-software-bugs-no-longer-a-
threat-to-ioc)

~~~
engi_nerd
Those two articles are talking about two different things.

ALIS is the logistics system for the program that has an off-jet and on-jet
component. ALIS takes the role of a traditional computerized maintenance
management system and expands it to include an on-aircraft prognostics and
health monitoring component. The idea is to have the jet tell you what's wrong
with itself.

The article you linked to is about the F-35's mission systems software, which
is really a software suite that encompasses all the computers on the aircraft
itself. The MS software load provides all the stuff necessary to make the
jet's hardware do what it needs to do to fight.

~~~
mrweasel
So, the plane will fly, it will just be hard to maintain and operate?

~~~
hluska
The best answer I can find is, "I don't know."

For years, the F-35 project has been calling ALIS absolutely critical. Now,
they seem to be saying that the F-35 can fly without it and that ALIS only
keeps the cost of ownership down.

I suspect that the real answer to that question is either classified or
closely held amongst a core group.

~~~
engi_nerd
[https://en.wikipedia.org/wiki/Operations_security](https://en.wikipedia.org/wiki/Operations_security)

------
jlarocco
This headline is really misleading.

TFA is about the "Autonomic Logistics Information System" (ALIS) system used
to automatically order replacements parts and schedule service, _NOT_ the
plane itself.

It's grounding the plane because they can't schedule maintenance and order
parts correctly now that they're moving to this new system, not because the
plane itself is buggy. The planes have been flying (mostly bug free) for over
10 years now.

A lot of people here are using this to scapegoat C++, but although the plane
is written in C++, I don't think the ALIS is...

~~~
drzaiusapelord
Don't bother. Threads like this just become mindless ragefests from people
with political axes to grind and sites like vice know how to exploit these
feelings for ad impressions. Liberal doves also want anything to attack new
weapon systems spending. Its an ugly scene altogether and accurate information
and commentary about military programs are lacking.

I've lived through a few major weapons systems changes and its all the same.
Heck, anyone who has ever worked on a large project knows this stuff is fairly
common. A lot of people don't want this thing in the air, especially Russia,
China, Iran, and North Korea. Its a very capable weapons system and fills a
much needed role.

Its crazy a lazily sourced article like this could make it to the front page
of HN. It reads like my grandmother wrote it, "THE AIRPLANE IS RUN BY A
COMPUTER! TECHNOLOGY IS SCARY!" To compete on this level, automation and
complexity are required. Its silly to think we are going to back to how planes
were designed 50 years ago.

~~~
hackuser
> Liberal doves also want anything to attack new weapon systems spending

I generally agree with your points, but not this one. I've seen no partisan
bias in the criticism, and John McCain is the biggest critic of this program,
which is being pushed by the Obama administration. In fact, IME most of the
criticism comes from the temper-tantrum-of-the-week branch of the GOP.

Also, Democrats have become hawkish; almost everyone says they are for a
strong military including Clinton and Obama. There aren't many of those knee-
jerk doves left - can you name a prominent one?

------
dsp1234
Appears to be a copy of
[http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-
cr...](http://www.pogo.org/straus/issues/weapons/2016/f-35-chief-critical-
logistics-software.html) from April 20th

------
_of
Why was the software not such a big problem with other jets like F-16 and
F-22? Is the F-35 software really so much more complicated?

~~~
PeCaN
1) The F-35 has VASTLY more software and several software systems that the
F-16 and F-22 didn't have.

2) The F-22 software was largely written in Ada, a sane language that's great
for real-time systems that cannot fail. Apparently the F-35 software is mostly
written in... C++. Writing fault-tolerant real-time C++ is like walking on a
tightrope while gripping a gun with the safety off. You can do it, but it's
not particularly pleasant.

3) Apparently the team was considerably expanded/replaced between the F-22 and
F-35.

~~~
PantaloonFlames
ps: the F-22 was also over budget and under-delivered on expectations. So much
that the Pentagon canceled the remaining production run after 195 aircraft.
[https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor](https://en.wikipedia.org/wiki/Lockheed_Martin_F-22_Raptor)

What the heck are we doing with these airplane projects? We build stuff we
don't need, that doesn't work, and is waaay more expensive than we thought,
and the starting cost point was extremely expensive to begin with.

~~~
hackuser
> the F-22 was also over budget and under-delivered on expectations. So much
> that the Pentagon canceled the remaining production run after 195 aircraft

It was canceled because it was expensive and no longer needed, because the
mission it was designed for - fighting a peer enemy - no longer existed. That
was 2009, when the U.S.'s primary military concern was fighting two
simultaneous wars against enemies who didn't own one airplane or air defense
system. Any plane could do the job; we didn't need F-22s. Also, our prior
generation planes were still the best in the world.

Seven years later, Russia and China have engaged in massive military military
buildups, including significantly improving their air combat capabilties, and
aggressive behavior. The peer enemy mission is back at the top of the list
(and some are now talking about re-starting F-22 production).

------
esaym
I don't really understand how the computer systems of the F-15 and F-14 which
were written in 1970's and probably in assembly seemed flawless yet with
todays tech we can't even do this?

~~~
alkonaut
> I don't really understand how the computer systems of the F-15 and F-14
> which were written in 1970's and probably in assembly seemed flawless yet
> with todays tech we can't even do this?

Programming languages haven't evolved much since the 70's, yet the number of
lines of code we have to write to fly a 5th generation fighter jet has
probably followed the same curve as the number of transistors inside it. And
scaling a CPU turned out to be easier than scaling a program. Imagine the
difference in complexity between the earliest computers controlling car
engines, and what the computers inside a new Tesla are doing. The same with
jets. Modern jets can barely stay airborne without a lot of computer power.

~~~
esaym
True, but can you even imagine being a software dev in 1970? Where TDD, unit
testing, or testing even in general isn't even a thing thought of? It just
seems so foreign.

Even now, someone in college and spend all of their free time reading blogs
and books for best practice development and can generally fit in pretty good
with any development group once graduated. I don't think people had those
luxuries in the 70's.

~~~
duncan_bayne
> ... or testing even in general isn't even a thing thought of?

That's really, really, REALLY not true.

[https://medium.com/@verne/margaret-hamilton-the-engineer-
who...](https://medium.com/@verne/margaret-hamilton-the-engineer-who-took-the-
apollo-to-the-moon-7d550c73d3fa#.86lu3dxcc)

^^ fairly sure that she was thinking a lot about testing

------
windexh8er
As a whole the F-35 is one of those big disappointments that was the direct
result of each service branch having a say in requirements of the aircraft.
There are more than a few documentaries and articles outlining this in
fantastic detail.

Ars ran a similar article to this one in January:
[http://arstechnica.com/information-
technology/2016/01/f-35-s...](http://arstechnica.com/information-
technology/2016/01/f-35-software-overrun-with-bugs-dod-testing-chief-warns/)

I was an employee at Lockheed during the time of when the "JSF" vendors were
competing for the contract. While I wasn't directly involved in any F-35
programs (I was on the MS2 side supporting NATO programs, particularly ASOC /
SCCAN - that's another story in itself...) I was friends with a few
programmers responsible for the main flight systems of the early aircraft (I'm
unsure if that system was ultimately used in the production version or not)
and the attitude seemed to be that the aircraft would never make it to
operational use anyway. I was a, relatively, new and naive employee (first
employment after college) and it struck me as odd that the plane was a joke
internally to the company, at least those "building" pieces of it and it makes
me wonder today if the lackadaisical attitude was ripe across all facets of
the program. If so it makes sense today of all the failures the program has
endured. The aircraft has always underperformed and been overbudget.

Ultimately the tax payers foot the bill for a next-gen aircraft that
continually gets beat by task-specific variants that are decades older
unfortunately. I used to joke around with people about how the $1000 toilet
wasn't far off once I learned about how every program's goal was to be cost
plus and then milk scope creep to the Nth degree.

The programs I was involved with were selling commodity Sun + Cisco + some
overpriced serial to Ethernet bridges
([http://spt.sunhillo.com/products.php](http://spt.sunhillo.com/products.php))
tied to some archaic war/peacetime ATC software. It was absolutely astounding
what those packages cost and what was actually delivered.

I saw some were asking what language for some of these systems and if it were
along the lines of what I was exposed to it was a mix of anything and
everything. Most critical systems were written in ADA, as suggested, but I saw
Java (think awesome Java in the early 2000's), TCL, shell scripts and the like
as major components to these systems. In fact I remember a dev having problems
getting working out reading serial data from a GPS receiver to set proper
system time - so he just bundled ntpd in the build as his own work.

Working there was an eye opening experience for both good, and bad reasons.
The stereotypical jokes were, unfortunately, true to an extent and I couldn't
continue to justify pretending I was working on cool, unspeakable things that
were shrouded in James Bond'esque mystique.

Ultimately I was proud walking in day one and couldn't get out of there fast
enough. The people were great, but the management and true colors of the
company were centered on, IMO, calculated enrichment of select groups of
people. It was never about the best product - it seemed it was about who was
next in line to get fed.

------
aburan28
We should just sell them to Saudi Arabia

~~~
aburan28
I am genuinely interested in who/why I was downvoted for suggesting we sell
F35's to Saudi Arabia

~~~
aeturnum
I didn't downvote you, but some things jump out at me.

The suggestion we sell them to Saudi Arabia is orthogonal to the subject of
the article (technical problems). Your post and the article do not, from a
naive perspective, have a connection.

My guess is that you think the F-35 project is not a worthwhile investment for
the US defense system, but that it cost a lot of money to make, and so we
should get some money out of it. Further, since Saudi Arabia has a lot of oil
money and wants weapons, we should sell them this plane (which it seems you
think is a failure) - even though Saudi Arabia is nominally an ally and it
isn't obvious its a good idea to sell an ally weapons we consider bad
investments.

My previous paragraph is a just a stab at the thinking that might inform your
post - I'm probably totally wrong. Whatever the motivating factors, there's a
lot you don't say - either because you don't think it's necessary or you don't
want to talk about those background assertions. Obviously, we all believe the
people here understand (at least some of) the context, but my guess is that
people found your post to not be forthcoming enough.

------
azinman2
How is this still getting funded?!

------
willtim
The software has apparently been written in C++, it all sounds like a bit of a
farce. EDIT: to clarify the above comment, the context is the use of C++ and
mainstream tools for 8 million lines of safety critical code.

~~~
duaneb
You can write safe C++ (although it's hard). You can write extremely buggy
haskell or ada. Are you really that much of a language bigot you can't see
past the damn language to provide any interesting conversation?

~~~
PeCaN
Speaking from experience, it's a hell of a lot easier to write buggy C++ than
buggy Haskell or Ada (both of which can in some cases be formally verified to
follow the spec).

C++ is unsafe by default and you have to write lots extra checks to make sure
your code is bulletproof. Ada is safe by default and you have to be verbose to
do powerful-but-dangerous things with pointers.

While yes, you can write secure and fault-tolerant code in any language and
insecure and buggy code in any language, the sheer amount of C and C++
software that has been demonstrated to be insecure and very buggy in recent
years raises some questions about whether that's the tool we want to use.

Languages are tools. C is a $5 wrench with a blade for a handle. Ada is a
pocketknife where you can pull out the wrench or blade if you need it. Both of
them can do the same things, but with one you have to be very, very careful to
not cut yourself.

~~~
duaneb
Sure. None of this is news to anyone. Why not discuss why C++ was chosen in
the first place?

~~~
Jtsummers
Ability to hire people to fill the role of programmer.

~~~
tremon
That sounds like they were looking to scrape the bottom of the barrel. If
programmer salary played a key role for such a critical piece of software,
they have bigger problems.

~~~
PeCaN
From what I understand from this thread, it's worse than that:

The problem wasn't programmer salary, it was that _their budget was so big
that they wanted to hire extra people and could not find enough Ada
programmers_. Yup. It got written in C++ because they wanted to spend more
money and there are more experienced C++ developers than Ada developers.

------
frandroid
Space-X: check. Tesla Motors: check. Hyperloop: in progress. Tesla fighter
jet: The missing project in Elon Musk's portfolio...

~~~
wavefunction
I think he prefers being called "Iron Man" rather than "Tony Stark."

------
sickbeard
No it's not

[http://www.defensenews.com/story/defense/air-
space/2016/04/2...](http://www.defensenews.com/story/defense/air-
space/2016/04/26/f-35-chief-software-bugs-no-longer-threat-ioc/83553372/)

~~~
oxryly1
Different software.

