> The F-35 requires more than 8 million lines of code, compared with about 2 million for the F-16 and less than 1 million for other fourth-generation fighter aircraft
The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.
The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.
For anyone who's unfamiliar with the term "Mythical Man Month," it's the title of a book which argues, amongst other things, that assigning more engineers to a late software project will make it even later.
The book can argue more convincingly and thoroughly than I can, but briefly: When new engineers join a project, they have to spend a lot of time familiarizing themselves with it, so they're not productive for a while. They take time away from the project's original engineers by asking a lot of questions, which actually means progress slows down.
Further, more people means more complexity to manage. There will be more miscommunication, more instances of engineers' work conflicting with other engineers' work, more differences in coding style, etc.. With more than 200 people on a project, I can only imagine the nightmares.
I'm not familiar with the F-35 codebase. But I have a hard time imagining it could benefit from 200+ people working on it simultaneously.
I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway? If not, how in the world can you have engineer A coding a given module on the dayshift, who then hands it off to engineer B on the nightshift? Software is not like laying bricks. You can't just come on the next shift and pick up where the last person left off. You lose all the context they had in their heads while coding, which is absolutely essential.
Frequently aerospace companies will have a limited number of licenses for various pieces of software. C++ as a language obviously doesn't have this limitation, but a limited number of licenses for various custom test suites and third party software (e.g. Matlab) is frequently a bottleneck. If this is a bottleneck, it makes sense to have these licenses in use 24/7.
Purchasing more licenses is always an option, but usually not a quick one.
Maybe it's really obvious, I mean it should be, but Lockheed clearly isn't much of a software company.
Office space and third party software licenses mandating a 24 hour work cycle for the most expensive defense project of all time?! Maybe it should not be surprising. Their peers seem to just appropriate rights and constitutional interpretation to meet their needs, can't these guys just appropriate more matlab or whatever compiler it is licenses? Just seems like it is being treated like a manufacturing problem.
This same class of folks who are making these decisions are all contracting for NSA and have access to your data to keep us all safe...
Good point. It does make me wonder though: Does the time and money cost of purchasing more licenses outweigh the effect on employee morale resulting from night shifts?
> I'm also skeptical about this idea of 24/7 shiftwork. Will each person will have their own little piece of the codebase that nobody else touches? If so, then why have a nightshift? Why not just have everyone work in the day, since it's all in parallel anyway?
Because that way you can fit more people in the same facility? Otherwise, it will be unused for 8 hours each day (being optimistic).
Why not? It's not like they can just go down to the local HackerNinjaRockStar DojoSpace and rent some Ikea standing desks. At Apple, for instance, there are sealed buildings and security requirements that impose a cost on expansion above espresso machines and foosball tables -- and they're not required to meet DoD legal standards.
This is probably true. Still, it seems like a red flag to me that on a project like the F-35, which presumably has adequate funding for office space and other such things, their only solution for providing offices for their own employees is night shifts.
There's a very good reason not to have engineers work nights. While some are undoubtedly night owls, many (most?) will have lives and families which will suffer from night shifts. That accelerates burnout and increases resentment--both of which are usually running high in a troubled project anyway.
The F-35 has multiple variants, one of which is a STOVL (Short Takeoff and Vertical Landing) aircraft with a vertical flight system. The amount of code necessary to deal with that is likely a large part of the difference in LOC, not to mention the more complicated systems for weapons, radar, communications, etc...
In other words, the fact that it is written in C++ tells you little more than the fact that it is written in C++ and not Ada, but that alone will tell you that they probably expanded their pool of potential programmers by a large margin.
Considering the number of job postings I've seen where they want N+ years of experience, I'd say it's easier to find that in C++ than Ada if they do that.
Oh, they crash. Software on currently fielded jets crash all the time, pilots call them "software anomalies."
Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.
Sorry, bad grammar. I meant that sometimes they reboot automatically. The systems are designed to always reboot without physically crashing the jet, but sometimes the pilot has to command the reboot themselves.
> Systems are designed to fail softly and reboot quickly (~1 minute) and sometimes do so automatically without physically crashing the jet. Sometimes the pilot won't even notice.
If that's the case, why not use something like Erlang?
I think the difference in LOC count is likely due to the much more complicated avionics in the F-35 than the technology stack, as compared with older aircraft.
I'm pretty sure no self-respecting engineer would ever attempt to design such a thing (although many might be deluded into thinking that e.g. advanced type systems might help achieve that goal).
> choice of using C++
Almost certainly not a good indicator.. much of those LOC are likely integrations from other projects. Who is to say the alternative didn't involve rewriting some more sensitive, costly-to-produce code in Ada?
This website has some information about serious bugs. There are a variety of languages being used, by different industries, with either expensive war machines or expensive civil aeroplanes full of passengers or impossible to service space-craft.
Any one who's worked at Martin Marietta in the past (now part of Lockheed) will recognize this. Martin had a long, long history of doing this sort of thing for projects in trouble.
When they get past this (and they will, they have the DoD over a barrel, they just have to not gloat too publicly) the fallout to the F-35 will last as long as F-35s are in operation. In the early 90s, Martin people hand-assembled Titan rocket guidance and control software because some time in the distant past, the assembler had some bugs. The F-35 software group will live with whatever horrific "best practices" the current group of reassigned workers come up with.
To be completely truthful, they compared binary dumps of hand-assembled, and machine-assembled software, to ensure that everything was exactly the same. But yes, they hand-assembled some weird instruction set. I can't recall asking if they assembled to ASCII hexadecimal word (24-bit) representations, or to something else.
I believe this was for "Commerical Titan", a short-lived, Titan 34D variant. Four flights, one of which left Intelsat VI in the wrong orbit. That wasn't software, but plugging the wrong cable in.
I'm hoping this will help justify limiting purchases of the F-35 and replacing it sooner with cheaper, more expendable, but roughly equally effective UCAVs and better missiles.
Not only have they added 200 engineers from other projects which will now suffer delays and quality issues, both also "They're running 24-7, what we call lights out." which will ensure massive burn out. Brilliant...
If you time it right, you don't have to hit burnout. In my view, "timing it right" basically means completing the project before burnout. A successful completion followed by a little R&R goes an incredibly long way to combating burnout.
Sounds lovely. Is that really a thing? I guess if you paid me enough money, I'd put up with that, but mostly, this makes me glad I declined their job offer several years ago.
There analogy is not good, but I would also assume that lockheed won a competitive bidding process to get the contract as well. They promised to build it at a certain price and in a certain time.
Yup, Lockheed's X-35 beat out a (horrendously ugly) Boeing airplane to win the contract, but my understanding was that the initial orders were cost-plus. That's pretty typical for large procurements of advanced aircraft. It's hard to know long it will take and how much it will cost to build something that bleeding-edge.
"Accountability". A conspicuously heroic effort (likely counterproductive long-run), because their +$6 billion drip is threatened by lawmakers who don't think they're taking this whole thing seriously.
For the sake of whatever competent programmers were already working on the system, I hope this is mostly PR noise.
Because loitering and enduring battle damage make so much sense in an era of countermeasure-immune SAMs and huge directed-shrapnel warheads. Spitting DU rounds onto a city block also isn't a great idea in today's conflicts. The A-10 is obsolete, the tankbuster role is obsolete, the F-35 isn't perfect but is far more appropriate for today's conflicts.
Yet we fight a lot of "obsolete" enemies. The effectiveness of the F-35 won't be known until it is battle-tested. Before then, it's all rainbows and unicorns.
The F-35 is programmed in C++, while the other aircraft are programmed in Ada, which is one of the reasons that there is so much more code. When you have to code, test, and maintain 8 million lines of C++ in an environment where a single bug can be deadly, you should expect delays.
The choice of using C++ to write a system that is designed to never crash and the obvious ignorance of the "mythical man month" does not give a good impression of Lockheed's management.