
The U.S. Air Force learned to code and saved the Pentagon millions - zeristor
https://www.fastcompany.com/40588729/the-air-force-learned-to-code-and-saved-the-pentagon-millions
======
yazaddaruvala
I've heard that the salaries of government employees are capped. I'm very
curious how this will play out.

The Air Force can easily afford to pay Silicon Valley salaries (sure the
officers who are being taught to code can be paid less), but if the Air Force
is artificially limited by red-tape to offer competitive salaries, wouldn't
these software developers choose to leave as soon as they've learned?

The other issue is paying these officers competitive salaries, would have
quite the ripple effect. Higher ranking officers would all need to get paid
more, or software developers would need inflated ranks, but with some
designation where rank did not determine command.

I have zero insight and am just speculating. If anyone has more insight I'd
love to hear it.

~~~
oliwarner
This is a bizarre thing that keeps popping up here. This assumption that as
soon as somebody learns how to code, they pack up and move to the Bay Area.
Since when was Silicon Valley the only place with competent software
developers? We exist all over the planet, often getting paid very different
amounts, and usually, yes, very much less than Bay Area developers.

In reality, the USAF here is competing against the cost of defence
contractors. Which it turns out, is pretty easy because they've been charging
megabucks for everything for two decades because _their_ only competition has
been other defence contractors. Thanks to politics, right or wrong, they've
had no capacity to ramp-up on in-house development and submit their own bids
to really compete on price.

I really hope we're entering into an age of reality where public organisations
(military, government, healthcare in the civilised world, etc) —and more, the
people funding them— realise that in-housing development work is so much
cheaper than farming it out.

~~~
mattmanser
To be fair on them, one of the reasons it got so expensive was probably also
because it all followed waterfall and the USAF would pile on every
conceivable, often contradictory requirement, rather than start with an MVP
and iterate.

Plus it would take huge amounts of salesman/consultant hours to get through
the bidding process, plus each successful bid has to effectively pay for all
the costs of other bids that failed for the consultants to be profitable.

~~~
mcculley
For the Air Force, the MVP is often something that can do a lot of damage.
Some things require expensive upfront design. One cannot just iterate through
two week sprints until the crew survives.

~~~
mgkimsal
however, that MVP - used in a test environment - can lead to much more useful
feedback from real people on the ground in days or weeks, vs years to build
then deploy then getting ignored because it doesn't do what the users need.

There's a balance to be found, and it sounds like some of these skunkworks
approaches are starting to get more formalized.

------
bdavis_
What a wonderful story. It all comes down to risk and reward. Would Northrup
go all 'agiley' and get this done quickly? Sure if you paid them Time and
Material for the work and let them off the hook for any operational failures.
No-one wants to do the work for 5% profit. No-one wants to hear the USAF want
compensation when your program gives them a wrong answer and you lose a plane.
So, we write a spec, and all agree. The we do a design, and all agree, Then we
deliver and requirements have changed. The nature of the world we live in.

~~~
w8rbt
When theory meets reality, requirements change. We thought this approach would
work, and our research suggested it would. Then the real world came along.

~~~
Nexxxeh
No plan survives first contact with the enemy?

~~~
tabtab
The more general version: "No plan survives first contact with reality."

Somebody once also said that planning is still very useful even if the plan
changes later because it helps those involved prepare for and understand the
changes better when they do come.

------
mattlondon
What is most interesting about this article is the comments here considering
what we've recently seen RE Google. Where are the calls from HN commenters
suggesting that the engineers refuse to work on this? Where are the commenters
questioning the morals and ethics of the engineers working on this? Where are
the comments on here calling for the engineers working on this to _quit_ in
protest, or be deemed a _collaborator_?

Or does that line of thought only apply when it is everyone's favourite
bogeyman Google that is contracted to replace manual work with automated
systems? If its not Google, then is it A-Okay to work on military systems?

Both this and the Google project are IT systems that save manual, error-prone,
and time-consuming work to enable the US military to more efficiently kill
people.

I know I'll get down-voted to hell on this, but please if you're going to try
and take the moral high ground at least do so consistently.

~~~
bogle
You should qualify which engineers you are refering to. From the article it's
clear that the majority of the coding is "by a fast-moving cadre of Air Force
engineers". They are Forces personnel, not civilians. Yes, Pivotal are
involved, equally it seems if they are practicing XP, perhaps now only as
Agile Coaches.

I also believe the applications this team are creating are substantively
different in intent to those the Google employees were complaining about.

~~~
Glide
Hey, I work for Pivotal and I'm on an Kessel Run Air Force engagement. We
teach things by doing them. That's why we're effective. So we are involved
heavily in the making of software, we've always been. Our main schtick is that
we try to make sure that we aren't necessary to the software development
process when we roll off.

There are actually a couple of things that are the reason why a lot of people
don't think it's too much of a problem (some people do and I work out of an
office that does a lot of government work so we hear more of this than others
do). The main reason is that there isn't pressure for a pivot to work on a
project. If the project makes you feel uncomfortable request to be on
something else. Leadership has been very thoughtful in addressing how people
feel about working on military and government projects (especially with the
new administration).

The other reason is that the applications are pretty different in intent, from
what I understand, to what was complained by Google employees. It honestly
feels like very standard enterprise issues. We're helping them cut through a
lot of the same issues anyone would find at large enterprises.

------
NiteCoder
This story brings back a lot of memories. Right before I separated from the
USAF in 1997, I was working with a tanker squadron deployed in support of
operations in Bosnia. Crews would spend hours parsing the Air Tasking Orders
to figure out when to take off, and with how much fuel. I had just gotten in
to programming and wrote a simple VB6 app to parse the huge file and produce a
simple report with the details they were after. It saved the crews hours, and
I was still getting feature requests on the app for a year after I separated.
It's simply amazing it took another 20 years for them to solve the problem.

~~~
Jtsummers
There is an, unfortunate, lack of continuity in many offices like that in the
military. This is due (as you're aware, this statement is more for others) to
the way that personnel get moved within the military. Officers, in particular,
as the ones who are typically the "decision makers" for selecting what
projects to work on will be in their job for 2-3 years at a time. If they
don't have a chief or a civilian staffer who acts as a continuity officer,
projects like what you did get forgotten.

"Why are we still using this VB6 thing after 15 years? We should put out a
contract and have it done properly." says some colonel or major who doesn't
know better. And so it happens, a contract is awarded and it's a mess. I saw
it more from the civilian side. Without strong continuity officers, when the
military leadership changed over the direction and intent of things is
forgotten.

Additionally, as this article points out, the military is set on Waterfall-
styled contracts and projects, given to the lowest bidders. Which is why it
becomes a mess once a contract is awarded. Waterfall only works under a few
circumstances, and I guarantee your lowest bidder won't satisfy them all
(expertise with the implementation technology, expertise in the problem
domain, expertise with how the system is actually used, ideally a project
that's been done 100x before by the same team).

------
forapurpose
Stories like this one should be understood not in isolation, but as one point
in a pendulum system:

Endpoint A: These rules are all "red tape" stopping us from getting things
done. Let's just get the job done and save lots of money, time, and overhead.

Endpoint B: Our systems are crap; they don't meet specifications - we don't
even have specifications so we don't know what's needed; they are unreliable;
they don't interoperate; there's no competition for the work, it just goes to
incompetent people who have friends in the right places and rip off taxpayers;
and military officers know nothing about procurement, bidding, or complex
system development. This is ridiculous and unprofessional: We need
specifications, competitive bidding, quality requirements, QA, and
professional procurement for our $600 billion budget.

....

The problem isn't the endpoints, it's the pendulum. If this is your
organization, you need to get off the pendulum and make expert, intelligent
decisions about trade-offs.

EDIT: Some clarifying edits

~~~
tabtab
Competent internal teams can produce relatively simple solutions within a
reasonable budget. However, it's hard to formalize and replicate such teams
and/or team habits. That's why the more formal specification & contract
approach is often used instead: It's a BIT more predictable.

The first approach may get a grade from A to F, while the second ranges mostly
from B- to D-. Predictable semi-crap is often easier to accept than something
that might easily be an A, F, and all between. The first's probability curve
is mostly flat from A to F, while the second peaks around C-, and thus it's
somewhat more predictable, and that's why managers and bureaucrats often
prefer it.

~~~
forapurpose
> Competent internal teams can produce relatively simple solutions within a
> reasonable budget. However, it's hard to formalize and replicate such teams
> and/or team habits.

Tangentially, that is exactly the military's central management challenge (per
my limited understanding): Combat depends on replicating competence in
internal teams, under extreme stress and extremely unpredictable situations,
and with unpredictable turnover of key personnel, including management (they
die). The U.S. military, for example, has to accomplish this across hundreds
of thousands of personnel.

Imagine you get a new manager the week before a major project is due, with
lots of critical decisions to be made. Now imagine you get a new manager the
day that person is supposed to lead you on an assault up an enemy hill, where
a less-than-optimal decision might kill you.

------
Shivetya
I enjoyed that, there mere mention of Shaw AFB brought back memories of
serving there in the late 80s leaving just before the first Gulf "war" broke
out.

anecdotal part goes here.

Being in IT back then was fascinating, the secure comm side systems were
Burroughs of the generation just after tube computers went out of style.
Complete with paper tape booting or switches and lights. The main system which
I was on was a Sperry 1100/70 using cards as the main input from on base
customers. Those transitioned into images on a 360k diskette when I was
leaving. The system data was mostly stored on tape (autoloading reel) and
there were three disk packs including one we joked was kick start (seriously,
if you swapped packs and it didn't spin you faced away with your butt on the
edge and swift flat to the side hit it with your boot).

It was that diskette that got me into programming. I started with Turbo C and
Turbo Pascal settling on the second. The tool we were provided by Sperry would
upload the card images one line at a time. On our Sperry branded PCs (same as
IBM/XT) it could take longer to upload a deck than to run one through a card
reader and the personelle group had the equivalent to three boxes. I ended up
digging into the emulator software, the arcane manuals we had, and whooped up
something in TP that did it in a tenth of the time. From that point on it was
off to the races for my desire to program.

~~~
tabtab
I once interned at a military base that used punched card readers. Some jobs
(processes) would refuse to run if any card in the deck was detected as being
bad (using counts and check-sums), such as misaligned holes. Being analog,
different cards often were "bad" on repeat reads. It drove everybody crazy.
Experienced card people knew all kinds of tricks to reduce such a problems. I
learned to respect experience.

------
inamberclad
> To do so, a person called “the Gonker” entered data into an Excel
> spreadsheet known as “the Gonkulator,” while “the Planner” arranged magnetic
> pucks and plastic laminated cards on the whiteboard to indicate how long
> planes could stay in the air.

I love the names and vocab that comes out of unofficial military slang.

~~~
AndrewKemendo
It came from Hogans Heroes and was generally adopted by the military as a
"black box" that nobody understands. In my experience, it's predominantly used
in the Air Force.

------
blahedo
> _“Just because you followed the software requirements spec doesn’t mean you
> met the capability need,” Kroger said._

Sage words.

I do wonder, though: becoming more agile (or agile AF as the case may be) is
one thing, but is it healthy for an actual _military_ branch to be moving in
the direction of an industry whose unofficial motto is "move fast and break
things"? That's one thing when the bad outcome might be that someone gets
delivered the wrong thing or gets charged twice and you have to refund them.
But broken things have much graver consequences when they involve guns and
bombs and airplanes.

~~~
Arwill
> "move fast and break things"

Doesn't mean that you deliver broken code, it means that you should not be
lazy to rewrite old code.

~~~
breakingcups
That's not what that quote means. It literally means it's okay to not get it
right, to make mistakes, as long as you can deliver quickly [1]. Not that
weird for an industry where time to market is everything.

[1] [https://www.sfgate.com/news/article/Mark-Zuckerberg-
Moving-F...](https://www.sfgate.com/news/article/Mark-Zuckerberg-Moving-Fast-
And-Breaking-Things-2461393.php)

------
danso
Speaking of stories of people coding in the military and in other U.S.
bureaucracies, these came to mind:

How I got a medal from the Army for writing code (2014):
[https://news.ycombinator.com/item?id=7950190](https://news.ycombinator.com/item?id=7950190)

How learning to code kept me sane when I was a diplomat (2014):
[https://news.ycombinator.com/item?id=8810589](https://news.ycombinator.com/item?id=8810589)

------
jacques_chester
A few months ago some Kessel Run folks gave (really fantastic)
presentations[0][1] at CF Summit which are worth watching for deeper dives on
the experience.

Disclosure: I work for Pivotal, though in R&D.

[0]
[https://www.youtube.com/watch?v=V8w4jf5clJk](https://www.youtube.com/watch?v=V8w4jf5clJk)

[1]
[https://www.youtube.com/watch?v=lCgxdzoJGqE](https://www.youtube.com/watch?v=lCgxdzoJGqE)

------
kerpele
From the article: 'In addition to the name, the team chose a hashtag for its
project that is a nod to both agile software development, and the fact that
Kessel Run’s culture is more startup than strictly military: #agileAF

“The AF stands for Air Force,” Furtado assured me.'

I wonder where the reporter's scepticism comes from...

~~~
sarabande
It could otherwise be read as "agile as f __* "

~~~
Glide
Yeah. I think they know that. They always chuckle when we bring up the
hashtag. I'm on a Kessel Run project now and they are great clients.

------
totaldick
There's a few people like Will Roper (former MIT prof) and Raj Shah in the
military that are introducing agile thinking into the military, which it
desperately needs. Other countries are catching up and it's fast becoming the
case that software is the only thing that really matters in warfare.
[http://www.foxnews.com/tech/2018/06/07/f-35-to-2070-air-
forc...](http://www.foxnews.com/tech/2018/06/07/f-35-to-2070-air-force-says-
software-will-decide-who-wins-future-wars.html)

~~~
notarealaccount
I call BS on this [0]

[0] [https://blog.codinghorror.com/boyds-law-of-
iteration/](https://blog.codinghorror.com/boyds-law-of-iteration/)

------
eriktrautman
It’s a very interesting look behind the curtain at what’s wrong with the
procurement and development process but also reads like a Pivotal Labs puff
piece.

------
maxxxxx
"“It hasn’t been peacetime in a long time,” I was told."

If we don't have peace right now, how can we ever have peace? Is eternal war
finally there?

~~~
borski
This is the current war:
[https://en.wikipedia.org/wiki/Operation_Inherent_Resolve](https://en.wikipedia.org/wiki/Operation_Inherent_Resolve)

------
UncleEntity
Seems like NASA's europa-pso code is right up the alley for what the tanker
scheduling needs and is already out there collecting bit-rot:
[https://github.com/nasa/europa](https://github.com/nasa/europa)

Think there was a follow-on project but can't remember the name of it.

------
fulafel
Interesting there is no discussion about Pivotal and the ethics of optimizing
war machines, given that only a couple of weeks ago there was a big discussion
about Google employees successfully stopping the company from working on
drones.

~~~
bogle
Possibly because the applications are qualitatively different; possibly
because the software is being produced by military personnel (aided by
civilian advisors).

------
jonhendry18
Soon: "This should be contracted out, we'd save a lot of taxpayer money".

Later: Contractors take over costing much more.

------
coldcode
No one commented yet on the title "... saved the Pentagon millions". Compared
to 700B a year (budgeted, more is spent), millions is sort of less
inspirational.

------
black6
The Army has Jyn Erso. The Air Force has the Kessel Run Experimentation Lab.
Is Disney sponsoring the DoD now?

------
exabrial
I had a literal Laugh out Loud at the #AgileAF (Agile Air Force) hashtag.

------
trumped
overall, did the Air Force spend more or less?

