Hacker News new | comments | ask | show | jobs | submit login
The U.S. Air Force learned to code and saved the Pentagon millions (fastcompany.com)
252 points by zeristor 7 months ago | hide | past | web | favorite | 139 comments

I've heard that the salaries of government employees are capped. I'm very curious how this will play out.

The Air Force can easily afford to pay Silicon Valley salaries (sure the officers who are being taught to code can be paid less), but if the Air Force is artificially limited by red-tape to offer competitive salaries, wouldn't these software developers choose to leave as soon as they've learned?

The other issue is paying these officers competitive salaries, would have quite the ripple effect. Higher ranking officers would all need to get paid more, or software developers would need inflated ranks, but with some designation where rank did not determine command.

I have zero insight and am just speculating. If anyone has more insight I'd love to hear it.

This is a bizarre thing that keeps popping up here. This assumption that as soon as somebody learns how to code, they pack up and move to the Bay Area. Since when was Silicon Valley the only place with competent software developers? We exist all over the planet, often getting paid very different amounts, and usually, yes, very much less than Bay Area developers.

In reality, the USAF here is competing against the cost of defence contractors. Which it turns out, is pretty easy because they've been charging megabucks for everything for two decades because their only competition has been other defence contractors. Thanks to politics, right or wrong, they've had no capacity to ramp-up on in-house development and submit their own bids to really compete on price.

I really hope we're entering into an age of reality where public organisations (military, government, healthcare in the civilised world, etc) —and more, the people funding them— realise that in-housing development work is so much cheaper than farming it out.

To be fair on them, one of the reasons it got so expensive was probably also because it all followed waterfall and the USAF would pile on every conceivable, often contradictory requirement, rather than start with an MVP and iterate.

Plus it would take huge amounts of salesman/consultant hours to get through the bidding process, plus each successful bid has to effectively pay for all the costs of other bids that failed for the consultants to be profitable.

For the Air Force, the MVP is often something that can do a lot of damage. Some things require expensive upfront design. One cannot just iterate through two week sprints until the crew survives.

however, that MVP - used in a test environment - can lead to much more useful feedback from real people on the ground in days or weeks, vs years to build then deploy then getting ignored because it doesn't do what the users need.

There's a balance to be found, and it sounds like some of these skunkworks approaches are starting to get more formalized.

Oh most definitely. The tender process is awful from start to finish, regardless of market. If nothing else, as you say, it funnels huge amounts of resource away from where it needs to go (the product, developers and engineers) and into the pockets of salesmen, managers and shareholders. It just doesn't make sense for public money.

But MVPs aren't always themselves a viable development target... Especially here in a military scenario where the finished product tends to be the MVP. Safe enough and hopefully effective.

Honestly, I'd much rather work for the US Govt. than some snazzy SV company, regardless of pay.

> This assumption that as soon as somebody learns how to code, they pack up and move to the Bay Area... In reality, the USAF here is competing against the cost of defence contractors.

Exactly. The airmen that can leave, will leave to defense contractors, not silicon valley.

The armed forces training people to go into private versions is nothing new. The difference here —that you don't see in military engineering or mercenary work— is the USAF can control both supply of and demand for developers on its internal projects and undercut contractors.

And if they have transferable skills on exit, all the better for them.

Why couldn't they do the same with any other kind of military contracting?

The cost to get up and running. You're taking hundreds of millions in capital expenditure for an engineering R&D lab. Double that if you want to actually produce something.

Compared to a pile of consultants, laptops and whiteboards. Software is cheap to make.

Government pay scales don't apply to government contractors, though. It would definitely be possible to develop things at a lower cost than defense contractors do, but not with the artificial limits on salaries imposed by the law.

I've had several reads of this now, and don't understand which way you're arguing.

The current US Government pay scales are more than enough to match average developer salaries in much of the US. No, they're not going to get super-rich but they could get paid more than enough, help their country and have a stable job that doesn't rely on finding ways to monetise faddy bullshit ideas.

One thing I was disturbed by going from the US Navy to working privately in software development in Seattle is how much of a secret pay is in the private sector.

You can pretty much look up how much money anyone in the military makes. Base pay is trivial to look up, and all the other allowances (housing, hazard, etc.) can be looked up.


The same is true in the government, AFAIK.


AF vet, and frequently frustrated by this taboo. It's so contrary to the interests of everyone except the employer, but folks become visibly uncomfortable whenever talk of comp comes up.

Well, the American folks anyway. I work with a lot of Europeans, and they don't seem to share the norm.

A strong case against being open with the numbers is it that it could limit your ability to negotiate a higher rate with a potential new employer.

In the air force / navy / army etc there is only one employee to choose from so this never comes up.

That's a weak case - friends/colleagues will most probably have 0 effect on negotiations on some new job. As written this hurts everybody except the employer, but if some folks across the pond are dumb enough to play this game, they deserve the effects

I've had multiple employers tell me that it's a firing offense to discuss pay with my coworkers. Most people don't know if that's legal or not however and just remember that HR or the employee handbook said not to discuss what they get paid or they'll be fired.

Europeans are open with their salaries?

European bank. Salaries were confidential and discretionary. Then they adopted a salary grid based on title and business area mostly to reduce the bonus component (a disproportionate amount of the comp was in bonus). Employees were happy, first because it meant a large base pay uplift for pretty much everyone, and because transparency means base pay ceases to be something to discuss about, everyone feels like he is being treated fairly.

Then I believe it triggered all sorts of problems. Front office base salaries is higher than middle office, no one will take a pay cut so it prevented front to middle office moves which is is an undesirable effect. It is also expensive, they liked having the ability to hire more cheaply. They dropped the grid after a couple of years.

But all the time the numbers were confidential (but when there is a grid it’s a matter of minutes before an accurate table circulates).

Some. Brits aren't.

I don’t know about the rest of Europe, but here in Germany people have openly talked about their salary with me in several occasions. Granted, it wasn’t any coworkers, it was always a friend or someone I knew.

In Norway this is an open database which gets published annually and anyone can search in it to see what other people make.

It's not a digital wild-west open style though:

1. You have to authenticate using something which identifies you specifically as a citizen.

2. Another caveat is that people who gets looked up will be notified about that, with information about who it was who looked them up. (so it can't be used for direct covert spying)

That said, this system allows for a certain degree of transparency, and makes it easier for journalists, press, researchers etc to dig into issues such as social inequality.

I understand the motivation (even though I fundamentally disagree with it), but I find this rather inconsistent with issues such as 'the right to be forgotten' and other privacy initiatives put out by the EU.

I can see this one being taken to the EU Court of Justice by someone.

In Finland you can go and check how much somebody else earned and paid taxes. You can’t do this online, must visit the tax office in person.

This is balancing between privacy and transparency of tax system.

I don’t think the EU privacy legislation applies to this kind of public functions.

Norway is not in the EU.

Technically correct is the best kind of correct ;)

Pragmatically correct is better than technically correct :)

Norway will be subject to EFTA court, which is almost de-facto Court of Justice.

Granted, how someone might sue and escalate an case is a different thing altogether.

But they adopt GDPR, nonetheless.

Here in Finland people are somewhat secretive (especially the older generation) about their pay but tax records are public information so you can calculate it from there if you really want to know how much someone is making. Though the system is kept offline on purpose and thus you have to go to the tax office and browse the paper records and take notes.

Keeping the pay secret is kinda puzzling on traditional fields as the salary structure is very rigid on those due to strong unions. Doesn’t apply much in IT though where it is just minimum wage set by unions and you negotiate for (much) more.

Guaranteed pension and benefits for life after 20 years with substantial subsidies are worth a lot.

Given the choice between a typical developer's life (outside a tech hub or hot spot most programmers aren't compensated much different from any other low social status white collar worker), moral issues aside, I'd take the military job any day.

A person can join the military at 18, retire at 38 and be in a prime position to capitalize on his network as a consultant or "small business owner" contractor, or else just sit back and enjoy his generous pension and health and other benefits. That's a big deal.

Military retirement after 20 years isn't all that generous, it's about $40K. It's nice but you're still probably getting a job if you've got a family.

$40k household income for life, and that's still above median for many parts of the US (though about 2/3rds median for the US as a whole). Are there other job opportunities, outside some police departments, that offer similar compensation and let you retire at age 40?

Yes it's great, it's equivalent to over a million dollars with a safe withdrawal rate. I'm just saying you're probably still going to be working, not actually retired.

$40K is the figure I got from a Navy commander who's been in over 20 years. He and his family are used to approximately triple that level of income. He doesn't feel he can retire.

Lower-ranking people will be used to living on less, but will also have proportionally lower retirement pay. If you're a lower-ranking person and can get significantly higher pay in the private sector, it's not necessarily a dumb choice to go for it.

Unless you have to work in a SCIF

Well then, take the military job if it is that simple. There's probably nothing stopping you joining.

You can work for the military as a civilian, too.


Good luck.

The article talks about enlisted doing the coding, not officers.

They cannot leave until their contract is up. They literally cannot leave as soon as they've learned if they still have years in their contract. Additionally, the military dangles a lot of incentives to re-enlist, and people who have spent a lot of time in the military sometimes have trouble adjusting to life outside it. But yes, people will leave. There's always more. You would be surprised how many bright people choose to join the military even though it doesn't make economic sense for them.

I'm on mobile and copy paste isn't working, but enlisted pay grades are publicly available. They are much lower than SV salaries and they are set in stone.

An unsurprising number of Navy nukes re-enlist from the Persian Gulf. That $60k+ re-enlistment bonus is tax-free if you sign in a war zone.

Also, for people who joined the military out of high school, the private sector is frightening.

You pay zero for housing, zero for medical for an entire family...not even a co-pay. After 20 years you get a pension for life. And for many, they are doing it for reasons other than to put more money into a CxO's pocket.

Oh, no doubt there are benefits. If I come across a little sour right now, a close family member recently graduated and signed up for the military without really talking to anyone, because the recruiter told him being a peon in the Army would guarantee him a job at Google.

In my day job, I work with enlisted (not Air Force) and a bunch of ex-military, so I'm pretty familiar with the situation. I have actually been wondering if military retirement drives down wages, because a bunch of guys take pretty low salaries that are only okay because of their retirement check. Great deal for them, and definitely well-earned - it's no cakewalk to earn military retirement, harder than anything almost anyone in the private sector has to do - but I wonder how it works out across the labor market.

I do feel like, outside of the officer corps (who for some reason seem to hold onto it more often), the sense of noble purpose often gets crushed out by the inanity and bureaucracy so many of them have to put up with. IME that's usually why they don't re-enlist.

You get specialist pay don’t you? An enlisted person who pilots a helicopter or works in a special operations unit gets paid more. You could just have specialist pay for developers.

But anyway this whole ‘you need to pay them the same as civilians’ is nonsense anyway - soldiers are already paid a fraction of what private security get now and it doesn’t seem to be a major problem causing people to leave to do that equivalent job.

People join and stay in the military for great reasons other than pay, such as comradeship, family history, sense of duty, the challenge - appeal to those.

Sort of, but not really. Being in SOCOM (special operations command) units, you'll often get extra training in the form of Additional Skill Modifiers. For instance I was a UAV Pilot trained to fly the Hunter TUAV who got put in a Shadow 200 TUAV unit. I went back to training for a few months and came back with a W2 ASI denoting Shadow 200 qualification. That made my job designation change from 96U to 96UW2, which depending on the role, can mean extra pay.

Generally socom operators get paid more due to hazard pay, which is a graded scale based on how likely you are to get maimed for life or killed. I can't speak for anyone but the US Army, but they only allow commissioned officers and warrant officers to fly helicopters. As a UAV pilot, I was in a group of the only enlisted pilots in the entire US Army. We were on the same flight status with the same awful yearly physicals of the helicopter / plane pilots.

> awful yearly physicals

I’m too curious not to ask what those encompassed.

The worst part of a Class 1 Flight Physical is a colon test, which involves a glove and some unpleasant moments. Funny story, we had a soldier who was a former gym rat and horribly homophobic. We were all up for this unpleasant experience and his best friend schemed with the flight surgeon who performs this physical. This could have likely gotten everyone in a lot of trouble but somehow everyone laughed it off.

Anyways, his best friend hid inside the room behind the gurney you bend over. When the surgeon did the colon check, his buddy came from behind the gurney. As the surgeon put his left hand on the soldier's left shoulder, his buddy put his right hand on his right soldier. This was right before the surgeon put his finger in/out for the colon check. Knowing what was going to happen we heard this big muscle bound guy scream like a 12 year old child and erupt in anger. Somehow by the grace of god he didn't assault the full bird colonel flight surgeon. Once he saw his buddy he realized he'd been had and got really mad but calmed down quickly. The rest of the 2 years he was in our unit that poor guy never lived it down. We were outside the room pretty much all on the ground crying laughing so hard.

I work in the public sector for a lot less than the offers I get from the private sector. I haven’t once considered leaving.

Well I have, but that’s when the right decision drowns in politics and red tape, not because of money.

A lot of people would consider me an idiot, but I get a sense of duty and purpose from public service that is worth more to me than money.

Of course it helps that my pay check isn’t exactly bad. So it’s also a choice between good money and better money, I’m not sure how I’d feel if it was a choice between shitty money and great money.

The housing allowance for an O-1 in San Francisco with no dependents is over $4000. That’s a good deal better than the rent spending most Bay Area bigco engineers feel comfortable with. I’d argue the military is quite competitive with the tech industry when location is taken into account.

To be 100% honest, if they offered $80k and military level benefits I bet you a ton of people would take that offer as long as it was guaranteed in writing they would be in the US writing code. Most developers are mediocre and paid in the $80-120k range depending on cost of living.

Honest question, how can one look at themselves unbiasedly and determine if they're a mediocre programmer?

If you don't:

A) Massively outperform compared to your peers.

B) Massively underperform compared to your peers.

You are almost certainly a mediocre programmer.

If you are in the B category, it is usually pretty obvious to _everyone_ and you've probably seen your performance reviews at _multiple_ employers be below average.

If you massively outperform your peers, you probably find your boss at _multiple_ employers going out of his way to retain you. (i.e. Massive salary bumps, bonuses, promotions, etc.)

If neither of these are true, you are almost certainly mediocre.

For instance, I made a really stupid decision for personal reasons to leave a job where I massively outperformed my peers (my boss was really bad at hiring people) and received a 30k raise a couple years before I left. But, realistically, that was the _only_ job I had that kind of outsized performance and it was more to do with my boss's skill at hiring people than my skill as a developer.

At my jobs before and after that one, I've fell safely in the mediocre range. Never been fired, never had a PIP, and generally got "Meets expectations-ish" performance reviews.

At my current job, I underperform my peers but that is largely because I'm their first hire in ~3 years. I have alot of catching up to do as a result, and I've been there ~90 days.

Mathematically the vast majority of programmers in the world are mediocre by your definitions there. Was that the intent of your post?

Actually, the definition is very localized (you could be mediocre at a big co or a rockstar at a tiny backwater company) and relative. I don't think it's a stretch to assume that the majority of programmers are mediocre, the distribution curve of talent/ability not withstanding.

Yes. He was asking how to be unbiased about self evaluation. You have to paint really broad strokes to do that.

Is it perfect? No.

Do I think the 70% of programmers that probably fit this definition are mediocre? Yeah.

Keep in mind I am calling myself a mediocre programmer.

The allure would be job/salary security for 10-20 years (albeit at a lower salary than SV), and retirement in 20 years with full pension and medical benefits for life.

The cap should be based on the private sector equivalent of a job. Having an absolute cap is counter productive.

I don't think the public is quite ready for just how much developers get paid. I doubt many would believe it.

We can't even afford fresh college graduates. But we can afford Oracle.

I'm not even sure all _developers_ are quite ready for how much they (can) get paid - if they choose the most lucrative jobs.


Facebook's _median_ salary is apparently over $240k...

And IBM's is around $50k.

Everyone hears about these $200k salaries, but most developers aren't making that. The median for the USA is around $100k, it's still a high salary for a job that requires only an undergraduate degree and no certifications, but it's not obscenely high like $240k.

Not to mention the median for a developer in the Bay Area is around $125-130k [1] [2]. Those $200k+ salaries often quoted here on HN are top outlier employees at top outlier companies, not rank-and-file mid-level engineers at average no-name companies.

1: https://www.glassdoor.com/Salaries/san-francisco-software-en...

2: https://www1.salary.com/CA/San-Francisco/Software-Engineer-I...

It doesn't require an undergrad. I was in the US Army as a UAV pilot for 4 years, got out and got a job as a Unix Admin. I'm now a successful software engineer. Also, $240k is not obscenely high. Top end google engineers make more and that's about normal starting comp for a very talented engineer in certain segments of finance.

It doesn't strictly require an undergrad, no, but it helps to have one.

And $240k is very high for a non-management job, and compared to the median income and the minimum wage for the USA is obscenely high. Median household income is ~$60k, and median personal income is ~$30k. Minimum wage in the USA is around $15k.

In my books, 8 times the median income, or 16 times the minimum wage, is obscenely high. That kind of income disparity is not good for society.

My point was that it is not high in the world of software development. There are ranges much higher than the GP's stated $240k. I work in an industry where that is the case (finance).

In one sense it's clearly not obscenely high - it's the median salary after all - _half_ of Facebook's employees make more than that.

But from the perspective of someone struggling to make minimum wage, I'll be it 100% is obscene...

Would be surprised if the average IBM developer salary is around 50k. My guess would be higher, proabably around 90k - 100k for US employees.

depends where and what

A resonably good developer doing backend in IBM Cloud is getting 150k+ in Dallas.

And the $50k IBMer in upstate New York lives about as well as the $240k Facebook employee in Menlo Park.

This keeps being brought up but I'm sorry, no. The IBMer maybe has a larger house that they own rather than rent. In basically every other respect, the $240k-earning employee comes out ahead. The latter earns more, can save _way_ more, lives in an area with better weather, and can afford to retire far earlier.

Exactly, they can put much more into a 401k and then move somewhere cheap to retire and have a much higher standard of living than the $50k guy would.

That's a sad statement about the cost of living near SF.

You can't make this shit up: the median home price in Ithica is $242k [0]. In Menlo Park, it's $2.42m [1]. A nice, even 10x.

[0] https://www.zillow.com/ithaca-ny/home-values/

[1] https://www.zillow.com/menlo-park-ca/home-values/

Wow, that makes my 160K 1500 sq ft house in the upper midwest (with easy access to 6-figure jobs) seem absolutely cheap by comparison. Only problem is property taxes are high (4K - 8K in this area). Oh, and typical corrupt Illinois politics.

So is California.

He doesn't save as much, though. The FBer could save the IBMer's gross income every year.

What good does that do him? One typically saves for:

- Income replacement in an emergency

- Income replacement in retirement

- A down payment on a home

- A home improvement project

- A child’s college expenses less need-based financial aid

All of these are inflated by living in a high-cost high-wage situation such that the larger balance has you merely keeping up in terms of the capabilities your savings buys you.

The extra savings only make a difference if you move somewhere cheap before you draw them down. That’s a great argument for doing a few years in SV in your twenties but less relevant for senior talent.

Having a higher income and higher gross savings allows you to invest more in:

1. Rental properties

2. Securities - stocks, bonds, derivatives, REITs etc.

3. Investment opportunities not available to others. The $250k-earning employee (sorry, yeah I bumped it up $10k to make this argument) is an accredited investor.

All of the above have compounding benefits over the years. Investing $25k today is better than investing $5k, even if both are 10% of gross income of our example employees. $25k savings allows you to put a down payment on a $125k rental property in a low-cost area every year.

> All of these are inflated by living in a high-cost high-wage situation

The "high-cost" only applies if you insist on living the exact same lifestyle as you would in a low-cost area: large house, boat, large vehicles etc. It's just common sense to limit your spending on overpriced things. In the Bay Area, housing is overpriced.

It’s just common sense to compare like with like. The appropriate comparison to a shared apartment with a long public transit commute in SF is the same lifestyle in another city. Which may not provide as much investment income, but is so incredibly cheap you could achieve FIRE or be an artist or whatever.

> It’s just common sense to compare like with like.

I guess we should agree to disagree :-). I think you should adapt your lifestyle to the area you live in. Here's an analogy: let's say I love fresh sushi. If I live near the coast and a healthy fishing industry, it'll be good and cheap. If I live in Las Vegas, it can still be had but it'll cost me. Do I keep up my expensive sushi habit in Vegas and complain about the cost-of-living? Or do I just eat less of it and accept that there are benefits to being in Vegas that outweigh the more expensive sushi?

Besides that, no $250k-earning employee is living in a shared apartment with a commute to make ends meet in SF. If they live that way, it's to maximize their savings.

Vegas provides plenty of dining and entertainment options to make up for lost sushi. It’s not worse, just different. The anolgoue in SF lifestyles would be a high-rise condo instead of a house, proximity to transit instead of a nice car, etc. The frugal options in SF are not just different, but straight-up worse.

When a majority of that $250k is in Monopoly money (i.e. most of the time), they absolutely are.

My family of four live on that FBer's income in a nice single family rental and still save over half that IBMer's gross income. I can mortgage a multi-family dwelling in the Midwest every 4-5 years or so, invest in stocks/index funds, and generally do a lot more than the IBMer will ever be able to.

if they choose the most lucrative jobs.

I don't think that's how it works.

You can only choose to apply.

When I graduated college around 1990 I recall that was what financial firms in NYC were offering for real time/hft programmers.

If that’s salary TC is significantly higher. That’s what a new grad can make there in their first year in TC.

Many folks who join the military did so out of a sense of purpose. Not all, to be sure, but many.

The second thing my acquaintances who enlisted cited was retiring from military with half pay after 20 years of service and access to veterans’ benefits which in hindsight looks pretty nice for most non professional career options otherwise.

It's not a bad deal, especially when you consider you could live with minimal expenses while in the military. Saving a lot is not hard even on modest military pay when your living costs are all covered. I can't think of any other opportunities where you could start working at 18, save a lot, retire at 38, and get a pension.

Living costs, food costs, most of your clothing, transportation, etc. You literally just pay for fun and off duty extra activities / extra food.

Working for the military can pay very very well -- if you're a consultant or work for a third party.

The best rank structure for this would be the warrant officer.

What a wonderful story. It all comes down to risk and reward. Would Northrup go all 'agiley' and get this done quickly? Sure if you paid them Time and Material for the work and let them off the hook for any operational failures. No-one wants to do the work for 5% profit. No-one wants to hear the USAF want compensation when your program gives them a wrong answer and you lose a plane. So, we write a spec, and all agree. The we do a design, and all agree, Then we deliver and requirements have changed. The nature of the world we live in.

Risk IS ALL it comes down to. Risk and who the the big-wigs have lunch with. Is the company big enough to sue when something screws up? Tick!

i worked on software like this as a contractor for ausDoD. The same shitty companies who were big enough to sue would get awarded contracts over and over again even though nothing they ever delivered was on time, budget or worked. Accenture didnt even have programmers, they were hire buisness grads and give them crash courses on writing software. so whatever was delivered, would need another billion dollar project to replace it. why? its a massive bureaucracy, where everyone is covering everyone elses ass, everyone is fighting for budgets that they will do anything to protect. 90% famine until the last month, when its a feast and all the desicions get made, because if they dont spend it, they loose it.

The reason why this is changing is not because its a light-bulb moment, its because its taken this long for the people up the chain to backed into a corner. Eric Smitch isnt the first person to think of this, his the first person with high enough clout to get it done (on a micro-level).

suffice to say, working in such a environment is toxic for your career in software development. unless u want to work on these secret networks all your life, on the same software, maintaining it, its just a dead end. Want to use that new web tech? Nup. The SoE takes years to update, you still need to support IE8 for the next decade.

For what it's worth, Pivotal Labs bills on a time & materials basis. But we're not billing to create software, we're billing to help you learn how to build it yourself -- and to teach your own to do the same.

Disclosure: I work for Pivotal in R&D.

When theory meets reality, requirements change. We thought this approach would work, and our research suggested it would. Then the real world came along.

No plan survives first contact with the enemy?

The more general version: "No plan survives first contact with reality."

Somebody once also said that planning is still very useful even if the plan changes later because it helps those involved prepare for and understand the changes better when they do come.

Sounds like more to do with not taking battlefield conditions/training into consideration.

When push comes to shove, if you give people cool tech, unless it's easy to mindlessly pick up and use they won't use it. When I was in, I had boxes and boxes of crap that was on the books that we never used (and didn't know how to use but had to inventory), then the stuff we actually used.

"Everyone has a plan 'till they get punched in the mouth." - Mike Tyson

What is most interesting about this article is the comments here considering what we've recently seen RE Google. Where are the calls from HN commenters suggesting that the engineers refuse to work on this? Where are the commenters questioning the morals and ethics of the engineers working on this? Where are the comments on here calling for the engineers working on this to quit in protest, or be deemed a collaborator?

Or does that line of thought only apply when it is everyone's favourite bogeyman Google that is contracted to replace manual work with automated systems? If its not Google, then is it A-Okay to work on military systems?

Both this and the Google project are IT systems that save manual, error-prone, and time-consuming work to enable the US military to more efficiently kill people.

I know I'll get down-voted to hell on this, but please if you're going to try and take the moral high ground at least do so consistently.

>Where are the calls from HN commenters suggesting that the engineers refuse to work on this?

shrug I find that the state monopoly on violence to be much more justified than some of the perfectly-legal ways people in the private sector choose to exploit their fellow humans for profit. -- At the very least, we treat the state's exercise of violence as a necessary evil to be minimized.

I found it darkly-humorus that nothing short of actual military contracting could wake up oh-so-progressive googlers to some "are we being evil" introspection.

I want software developers to collectively adopt an ethical standard and stick to it. I really, really do. But the only way I'm able to cope with what I've been witnessing for the last 15 years is to aim scorn at us all for being rationalizing frauds.

> I know I'll get down-voted to hell on this, but please if you're going to try and take the moral high ground at least do so consistently.

I feel that saving money on tanker planning is on a different ethical level from creating drones that can kill people upon facial recognition.

Disclosure: I work for Pivotal in R&D.

You should qualify which engineers you are refering to. From the article it's clear that the majority of the coding is "by a fast-moving cadre of Air Force engineers". They are Forces personnel, not civilians. Yes, Pivotal are involved, equally it seems if they are practicing XP, perhaps now only as Agile Coaches.

I also believe the applications this team are creating are substantively different in intent to those the Google employees were complaining about.

Hey, I work for Pivotal and I'm on an Kessel Run Air Force engagement. We teach things by doing them. That's why we're effective. So we are involved heavily in the making of software, we've always been. Our main schtick is that we try to make sure that we aren't necessary to the software development process when we roll off.

There are actually a couple of things that are the reason why a lot of people don't think it's too much of a problem (some people do and I work out of an office that does a lot of government work so we hear more of this than others do). The main reason is that there isn't pressure for a pivot to work on a project. If the project makes you feel uncomfortable request to be on something else. Leadership has been very thoughtful in addressing how people feel about working on military and government projects (especially with the new administration).

The other reason is that the applications are pretty different in intent, from what I understand, to what was complained by Google employees. It honestly feels like very standard enterprise issues. We're helping them cut through a lot of the same issues anyone would find at large enterprises.

You're talking about a company that's motto use to be "Don't be evil" and who has been referred to as the Chocolate Factory (a la Willy Wonka). Google isn't historically a military contractor or a systems contractor in general. All of their advertising, keynotes and press conferences center around connecting and helping people, not helping to develop weapons. Culturally they're a friendly company trying to make the world a better place. Presumably their employees didn't seek employment with Google under the impression they'd ever be developing technologies to aid the military in carrying out it's mission. So why is it so difficult for you to comprehend the backlash or cultural resistance to this?

This would be like Coca Cola announcing it's new chemical weapons division.

They are military personnel, writing code for military applications. They are under the obligation to refuse to execute orders that go against international conventions and have an obligation to respond to Congress/Senate inquiries. Their superiors, who might want to pressure them into doing things that are illegal or don't align with the US stance can be court-marshaled.

Good luck getting anywhere near that level of responsibility from Google execs or managers.

This story brings back a lot of memories. Right before I separated from the USAF in 1997, I was working with a tanker squadron deployed in support of operations in Bosnia. Crews would spend hours parsing the Air Tasking Orders to figure out when to take off, and with how much fuel. I had just gotten in to programming and wrote a simple VB6 app to parse the huge file and produce a simple report with the details they were after. It saved the crews hours, and I was still getting feature requests on the app for a year after I separated. It's simply amazing it took another 20 years for them to solve the problem.

There is an, unfortunate, lack of continuity in many offices like that in the military. This is due (as you're aware, this statement is more for others) to the way that personnel get moved within the military. Officers, in particular, as the ones who are typically the "decision makers" for selecting what projects to work on will be in their job for 2-3 years at a time. If they don't have a chief or a civilian staffer who acts as a continuity officer, projects like what you did get forgotten.

"Why are we still using this VB6 thing after 15 years? We should put out a contract and have it done properly." says some colonel or major who doesn't know better. And so it happens, a contract is awarded and it's a mess. I saw it more from the civilian side. Without strong continuity officers, when the military leadership changed over the direction and intent of things is forgotten.

Additionally, as this article points out, the military is set on Waterfall-styled contracts and projects, given to the lowest bidders. Which is why it becomes a mess once a contract is awarded. Waterfall only works under a few circumstances, and I guarantee your lowest bidder won't satisfy them all (expertise with the implementation technology, expertise in the problem domain, expertise with how the system is actually used, ideally a project that's been done 100x before by the same team).

Stories like this one should be understood not in isolation, but as one point in a pendulum system:

Endpoint A: These rules are all "red tape" stopping us from getting things done. Let's just get the job done and save lots of money, time, and overhead.

Endpoint B: Our systems are crap; they don't meet specifications - we don't even have specifications so we don't know what's needed; they are unreliable; they don't interoperate; there's no competition for the work, it just goes to incompetent people who have friends in the right places and rip off taxpayers; and military officers know nothing about procurement, bidding, or complex system development. This is ridiculous and unprofessional: We need specifications, competitive bidding, quality requirements, QA, and professional procurement for our $600 billion budget.


The problem isn't the endpoints, it's the pendulum. If this is your organization, you need to get off the pendulum and make expert, intelligent decisions about trade-offs.

EDIT: Some clarifying edits

Competent internal teams can produce relatively simple solutions within a reasonable budget. However, it's hard to formalize and replicate such teams and/or team habits. That's why the more formal specification & contract approach is often used instead: It's a BIT more predictable.

The first approach may get a grade from A to F, while the second ranges mostly from B- to D-. Predictable semi-crap is often easier to accept than something that might easily be an A, F, and all between. The first's probability curve is mostly flat from A to F, while the second peaks around C-, and thus it's somewhat more predictable, and that's why managers and bureaucrats often prefer it.

> Competent internal teams can produce relatively simple solutions within a reasonable budget. However, it's hard to formalize and replicate such teams and/or team habits.

Tangentially, that is exactly the military's central management challenge (per my limited understanding): Combat depends on replicating competence in internal teams, under extreme stress and extremely unpredictable situations, and with unpredictable turnover of key personnel, including management (they die). The U.S. military, for example, has to accomplish this across hundreds of thousands of personnel.

Imagine you get a new manager the week before a major project is due, with lots of critical decisions to be made. Now imagine you get a new manager the day that person is supposed to lead you on an assault up an enemy hill, where a less-than-optimal decision might kill you.

I think it's a really tough problem, and maintaining all the capability in house may not be the right move, but there must be some in house capability to use for prototyping and technical decision making, otherwise your procurement gets disastrous because you have no ability to evaluate what you're buying.

I enjoyed that, there mere mention of Shaw AFB brought back memories of serving there in the late 80s leaving just before the first Gulf "war" broke out.

anecdotal part goes here.

Being in IT back then was fascinating, the secure comm side systems were Burroughs of the generation just after tube computers went out of style. Complete with paper tape booting or switches and lights. The main system which I was on was a Sperry 1100/70 using cards as the main input from on base customers. Those transitioned into images on a 360k diskette when I was leaving. The system data was mostly stored on tape (autoloading reel) and there were three disk packs including one we joked was kick start (seriously, if you swapped packs and it didn't spin you faced away with your butt on the edge and swift flat to the side hit it with your boot).

It was that diskette that got me into programming. I started with Turbo C and Turbo Pascal settling on the second. The tool we were provided by Sperry would upload the card images one line at a time. On our Sperry branded PCs (same as IBM/XT) it could take longer to upload a deck than to run one through a card reader and the personelle group had the equivalent to three boxes. I ended up digging into the emulator software, the arcane manuals we had, and whooped up something in TP that did it in a tenth of the time. From that point on it was off to the races for my desire to program.

I once interned at a military base that used punched card readers. Some jobs (processes) would refuse to run if any card in the deck was detected as being bad (using counts and check-sums), such as misaligned holes. Being analog, different cards often were "bad" on repeat reads. It drove everybody crazy. Experienced card people knew all kinds of tricks to reduce such a problems. I learned to respect experience.

> To do so, a person called “the Gonker” entered data into an Excel spreadsheet known as “the Gonkulator,” while “the Planner” arranged magnetic pucks and plastic laminated cards on the whiteboard to indicate how long planes could stay in the air.

I love the names and vocab that comes out of unofficial military slang.

It came from Hogans Heroes and was generally adopted by the military as a "black box" that nobody understands. In my experience, it's predominantly used in the Air Force.

> “Just because you followed the software requirements spec doesn’t mean you met the capability need,” Kroger said.

Sage words.

I do wonder, though: becoming more agile (or agile AF as the case may be) is one thing, but is it healthy for an actual military branch to be moving in the direction of an industry whose unofficial motto is "move fast and break things"? That's one thing when the bad outcome might be that someone gets delivered the wrong thing or gets charged twice and you have to refund them. But broken things have much graver consequences when they involve guns and bombs and airplanes.

> in the direction of an industry whose unofficial motto is "move fast and break things"?

At Pivotal we have a different version: "go fast, forever". I believe Facebook updated theirs from "move fast and break things" to "move fast".

> But broken things have much graver consequences when they involve guns and bombs and airplanes.

We aren't to my knowledge participating in weapons systems or flight avionics (and I hope we never get involved in weapons systems).

Our work demonstrates a principle discovered in other industries a while ago: relentless focus on improvement breaks the iron triangle. Fast, affordable, good: pick any three.

But you can't do this by mouthing some slogans, skimming a book and then burping up code as fast as you can type. It takes a genuine and sustained discipline to keep to the core practices: pairing, TDD, CI/CD, IPMs, retrospectives, user-centred design, lean product management, balanced teams etc. But keep to it and the payoff in sustainable pace is remarkable.

Disclosure: I work for Pivotal in R&D.

I would have thought "move fast and break things" could almost be an Air Force motto!

The traditional outsourced approach seems to have been "Move slowly, waste money, and break everything."

My concern is that security is a potential issue, and short term savings could easily be offset by a massive security disaster later.

Enthusiasm isn't enough. I hope there's a standard, professionally tested, security framework being developed by experienced and capable security experts which other projects can rely on, as required.

> "move fast and break things"

Doesn't mean that you deliver broken code, it means that you should not be lazy to rewrite old code.

That's not what that quote means. It literally means it's okay to not get it right, to make mistakes, as long as you can deliver quickly [1]. Not that weird for an industry where time to market is everything.

[1] https://www.sfgate.com/news/article/Mark-Zuckerberg-Moving-F...

Speaking of stories of people coding in the military and in other U.S. bureaucracies, these came to mind:

How I got a medal from the Army for writing code (2014): https://news.ycombinator.com/item?id=7950190

How learning to code kept me sane when I was a diplomat (2014): https://news.ycombinator.com/item?id=8810589

A few months ago some Kessel Run folks gave (really fantastic) presentations[0][1] at CF Summit which are worth watching for deeper dives on the experience.

Disclosure: I work for Pivotal, though in R&D.

[0] https://www.youtube.com/watch?v=V8w4jf5clJk

[1] https://www.youtube.com/watch?v=lCgxdzoJGqE

From the article: 'In addition to the name, the team chose a hashtag for its project that is a nod to both agile software development, and the fact that Kessel Run’s culture is more startup than strictly military: #agileAF

“The AF stands for Air Force,” Furtado assured me.'

I wonder where the reporter's scepticism comes from...

It could otherwise be read as "agile as f*"

Yeah. I think they know that. They always chuckle when we bring up the hashtag. I'm on a Kessel Run project now and they are great clients.

There's a few people like Will Roper (former MIT prof) and Raj Shah in the military that are introducing agile thinking into the military, which it desperately needs. Other countries are catching up and it's fast becoming the case that software is the only thing that really matters in warfare. http://www.foxnews.com/tech/2018/06/07/f-35-to-2070-air-forc...

It’s a very interesting look behind the curtain at what’s wrong with the procurement and development process but also reads like a Pivotal Labs puff piece.

"“It hasn’t been peacetime in a long time,” I was told."

If we don't have peace right now, how can we ever have peace? Is eternal war finally there?

Seems like NASA's europa-pso code is right up the alley for what the tanker scheduling needs and is already out there collecting bit-rot: https://github.com/nasa/europa

Think there was a follow-on project but can't remember the name of it.

Interesting there is no discussion about Pivotal and the ethics of optimizing war machines, given that only a couple of weeks ago there was a big discussion about Google employees successfully stopping the company from working on drones.

Possibly because the applications are qualitatively different; possibly because the software is being produced by military personnel (aided by civilian advisors).

Soon: "This should be contracted out, we'd save a lot of taxpayer money".

Later: Contractors take over costing much more.

No one commented yet on the title "... saved the Pentagon millions". Compared to 700B a year (budgeted, more is spent), millions is sort of less inspirational.

The Army has Jyn Erso. The Air Force has the Kessel Run Experimentation Lab. Is Disney sponsoring the DoD now?

I had a literal Laugh out Loud at the #AgileAF (Agile Air Force) hashtag.

overall, did the Air Force spend more or less?

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact