
Inside the Obama Tech Surge as It Hacks the Pentagon and VA - brensudol
https://backchannel.com/inside-the-obama-tech-surge-as-it-hacks-the-pentagon-and-va-8b439bc33ed1
======
cs702
The federal government hires a team of young, talented, motivated engineers
and managers, puts them in charge of failed software projects, gives them the
resources and authority they require to turns things around, and -- surprise!
-- it turns out they do a GREAT job.

In hindsight, this shouldn't be too surprising. What might be surprising is
that the same logic should apply to ALL government functions, not just
software development. Who says government projects and agencies _have to be_
poorly run? Who says we can't devise better systems and/or mechanisms (e.g.,
market-based mechanisms) for attracting and retaining great people in
government?

Throughout history, there are examples of government bureaucracies that have
been reasonably well run, in some cases for centuries. The Habsburg empire, in
particular, comes to mind:
[http://www.independent.co.uk/news/world/europe/habsburg-
empi...](http://www.independent.co.uk/news/world/europe/habsburg-empire-
territories-europe-have-higher-trust-in-justice-systems-a6893946.html)

~~~
liyanchang
Disclosure: I'm an engineer at USDS and these are my own opinions.

So in my admittedly short time in the government [0], I've witnessed how all
of these problems are due to good intentions. That's what makes this all
really tough because everything you think is bonkers actually has a reason.

The 1400 page travel regulations is a result of trying to prevent fraud -
every single issue that comes up results in a new rule.

The fact that it takes some projects years to deploy is that we would like to
plan and make sure that every resource is well-spent, that it's in a number of
languages and accessible to the blind.

It makes it hard for everyone - I've met lots of smart talented civil servants
and government contractors who want to do things differently but have their
hands tied behind their back.

[0] 2 years feels like forever to me but flash in the pan to many of the
dedicated civil servants I've met.

~~~
DannyBee
"The 1400 page travel regulations is a result of trying to prevent fraud -
every single issue that comes up results in a new rule."

This seems like a serious inability to understand that no process designed to
prevent future things you can't forsee is 100% effective (by definition). At
some point, you have to declare "good enough", and live with it until the
error rate becomes unacceptable overall again, then modify it.

IE it's likely 50 pages of those regulations gave them a 99.9%+ rate of
avoiding fraud. They then added 1350 pages to get to probably 99.99%

This is unlikely to be worth it.

(and yes, before someone points it out, i'm likely being generous with the
numbers)

~~~
sliverstorm
People are pretty sensitive about government financial workers committing
fraud, similar to how they are rather sensitive to government police
committing murder.

~~~
DannyBee
Sadly, in neither case will you ever have 100% compliance. Pretending it's
achievable, and trying to achieve it, is IMHO, silly.

Remember the _regulations_ do not prevent fraud, enforcement prevents fraud.
There already exist plenty of things saying it's not okay, etc. Saying "and
also, don't do that" is probably not actually necessary most of the time, in
the same way saying "don't shoot people" is sufficient. Saying "and also don't
shoot them while they are handcuffed" isn't necessary. Crappy post-
justification does mean the regulation was written wrong, and changing the
regulation to account for the post-justification will not actually improve the
process most of the time.

~~~
sliverstorm
I don't think we can take this much further without knowing what's actually in
the regulations, but I imagine they consist more of "officer's dash cam will
be run 24/7 and backed up in triplicate", "officer will learn proper gun
handling techniques X, Y, & Z", etc rather than "don't shoot people", "don't
shoot handcuffed people", "don't shoot clowns", "don't shoot children".

Or, in the fraud case, "books will be audited at frequency X", "Y behavior
makes it too easy to hide fraud and is not allowed". Rather than "fraud is
illegal on Monday", "fraud is also illegal on Tuesday", "fraud is even illegal
on holidays"...

Of course we can never achieve 100% with more regulation, but we make it more
of a priority to make abuse harder to get away with than elsewhere, presumably
increasing overhead in exchange for lowering abuse (yes, this is probably not
a strictly linear curve)

~~~
Bartweiss
In the travel regulations case, we _can_ see, and horrifyingly it's more of a
"fraud is illegal on Monday" situation:
[https://www.defensetravel.dod.mil/Docs/perdiem/JTR.pdf](https://www.defensetravel.dod.mil/Docs/perdiem/JTR.pdf)

There are some sensible regulations there, like having someone approve travel
requests, but there are also a lot of very narrow restrictions obviously added
by someone who wanted to prevent Fraud X, but lacked the authority to change
what was already written. The result is that you get more overhead with
depressingly little payoff.

In principle you're right about the trade-off, but that's only the case when
rule-writers have the authority to sensibly restructure what already exists.

------
glup
A note on personal experience as a tech contractor at a federal agency: I
worked as a contractor at the US Geological Survey (USGS), in the Department
of the Interior each summer between 2006 and 2010.

The most useful thing I worked on was a program for calculating air-sea carbon
flux: I did the OS X GUI and helped with the core logic (computing air-sea
carbon flux is nontrivial). At the time I was appalled that we spent $10k on
what could have just been an R package around a wrapper for the C++ core. But
since then the program has become a very common tool for ocean acidification
research, with ~175 citations, many of these have since been cited in all
sorts of climate change policy documents. It's admittedly hard to assess
value, but it's definitely more than 10k. And definitely a public good that
the market cannot be counted on to deliver.

I also found out at some point that the contractor I was working for (a
Fortune 500 engineering and defense company) billed the USGS more than twice
what I received (I got no real benefits). The federal employees I worked with
_detested_ the contracting company because they knew how much it was skimming
from the contract employees, many of whom were full-time and had been there
for years. When I figured this all out, I quit, started my own company, and
was hired directly as an independent contractor. Unfortunately, this was not
an option for most people, as this wouldn't have worked for a full-time, year-
round position.

~~~
liyanchang
Disclosure: I'm an engineer at USDS and these are my own opinions.

I'm personally appreciative of the work that you've done and your continued
work, in spite of the many challenges and frustrations. There are many ways to
serve. It's always incredibly rewarding to be working along side dedicated and
talented public servants and contractors. Thank you.

------
pjungwir
I'm curious if anyone here from USDS or 18F has deployed Postgres in a FIPS
140-2 environment, and if so what challenges they had. The VA seems to be
saying that Postgres should not be used:

[http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2](http://www.va.gov/TRM/ToolPage.asp?tid=5692&tab=2)

contrast that with Oracle:

[http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2](http://www.va.gov/TRM/ToolPage.asp?tid=9&tab=2)

(Don't miss the difference in tone on the "Analysis" tabs.)

I'm sure some of that is due to lobbyists, but nonetheless it seems to me that
there are legitimate challenges in making Postgres meet FIPS 140-2
requirements. I've been able to recompile Postgres to use the FIPS OpenSSL
wrapper, but storing passwords with MD5 is a harder issue to fix, and of
course there is no definitive list. Does anyone have some experience with
this?

~~~
saosebastiao
I'm pretty sure the only OS that is approved for electronic voting systems is
Windows XP, which has obviously become a problem, but a new OS would likely
take two or more years to go through the approval process. Ubuntu changes so
much in two years that by the time it is approved, it would have lost 2/3rds
of its life even as an LTR.

I would assume that the main reason that Oracle is preferred is due to
lobbying, but also the fact that they don't really innovate anymore so it is
non-trivial for them to provide support and bug fixes for a decade or two.

~~~
nickparker
Does anyone else find it extremely discomforting that the voting machines even
have a full fledged OS?

It's a throwback to this xkcd [https://xkcd.com/463/](https://xkcd.com/463/)

In my mind, the ideal electronic voting machine:

\- Has mechanical buttons which simultaneously punch a paper ballot in a
manner observable to the voter

\- Runs on a small (open source) microcontroller that's been audited for
backdoors.

\- Runs (open source) crypto directly on the metal

\- Publishes the results in a cryptographically verifiable way.

IMO, anything else is evidence of at least government/contractor ineptitude,
and potentially malfeasance.

~~~
saosebastiao
In terms of security, that is probably exactly correct (I'm a complete
security novice FWIW). Way too high of an attack surface area. That being
said, as soon as you get into microcontroller territory, you turn it into a
niche thing that makes it inaccessible to most. What are the chances of
finding talent that can write user-facing code that runs bug free on
microcontrollers while simultaneously writing/proving the crypto? Maybe that
talent is more common than I suspect, but I certainly haven't seen it. I would
be happy with rust/ocaml on a unikernel or rump kernel though :)

~~~
vonmoltke
Why does dropping the OS mean you have to run on microcontrollers and write
everything from scratch? All it means is your software needs to incorporate
the OS functions that it needs for interacting with the hardware. You can
still use any hardware platform you want and any third-party libraries
available for said platform.

~~~
di4na
Which is exactly what micro controllers programming is.

I worked and still do some work for embedded programming. If you never did it
before : OS make life super super easy. Really.

~~~
vonmoltke
Well, yeah, most microcontrollers require bare-metal programming. That's not
my point. My point is any device can be developed for that way, and that going
OS-less does not mean having to write everything from scratch. After all,
that's what the OS _is_ : a bare-metal application that functions as
middleware (amongst other things) for the hardware platform.

Also, I know having an OS makes it _easy_. If you have full control over the
hardware platform and environment, and user interaction with the system is
limited to a relatively small set of UI objects, you shouldn't need a full OS.
Something bare-bones like DOS or FreeRTOS should be fine. Even on the box I
worked on that used RHEL we stripped almost everything out. I'm still not sure
why they decided to go with RHEL as the base, considering the amount of work
that went into customizing it.

------
saosebastiao
What I would love to see is an 18F-like organization that creates open source
software that helps user-facing government organizations at the state and
local level. All of these things should be simple but (typically) are not:

* Paying a utility bill online

* Registering to vote

* Finding out how to get from place to place using public transportation

* Paying for public transportation passes

* Signing up for unemployment benefits

* Proving residency

* Getting an ID card

* Registering a vehicle

* Paying taxes

* Entering pleas to local courts for minor offenses and infractions without taking a day off work

* Reporting non-emergency code violations (like construction noise after hours)

* Knowing who your representatives are

* Communicating with your representatives when you do know who they are

We actually use and interact with local and state governments far more than we
do with the federal government. And yet when it comes down to it, they are the
most difficult to work with because their resources are so constrained. I
honestly believe that local governments could benefit from some resource
allocation on their behalf at the federal level for the creation of highly
scalable technological public goods that will only ever be used at the local
level.

~~~
ondrae
I'm an engineer at Code for America -
[https://www.codeforamerica.org/](https://www.codeforamerica.org/) \- a non-
profit that does this kind of work with cities, counties, and states.

Its still pretty early days for civic technology. I think we'll see tech
provide all the services you listed, eventually. It'll probably be a mix of
outside vendors and local governments running their own engineering teams.
There are still some legal barriers in the way, procurement for example.

Its happening, just slowly.

If you are interested in speeding up the progress of your city, I'd recommend
checking out if there are any civic tech volunteer groups near you.
[https://www.codeforamerica.org/brigade/](https://www.codeforamerica.org/brigade/)

------
GVIrish
It's heartening to see some new approaches government IT actually getting high
level support to make a positive impact. Funnily enough I see someone I used
to work with in one of the pictures.

Hopefully this will lead to more substantive change to how government IT is
done. It's great to have teams that can drop into problem areas with a clear
mandate to get things moving, but the whole system needs some serious reforms
so that these IT quagmires don't get created in the first place.

This article heaps a lot of blame on contractors but fact of the matter is,
contractors are often hamstrung by what the customer wants them to do and what
the customer allows them to do. And some of that comes back to what was
specified in the original contract.

Ultimately there are multiple components to the abysmal state of government IT
projects:

* The government has moved to a model where very few full time employees are technical people. The idea is that almost all IT will be outsourced to the private sector

* Because fewer and fewer FTE's are technical, procurement and project management decision-making is compromised.

* There is little accountability for government employees, and by extension, contractors. Botched projects where millions of dollars are waste rarely have major career consequences for FTE's, and the bigger contracting companies don't have their ability to get new contracts affected at all.

* Building software for government often involves arduous interpretation of regulations, trying to build to undocumented and byzantine business processes, or waiting an inordinate amount of time for the bureaucracy to make important decisions (then change their mind!).

I feel like empowering the technical people is a very important first step,
and getting them executive backing may create the momentum needed to change
some of the common pitfalls. But I think there are some big changes that have
to happen to the whole ecosystem to really get things to where they need to
be.

It's all well and good to give small teams exceptional power to get around the
bureaucracy but that approach doesn't fix the underlying problems.

~~~
liyanchang
Disclosure: I'm an engineer at USDS and these are my own opinions.

You make a great point - USDS is just one part of the solution. In fact, in
almost all of the USDS projects, we work very closely with agency employees
and contractors (many of which are just as talented and have chosen to serve
their country). Most of the times, I spend very little time hands-on-keyboard
and helping empower the existing team.

As for the longer term solution, there is a less publicized version of what
folks are doing. USDS has a number of contracting officers who are helping
teach others in government how to be savvy customers of technology. 18F and
GSA have been doing a lot on this front as well to help bring in really good
contractors and writing agile purchasing agreements. The Office of Federal CIO
is rewriting and simplifying tech policy and the Office of Federal Procurement
Policy has been working on the procurement side. These are the long term
changes that I'm personally excited about.

~~~
GVIrish
That is great news, I think engaging and training COR's across government
could yield some real improvements to how these projects go.

I'm personally seeing some 18F projects come down the pipe that are truly
innovative and hopefully will encourage organizations to think differently
about IT procurement. Still a lot of silly stuff out there but at least the
ship is starting to change heading.

~~~
zacharycohn
What 18F projects are you excited about?

[Disclaimer: I work for 18F and will probably point those teams at this
comment so they can do a happy dance!]

------
dkrich
Unfortunately, there's quite a bit of misinformation in this article.

The author makes it sound as simple as hiring fresh hackers from Silicon
Valley to take over failed projects from stodgy software developers from the
large D.C. contracting firms.

Having worked on government projects like those cited in this article for
several years, I can tell you that it is the government agencies themselves,
and not so much the developers who are to blame. To build a cool iPhone app
for a private service, you can pretty much use whatever tools and timetables
you want.

Not so with government tech. With even small projects, you are beholden to
policies that were written several levels up. Things like no cloud-based
hosting, no sites that don't use government-written style guidelines, and in
many cases, no development that occurs outside the government facility. If you
hamstrung pretty much any private tech company with the same restrictions
government contractors have to deal with, I can guarantee you they would look
a lot different.

That's not to say that change can't occur, but to attack the contractors is to
misunderstand the root cause of why government software is historically so
bad. There are actually many, many very talented developers and software
engineers steeped in the latest lean startup theories inside the beltway, but
they can seldom use any of that because government policies stand in their
way. For most projects, you are dealing with literally thousands of
regulations throughout the development process, and most software apps have to
account for ever-changing laws and policies that are specific to each
application.

However I have already seen a great deal of positive change, with many
agencies opening their minds to newer, faster methods of development and
allowing better tools to be used.

~~~
jessriedel
So it's an accident that this involved fresh hackers from Silicon Valley?

~~~
dkrich
Not an accident- selectivity bias. Ask those hackers to build an app for the
USPS, FDA, the DOE or any other agency that imposes strict regulations and has
to implement business logic to account for tens of thousands of rules (not an
exaggeration) and see how agile and innovative they are. There's a reason that
most of these contracts are worth between 7 and 8 figures. That's the kind of
manpower required to engineer them.

~~~
jessriedel
The point is that you went from v1 to v2, and the major thing that changed was
the age and attitude of the team. If that sort of dramatic improvement was
only conditional on other aspects of the situation, fine, but that doesn't
invalidate the article.

------
gumby
Great article. For me this is the money quote: “The culture is very
persecutory. If something goes wrong it’s usually followed by a witch-hunt,
and the result is very chilling. People are afraid to take risks.”

This applies top to bottom (and has nothing to do with the current rage
between republicans and democrats). The 1400 pages of travel regulations are a
classic symptom.

------
impostervt
"U.S. Digital Service salaries are set in accordance with the General Schedule
Pay Scale, which is capped at $160,300."

That's not bad, but not great for really high talent employees. I wonder how
many people get that rate though.

[https://www.usds.gov/join#compensation](https://www.usds.gov/join#compensation)

~~~
liyanchang
Disclosure: I'm an engineer at USDS and these are my own opinions.

That's right. The US Government is probably not going to win on compensation -
salary limits, no stock options, no lunch. However, while it's not a money
making enterprise, it's enough to do just fine. I recognize the sacrifices
that many of my colleagues have made to be here, not to mention the sacrifices
that federal employees and contractors have made (some who are very good and
could be making more in the private sector) and that makes this even more
worthwhile.

The thing that government can offer is impact. I've always known that
government has a big impact on people's lives but not sure if I could
personally make an impact. That's what USDS, 18F, a number of other
opportunities are offering.

This is not a job for everyone. But if you're a certain type, there's nothing
quite like it.

~~~
mattmcknight
The government could do a lot more by adjusting the differentials for
technical competence on the GS scale to be somewhere near market (up and
down). However, assessing technical competence isn't exactly straightforward.

------
elicash
I love this part:

Another project involves working on the Global Positioning System. Lynch still
can’t believe that he can work on something that cool while serving his
country. “I didn’t even know that we ran GPS, right?” he says. “That’s crazy
shit.”

~~~
MrMullen
It is funny to explain to people that don't know anything about GPS that the
government not only runs it, the Department of Defense is the steward of it. I
guess people assume that something as "cool" as GPS system can't be owned and
operated by the US government.

------
arh68
I'm excited, skeptical, and interested all at once.

When the hackers leave, who's going to end up doing maintenance on these
projects? It's not unusual for a gov't developer to work on a project just
long enough to be productive, only to have them leave for greener pastures
(often a promotion or relocation). In this case, most will want to stop
suffering at lower salary and go back to SV. So it seems like it's going to be
USDS holding the bag; unless there's a solid plan to transition the workload
back to the VA (either organic or contractor), then USDS will have to maintain
every project they start (forever).

On the other hand, I bet there are more than a couple good developers within
the VA that would love to take over the projects, if only there were a
permanent position they could do it from. Why can't Ash Carter set up more
permanent positions?

Worst case scenario, a new President comes in and sunsets USDS, requiring VA
to maintain the workload. VA balks, they hand it to Deloitte/BAH/etc, and
we're back to square one.

Best case scenario, USDS build quality solutions that last decades and can
become the next legacy system that VA builds on. And maybe USDS (if it's still
around) can re-revamp the system, etc etc.

Also, I really don't understand the lobbyist's angle on open source: isn't all
code written by a federal employee already in the public domain? (with obvious
exceptions re: security etc)

------
mbesto
My favorite part of this whole situation:

"Start Up Life Meets the Compliance Riddled Federal IT Market for 18F and
USDS" by A.R. "Trey" Hodgkins (ITI)

[https://www.itic.org/news-events/techwonk-blog/start-up-
life...](https://www.itic.org/news-events/techwonk-blog/start-up-life-meets-
the-compliance-riddled-federal-it-market-for-18f-and-usds)

 _18F’s and USDS’s heavy reliance on open source and custom coding raises
serious concerns. While open source is an entirely appropriate option for
agencies to consider when looking to acquire software solutions for mission
needs, we need to ensure that these are not the only options available as
different missions have different demands. Making open source software the
only preferred solution doesn’t adhere to the tenets of tech neutrality. It
also limits options and competition, which means we may not be achieving best
value for the taxpayer._

------
ryanmarsh
It's my _dream_ to work at USDS.

First I need to save enough money to cover the difference in salary for 2-3
years.

~~~
Klonoar
Mmmm, I've been through the process and gotten an offer, and it wasn't that
far off what the industry standard is (and honestly beats regular DC rates).
They're really solid overall.

Granted everyone's got their own experiences, so YMMV I guess. Just curious
where your data comes from?

~~~
ryanmarsh
I read (sorry I don't recall where) a comment on GS levels for USDS. It was
like half my current income. Perhaps I should just apply and find out for sure
though.

~~~
knowaveragejoe
[https://www.usds.gov/join#compensation](https://www.usds.gov/join#compensation)

------
kkhire
how cool would it be if this obama tech team really bolstered recruitment of
more software engineers into gov't?

I'm talking federal, state, and local level. Just like each state has senators
and reps; 1 engineer for every 1M residents that live in the state.

each state's teams would tackle problems with software, share solutions to
widespread, formulaic issues with one another. it'd be a iconic and fresh-
breath-of-air kind of way to speed up efficiency and effectiveness (government
and big corps are SO SLOW)

------
Klonoar
This article does a good job of showing it, but everyone talked about is
seriously amazing. I made it through all the interview pieces and ran into
some other life complications so wound up not getting to work with them, but
having been by their space in the Pentagon they're doing some of the coolest
stuff imaginable right now.

If you're looking for a new job, I'd highly recommend giving it a shot.
Definitely felt like an opportunity unmatched elsewhere right now.

------
vonmoltke
Great to hear Ash Carter is so behind this. That makes me want to sign up for
the Defense Digital Service, assuming they have work that isn't frontend-
heavy.

------
pp19dd
> These entities cannot continue to operate as both buyer and seller. There is
> an inherent conflict of interest when the entity that consults on how to
> solve a problem then also offers to sell you the solution. The government
> outlawed such arrangements for contractors, so they should not be permitted
> to create such a dynamic internally.

That's a very strong accusation from Hodgkins (linked blog post in the
article).

------
knowaveragejoe
Some really heartening news here. I hope it continues under the next president
and doesn't succumb to the lobbying efforts by the larger contractors.

------
infinite8s
There's something about the second photo showing their 'funky' office. That
looks like the team member was stuck in an unfinished basement (evoking
memories of Office Space).

[https://cdn-images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cG...](https://cdn-
images-1.medium.com/max/600/1*Og10aHFHpbFXrwD7cGYzkw.jpeg)

------
mtgx
I wonder how many of these people will end up being recruited by the NSA.
Sounds like the "NSA boycott" is about to be over soon.

[http://www.npr.org/2015/03/31/395829446/after-snowden-the-
ns...](http://www.npr.org/2015/03/31/395829446/after-snowden-the-nsa-faces-
recruitment-challenge)

