
How a team of young people helped rebuild healthcare.gov (2015) - rmason
http://www.theatlantic.com/technology/archive/2015/07/the-secret-startup-saved-healthcare-gov-the-worst-website-in-america/397784/?utm_source=atlfb&amp;single_page=true
======
JustSomeNobody
As a developer in my 40's, I see things like this and worry that the software
industry will come to rely only on the young because they don't ... I don't
want to say care ... but put as much emphasis maybe? ... on work-life balance.

Developement-wise, I can hang with the 20 something crowd, no problems. I just
can't compete with the single/no kids thing.

~~~
ryanSrich
As a married person with no kids I feel the same way. I'm not 40, or even 30
yet, but I look at my peers and wonder if they'll grow to regret putting in 60
hour work weeks just to make someone else rich. I'm guilty of doing it too,
for sure, but it wears you down quick.

~~~
andyidsinga
Being in my 40s and having been there (in my 20s) I can tell you they _will_
grow to regret it.

That said, its up to people like me running teams and businesses to set the
tone - encourage balance while not squandering inspired work.

Re inspired work: I once made the mistake of telling one of my employees "slow
down a little, i don't want you to burn out and hate working here". He had an
astute response: "Andy, I'm inspired I have to keep working on this until its
done!" He worked weird hours - night and day for 2 or 3 days ..and when it was
done I remember him needing some well deserved R&R.

The message from there on out to my teams is - if you're inspired, work. When
you're done, rest. I'm not watching when you roll in in the morning, I'm
watching the team's accomplishment and morale.

~~~
zghst
You must be one awesome boss

~~~
andyidsinga
well -- unfortunately not. I have plenty of bad habits and ways I drive my
coworkers nuts.

------
rmason
The entire culture has to change. Had a female programmer who works in
Washington D.C. tell me a few years back the unwritten rule is that unless
you're at one of the top rungs of the GS federal pay scale you aren't supposed
to talk at all.

She had a boss who met with developers and told them nobody cares about your
opinions. You see million dollar mistakes being made and you either accept it
or leave government which is what she did.

~~~
dforrestwilson1
I too experienced several million dollar efforts that went nowhere.
Governments in general are not geared to think about returns on investment,
but our current one is quite terrible.

~~~
wslack
I think government, in general, is built to be slow and deliberate, but as the
pace of tech change has accelerated, the responsiveness of government hasn't
kept up. Git was invested in 2005, for example.

ref: [https://bhc3.files.wordpress.com/2008/05/consumption-
spreads...](https://bhc3.files.wordpress.com/2008/05/consumption-spreads-
faster-today.png)

~~~
vaadu
The government is geared towards risk aversion and ensuring that the friends
of the people at the top are taken care of.

Results are secondary. This is possible because there's no accountability for
not getting results.

------
whamlastxmas
Since I find non-descriptive headlines like this annoying (edit: looks like
it's been changed on HN to be much better), here is a quote from the article
that surprisingly provides its own TL;DR:

>Here is the tl;dr version of their story: Marketplace Lite, or “MPL” as they
came to be known, devoted months to rewriting Healthcare.gov functions in
full, working as a startup within the government and replacing contractor-made
apps with ones costing one-fiftieth of the price. And when, nearly a year
after the initial launch of Healthcare.gov, the website’s second open-
enrollment proved much healthier than its first, it was the MPL team who
celebrated.

~~~
dak1
And perhaps the biggest takeaway of the whole article, buried in the middle of
it:

> Instead, if the successes of the MPL team confirm any guiding principle for
> the future, it is this: Technical workers—not only engineers but
> designers—have to be involved with a process from the beginning. They will
> know that features must be described separately from needs, and that, when
> building software, smaller teams often perform better than larger teams.

------
lostcolony
Speaking as someone who worked in the past for a government contractor (DoD,
so these issues may not be entirely the same with the feds, but I bet they're
very similar), the issue isn't solely that bureaucrats don't get tech, it's
also that the bureaucratic red tape that evolved to try and keep costs down
has instead led to them rising, -especially- with software.

From what I saw, government contracting requires all requirements up front. It
then floats a bid, and the cheapest bidder wins. Whether or not the contract
has been met is determined by that list of requirements; even if you fail to
deliver on a non-functional requirement that is obvious ("the system should be
able to scale out and handle X simultaneous users"), you aren't penalized if
it wasn't one of the requirements. In fact, delivering on obvious things that
aren't part of the contract -hurts the contractor-; it requires development
time and effort (and thus eats into profits), and the government is legally
prevented from taking such things into account on future contracts (that is,
they can't say "Well, company A is the cheapest, but the last contract they
took was a broken piece of shit; company B is a little pricier, but they
delivered quality, we'll go with company B". No, you go with the cheapest,
period, because if they fail to deliver what you specify you can sue, and if
they fail to deliver what you -wanted-, but failed to specify, too bad). It's
possible to renegotiate the contract mid-stream, but it's expensive and time
consuming.

So the contractor has every incentive to make sure that there is no time spent
on quality, security, stability, scalability, etc, unless that's -explicitly-
what was asked for. This article points out that having developers involved
from the get go on the Healthare.gov rework led to it being successful, and
that's very much true; you need technical people who care about more than just
making bank to point out non-functional requirements that are missing. And, in
general, trying to list all requirements up front is a fool's game, but such
is the government's bidding process.

~~~
vonmoltke
I was a little too quick on the upvote for your comment. Your first paragraph
is spot-on, but your second is not.

> the government is legally prevented from taking such things into account on
> future contracts (that is, they can't say "Well, company A is the cheapest,
> but the last contract they took was a broken piece of shit; company B is a
> little pricier, but they delivered quality, we'll go with company B". No,
> you go with the cheapest, period

This is absolutely wrong. There is no requirement that the federal government
must go with the cheapest bidder or that they cannot take performance into
account. The government can and does regularly reject bids because they are
unrealistically low; in fact, I was on one DOD program where both bidders were
told to go home and come back with something realistic. Past performance
definitely factors in to what the government considers reasonable. It's one of
the reasons the government requires labor tracking even in situations where
the labor expended doesn't affect the final cost, such as fixed-price
contracts and uncompensated overtime.

~~~
lostcolony
That's some comfort then. I never saw any indication of it; across multiple
projects I never saw any drive to refactor, or any of the signs I associate
with developers taking pride in one's work (documentation above what is asked
for, root cause analysis even when a surface issue has been taken care of, or
really, any addressing of non-functional requirements that were obvious, but
which had not been demanded in the contract). Perhaps because they were mostly
long running projects that had already been bidden on, and just having the
domain knowledge meant that they'd continue to have the advantage on any
future bids to extend it.

Also, while I've seen bid wins/losses talked about from the perspective of the
quality of the solution, I've not seen them talked about having been won/loss
due to past performance. I hope you're right that that's a consideration, but
the lack of controls around quality, and the obvious apathy I saw, lends me to
think that if that's a consideration, it's not one the government knows how to
actually judge.

------
gabesmed
Hey all, i'm one of the team members of Nava, the public benefit corporation
that emerged from the MPL team to continue to improve how the government
serves its people...happy to answer any questions!

~~~
xir78
Are you able to talk about the changes you had to make to the login service?
Would be interested in knowing what the scaling issues were.

~~~
gabesmed
The core of the issue was that EIDM, the old system, was an assembly of
products designed for corporate intranets on the orders of 1000s of users --
it was never meant to be used for a consumer system. So all this super
complicated permissions logic was bogging everything down.

They mitigated the effects for a while by running it on bigger computers -
when we came in it was running on Oracle's BIGGEST computer, the Exadata. $6MM
per environment. Terabytes of ram. But still just completely bogged down by a
set of software not designed for its use case.

~~~
tptacek
Thanks for this detail; it's vivid and upsetting. One assumes that VAR pricing
was at play here as well: not only did the government get sold a pointless
super-computer, but the contractor made points on that sale.

~~~
gabesmed
Exactly! The root of all this (I think) is the contracting structure for
building jet fighters -- cost plus percentage -- was applied to software,
which has such a different cost structure. With that structure there is no
motivation to ship working software, or work efficiently at all. So people who
value that won't work for any company that behaves like that, hence the shift
of talent elsewhere.

~~~
poof131
God. I thought cost plus contracts were only used in exigent circumstances
such as war, but apparently they are getting used more now, such as for the
F-35.[1] Are software contracts really cost plus? These contracts shouldn’t be
used for fighter jets or anything else. They should be banned except in
critical situations and even then used sparingly. I remember stories about the
boondoggle in Iraq, $20 coke cans and such.[2] Really sad if these are being
used more now. Indicative of a lazy, corrupt government contracting process.

1\. [https://en.wikipedia.org/wiki/Cost-
plus_contract](https://en.wikipedia.org/wiki/Cost-plus_contract)

2\.
[https://en.wikipedia.org/wiki/Iraq_for_Sale:_The_War_Profite...](https://en.wikipedia.org/wiki/Iraq_for_Sale:_The_War_Profiteers)

------
bmmayer1
The sad thing about this is that said young people didn't get paid anywhere
near the $250-500 million paid to the contractors and cronies[1] who built the
broken healthcare.gov in the first place.

[1][http://www.bloomberg.com/news/articles/2014-08-28/all-the-
co...](http://www.bloomberg.com/news/articles/2014-08-28/all-the-companies-
making-money-from-healthcare-dot-gov-in-one-chart)

------
rev_null
The new site may be easier to use, but is also fraught with privacy
violations.

[https://www.eff.org/deeplinks/2015/01/healthcare.gov-
sends-p...](https://www.eff.org/deeplinks/2015/01/healthcare.gov-sends-
personal-data)

------
mcorrand
This is a hard problem to solve perfectly. I'm still stuck at the ID
verification step, after sending in my documents. Never heard back. I'm the
perfect corner case though, having gotten a green card recently, left a job,
moved to another state, so on so forth...

~~~
waterphone
Same, I was never able to get past this point, apparently because I don't
exist in Experian's system due to having no credit history. Attempts to
proceed beyond have never been successful the few times I've tried since, so
I've given up.

------
imh
>AWS is a data center of supercomputers

How the hell was this published?

~~~
ryanmarsh
How would you explain AWS to the layperson in a phrase?

~~~
slapresta
With words that have meanings? To the layperson, "data center" and
"supercomputers" translates to "tech stuff".

"AWS is a service by Amazon that runs software for you in computers connected
to the internet" is still inaccurate, but has a meaning for the layperson
other than "tech buzzwords with superlatives".

~~~
jaibot
Eh, it's not that inaccurate. AWS is a bunch of data centers full of big
computers that simulate a bunch of smaller computers.

~~~
killbrad
Can you say that in a generally digestible way?

~~~
boyaka
AWS is a computing service built on "datacenters" located all around the
world, which are large warehouses packed tightly with racks of computers that
are networked together and efficiently powered and cooled.

I'm missing a description of virtual machines / containers but that might be
going into too much detail.

------
alexmingoia
It's a disgrace that no one is held responsible for healthcare.gov. We get
article after article about "Silicon Valley" and their noble role in this shit
show, yet not a single article about the criminally idiotic and corrupt
politicians and their contractors.

------
tuna-piano
I was struck by this part

"The MPL team was not the ideal workforce: They were (and remain) contractors
to contractors. They were not protected by a union, nor did they enjoy the
many benefits of working as public employees. They are coders-for-hire who
could relocate across the country quickly, and they reflect the larger
industry they work in: Mostly young, mostly male, and highly educated. It does
not have to be this way. This kind of precarious employment—lucrative, quasi-
nomadic—is as much the result of poor planning as a natural consequence of
writing code for a living."

Why are any of those things not ideal? What's wrong with being contractors to
contractors,non-unionized, young, male or educated? Or am I misunderstanding
the author's point?

~~~
dastbe
I think the focus was on the temporary employment part. The government does
not need fly-by-night developers who come in when things are completely
broken. they need a strong technical core that builds things well and with low
costs the first time.

------
donpdonp
Upvoted for not using the term whiz-kids.

------
Tiquor
I'm curious how this team got the call for the project.

~~~
gabesmed
Todd Park, the ex-CTO of the United States, started to process of pulling in
Silicon Valley talent. We were called "Todd's kids" for the longest time :)

Todd reached out to the CEO of Civis Analytis (Dan Wagner I believe?), who had
pitched in with the Obama campaign. Through him Todd reached Mikey Dickerson,
who was at Google at the time (Mikey is now the head of USDS). Mikey looped in
a few other googlers...and so it spread.

I found the MPL team through friends in the startup world, many of whom were
ex YC founders

~~~
dpritchett
It's pretty wild to think they had to go to the opposite coast to source a
node/mysql/nginx solution. I know people who were doing such things much
closer to DC. Granted, we are lower profile and maybe less pre-certified as
"sure things". I wonder if borrowing Googlers is the government tech fire
fighting equivalent of "nobody ever got fired for choosing IBM"?

~~~
serge2k
> I wonder if borrowing Googlers is the government tech fire fighting
> equivalent of "nobody ever got fired for choosing IBM"?

probably. Reasonably high chance of getting the smarts you need, plus you
borrow a name.

------
tssva
The site can only be considered a success compared to what it replaced. It is
still slow, the workflow is confusing and this past enrollment period actually
being able to login was still an issue.

------
trhway
the original healthcare.gov was made using SCRUM. SCRUM fails by design on
projects involving more than one team with dependencies between them, and the
result in such cases is usually complete devastation like the original
healthcare.gov. There is no surprise that the [small] "team of young people"
were able to incrementally build a [simple] webapp and an app using SCRUM -
that is the sweetspot of SCRUM, pretty much the only situation where it
doesn't screws things.

------
kuharich
Previous discussion:
[https://news.ycombinator.com/item?id=9857662](https://news.ycombinator.com/item?id=9857662)

------
bogomipz
Interesting, does anybody know how or if this relates to 18F? Is there a
correlation? I ask because there is currently another post on HN today about
18F.

------
lllorddino
Half way into the article and still no mention of what the back end was like.
Sigh. Also I wonder why they weren't allowed to use AWS.

~~~
fucking_tragedy
I'd have serious problems if my healthcare information was processed and
stored on servers controlled by a company like Amazon.

~~~
killbrad
So, is it that you don't get how encryption works, or you think Amazon can
access that data somehow?

~~~
pjscott
If I ran servers which processed temporarily-plaintext health care
information, and I couldn't snoop on it even with effort, I'd be ashamed of my
poor skill at villainy. Encryption is not all-powerful.

------
beckler
I saw Mikey Dickerson talk about his experience in the initial recovery phase
of healthcare.gov at Velocity in 2014. I think it shows just how insane the
situation was.

[https://www.youtube.com/watch?v=7Vc8sxhy2I4](https://www.youtube.com/watch?v=7Vc8sxhy2I4)

------
shitgoose
101 comments into discussion and no one mentioned CGI? the actual morons that
fucked up the project and wasted 100's of millions of taxpayers' money? how
many times should they piss in our face till we actually start noticing
things?

~~~
annoncoward
I mentioned CGI..

------
mmaunder
'Young people' are so cute when they're productive. Who knew?

~~~
serge2k
But... entitled millenials...

------
cmonaghan
I'm a member of the Marketplace Lite team, and of the resulting public benefit
corporation, Nava. I'm happy to answer any questions!

~~~
rvanlaar
Do you think the success was caused by the team being young? Or just building
what the customer, i.e. the people really needed in an agile way?

~~~
cmonaghan
I think a key factor for our success was in being a small team, using a modern
technology stack, and building in an agile way. I think the article
overemphasizes the youth of the team. While many of us were quite young, our
team included individuals in their 40s and 50s. I think the skew towards youth
is more a byproduct of the risk profile of the project, and the need for us to
spend significant time in Baltimore/DC.

------
exabrial
After how many millions was given away though?

------
superswordfish
This article was written by Robinson Meyer, for anybody else confused by the
lack of a byline.

------
joshkpeterson
This team became Nava: [http://navahq.com/](http://navahq.com/)

