
How Healthcare.gov Botched $600M Worth of Contracts - petethomas
http://www.bloomberg.com/news/articles/2015-09-15/how-healthcare-gov-botched-600-million-worth-of-contracts
======
brandonb
Hey HN! I worked on the "tech surge" to save healthcare.gov, and as many of
you have noticed, this report gets everything backwards. A 96-hour training
course would not have saved healthcare.gov, and you can't fix bureaucracy with
more bureaucracy.

If you want to understand what went wrong on healthcare.gov and how it was
fixed, Mikey Dickerson's talk here is the closest thing we have to a
postmortem:

    
    
      https://www.youtube.com/watch?v=0albm_hhQzM
    

healthcare.gov has also given rise to new initiatives to fix the way the
government writes software. Within the government, there is the US Digital
Service, 18F, and the Presidential Innovation Fellows:

    
    
      https://medium.com/@USDigitalService/an-improbable-public-interest-start-up-6f9a54712411
    
      18f.gsa.gov
    
      http://www.jasonshen.com/2015/when-did-you-do-your-tour-of-duty/
    
      http://www.jasonshen.com/2014/everything-wanted-know-presidential-innovation-fellowship-afraid-ask/
    

There are also new types of government contractors, including Nava and Ad Hoc
LLC:

    
    
      http://www.fastcodesign.com/welcome.html?destination=http://www.fastcodesign.com/3047856/innovation-by-design/meet-nava-a-startup-that-wants-to-fix-the-governments-crappy-design
    
      http://adhocteam.us/

~~~
tssva
This is one of many audits conducted as part of the IG's review of the
healthcare.gov rollout. The purpose of this audit was to determine if existing
rules regarding contract management were followed.

It in no way claims that taking a 96 hour training course would have fixed the
rollout of healthcare.gov or even that the contract management failings were
the primary reason for the failure of the rollout.

~~~
powera
I hate to say that the (lousy) article claims exactly what you say the audit
doesn't claim. It specifically describes "how shoddy oversight by the Centers
for Medicare and Medicaid Services (CMS), which manages federal health
programs including Obamacare, contributed to the website’s early struggles."

~~~
jcranmer
"contributed to" ≠ "caused"

------
Fede_V
The solution proposed to fix this mess is insane. A '96 hour training course'
at the end of which you are able to handle 100+ million projects? That's 2
weeks of work + change. Does anyone think that they could end up qualified to
handle contracts of that size after a generic 2 weeks course?

Of the people that enroll in this program, what % fail? If it's serious, the %
of people qualified should be at most 50% - yet it's obviously going to be
just a rubberstamp where you put in the hours, then get the certification.

As someone who firmly believes in the power of the state to do good, these
things make me go insane. The solution to failures of this magnitude are not
to pile on more useless regulations and certifications, or to put even more
emphasis on useless credentialism. Healthcare.gov got fixed when they gave
carte blanche to highly qualified engineers and managers that gave up very
lucrative jobs to help fix things.

I am convinced that lots more people would be willing to give up their cushy
tech jobs for a job where they genuinely made a huge difference in the lives
of people, provided you got rid of all this completely meaningless hoop
jumping. Give clear lines of responsibility, take a serious look at your
hiring process, and jesus, if your solution to a 600+ million dollar failure
is to mandate a 2 week course... you really need to think things over.

~~~
droopyEyelids
Glad I wasn't the only one who balked at the idea it would have gone well if
everyone had more training.

I think we really need intelligent, experienced people who have a proven
record in large product management. However, they're rare and more expensive
than the government will pay for. Compounding the failure is the fact that you
can't get fired from government jobs for incompetence. These projects aren't
going to get any better.

~~~
pbhjpbhj
>you can't get fired from government jobs for incompetence. //

This seems highly unlikely, can you expand further, some proof? [I assume
you're talking about US state and federal government?]

~~~
protonfish
It seems to be a property not just of government, but any large organization
that becomes driven by politics over results. I just quit a job in a large
corporation where I had the displeasure of observing this up close. I have
noticed several key dysfunctions in a political hierarchy.

1\. You cannot be smarter than your superiors.

After several layers of this, this cumulative effect is that the people on the
bottom (those that do the actual work) are not allowed to be smarter than a
sea sponge.

2\. Reporting bad news to superiors is punished.

Problems are covered up instead of solved. Reports up the chain must hide
problems without being provable lies so they are incomprehensible - they must
appear legitimate but be devoid of all fact. Using as much jargon and acronyms
as possible is effective.

3\. Blame for failure will be placed upon the least popular person.

Typically this will be whoever is drawing attention to problems. The reason
for project failure will be because of "bad attitude" and not being a "team
player" leaving the next project with a greater proportion of ineffectual suck
ups.

~~~
gaius
_1\. You cannot be smarter than your superiors._

At one company I worked for, you couldn't be taller than your superiors
either.

~~~
protonfish
I've seen that effect, but I don't think I've ever seen it as rigidly enforced
as this. It does illustrate how most hierarchical organizations are nothing
more than adult versions of king-of-the-sandbox.

~~~
gaius
It was never a formal policy - but it was common knowledge that you shouldn't
even hire someone taller than yourself because they'd surely leapfrog you for
promotion.

------
OliverJones
I find it interesting that this audit only covers administrative contract
oversight. The failings uncovered seem to have been of the variety "Contractor
A didn't submit monthly reports for February and March". I don't claim that
contracts of that size should have poor administrative oversight, not at all.
Even movies with big budgets mention the assistant producers and accountants
in the credits.

But what about technical oversight? Were there design-review meetings where
somebody said "our system emits data in HL7" and somebody else said, "we
haven't defined the XML schema for our data acceptance module yet." In a
meeting like that somebody in tech oversight should have said "wait! is it HL7
or XML? Who is going to sort this out?"

(I have no idea whether this actual issue came. Probably the stuff that
actually came up, or should have, is far more intricate.)

The purpose of administrative monthly reports' "issues" section is to track
stuff like like this. That's why contracts require them. If the government can
make the Apollo program work, the government can figure out technical
oversight. It just takes the will to do so.

~~~
viraptor
I see it as a high/low level oversight. If people don't know who controls the
project and how to measure the progress on it, then who would even care about
technical reports? That's like checking the details on a body with a head
missing - maybe the guts are OK, maybe not. He's dead anyway.

------
jakozaur
Lesson for government: Software is not a typical infrastructure project (e.g.
bridge). It is almost impossible to specify all requirement upfront. That's
why waterfall methodology is prone to spectacular failures.

Training government officials how to manage those project can help, but I
doubt it will make a difference. It can rather give false sense of "fixing it"
without actually fixing it.

I would rather focus on hiring competent ppl and give them freedom to do that
right. Such as: [https://www.whitehouse.gov/digital/united-states-digital-
ser...](https://www.whitehouse.gov/digital/united-states-digital-service)

~~~
threeseed
I wouldn't lay blame at the government over this. I have seen countless large
implementations fail within enterprise companies i.e. massive overrun in
budgets, not meeting requirements.

And unfortunately waterfall is really the only approach for projects like this
i.e. you are having to select different third party vendor products up front
and integrate them with custom applications. You can't iterate in sprints when
you don't control the software.

And if only hiring competent people was so easy. Really talented IT
professionals are very hard to find and keep. The Digital Service you
mentioned really is a great step towards retaining talent and long overdue.

------
smoyer
I'm a bit shocked that anyone could believe there was actually $600M of
software to build. I know prices are inflated by the whole federal contracting
guidelines but I'd say these contracts were at least two orders of magnitude
greater than what they would have been in the private sector. And nothing
against our friends to the north, but why wouldn't we bid more of this out to
US companies? Perhaps those that satisfy the WO-MO requirements?

~~~
rwallace
Okay but aren't you implicitly assuming a sane set of specifications? I've
seen it claimed that the input legislation was forty thousand pages. Can't
confirm that of my own knowledge, but if it's true, far from criticizing the
management of the implementation, we should be holding it up as a miracle that
it was ever successfully implemented at all for any price.

~~~
smoyer
I've seen that too ... and it probably cost many millions of dollars to come
up with the ambiguous and likely contradictory specification. People always
want to blame the waterfall method for this, but there are a couple aspects to
BDUF that I think can help even the most agile project. From the DOD world,
Interface Control Documents (ICDs) are a great way to make sure components
(especially those developed by different teams) inter-operate. There should
also be a reasonably cohesive architectural vision. From that starting point,
write some user stories and adapt as needed for each sprint.

You need to know approximately where you're headed, but you almost certainly
_CAN 'T_ know everything you'll learn ahead of time (as this sort of
specification implies.

As an aside, contractors love ambiguities in specifications and it's actually
in their best interest (unless operating on a fixed-price contract) to
purposely produce something that meets the specification but is obviously not
what the customer needs. There's good money in ECRs.

------
reustle
This seems to put most of the blame on the gov workers managing the contracts,
and very little on the contracted companies themselves. I would find it hard
to believe that they also weren't absolutely sure what they were milking.

~~~
kctess5
It sounds like the people hiring the contractors didn't exactly do their due
diligence.

> unclear who is really in charge, sometimes it's no one

> vague deliverables from people who don't know what they're talking about

> tons of cash, regularly, with no end in sight

Sounds like an incredibly profitable and easy contract. It's an ethical grey
area, but it's not surprising that companies took advantage.

~~~
OliverJones
"easy contract?" I doubt it. Ten million users on the go-live date?
Interchange of health-care data from system to system? Getting Oracle's
identity / authentication management system integrated as a door-one
requirement? Hostile congresscritters doing everything in their power to
torpedo the project? The only difference between this and a typical giant
cluster#uck is the vast and visible scale.

------
devit
Looks like they should look at who stole the money.

Even a very generous 100 engineer team paid at 300k/year for 3 years only
costs 100 millions.

If no one is looking into it, it's reasonable to assume that whoever is
supposed to look into it is the one who stole the money.

~~~
thaumasiotes
> If no one is looking into it, it's reasonable to assume that whoever is
> supposed to look into it is the one who stole the money.

While this isn't exactly unreasonable, I don't see why it's such a slam dunk
compared to "whoever is supposed to look into it is lazy" or "there is no one
who's supposed to look into it".

------
bingobob
its a joke software and governments here in Australia checkout out this mess.

1.5 billion on a Public transport metro card system
[https://en.wikipedia.org/wiki/Myki#Issues_and_criticisms](https://en.wikipedia.org/wiki/Myki#Issues_and_criticisms)

1.2 billion on a payroll systems WTF!!
[http://www.theaustralian.com.au/business/technology/queensla...](http://www.theaustralian.com.au/business/technology/queensland-
government-to-sue-ibm-over-health-payroll-
disaster/story-e6frgakx-1227140553687?nk=43303f5167de2715cd9df22efb653154-1442751558)

~~~
noir_lord
UK's Connecting for Health System

> Originally expected to cost £2.3 billion (bn) over three years, in June 2006
> the total cost was estimated by the National Audit Office to be £12.4bn over
> 10 years

Which is roughly 27bn Australian Dollars, when it comes to this stuff our
Government is in a class of it's own.

~~~
pbhjpbhj
IIRC the system was then abandoned as unworkable?

~~~
noir_lord
Most of it was, some bits where salvaged (I think, I followed it at the time)
at huge cost and then they built a newer version which sorta works.

In any base £20bn over 10 years seems like quite a lot on an initial budget of
2.4bn over 3.

------
DanielBMarkham
This will go into the big stack of incredibly large technology project
failures, including the FBI case management system and the IRS modernization
program.

From a systems point of view, the problem here is the publicity. For each
public disaster, there are many more that never get that much attention.

Just got through reading a biography of John Boyd. Great narrative about how
huge systems of people act politically to resist change and make decisions.

------
DanBC
Botched health IT projects aren't new The UK has had some horrifically
overspent and underdelivering health IT projects. It's a shame that people
don't appear to learn from other people's earlier mistakes.

[http://www.telegraph.co.uk/news/health/news/8780566/Disastro...](http://www.telegraph.co.uk/news/health/news/8780566/Disastrous-11.4bn-
NHS-IT-programme-to-be-abandoned.html)

£11.4bn !

~~~
gaius
This is why I laugh when people complain about the cost of a Trident
replacement... You could pay for it out of the NHS IT budget and neither
patients nor medical staff would even notice.

------
sylvinus
You can watch a very entertaining talk from one of the Google engineers who
went to fix the website at the time: [https://www.youtube.com/watch?v=GLQyj-
kBRdo](https://www.youtube.com/watch?v=GLQyj-kBRdo)

~~~
nodesocket
The most ridiculous part, they didn't have a way to run distributed commands
over ssh. When a filesystem mounted in read-only mode, somebody had to ssh
into each of the 175 servers manually and run commands to fix it.

~~~
psykovsky
What do you mean by "distributed commands over ssh"?

~~~
garrettgrimsley
Say you want to run a command on a group of machines using SSH. Rather than
manually running the command on each remote machine you can automate their
execution over the group using what is known as distributed commands.

[http://www.tecmint.com/using-dsh-distributed-shell-to-run-
li...](http://www.tecmint.com/using-dsh-distributed-shell-to-run-linux-
commands-across-multiple-machines/)

[http://www.ibm.com/developerworks/aix/library/au-
satdistadmi...](http://www.ibm.com/developerworks/aix/library/au-
satdistadmin/#N100E1)

------
PretzelFisch
600m and the gov still has to take the contractor with the lowest bid still?
Makes me think the requirements were so broken no one really wanted the work.

------
dangero
Tangential question, how do software firms get contracts like this? Seems like
a good business to be in.

~~~
thrownaway2424
Navigating the byzantine contracting system is the only skill these companies
have. They can't do the job, but they can fill out the forms, which is a rare
skill.

~~~
psykovsky
You mean reading and following the instructions manual is hard work beyond the
reach of a normal human being??

~~~
thrownaway2424
If you think it's easy to win government contracts then I invite you to do so.
It can be lucrative.

------
pbreit
This is why I wonder how much we need more and better coders versus better
managers?

~~~
ownagefool
You need both.

The biggest issue I take with good coders lead by non-technical managers is
they invariably convince managment they need to rewrite everything in the
flavour of the month or architect in requirements so they can use technologies
they're interested in.

Inevitably, these guys are now qualified with a whole new bunch of tools, so
they move on and hopping about the industry writing a whole bunch of software
that relies on a datastore that doesn't actually save data consistently, for
example.

------
mentos
Could this project have been completed by the likes of Google?

------
nolepointer
I expect nothing less from the federal government.

------
spoiledtechie
Disclaimer: Highly political rant.

You mean to tell me a company that got selected a no bid contract from their
connections in the Obama administration couldn't do an adequate job?

Crony politics at its best.

