
Poorly Managed HealthCare.gov Construction Cost $840M, Watchdog Finds - malchow
http://online.wsj.com/articles/poorly-managed-healthcare-gov-construction-cost-840-million-watchdog-finds-1406751529?mod=WSJ_hpp_MIDDLENexttoWhatsNewsTop
======
brandonb
I was part of the "tech surge" brought in to help patch up healthcare.gov in
December. The GAO report gives a decent idea of what went wrong in 2010-2013
from a high-level contracting and policy perspective, but it doesn't really
give much technical detail or describe what was broken about the organization,
software architecture, and culture.

A couple of my colleagues have given talks on healthcare.gov from the
perspective of the engineers who worked around the clock to help get the site
running again:
[http://youtu.be/0albm_hhQzM?t=3m40s](http://youtu.be/0albm_hhQzM?t=3m40s)
[http://www.youtube.com/watch?v=GLQyj-
kBRdo](http://www.youtube.com/watch?v=GLQyj-kBRdo)

A small team is also re-writing the site, starting with the most troublesome
components for the next open enrollment period. Wired covered some of what is
going on here: [http://www.wired.com/2014/06/healthcare-gov-
revamp/](http://www.wired.com/2014/06/healthcare-gov-revamp/)

If any of you out there want to help, please email jobs@hcgov.us!

~~~
butner
Fair perspective, but the government is still trying to solve a lot of the
problems with closed-doors and the same contractors. There's a lot of startups
that are focused on solving the problems of getting people health coverage.

A bigger challenge is ensuring people get the right coverage for their
situation and understand the plan and provider network they're buying.

Disclaimer: I work at Stride Health, we help individuals buy plans in-context
of their doctors, drugs, and health conditions they're treating. We're also
hiring, sans the $840MM budget
([https://stridehealth.com/jobs](https://stridehealth.com/jobs))

------
vnchr
Consulting firms like Deloitte were taking payments of +$70 million[1] to do
"analysis" that "informed development and decision making." The associated
waste of resources is unfathomable in the startup world but seems to be common
with federal contracting.

1\. Tipsy Deloitte associate chatting at a Chicago bar

~~~
doubt_me
Why the hell does everything the government do have to go through a contract
and outsourced/ bid on?

Why don't they contract someone to come up with a better solution?

~~~
gregd
It's essentially a broken attempt to avoid nepotism and gives the illusion of
'fairness'. Having worked in two different state governments for over 10 years
of my IT career (some of it in management) I can assure you that most of the
people in positions of power aren't really forward thinking enough with
technology and IT.

Also, most governments have made a feeble attempt to move toward a 'one size
fits all' type of IT structure. This includes state run data centers, state-
wide Active Directory structures, procurements services, buying from approved
hardward and software vendors, yada, yada, yada. They didn't care that
software from approved vendor A met 48% of my needs list whereas software from
vendor B met 98% of my needs list at a fraction of the cost. Vendor A had been
approved by procurement and vendor B hadn't. How are we to know your Cousin
Billy doesn't own and run vendor B? How do we know you aren't receiving a
kickback for buying from vendor B?

Which leads me to another ridiculous rule under State Government...I wasn't
allowed to receive anything from a vendor (worth > $10) as a gift, if it
wasn't also available to the general public. So say I am interested in buying
a server for $4500 and I'm on the fence. The vendor wants to throw in 2 years
of 7X24 technical support for 2 years if I buy the server from them. NOPE. If
the general public can only buy the same server and get 5X8 for 1 year. I
wasn't allowed to even (on the fly) negotiate a better deal.

~~~
dragonwriter
I've spent quite a lot of time in state government, and agree with much of
what you say (but see edit, below) -- but your description in the last
paragraph seems to conflate common kinds of ethics rules on what _you_ as a
state employee are allowed to receive with rules for what the _state_ is
allowed to receive. Usually, the type of thing you discuss wouldn't fall under
the no-gift type of ethics rules that would usually have a token exception
like the under $10 one you mention, which usually apply to gifts to state
employees, especially contract decision makers. It might be problematic for
other reasons, though, under contract bidding rules (since essentially the
vendor would be bidding on a contract with a different scope of work than the
publicly announced RFP/RFO/IFB, and thus there wouldn't be fair competition
between bids for the same thing.)

EDIT: But not the initial comment about anti-corruption/anti-nepotism --
contracting out everything _isn 't_ a fix for nepotism/corruption -- indeed,
statutory rules _restricting_ contracting out (which, IME, state agencies are
_very_ good at working around when they aren't just ingoring them outright)
are attempts to restrict nepotism/corruption. More often, the contracting out
is because budget rules make it administratively simpler in practice, even if
the theoretical conditions that should apply are harder, to contract out then
to get additional state positions. (An example of how this can manifest is
rules which require legislative action _both_ to appropriate funds _and_ to
assign positions to an agency to perform that function, but where using
appropriated funds to contract out doesn't require separate legislative
action, even if it requires the agency to certify that the work cannot be done
by workers in any civil service classification.)

~~~
gregd
To put my comment into context, I'm already forced to purchase a server from
say Dell, off of their state website because they've _already_ been put on
price agreement, which means they've already bid on and _won_ the right to
sell to the state. There is no RFP, there is no other procurement process
other than to order it off of price agreement.

But having said that, even with companies on price agreement, I wasn't
_allowed_ to take anything else that they might offer me if it wasn't also
being given to the general population. A quick story: I received a really nice
backpack from HP when I bought one of their very first "convertible" tablets
(this would have been around 2008-09). When I purchased it, the site made no
mention of the "extras" and I had to tell my boss that I received these
without being aware that I was going to get them.

------
Ecio78
Each time I read about this I think about our dear Italian tourism portal
www.italia.it that costed us more than 58mil€ in 2004-2008 and was a huge
failure. According to wikipedia[1] we have then spent additional 10mil for
redoing it, plus 20mil for managing it in these years. Interesting for the HN
designers is probably the "nice" logo[2] that won a 100.000Euro logo
competition in 2006 (it was created by Landor, a US or UK brand design
company...) There were jokes about the green "t" that looked like a cucumber
(and in Italy the cucumber - "il cetriolo" \- has sometimes a double
meaning)..

[1]
[http://it.wikipedia.org/wiki/Italia.it](http://it.wikipedia.org/wiki/Italia.it)
[2]
[http://www.marcosansalone.com/creativeantblog/?p=310](http://www.marcosansalone.com/creativeantblog/?p=310)

------
malchow
Here is the full testimony to be delivered tomorrow:
[http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-...](http://docs.house.gov/meetings/IF/IF02/20140731/102587/HHRG-113-IF02-Wstate-
WoodsW-20140731.pdf)

------
colmmacc
Maybe this is facile, maybe it is illustrative, but I find it interesting to
compare HealthCare.gov and Twitter. Twitter has raised over a billion in
funding so far, has been going 8 years and has over 3,000 employees.

Healthcare.gov is at about 3 years, $840M, and I can't find a number for how
many people were involved, but it's likely also in the thousands.
Healthcare.gov has delivered life-altering value to about 6 million people and
cost about $150 per user (soon amortizable over at least two years). Twitter
has about 200 million active users getting everything from minor titillation
to revolutionary aid and has cost about $6 per user. Neither of those "prices"
seems shocking to me, and I'd pay $6 to use twitter and $150 to use
Healthcare.gov.

If anything, healthcare.gov seems cheap in that context. In terms of scale of
effort; if I took a cursory look at twitter's functionality, and even taking
scale into account [1], I'd spitball it as taking maybe 3 or 4 small dev teams
for the website and API, and another dev team per app; maybe 100 engineers
total. I'm sure they have more than that though; because a cursory spitball
guess is an insult and there's likely a lot of hidden complexity and business
logic discovered only you if try to do something like twitter very well. I'm
also sure there's a huge mass of employees dedicated to wet, human, problems
too like sales acquisition and support and what direction the shading should
be on the twitter eggs and why the logo should just be a bird.

And so it likely is with Healthcare.gov, interfacing with hundreds of
different healthcare providers, handling different administrations and so on;
probably a real deep mess. I'm impressed they got it working at all;
especially as when they set out it was never intended to be used much (as I
understand it the idea was that the state exchanges would handle the larger
burden). The frontend seems to have been shoddy, but how much of the expense
was that?

[1] I have built and operate services at the same scale as twitter.

~~~
chatmasta
You're neglecting to consider an important difference between Twitter and
Healthcare.gov, which is that they have completely different usage patterns.
Even if we assume Twitter and Healthcare.gov serve the same number of users,
the engineering and scalability challenges of Twitter far outweigh those of
Healthcare.gov. Active Twitter users engage with the platform daily, often
simultaneously with millions of other users, and expect state changes to
propagate across the platform immediately. Healthcare.gov, on the other hand,
barely even has a concept of "active" users. For the most part, each user only
engages with the website once at sign up, and then periodically thereafter.
Users do not have an expectation of immediate propagation of state change. As
long as the site works, nobody cares how long it takes for the accounting
system to update.

My point is that you're comparing apples and oranges. Equating raw user
numbers alone is not a fair comparison, because that's not what causes the
engineering problems that make Twitter hard to scale. It's the _usage
patterns_ of those users that create the challenges.

Healthcare.gov is one step up from a static website, and customers rarely
engage with it throughout their lifecycle. It's absurd to suggest that
developing it should cost anywhere near the same as developing Twitter.

~~~
colmmacc
> Even if we assume Twitter and Healthcare.gov serve the same number of users,
> the engineering and scalability challenges of Twitter far outweigh those of
> Healthcare.gov.

I may have a bias because of my background, but I absolutely don't agree.
Twitter is a very busy service, a planet-scale multi-way messaging bus isn't
trivial by any means, but that part is still a relatively well defined, small
in scope, engineering problem and something a small team of talented engineers
could tackle. That's probably the easiest part to account for.

Healthcare.gov on the other hand is gigantic in scope and faces many ill-
defined problems; "integrate" with 50 states and various other administrations
and insurance providers and so on to deal with, all of which are a moving
target.

In my experience the cost of having things poorly defined is something like
100x having something well-defined, and increasing scope is a lot more costly
than increasing scale.

~~~
chatmasta
Ah, I understand your point better now. You're comparing scope vs. scale as an
engineering bottleneck. That does make sense, simply because number of
engineers need to increase linearly with scope, but not necessarily with
scale. Put another way, scalability problems are a subset of scope problems.

Still, I get the sense that the government threw way more money and engineers
at this than necessary. The fact that a few relatively small "repair teams"
could join the project after development and fix all the problems, makes me
wonder if those problems really needed to exist in the first place.

Perhaps it would have been better to group these 1000's of engineers into
smaller groups of ~100, and have them compete with each other to implement
specs in the fastest/best way. Pay each engineer a base salary, and then
allocate the remaining money to a bonus system that incentivizes fast and
quality development.

All that said, I'm talking out of my ass. :)

------
Daishiman
It's amazing the amount of downvotes on posts that comment just how
_tremendously_ difficult it is to plan, design, code, test, and support a
website that has to serve literally hundreds of use cases and is meant to last
multiple decades and integrate with thousands of other government systems that
have been standing since the 60s and all have their own idiosyncracies in
protocols, documentations, and bureaucracy.

It looks like most people in HN have never actually worked on large systems
and have no idea just how costs increase exponentially, how internal politics
and policies affect those projects, and how infernally hard it is to think in
terms of systems that have to be used by more than a handful of use cases.

There's not doubt that government projects have a huge amount of internal
overhead, but if you compare them to the stuff that goes on in multinationals
that have working systems in the 60s, it's really nothing out of the ordinary.

I know a guy who worked in a project related to one of the largest insurance
companies in the US. It's a 60 year-old system running on IBM System Zs,
AS/400s, Unix and Windows systems, with varying degrees of hardware and
software obsolescence, running custom COBOL builds and compilers, integrating
with thousands of external APIs, on half a dozen data centers with completely
different specs. There was a literal library of documentation spanning
millions of pages of requirements to describe how policies flowed through the
systems, how batch processes should run, and what tests should be used to make
sure a lady who took out an insurance policy on her TV in the 70s can still
cash out on it now and withing 20 years in the future.

All this does not even include the network topologies spanning TCP/IP, custom
frame relay protocols, proprietary NICs, SANs for storage and backup, tape
libraries, electrical layouts for data centers, etc.

There's more than a few reasons why large software projects fail; I have not
even begun to include the intricacies of the aforementioned industry and have
not even counted the regular issues of any project.

~~~
lifeformed
Sure, it's quite an engineering challenge, but... $840M? That's a staggering
figure.

~~~
Daishiman
Have you seen the costs of similarly large software projects? It's actually a
figure well within those margins.

For the same insurance company I quoted, the people in question estimated that
modernizing their system would cost $500 million USD in 1997 dollars and take
6 years without considering continuous upgrades of code and infrastructure.
And it's likely that this company's infrastructure was probably much cleaner
and uniform that goverment agencies'.

------
Acen
Is there a way to bypass the pay wall to read this?

~~~
nostromo
Click here:
[https://www.google.com/search?q=Poorly+Managed+HealthCare.go...](https://www.google.com/search?q=Poorly+Managed+HealthCare.gov+Construction+Cost+%24840+Million&tbm=nws)

Then click the story.

~~~
click170
IMO you shouldn't have to do that, so I flagged it.

------
rakoo
So, where's the github repo ?

------
Daishiman
Is it me or is that not that much money for a system that serves a 350
million-person nation?

~~~
nostromo
The federal site only had 5.5 million people sign up for healthcare.

That's $150 per registered user.

[http://www.businessinsider.com/how-many-people-signed-up-
for...](http://www.businessinsider.com/how-many-people-signed-up-for-
obamacare-2014-5)

~~~
jjoonathan
So less than 2 weeks of health care coverage per user.

While we're on the subject of waste, the USA's ongoing experiment with market-
based health care costs ~20 weeks of coverage per user per year compared to
its social counterparts.

[http://healthcarereform.procon.org/view.resource.php?resourc...](http://healthcarereform.procon.org/view.resource.php?resourceID=005522#III)

~~~
beat
Market-based isn't the problem. Switzerland, Germany, and Singapore all have
mostly-private competition-driven systems (more private than the American
model, even), and they are inexpensive and provide high-quality care, along
with universal coverage. So private insurers aren't why the American system is
so uniquely incompetent and inefficient.

So what is the problem? The way I see it, there are two features that really
distinguish the American system structurally. First, there's the state-by-
state laws and regulations, resulting in effectively 51 different legal
regimes. Second, American insurance is provided primarily through employers,
rather than purchased individually. In fact, non-employer insurance is
prohibitively expensive and often impossible to get.

I think the employer role is a _much_ more likely cause of problems than the
state laws, which are mostly consistent. Worse, tying insurance to employment
traps people in jobs they hate - a distinctly un-American outcome.

So anyway, I used to think a more socialist system was the right answer too,
until I looked more closely at the working systems in the world. It's not that
one particular model works, because there are lots of models in use in other
countries and they all work fine. It's that one model _doesn 't_ work - what
we have in America.

~~~
jjoonathan
> Switzerland, Germany, and Singapore all have mostly-private competition-
> driven systems (more private than the American model, even)

I'd call them "social".

They're all two-tier systems. The basic tier is heavily regulated and meets my
definition of social: government dictates terms, price, profit margins, and
provides funding such that everyone is guaranteed to be able to afford it.
It's no more private than your typical govrnment contractor. The supplementary
tier is private in name and practice. I agree that this is the best
arrangement and I suspect we mean the same thing and are just defining
"social" differently.

[http://www.commonwealthfund.org/~/media/Files/Publications/F...](http://www.commonwealthfund.org/~/media/Files/Publications/Fund%20Report/2011/Nov/1562_Squires_Intl_Profiles_2011_11_10.pdf)

[http://en.wikipedia.org/wiki/Healthcare_in_Singapore](http://en.wikipedia.org/wiki/Healthcare_in_Singapore)

[http://en.wikipedia.org/wiki/Healthcare_in_Switzerland](http://en.wikipedia.org/wiki/Healthcare_in_Switzerland)

[http://en.wikipedia.org/wiki/Health_care_in_Germany](http://en.wikipedia.org/wiki/Health_care_in_Germany)

> The way I see it, there are two features that really distinguish the
> American system structurally. First, there's the state-by-state laws and
> regulations, resulting in effectively 51 different legal regimes.

Agreed. This is a huge problem that needs to be fixed by regulating at least
the bottom tier of health care provision at the federal level.

> Second, American insurance is provided primarily through employers

Employer-provided insurance does have a very strong purpose in the free market
that adds value vs individual insurance: it dramatically decreases the risk of
the bandwagon effect (people with need X pile on to policies that favor need X
and sink them). Put differently, it chops off the long tail of P(claim rate in
insured population deviates significantly from that of national population).
Risk=money, so employer-provided care actually does decrease the cost of
providing health insurance. It also increases the effective bargaining power
of employees by allowing them to act as a bloc and making it economically
viable to actually analyze and compare the nuts-and-bolts of a large number of
alternative plans.

Of course, there are other ways to achieve the same thing, but they're all
"social": the government can dictate terms for the bottom tier of care (so all
insurance plans look the same to someone maximizing personal utility of an
individual plan) or the government can outright provide the insurance (no
choice = no bandwagon effect).

I agree that the freedom-reducing effects of employer-provided care are
decidedly un-American, but I contend that they occur as a direct result of
allowing insurance companies the freedom to set different terms in the basic
tier of their individual plans.

~~~
beat
Ah, but look at the results! If you accept my contention that the primary
distinguishing feature of the American system is the employer involvement, and
the demonstrable facts that a: American health care is twice as expensive as
the rest of the civilized world, and b: America fails to cover 15% of the
population... well, then it seems that employer-based insurance is a
significant contributor to the high cost and coverage gaps.

So if the purpose in the free market is to add value, it's pretty much a
dismal failure.

The thing is, though, that employer-based insurance wasn't there for some
dream of free market efficiency. Rather, it's a legacy from mid-20th century
labor negotiation. Back when insurance was largely nonexistent, labor unions
negotiated for it as a benefit, and it became widely accepted practice. This,
along with commie-phobia, kept America from adopting a more socialist model.
Today, it's a political problem, because unions are actually against
eliminating employer-based insurance, as it's a powerful negotiating tool for
them. So it's hard for Democrats to attack it.

~~~
jjoonathan
> well, then it seems that employer-based insurance is a significant
> contributor to the high cost

That conclusion would follow only if employer-based insurance were the only
plausible explanation we had to explain the increased cost. Off the top of my
head, here are a few alternatives:

1\. Pharma lobby has neutered efforts to centralize bargaining for drug
purchases for private insurance companies (what is the bargaining power of
BCBS Wyoming vs the NHS? Nothing.)

2\. Doctor's guild has created an artificial scarcity of doctors by squeezing
the training pipeline.

3\. Differences in entry-level health plans prevent price competition because
the valuation process is too difficult for consumers (allowing insurance
companies to differentiate based on ads and such).

> So if the purpose in the free market is to add value, it's pretty much a
> dismal failure.

No kidding. Sometimes it works miracles, sometimes it falls flat on its face.
Here it has fallen flat on its face. Repeatedly. And we keep throwing money at
it hoping it will fix itself :/

> employer-based insurance wasn't there for some dream of free market
> efficiency. Rather, it's a legacy from mid-20th century labor negotiation.

That's the narrative they taught it a health care economics class I took. I
don't buy it. Say what you will about the market, but it does not hesitate to
tear down economic structures that have outlasted their utility. In
particular, if you think that individual insurance plans are more expensive
due to cultural legacy rather than due to economics, I invite you to create
your own insurance company and undercut the incumbents ;)

> Today, it's a political problem

Yup. The only real question is how long the circus will continue. ACA was a
step in the right direction, but it doesn't go far enough. Time will tell. My
fear is that stopgap solution is the enemy of the good solution.

------
known
Mediawiki is better for storing/retrieving UNSTRUCTURED data

------
alien3d
Wow.quite cheap...

------
sytelus
This number may look big but remember average per head cost of a software
engineer is about 230K/yr. Any software project that takes 3 years and
requires ~1000 engineers would approximately cost similar amount.

On the side, I'd first hand experience with healthcare.gov to help someone
else. I can imagine myriads of integration points between insurance companies,
keeping track of their plans, state level requirements, credit checks, other
govt agencies like social security, customer service backend etc. From my
experience in working with this kind of complexities I think 200-300 engineers
may be more than enough for 2 year execution plan. So still it's about 3X-4X
waste when government gets involved.

PS: Per head cost needs to include base salary + bonuses + stock grants +
hiring fees + employee events + office expenses + subsidized cafeteria +
health benefits + 401K etc etc.

