
HealthCare.gov’s glitches prompt Obama to call in expert troubleshooters - cgtyoder
http://www.washingtonpost.com/national/health-science/experts-working-to-fix-healthcaregov-insurance-marketplace-officials-say/2013/10/20/1e1a35ce-39b4-11e3-a94f-b58017bfee6c_story.html?hpid=z1
======
smoyer
I live about three hours away from D.C. and the suburbs where much of the
government contracting takes place, and I'm going to make an offer that
doesn't throw more of the tax-payer's money into the fire.

I will, at my own expense drive to D.C. and spend two days of my vacation
evaluating the architecture of the failing system, and spot check the source
code quality. Additionally, I'll add the project to my Sonar source code
quality analyzer so that there's a focus on what code isn't up to industry
standards and which sections should be addressed first.

It's still astounding to me that we spent $700M on such a pathetic _LITTLE_
system ... there simply isn't that much code to write. If there are others
who'd like to show the government how it should be done, I'd be happy to start
an open-source project to rewrite the whole project, as well as a CI/CT/CD/CM
system to make sure that the code is maintainable, that changes are delivered
in a reasonable amount of time and that servers are constructed and clustered
in a consistent way.

This really isn't rocket science (and it's a good thing ... there would have
been deaths).

EDIT: If someone in the government a) reads my offer and b) is willing to let
"the industry" voluntarily help, you can contact me via the e-mail address on
my profile page.

~~~
dangrossman
Nobody spent $700M on a website. Over 3 years, the Centers for Medicare &
Medicaid Services (CMS) spent $394 million in total to establish and operate
the Federally Facilitated Exchanges. That includes building and staffing call
centers, media outreach programs, consumer complaint tracking systems, tech
support, employer support programs, all the integration and portals for the
health insurance industry, integration with a dozen departments of the federal
government, administrative staff, etc.

The "Federally-facilitated Exchanges IT" contract awarded to CGI Federal Inc,
which is the one that includes building the Healthcare.gov website among other
services, was $55 million. The numbers are directly from the Government
Accountability Office's report:

[http://www.gao.gov/assets/660/655291.pdf](http://www.gao.gov/assets/660/655291.pdf)

That's still a big number, but it's over an order of magnitude less than
yours. It should also be noted that not all of this money is coming from
taxpayers. The GAO estimates that in fiscal year 2014, $450 million in fees
will be collected from the insurance companies participating in the exchanges,
and this money is directly credited to the account used to manage the FFE
program.

~~~
nhebb
> It should also be noted that not all of this money is coming from taxpayers
> [...] $450 million in fees will be collected from the insurance companies
> participating in the exchanges

The ACA was upheld in the Supreme Court on the basis that it was a tax. Unlike
other taxes, which are paid to the government, you will be paying the tax to
an insurance provider. They, in turn, will be using the money to partially
fund the exchanges. So technically you're correct, but it's just a redirection
of tax dollars.

~~~
fpgeek
Insurance companies are not required to participate in the exchanges (and many
don't). And while people are required to buy insurance, they aren't required
to buy it from the exchanges (many will still get insurance via their
employers, for example).

Both sides are choosing to use the exchanges rather than one of their
alternatives, so the fees paid are user fees, not disguised taxes (unless you
consider all user fees disguised taxes, which is a separate philosophical
discussion).

------
JumpCrisscross
Politics were allowed to trump project management.

“One highly unusual decision, reached early in the project, proved critical:
the Medicare and Medicaid agency assumed the role of project quarterback,
responsible for making sure each separately designed database and piece of
software worked with the others, instead of assigning that task to a lead
contractor.

Some people intimately involved in the project seriously doubted that the
agency had the in-house capability to handle such a mammoth technical task of
software engineering while simultaneously supervising 55 contractors. An
internal government progress report in September 2011 identified a lack of
employees ‘to manage the multiple activities and contractors happening
concurrently’ as a ‘major risk’ to the whole project” [1].

It is speculated that "facing intense opposition from congressional
Republicans, the administration was in a bunker mentality as it built the
enrollment system, one former administration official said. Officials feared
that if they called on outsiders to help with the technical details of how to
run a commerce website, those companies could be subpoenaed by Hill
Republicans, the former aide said” [2].

[1] [http://www.nytimes.com/2013/10/13/us/politics/from-the-
start...](http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-
of-trouble-at-health-portal.html?hpw&pagewanted=all&_r=0)

[2] [http://www.politico.com/story/2013/10/obamacare-aca-
website-...](http://www.politico.com/story/2013/10/obamacare-aca-website-
broken-98412_Page2.html#ixzz2iL0ClGKC)

~~~
nhebb
[2] doesn't make sense to me.

> "Officials feared that if they called on outsiders to help with the
> technical details"

Aren't the 55 contractors outsiders? This sounds like typical DC finger
pointing, blame game, and the-buck- _doesn 't_-stop-here tactics.

------
DigitalSea
Sounds like a standard run-of-the-mill Government contract job to me. It's a
known fact getting a Government contract is like winning the lottery for a lot
of businesses. Ramp up your usual price by 1000%, take twice as long as you
should and when SHTH proclaim the problem is a lot more complex and bigger
than you thought and all is forgiven. Having said that, not all contractors
deliberately waste money and drag out projects, but it is a known aspect of
contract work. Whether or not that is the reason in this instance has not been
proven and probably never will be.

The Obama Care fiasco continues, almost like it was deliberately on purpose in
a way.

------
ams6110
So, the fiasco continues.

A cynical person might think first contractor failed on purpose, so that the
second bunch of "experts" could be brought in to fix it. One system for the
price of two (or more, "expert troublshooters" don't come cheap). I'd really
like to see a transparent accounting of who is getting paid, and how much, to
implement this so-called website.

~~~
mlangdon
I'm perpetually puzzled that people in tech keep seeing nefarious purposes
behind this failure when it sounds like every failed project we've heard of.
Moving targets (how many states, how many users), feature creep, managers with
no understanding of the undertaking, too many managers, and no time for
testing and debugging.

It's not evil, it's just incompetent and embarrassing.

~~~
siculars
all the states. all the citizens. how is that not obvious? these are known
knowns.

~~~
Spooky23
No it's not.

All states were supposed to have their own exchanges. 14 ended up doing it.
New York after all sorts of political machinations agreed to run their own
exchange at the 11th hour. (Supposedly "unprecedented" traffic also screwed up
the NY exchange, which was also built by some big government contracting firm
-- CSC)

------
napoleond
Does anyone have more information about what exactly is going wrong with this
site? Apparently Development Seed (the company behind
[http://mapbox.com](http://mapbox.com)) was (one of?) the contractors behind
the project[0] and they've always seemed (at least judging from their product
offerings) quite competent.

0\. [http://developmentseed.org/blog/2013/06/25/healthcare-
launch...](http://developmentseed.org/blog/2013/06/25/healthcare-launches-in-
the-open/)

~~~
apendleton
Dev Seed did the user-information/knowledge-base component, which has worked
fine. They didn't work on the healthcare exchange, which is what has been
broken.

------
DanielBMarkham
IIRC, the site was handed out to 55 (!) contractors to create?

Maybe #56 is the charm.

I take no pleasure in watching this train wreck. It could, however, be a
teaching moment for the country on what's wrong with Federal IT. Probably
won't be, but it could be. So the story is worth following.

~~~
forkandwait
I think 3 really good programmers, 2 sys admins, 1 non-clueless PM, and 1
person to arrange their travel would have gotten this done faster.

~~~
brazzy
No - they'd by now have read through about 20% of the regulations the system
needs to fulfill.

------
microcolonel
What a pile of BS.

They're so moronic that they built the thing such that registration of the
userbase would take more than a hundred years.

Stop coddling the state, they're not going to do it right, they don't welcome
the debate, they don't care, they won't present a balanced budget, the debt
default is constitutionally prohibited, the war on drugs is a war on people,
their wars are so obviously wrong that there are more soldier suicides than
deaths in combat, the military burns more oil than the entire population of
india, socialism doesn't suddenly work when the current regime tries it.... Oh
never mind, go ahead and keep wasting your time listening to sociopaths and
participating in silly, futile politics; I'll just make a greasemonkey script
to filter out all stories with "Obama" in them, and be on my merry way.

------
vaadu
Why are there no consequences for this epic failure? Someone should go to jail
for lying about the costs and the schedule. Many more should get fired and no
longer permitted to work on government projects.

~~~
CamperBob2
_Why are there no consequences for this epic failure?_

Because government. They get paid the same, by you and me, whether it works or
not.

------
pmorici
What is the point of healthcare.gov anyways? You can go and buy an ACA
compliant plan on a comparison shop website like ehealthinsurance.com that
works pretty well why did the government need to create it's own?

~~~
jjoonathan
The idea was to attack fragmentation, which is a huge cost center in the
health industry. A smaller individual insurance plan

* Has to pay more for drugs (less bargaining power vs a big plan)

* Has more statistical risk (# people goes with N, stdev in # people with a given illness goes with sqrt(N), so risk/person goes with sqrt(N)/N and larger plans have less risk)

* Has larger legal/consulting overhead

* Forces doctors to incur overhead with nonstandard forms

* Is vulnerable to the bandwagon effect (if, by accident, it happens to favor X group of costly individuals, the costly individuals proceed to pile on and sink the plan)

* Has the ability to discriminate based on fine-grained geographical knowledge and other factors

* Has an under-leveraged customer base (if the plan is "secretly evil" and dumps people who need expensive treatment and 1% of people require expensive treatment, a 100 person plan will have 0 or maybe 1 person being screwed at any given time, while a 10000 person plan will have ~100 people being screwed who are much more able to band together and fight a legal/PR battle)

Previously, small plans were able to flourish because of #6 and #7, incurring
#1-5 as costs of doing business. This was bad for everyone except insurance
companies. The best way to turn a profit in the insurance industry is to sell
insurance priced for a medium-risk person to a low-risk individual(or hi-risk
price to a medium-risk person). If you look closely, this kind of "innovation"
is actually just restoring the pre-insurance reality of charging sick people a
lot and healthy people very little. It defeats the purpose of insurance while
increasing costs (#1-5) and letting the people in charge of the purpose-
defeating process pocket a hefty cut of the proceeds by fooling people into
buying overpriced coverage.

Exchanges are designed to put an end to the tragedy of the commons. The
savings from #1-5 can be recouped at the expense of eliminating #6 and #7,
which were perverse incentives anyway.

Even if you disagree with my explanations, it is objectively true that
employer plans are much more efficient than individual plans
(E(payout)/E(cost) is larger). You can then see the ACA exchanges as an
attempt to reform individual plans to emulate employer plans. Of course, it
remains "cargo cult economics" unless you can figure out why you expect the
emulation to close the gap in value, which is what I was doing with the list
in the first place. In other words: I have provided 7 possible explanations
that I think make sense but there are even more out there which I don't agree
with and haven't listed.

~~~
fpgeek
You forgot one of the biggest reasons for the exchanges: they're the mechanism
through which the government provides premium subsidies to people who qualify.

I think there are a variety of reasons for that (e.g. keeping better tabs on
the large amount of money spent on subsidies, easier administration of the
subsidy tax credit), but the bottom line is the law says people only get the
subsidies if they go through an exchange, so...

~~~
jjoonathan
The government has many mechanisms through which it distributes subsidies, and
while I'm sure the centralized exchange is going to be one of the most
efficient and convenient, I'm not nearly so certain that the difference will
be larger than the effects it has on the market.

Let me be concrete. According to [1] (which gets its numbers from the CBO),
individual plans have 29% administrative overhead as opposed to 12% for
employer plans. If the exchanges successfully emulate employer plans, they
save 17%*$300b = $51b per year in health care costs. Unless the exchange
program is 10,000% more efficient than its alternatives (tax subsidy, payments
to companies, etc) then the effects I listed dominate.

In any case, the exchanges were designed with the explicit intent of making
the individual insurance market more efficient, not merely making subsidy
disbursement more efficient.

[http://www.americanprogressaction.org/issues/healthcare/repo...](http://www.americanprogressaction.org/issues/healthcare/report/2009/03/23/5802/the-
inefficient-individual-health-insurance-market/)

------
spoiledtechie
What I am more worried about than any other HealthCare.Gov glitch, is if this
website was so badly designed, that the data it holds, the functions it does
would also be extremely badly designed and secured. Yet NO ONE is talking
about this angle.

Thats what keeps me up at night.

------
chaseideas
What did anyone really expect? Sounds like a typical government contract.

I can only imagine how these conversations go...

"We need more QA! STAT! It's going to take at least $50 million to test this
system... it's awfully broken!" _chuckle_

 _face /palm_

That said, I think there should be a lot more concern about the security of
this site if there's been such a mishandling of the basic functions and
operational flow.

Seems only reasonable to suspect this will be leading to a horrible leak of
many people's private records at some near point in the future.

~~~
sseveran
This one doesn't even seem that bad yet. I mean its not like the new bay
bridge which was a decade late.

------
byoogle
Obama didn't read The Mythical Man-Month.

~~~
mangala
(Chuckle) Well seeing as how half the tech managers I talk to at private
companies haven't ever heard of it, I think he should be given a little bit of
a break.

------
pyalot2
The solution was built in 2/3rds by Accidenture, who got awared a $400m
contract (out of the total of $600m awarded) with the bulk of the remainder
going to IBM and the 50 other contractors.

If I know one thing, and I say this from firsthand experience, it's that
Accidenture is absolutely the worst choice for any kind of contract.

Accidenture isn't an "outsourcing" company. They're an oversized roleplaying
project, where the overlords (controllers) have the power and whim to kill
projects and just pay the termination penalty, and where everybody strives to
become a partner because it's up or out. Any problem inside Accidenture is
seen as a nail, and every solution is a hammer, meaning a combination of ISS,
MSSQL, Sharepoint, Windows Server and copious application of ridiculously
overpriced "consultants".

------
jowiar
The magic quote:

    
    
        One major problem slowing repairs, people close 
        to the program say, is that the Centers for Medicare 
        and Medicaid Services, the federal agency in charge 
        of the exchange, is responsible for making sure that 
        the separately designed databases and pieces of 
        software from 55 contractors work together.
    

We're talking about integrating a slew of both legacy (think decades old) and
new (think untested) systems, many of which don't have half-decent APIs to
work with (I'd be willing to bet there's at least one "database" in there
somewhere that consisted of an Excel workbook on a shared drive, perhaps with
an "API" of a paper form and a fax number), and many of which almost certainly
weren't built to scale to consumer web traffic. Steve Yegge's platforms rant
comes to mind here, and from talking with some of my federal employee friends
about the challenges they face, get to root of the problem. Building protocol-
centric services from the beginning goes a long way towards fostering
interoperability.

More broadly (and apologies for the non-HN bent to this), the project serves
as an example for the utter shitshow that is the United States healthcare
system. The exchange is intended to serve as an abstraction for the process of
figuring out what public and private options are available to you. The
motivation for building the exchange was "Hey! Figuring out what options exist
for healthcare is complicated!". Of course, building such a system doesn't
make the complexity go away - it just makes it someone else's job. Perhaps
that this determination is so complicated that it requires a ridiculous system
is an indication that we're "doing it wrong".

~~~
joemaller1
I've seen this "ancient systems" idea go by a few times. Honest question: If
these platforms are so old, they likely aren't terribly complex, from the
perspective of what's now computationally cheap. So, how hard would it be to
drop a computer on their local network and just dumb-clone the whole thing?

------
nbevans
They could learn a lot from how UK.gov
([http://digital.cabinetoffice.gov.uk/](http://digital.cabinetoffice.gov.uk/))
operates these days. It's a true agile shop, that's been ridiculously
productive and forward-thinking for at least the last 5 years.

------
bnolsen
So in order to reduce health care costs the government declares itself as all
knowing in the medical industry, inserts itself into the system with all the
beauracracy. Sounds to me like adding more middle men just raises costs and
also happens to kick open the door for typical government graft, corruption
and embezzlement. Better yet the government sets itself up to decide who gets
treatment, and data mining by the NSA and IRS can help identify polical
opponents or non supporters to determine on the fly available coverages and
rates.

Perfect for any tyranical statist type.

Does anyone really believe the whole website thingy wasn't full of graft,
corruption, kickbacks and paybacks? Come on, seriously.

------
mortdeus
All I know is that they need to start making decisions like minimizing their
css and js scripts and fetching jquery from Google before I start taking them
serious when they say the best engineers in the fed are working on it.

<script src='//assets.healthcare.gov/global/js/lib/jquery-1.8.2.js'></script>

[https://www.healthcare.gov/css/all.css](https://www.healthcare.gov/css/all.css)
[https://www.healthcare.gov/js/all.js](https://www.healthcare.gov/js/all.js)

------
dodyg
This type of integration project is always challenging regardless whether it
is government/private sector. Then add tons of required-by-law requirements
and you are talking about massive tasks to get things right.

Their main problem is the hard deadline for October 1st launch. Things should
have been rolled out state by state - but I am not sure whether this is in the
law or just a political decision.

------
ck2
Shouldn't they have been called in a year ago? Six months ago? A month ago?

$95 Million apparently buys everything but new management and proper coders.

------
devx
Now what? Another $600 million down the drain to fix the problems of the first
$600 million?

------
yeukhon
What kind of computer programmers did the government hire to oversee the
progress? I am going to question this person's ability. While I am just a
senior in college, I probably can hammer a lot more practical issues before
its launch. I am sure half of the HNers can do that too.

 _sigh_

~~~
yeukhon
Thanks for downvoting me, _government employees_.

~~~
tmorton
If you're lucky, you'll learn this lesson early in your career: when you think
"oh, that's easy, those people must be idiots" it's a sign that you don't
understand the problem.

People are not idiots. Large problems are never easy. Politics, management,
and salesmanship are genuine parts of the problem, not useless distractions
from technical stuff.

I'm not saying that improvement is impossible, I'm saying that improvement is
hard. The healthcare.gov system could have gone live, and been functional, on
October 1 - if you could hire better technical people. Can you hire a top-
notch team of techies in this environment? Ever tried it? It could be working
if the political appointees had nailed down the requirements earlier. Have you
ever tried getting hundreds of legally-mandated decisions pushed through the
federal bureaucracy? Many of those decisions were pushed back for political
reasons, which sucks - but could you made the unpopular decisions early, then
won enough elections to keep Republicans from dismantling the entire thing?
The system could be working if the law made more sense in a dozen different
ways, but all of those compromises were necessary to get the law passed in the
first place. Have you ever tried to write legislation, and get it through
congress?

Don't get me wrong - the healthcare.gov system is a clusterfuck, and there are
easily identifiable reasons for that - in retrospect. And hopefully we can
learn from the failures and move forward. But laying it all at the feet of
some nebulous government employee is completely unhelpful.

