
$90 million in maintenance per year for healthcare.gov - jessaustin
http://blogs.law.harvard.edu/philg/2014/01/12/90-million-in-maintenance-per-year-for-healthcare-gov/
======
guelo
I wonder if they also fired the government people that insisted that the site
had to use a weird proprietary XML database called MarkLogic, which apparently
was one of the reasons for the concurrency problems. (As described here
[http://www.nytimes.com/2013/11/23/us/politics/tension-and-
wo...](http://www.nytimes.com/2013/11/23/us/politics/tension-and-woes-before-
health-website-crash.html))

A SQL server, probably Oracle, should almost always be the answer for a large
bureaucratic product like this.

~~~
hga
I find it difficult to judge the quality of the database given what we've
learned, especially since the initial provisioning was entirely inadequate for
any database; about when that article was written it was moved off
government's Centers for Medicare and Medicaid Services' (CMS) shared VM
infrastructure to 12 dedicated servers.

However, there's absolutely no doubt the contractors weren't familiar with it
and its paradigm. A SQL RDBMS, which pretty much everyone is familiar with,
would have been a much less risky bet. But the CMS had had success with the
XML database, and since they arrogated to themselves the roles of general
contractor and system integrator (and in fact were "fired" about the time that
article was written), the consulting companies had no choice but to use it.

See my other comment where people are suspecting Accenture might just throw
away a lot of the current system and replace it with what they did for
California.

EDITED: to make more clear the distinction between the unit of the Federal
government CMS and CGI Federal, the fired contractor.

~~~
rhizome
_since they arrogated to themselves the roles of general contractor and system
integrator_

...and also attempted to arrogate to themselves those _future_ roles. Would
that there were taxpayer protections against that kind of anti-competitive (on
face, at least) strategy.

~~~
hga
Not entirely sure that's necessary when those who do it fail so hard, i.e.
there's at least some feedback in the system. The other exchange example where
this happened is Oregon, which isn't even pretending that their site works.

~~~
rhizome
Well, I'm sure there are ton of gov't contractors who have locked themselves
in successfully. The feedback only factors in negative cases, but it's still
harmful in the positive one.

~~~
hga
I think you may have made a misapprehension based on CMS being a similar
acronym to CGI, the fired contractor. The former is the Federal government's
Centers for Medicare and Medicaid Services, and those government bureaucrats
are the ones that arrogated to themselves technical management roles they were
completely inadequate to handle.

Ditto Oregon, when they hired a new IT person who though they could save
something like $32 million by being the prime contractor (as I recall; don't
remember what happened with systems integration, except of course it's still
totally failing).

We could only wish that government contractors had locked themselves into the
prime contractor and systems integrator roles for CMS or HHS; who knows, maybe
they would even have had the stones to tell the politicians NO! when the
latter continued making requirements changes too late for implementation.
Compare to Kentucky, which won by hiring a prime (Deloitte) who maintained a
ruthless focus on what we call the MVP.

------
cjoh
The problem here isn't with "government," but with the failed ERP-style
methodology of development that federal and local governments, along with very
large corporations use to develop technology. Hewlett Packard, Hershey, along
with a string of other large corporations have lost billions in value from
these kinds of integrations. Hershey even missed Halloween one year because of
a failed integration.

And while Silicon Valley and the startup community has largely moved away from
this methodology, large bureaucracies, ever afraid of risk, seem drawn to
them. But as far as I can tell, this idea that "nobody ever got fired for
choosing IBM" is a function of marketing, not of a track record of success
(IBM's latest for the feds: [http://sam.gov](http://sam.gov). $171 Million).

What we can do to solve this problem isn't to scoff and laugh at government.
While some want larger or smaller government, we all should want it to do what
it is doing affordably.

The opportunity for smart technologists to take on the idea of working with
government _commercially_ is the next step in the Civic Hacking movement. I
hope those invited to go to their next hackathon for the city or invited to
give their free advice and labor to the city out of a sense of civic duty,
begin to ask the question: why are you asking me for stuff for free with one
hand, whilst handing out million dollar contracts with the other. How can you
fix procurement in a way that allows me to compete for the serious work, so
that I can save you from this mess?

------
skywhopper
I'm surprised Philip Greenspun is acting so naive here. "If it works, why will
it take $90 million to keep it running?" This from a computer scientist?
Software is never "done", and maintenance on existing computer systems is
quite often far more expensive than the initial development. Do you want it to
keep updated with correct information about the various plans that are
available? To keep up with new tweaks to the regulations? To address new bugs
that come up when it's annual renewal time? To stay secure over time? All of
these things require a lot of work. And how about fixing the bugs that still
remain? Or improving the system?

Yes, government projects of this sort are inevitably going to cost far more
than a startup company, and mistakes will be made. But $90 million out of the
federal budget isn't even worth blinking at. The Defense Department spends
that much in about an hour and a half.

But if you really want to save money in this area, you should argue for
eliminating the exchanges and healthcare.gov altogether. Just sign everyone up
for Medicare. Save a few hundred billion over the next few years give or take.

------
nikatwork
Everytime there's a "government app costs $x million" story, someone always
chimes in with "Lol I could build that in PHP for $2."

It seems a fair percentage of HN'ers have not experienced the sheer
Khafkaesque deadlock that is a large corp or govt department.

The cost is not in the dev effort, quite often there is actually a very small
team of devs doing the "real" work. Outsourcing the dev work is not going to
help because it is the massive bureaucratic pyramid which is crushing the
devs, and that pyramid can't be outsourced.

This is not a technology problem. It's an enterprise management problem.

~~~
beat
Worse, it easily enters Mythical Man-Month territory where more staff
(especially more managers) are thrown at the problem, which actually makes it
worse.

------
pmorici
The really disturbing thing is this is almost certainly _not_ an isolated
incident. It's just that this is the first time that a government IT project
has received so much scrutiny. Makes you think when politicians insist that
taxes and fees need to be raised.

~~~
VladRussian2
well, the healthcare.gov project being the spearhead of the most important
policy push by the current President and his party - i'd guess that it is
actually the most efficient (less than half a billion) and successful
(developed in 2 years, fixed in 3 months) one among myriad of government
projects and budget sinks, most of which isn't auditable and hasn't been in
years (for example, see Pentagon budget audit (i.e. lack of it))

------
Aloisius
Sometimes I think about creating an organization that's sole purpose would be
to enter bids in on government tech contracts with upper and lower cost
estimates. It wouldn't actually _do_ any of the work (to maintain neutrality),
but at least put in a reasonable set of figures for how much a project
_should_ cost. Or maybe just a company that reviews proposals for
local/state/federal government agencies to ensure they aren't being ripped
off.

Frankly, tech contracts aren't the only thing that goes wrong. Civil
engineering contracts are just as awful with people underbidding and then
having massive cost overruns because there was no expert there to say, hey...
this isn't reasonable.

~~~
VladRussian2
>Sometimes I think about creating an organization that's sole purpose would be
to enter bids in on government tech contracts with upper and lower cost
estimates. It wouldn't actually do any of the work (to maintain neutrality),
but at least put in a reasonable set of figures for how much a project should
cost. Or maybe just a company that reviews proposals for local/state/federal
government agencies to ensure they aren't being ripped off.

so one more consulting participant and review layer to dig teeth into
government money without actually delivering on the project? And with the best
intentions like everybody else :)

------
jayd16
Misleading title. The $90m cost is a 12 month contract with a new company, not
the amount it might cost each year indefinitely.

~~~
hga
Bingo.

Given how bad the system still is (e.g. 5% error rate in sending info to
insurers
[http://www.washingtonpost.com/blogs/wonkblog/wp/2014/01/09/a...](http://www.washingtonpost.com/blogs/wonkblog/wp/2014/01/09/a-health-
industry-expert-on-the-fundamental-problem-with-obamacare/) and also see his
very high signal to noise ratio blog:
[http://healthpolicyandmarket.blogspot.com/](http://healthpolicyandmarket.blogspot.com/)),
and how much of it is left to be done (paying insurers, change in status
updates, etc. etc. etc.) this is not just cheap, but some are suspecting that
when Accenture fully takes over, right at the end of the spring enrollment
period, they'll just throw much or most of the existing system away and start
with what they did for California, which isn't hardly as bad.

------
3pt14159
I once worked on a project for the American government. The company I was
working for was the 4th and last in a line of companies. We were the ones
actually writing the software and it was still 5 times more money than I
thought was reasonable.

------
joelrunyon
I have zero technical skills and I would do this for a mere 85 million.

I'm pretty sure for 85 million, I could assemble a crack team of devs to knock
this out as a more competitive price.

~~~
pinaceae
nope.

this is highly sensitive data, you can't treat it like a private business
does, say like Target.

this needs to survive countless audits, from code, to docs, to various
accessiblity and other norms.

a large number of data interfaces in the background to byzantine and opaque
systems.

etc etc etc.

COMPLIANCE is the cost driver. and not necessarily for a bad reason.

~~~
erichocean
_this is highly sensitive data, you can 't treat it like a private business
does, say like Target._

You're joking, right? Because the lack of care that was put into security is
WIDELY documented. They couldn't even get _internal_ signoffs on security and
decided to launch anyway.

It's simple, really: the government doesn't care if they break their own laws.
The laws are for everyone else.

~~~
pinaceae
that's launch, in a rushed way. political decision. but as you see, highly
visible.

some new business site/app does not fall under the same public and
governmental scrutiny. the ones that do with the same scope as this site -
cost as much.

------
negamax
It's the same story everywhere. Heard this anecdote about installation of ERP
in one of India's major ports. The programmers were paid in gold to delay
project just so old ways of funnelling money remains intact.

------
DanBC
Hn readers might be interested to read about some of the UK government IT
projects. I'm sure some of them went well, but many had huge budgets and were
then scrapped because they were useless.

[http://www.computerweekly.com/blogs/public-
sector/2013/05/pu...](http://www.computerweekly.com/blogs/public-
sector/2013/05/pull-out-and-keepyour-guide-to.html)

Connecting for health cost £11bn - billion - and is in various states of
abandoned.

------
cryoshon
Politics, bureaucracy, mismanagement-- this policy has been a failure.

Time to join the 21st century and push for universal healthcare.

~~~
whatevsbro
> Time to join the 21st century and push for universal healthcare.

No it's not. Source: I'm a Finn.

Compare two scenarios:

    
    
      A) Healthcare is run with money that's just taken from people.
      B) Healthcare is run as a business, with people paying for it voluntarily.
    

Which one do you think will be better quality? Suppose you're running a car
repair shop, but your "customers" are just forced to come to you and pay for
your services. Are you motivated to: 1) provide them with good service, 2)
keep your costs low, 3) maintain high standards?

The obvious answers are: No, No, and No, and that's exactly what happens with
_everything_ that's run by any government, because it's always with someone
else's money and the "customers" aren't actually customers - they have no
choice.

It's rather unsurprising, then, that Finland's public healthcare system is of
poor quality, and that we have to wait for weeks or months for an appointment.

~~~
nwh
I live in Australia, our healthcare is universally free and of very high
quality. I don't have a problem walking into any clinic in my area and getting
free, prompt medical service. Just because yours sucks doesn't mean it's a bad
idea to begin with.

~~~
whatevsbro
> Just because yours sucks doesn't mean it's a bad idea to begin with.

I didn't say it's a bad idea because ours sucks. I said it's a bad idea
because if you're using money that's just _taken from other people_ , you're
not motivated to provide them with good service, and so on.

The factors I listed cannot be escaped. You can rest assured that your system
is slowly turning to shit too.

~~~
nwh
If you're going to have that attitude you'd better not be using our roads,
parks, public transport or indeed any other government services that have
"stolen" money from you. I think you'd be a lot worse off if you decided not
to pay tax and go without everything being a taxpayer entitles you to.

~~~
programmarchy
Hm, that's not a very good argument.

If a slave takes a meal from his master, it doesn't mean he must condone
slavery.

------
jgalt212
That may be a lot of money, but it's way cheaper than Nest.

[http://news.cnet.com/8301-11386_3-57617153-76/google-to-
buy-...](http://news.cnet.com/8301-11386_3-57617153-76/google-to-buy-nest-
for-$3.2-billion/)

------
semerda
Without knowing what lies beneath the mess.. isn't $90m in maintenance fees
extremely excessive? Or have they factored in a full rewrite? What are the
typical gov tech maintenance fees?

------
shit_parade
why anyone believed the government could sell a product is beyond me. It's as
if all the tech professionals have a blind faith untested by reality.

