
Troubled Obamacare website wasn't tested until a week before launch - ericcumbee
http://washingtonexaminer.com/troubled-obamacare-website-wasnt-tested-until-a-week-before-launch/article/2537381
======
JPKab
I've commented on HN recently on how ABSOLUTELY EYE-GOUGE INDUCING AWFUL it is
to do any kind of software work for the Federal Government. Due to a broken
hiring process and union-like job security, it is filled with incompetent
morons constantly scheming to find any desperate way they can to demonstrate
value. (I of course am talking about the GS-12 and up levels)

The men and women who work hard and care are like the proverbial Atlas: they
do the work of 15, with the other 14 people in their group using their exurban
DC commute to excuse their 7AM - 3PM schedule, of which the first 3 hours of
their day is spent surfing the net doing jack shit. And then these fucking
assclowns get promoted.

These are the people who do things like make major interface and usability
changes to a massive healthcare website and just magically expect the
contractor (CGI in this case) to be able to do it. They don't give a shit
about wasting taxpayer money, because it has no effect on their career at all.
Failed projects don't get you demoted or fired, schedule slips are so normal
that nobody even bats an eyelash....

This causes any sane person who points out the cost in time and money of
bullshit bikeshed type changes to be rolled over by the horde of overpaid
lemmings who are trying to yell as loud as they can before they head out at
3pm for their kids volleyball game in Manassas.

If this looks like a rant, it is. I hate the GS system. It gives government a
bad name, and these overpaid assclowns do nothing but fail, supplying ammo to
every Rush Limbaugh in the world advocating for elimination of government.

Please, please, if somebody with power is reading this: fix the government
hiring system. Fire the lazy assholes who work 5 hour days and pull six
figures. Fire the people who think not knowing how to use a computer beyond
Outlook doesn't interfere with their ability to perform as CIO of an agency.
Fire people who are in charge of projects that go over budget and late.
Fucking fire people.

Next week I'll be going in and meeting a GS-14 (just promoted for 13) who does
NOTHING at work but talk about his motorcycle and martial arts. When tasks are
handed to him, everyone else groans, knowing that it will simply not be done.
Instead they go behind him and perform the tasks and hope he doesn't find out,
cause if he does he will pull rank and humiliate them.....

Venting over.

~~~
bjornsteffanson
This is not a rant, because it's the truth. To outsiders, this attitude seems
hyper-negative because it's difficult to comprehend the level of ineptitude
found in federal government IT. As a non-national who has done some (very non-
sensitive) on-site IT work for the U.S. government, I experienced this
firsthand for several years. (And before I get accused of elitism, my home
country doesn't necessarily have the answers either.)

When all of the Snowden and PRISM stuff broke, my first thought was to chuckle
and ask, "How?" Based on my experiences with federal IT workers, it's hard to
imagine there is a room of crack coders underground in some basement or deep
inside some mountain just knocking it out of the park.

~~~
RougeFemme
The NSA recruits at some of the best graduate schools. In some quarters, the
joke is that if you have a PhD in math, comp sci or software engineering and
you're not teaching or doing post-doc work, you must be working in the _black_
world, which includes NSA.

~~~
JPKab
Agreed. A young woman sitting next to me at a Hadoop oriented class during the
summer (right as the Snowden info was coming to light) had just left the NSA.

She was brilliant. Little coding experience outside of Perl, but was able to
keep up on the exercises in Java with only minor hiccups due to her lack of
familiarity. She had been recruited out of graduate school.

On the other hand, she became extremely hostile when the Snowden incident was
mentioned by people next to us (they didn't know her former employer) and was
stating repeatedly that "None of you know what you're talking about, and if I
was allowed to I would correct you, but I can't." and "Do you really think
people can just sit down at a computer and spy on people without their
supervisor's approval?" (as it turns out, we were right, and she was
profoundly wrong)

It was a revealing moment, because it demonstrated the group think that can
permeate even the most brilliant mind. Of course, she also thought there was
nothing wrong with mathematics education in the U.S., which in my opinion
makes her clearly deluded.

------
atwebb
Shocked! I am shocked to hear that a major tech project with a hard deadline
was pushed through at the last minute.

Kidding aside, a selfish part of me likes things like this because it lifts
the veil and shows me that plenty of other people screw up. It's easy to
dismiss it as a botch or idiot contractors but I'm guessing there's a lot of
smart people, who worked long hours, and it's just hard to do right in these
circumstances.

------
wikiburner
There was a $600+ Million price for the website that was previously mentioned
in HN comments on this topic, and others were claiming that this number was
debunked, and the actual price was $30 Million or so.

NBC News has now released a report on what the actual numbers are. It's $196
Million as of 5/15/2013, and the price appears to be on its way to $292
Million: [http://www.nbcnews.com/video/nightly-
news/53309356/](http://www.nbcnews.com/video/nightly-news/53309356/)

Anyway, I apologize if it looks like I'm spamming the thread with it (I posted
at the bottom of the thread as a reply). That's not my intent, but I think
it's important information.

------
alphakappa
Why would this be surprising. It's a project with huge scope, with
dependencies that they can't fully control (IRS, Social Security etc), with
likely changing requirements, and a tight deadline.

It's not like they could have released in beta and iterated, or waited until
it was perfect. It's the nature of the beast - if the major annoyances don't
get fixed in a reasonable [1] amount of time, then it's worth writing reams
over.

1\. Undefined.

~~~
nhebb
The IRS interface is working fine [1]:

 _Sarah Hall Ingram, the director of the IRS 's Affordable Care Act office,
told a House panel Wednesday that the IRS hasn't experienced similar problems.

"The portion of the responsibilities the IRS is in charge of is going fine,"
Hall Ingram said._

[1] [http://thehill.com/blogs/healthwatch/health-reform-
implement...](http://thehill.com/blogs/healthwatch/health-reform-
implementation/327483-irss-obamacare-implementation-going-fine-official-says)

~~~
snowwrestler
According to the IRS.

~~~
hga
Well, they could be telling the truth. E.g. " _(If and) When_ we get a query,
we respond to it in our N second allowed window" (as I recall the total time
allowed is 7 seconds).

I'm pretty sure some decent fraction of the entities Healthcare.gov queries
behind the scenes are doing their job.

Well, most of the time. One thing that occurred to me is that if certain
actions require a _lot_ of backend queries to different sources, multiplying
the probabilities each could outright fail or blow the window could result in
an overall flaky system.

Which doesn't directly explain why the offline part of the registration system
is failing so often.

------
lvs
I have no particular opinion about this. However, in the context of an article
that relies heavily on unnamed sources, it's worth pointing out that the
Washington Examiner is a conservative tabloid founded in 2005 by Philip
Anschutz.[1]

[1]
[http://www.politico.com/news/stories/1009/28355.html](http://www.politico.com/news/stories/1009/28355.html)

~~~
jerf
I will make no bones about the fact I don't think Obamacare was a good idea on
any level. But I say that just to put my cards on the table. Forget politics.
I'm also a software engineer. Take off your politics hat and look at this with
your software engineer hat. Tell me, with a straight face, that this shows any
sign of being functional any time soon. Tell me that every software engineer
alarm bell you have isn't going off. Tell me that this website, which consists
of multiple pieces (insurance info in, user info in, user selection
mechanisms, user information flowing back out to the insurance companies, and
with every last one of those bits having endless, endless business rules
associated with them, and this a gross oversimplification of what's there)
only has problems with the user parts, coincidentally isolating all the
problems into the only part we've really collectively looked at, and once
those are ironed out it'll all go perfectly smoothly because this is an
isolated occurrence and not indicative of the whole project.

Republicans and Democrats and bureaucrats and judges and media and internet
commenters and everybody else can say whatever they like about whatever they
like in whatever forum they like with as much or as little truth as they like,
but the software doesn't care. It either works or doesn't. It can't be
cajoled, can't be bargained with, can't be threatened, can't be ideologically
swayed, and it's too stupid to be capable of any kind of cover up regardless
of what its motivation may be. It is what it is, no matter which way you voted
in the last election or which way you vote in the next.

~~~
hga
We're already hearing about backend nightmares. Can't find the _Washington
Post_ article where I first read it, but this UPI one citing them has the
details I remember ([http://www.upi.com/Top_News/US/2013/10/12/Obamacare-
website-...](http://www.upi.com/Top_News/US/2013/10/12/Obamacare-website-bugs-
vex-insurance-companies/UPI-75911381598024/)):

" _Several industry executives told The Washington Post the daily reports on
new enrollments since the roll-out of the Affordable Care Act have often
included duplicative information, such as people enrolling and canceling
several times in a day._ "

And you left out e.g. connecting to Experian to get the consumer's credit
score, which is part of plan pricing (a general trend with deductibles going
up). That one, I've noticed, pretty much requires the registration ... but
doesn't explain why the site forbids under penalty of perjury playing "what
if" to determine how you might best arrange things for you and yours.

~~~
hga
And referencing a _Wall Street Journal_ article that's behind their paywall,
[http://www.marketwatch.com/story/obamacare-woes-widen-as-
ins...](http://www.marketwatch.com/story/obamacare-woes-widen-as-insurers-get-
wrong-data-2013-10-18) says:

" _Emerging errors include duplicate enrollments, spouses reported as
children, missing data fields and suspect eligibility determinations, say
executives at more than a dozen health plans. Blue Cross & Blue Shield of
Nebraska said it had to hire temporary workers to contact new customers
directly to resolve inaccuracies in submissions. Medical Mutual of Ohio said
one customer had successfully signed up for three of its plans._"

On the other hand, _if_ they get _something_ , this tells us one simple method
the insurance companies can generally use to resolve issues. Although it
sounds like they'd best contact everyone who maybe signs up, and no doubt the
system will be telling two or more companies that some customers have signed
up....

A lot messier than CGI Federal's Medicare.gov's Part D prescription plan
signup system, where in all fairness the info is binary (I've signed up with X
plan) and both sides can depend on the site doing the right thing.

------
jreed91
Surprise, surprise the government is still using waterfall.

~~~
marcc
Well, read the article. The claim is that requirements were still changing 4
days before the launch. That's not waterfall, that's just no process at all.

~~~
RougeFemme
I wonder if the requirements had already been finalized and it's just the
"final" signed off documentation that was delivered so late. . .yes, still
broken/no process.

------
lancewiggs
The fundamental issue here, like many projects of this size, is that it was
trying to deliver a solution to a ridiculously complicated process. Moreover
the way of procuring that solution is also fundamentally broken. Simplifying
the underlying process would take a level of politics that is sadly well
beyond even one party, let alone two.

------
jackmaney
Wait...it was tested?

------
RougeFemme
I find it hard to believe that CMMS actually performed system integration,
such as it was. I find it easy to believe that CMMS hired a contractor to
perform system integration. And just as the individual contractor employees
saw issues, there were probably government employees within CMMS who also saw
issues. . .and just enough senior executives - government and contractor - to
ignore the "naysayers". . .and insist on having their fingers deeply in the
pie of such a high-visibility project.

And, of course system integration was not the only project
management/technical issue with this project.

~~~
hga
I don't see why you find it so hard to believe. Hubris is a pretty common
human flaw, perhaps it was felt this was too important to leave to the
professionals.

In this article, we have two sets of sources using two different methods
telling us this, and one of them is able to go on record, true name and all.
It's also entirely consistent with the observed results (of course, so is
having an outside integrator).

~~~
RougeFemme
Climb down. . .I'm not accusing anyone of lying or fabricating. And yes, I'm
well aware of the prevalence of hubris. I find it hard to believe because,
having worked on both sides of the fence, I've never known or heard of any
agency of any size, even DoD, doing the integration themselves. They almost
_always_ have contractors doing it, though they may manage the contractors and
call themselves integrators. I believe they may have referred to CMMS as
integrators without giving any thought to the contractor who actually
performed the integration - at the contracted direction of CMMS.

~~~
hga
Errr, I didn't think you were, and am sorry if I gave that impression.

But I've stumbled across even more confirmation, first from a _New York Times_
article on the 12th that I'm amazed we missed
([http://www.nytimes.com/2013/10/13/us/politics/from-the-
start...](http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-
of-trouble-at-health-portal.html?_r=2&pagewanted=all&)), which says:

" _One highly unusual decision, reached early in the project, proved critical:
the Medicare and Medicaid agency assumed the role of project quarterback,
responsible for making sure each separately designed database and piece of
software worked with the others, instead of assigning that task to a lead
contractor._ "

And then a _National Review Online_ (flagship publication of the Right, going
back to 1955) article by a non-specialist who got information from 5 CMMS
officials who also aren't software engineering types (but of course all
concerned are learning a lot now, as I'm sure most of us did once upon a
time). In [http://nationalreview.com/corner/361577/assessing-
exchanges-...](http://nationalreview.com/corner/361577/assessing-exchanges-
yuval-levin) he says they confirm most everything in the NYT article _but_ the
also disputed here by eyewitnesses claim that coding didn't start until
February. For integration:

" _The people I spoke with did all confirm the importance of one other detail
in the Times story: that CMS did not hire a general contractor to manage the
exchange project but handled that overall technical management task itself.
None of the people I spoke with wanted to get into how this decision was made
or at what level, but all of them agreed that it was a very bad idea and was
at the core of the disaster they have so far experienced._ "

The above also might have input from the 3 insurance industry people he talked
to.

And it's refreshing to see a lot of people put politics somewhat to the side
to figure out just what went wrong in project management and software
engineering land. The whole nation is getting an education in this, or as the
NRO author put it:

" _But [the CMMS officials] are policy and management people, not information-
technology experts.

This latter point turns out to be quite important. The reaction of these
individuals to what has happened in the last two weeks is the reaction of
people who are coming to realize that their expectations and understanding of
web development were mistaken. They believed (as I did too, I admit) that
whatever technical problems the exchange sites encountered at first could be
cleared up quickly and simply once things got going—that the contractors
developing the websites could just respond to problems on the fly, as they
became apparent. It is now increasingly obvious to them that this is simply
not how things work, that building a website like this is a matter of
exceedingly complex programming and not “design,” and that the problems that
plague the federal exchanges (and some state exchanges) are much more severe
and fundamental than anything they imagined possible. That doesn’t mean they
can’t be fixed, of course, and perhaps even fixed relatively quickly, but it
means that at the very least the opening weeks (and quite possibly months) of
the Obamacare exchanges will be very different from what either the
administration or its critics expected._"

Well, its policy type critics. A lot of us expected the common big project
screwups, which are hardly confined to the government.

------
puppetmaster3
$500mm web site. 1/2 of a billion. for a website.

~~~
dingaling
> for a website.

No, for a complex web of third-party integrations underlying byzantine
business logic. With a web site as the end-user interface.

~~~
hga
And with the powers that be _constantly_ changing the requirements.

According to the first anonymous source of in this article, up to the 4th or
6th day before launch.

No matter how good CGI Federal is, if they were being forced to constantly
rework stuff that should have been frozen months ago that'll bust the budget
and overall decrease quality as other stuff doesn't get the time needed for
polishing and testing as well.

