Hacker News new | past | comments | ask | show | jobs | submit login
Troubled Obamacare website wasn't tested until a week before launch (washingtonexaminer.com)
38 points by ericcumbee on Oct 18, 2013 | hide | past | favorite | 53 comments



I've commented on HN recently on how ABSOLUTELY EYE-GOUGE INDUCING AWFUL it is to do any kind of software work for the Federal Government. Due to a broken hiring process and union-like job security, it is filled with incompetent morons constantly scheming to find any desperate way they can to demonstrate value. (I of course am talking about the GS-12 and up levels)

The men and women who work hard and care are like the proverbial Atlas: they do the work of 15, with the other 14 people in their group using their exurban DC commute to excuse their 7AM - 3PM schedule, of which the first 3 hours of their day is spent surfing the net doing jack shit. And then these fucking assclowns get promoted.

These are the people who do things like make major interface and usability changes to a massive healthcare website and just magically expect the contractor (CGI in this case) to be able to do it. They don't give a shit about wasting taxpayer money, because it has no effect on their career at all. Failed projects don't get you demoted or fired, schedule slips are so normal that nobody even bats an eyelash....

This causes any sane person who points out the cost in time and money of bullshit bikeshed type changes to be rolled over by the horde of overpaid lemmings who are trying to yell as loud as they can before they head out at 3pm for their kids volleyball game in Manassas.

If this looks like a rant, it is. I hate the GS system. It gives government a bad name, and these overpaid assclowns do nothing but fail, supplying ammo to every Rush Limbaugh in the world advocating for elimination of government.

Please, please, if somebody with power is reading this: fix the government hiring system. Fire the lazy assholes who work 5 hour days and pull six figures. Fire the people who think not knowing how to use a computer beyond Outlook doesn't interfere with their ability to perform as CIO of an agency. Fire people who are in charge of projects that go over budget and late. Fucking fire people.

Next week I'll be going in and meeting a GS-14 (just promoted for 13) who does NOTHING at work but talk about his motorcycle and martial arts. When tasks are handed to him, everyone else groans, knowing that it will simply not be done. Instead they go behind him and perform the tasks and hope he doesn't find out, cause if he does he will pull rank and humiliate them.....

Venting over.


This is not a rant, because it's the truth. To outsiders, this attitude seems hyper-negative because it's difficult to comprehend the level of ineptitude found in federal government IT. As a non-national who has done some (very non-sensitive) on-site IT work for the U.S. government, I experienced this firsthand for several years. (And before I get accused of elitism, my home country doesn't necessarily have the answers either.)

When all of the Snowden and PRISM stuff broke, my first thought was to chuckle and ask, "How?" Based on my experiences with federal IT workers, it's hard to imagine there is a room of crack coders underground in some basement or deep inside some mountain just knocking it out of the park.


The NSA recruits at some of the best graduate schools. In some quarters, the joke is that if you have a PhD in math, comp sci or software engineering and you're not teaching or doing post-doc work, you must be working in the black world, which includes NSA.


Agreed. A young woman sitting next to me at a Hadoop oriented class during the summer (right as the Snowden info was coming to light) had just left the NSA.

She was brilliant. Little coding experience outside of Perl, but was able to keep up on the exercises in Java with only minor hiccups due to her lack of familiarity. She had been recruited out of graduate school.

On the other hand, she became extremely hostile when the Snowden incident was mentioned by people next to us (they didn't know her former employer) and was stating repeatedly that "None of you know what you're talking about, and if I was allowed to I would correct you, but I can't." and "Do you really think people can just sit down at a computer and spy on people without their supervisor's approval?" (as it turns out, we were right, and she was profoundly wrong)

It was a revealing moment, because it demonstrated the group think that can permeate even the most brilliant mind. Of course, she also thought there was nothing wrong with mathematics education in the U.S., which in my opinion makes her clearly deluded.


I agree 100% - however the alternative is just as worse. Going private means budget overruns, back room deals, and long delays.


Because in-house contracts never go overbudget.


I've known a couple of NSA programmers in the late-80s and through the 90s, and they were very good to the very best. I don't find it difficult to believe they can manage to get good people ... well, before Snowden et. al.

Whoever they got to do the power system for their new Utah site ... failed rather hard.


> Whoever they got to do the power system for their new Utah site ... failed rather hard.

Can you elaborate on this?


> The report blames the NSA “fast tracking” the Utah project and thus bypassing “regular quality controls in design and construction.”

http://www.forbes.com/sites/kashmirhill/2013/10/07/the-nsas-...


From memory there are regular power disruptions that cost "$100,000" each time. Now checking with Google: https://www.google.com/search?q=nsa+utah+power I see/remember this is delaying the opening of the site.


Wait, what's wrong with a 7-3 schedule? I mean, the "blowing off the first three hours of work" is one thing, but what's wrong with time-shifting so you can spend your after-work time with the kids instead of traffic purgatory?


I think the concern with it is that they were likely coming in early just so they could spend time goofing off before their boss or coworkers arrived and was able to check up on them.


At least, as you indicate, there are some who care and work hard and people groaning and performing the tasks of your GS-14. . .but I feel your pain. . .I've worked on both sides of the fence.


That sounds like an awesome blog, if you expand a bit on the characters ;)


What motivates you to continue to work in that environment?


What motivates me is that I USUALLY don't have to deal with that stuff anymore. I generally deal with other (private sector) clients lately, and one project that is Federal I'm working has such a hard-to-solve problem with such a big payoff that it keeps me motivated.


"every Rush Limbaugh in the world advocating for elimination of government"

As far as I know a strawman, unless you can come up with some telling quotes. You really think he's an anarchist???

Now, a return to the Founder's vision of a limited government, the pre-Hoover, not counting e.g. Woodrow Wilson during the war era and other Progressive excesses, that's another thing altogether.

Which your vent strongly supports. Basically, seriously reduce the size, scope, and most especially control of the Federal government, and return to the spoils system. No matter how bad it was and would be, it's still accountable in a way we can see the Civil Service isn't. And if the Federal government isn't trying to control just about everything, the need for a huge, "professional", "apolitical" (like the National Park Service, anyone???) "Civil" (again, the NPS is our current poster boy) Service is much less compelling.

If you can't admit to yourself that the Rush Limbaughs of the country have some good points, you are part of the problem, and will remain so.


>"every Rush Limbaugh in the world advocating for elimination of government"

As far as I know a strawman, unless you can come up with some telling quotes.

"Conservatives want to eliminate as much government from people's daily lives as possible."[1]. I suspect you'll claim the "as much.. as possible" means the OP is wrong, and I'll claim that the OP said "every Rush Limbaugh" - clearly making a rhetorical point rather than claiming Limbaugh wants to eliminate all government.

But then we'd both be missing the OP's point.

He has proposed (albeit in a rant) a solution that both progressives and conservatives should agree on: Incompetence should not be excused in government.

I'd suggest a fruitful discussion would be to concentrate on that rather than come to the predictable conclusion that people have different views on the role (and hence size) of government.

[1] http://www.rushlimbaugh.com/daily/2013/01/14/obama_and_moder...


Erm, no, he's not proposed a solution, nor do I think he thinks he has.

He's proposed a necessary tactic, mass(ive) firings, which I agree 100% with, but he hasn't proposed how to change the system so that all those ... deserving people could get fired.

I've put something on the table, the spoils system, however problematic, that will enable his desired tactic. I'm all ears for others.


How is the spoils system better?


There are certainly problems with the federal civil service, but how could a return to the spoils system be better? And to a large degree, the civil service still operates very much like a spoils system under the hood, despite all the documentation and attempts to the contrary.


Well, if it really operates very much like one, why not make it explicit?

As for how it could be better, as I said about, accountability.

Let's take a recent partial example: when George H. W. Bush became president, he immediately got rid of every "Reaganite" in the politically appointed/staff/whatever parts that a president has control over.

And the nation didn't like the results and fired his ass in the next election.


Shocked! I am shocked to hear that a major tech project with a hard deadline was pushed through at the last minute.

Kidding aside, a selfish part of me likes things like this because it lifts the veil and shows me that plenty of other people screw up. It's easy to dismiss it as a botch or idiot contractors but I'm guessing there's a lot of smart people, who worked long hours, and it's just hard to do right in these circumstances.


There was a $600+ Million price for the website that was previously mentioned in HN comments on this topic, and others were claiming that this number was debunked, and the actual price was $30 Million or so.

NBC News has now released a report on what the actual numbers are. It's $196 Million as of 5/15/2013, and the price appears to be on its way to $292 Million: http://www.nbcnews.com/video/nightly-news/53309356/

Anyway, I apologize if it looks like I'm spamming the thread with it (I posted at the bottom of the thread as a reply). That's not my intent, but I think it's important information.


Why would this be surprising. It's a project with huge scope, with dependencies that they can't fully control (IRS, Social Security etc), with likely changing requirements, and a tight deadline.

It's not like they could have released in beta and iterated, or waited until it was perfect. It's the nature of the beast - if the major annoyances don't get fixed in a reasonable [1] amount of time, then it's worth writing reams over.

1. Undefined.


The IRS interface is working fine [1]:

Sarah Hall Ingram, the director of the IRS's Affordable Care Act office, told a House panel Wednesday that the IRS hasn't experienced similar problems.

"The portion of the responsibilities the IRS is in charge of is going fine," Hall Ingram said.

[1] http://thehill.com/blogs/healthwatch/health-reform-implement...


According to the IRS.


Well, they could be telling the truth. E.g. "(If and) When we get a query, we respond to it in our N second allowed window" (as I recall the total time allowed is 7 seconds).

I'm pretty sure some decent fraction of the entities Healthcare.gov queries behind the scenes are doing their job.

Well, most of the time. One thing that occurred to me is that if certain actions require a lot of backend queries to different sources, multiplying the probabilities each could outright fail or blow the window could result in an overall flaky system.

Which doesn't directly explain why the offline part of the registration system is failing so often.


I have no particular opinion about this. However, in the context of an article that relies heavily on unnamed sources, it's worth pointing out that the Washington Examiner is a conservative tabloid founded in 2005 by Philip Anschutz.[1]

[1] http://www.politico.com/news/stories/1009/28355.html


I will make no bones about the fact I don't think Obamacare was a good idea on any level. But I say that just to put my cards on the table. Forget politics. I'm also a software engineer. Take off your politics hat and look at this with your software engineer hat. Tell me, with a straight face, that this shows any sign of being functional any time soon. Tell me that every software engineer alarm bell you have isn't going off. Tell me that this website, which consists of multiple pieces (insurance info in, user info in, user selection mechanisms, user information flowing back out to the insurance companies, and with every last one of those bits having endless, endless business rules associated with them, and this a gross oversimplification of what's there) only has problems with the user parts, coincidentally isolating all the problems into the only part we've really collectively looked at, and once those are ironed out it'll all go perfectly smoothly because this is an isolated occurrence and not indicative of the whole project.

Republicans and Democrats and bureaucrats and judges and media and internet commenters and everybody else can say whatever they like about whatever they like in whatever forum they like with as much or as little truth as they like, but the software doesn't care. It either works or doesn't. It can't be cajoled, can't be bargained with, can't be threatened, can't be ideologically swayed, and it's too stupid to be capable of any kind of cover up regardless of what its motivation may be. It is what it is, no matter which way you voted in the last election or which way you vote in the next.


We're already hearing about backend nightmares. Can't find the Washington Post article where I first read it, but this UPI one citing them has the details I remember (http://www.upi.com/Top_News/US/2013/10/12/Obamacare-website-...):

"Several industry executives told The Washington Post the daily reports on new enrollments since the roll-out of the Affordable Care Act have often included duplicative information, such as people enrolling and canceling several times in a day."

And you left out e.g. connecting to Experian to get the consumer's credit score, which is part of plan pricing (a general trend with deductibles going up). That one, I've noticed, pretty much requires the registration ... but doesn't explain why the site forbids under penalty of perjury playing "what if" to determine how you might best arrange things for you and yours.


And referencing a Wall Street Journal article that's behind their paywall, http://www.marketwatch.com/story/obamacare-woes-widen-as-ins... says:

"Emerging errors include duplicate enrollments, spouses reported as children, missing data fields and suspect eligibility determinations, say executives at more than a dozen health plans. Blue Cross & Blue Shield of Nebraska said it had to hire temporary workers to contact new customers directly to resolve inaccuracies in submissions. Medical Mutual of Ohio said one customer had successfully signed up for three of its plans."

On the other hand, if they get something, this tells us one simple method the insurance companies can generally use to resolve issues. Although it sounds like they'd best contact everyone who maybe signs up, and no doubt the system will be telling two or more companies that some customers have signed up....

A lot messier than CGI Federal's Medicare.gov's Part D prescription plan signup system, where in all fairness the info is binary (I've signed up with X plan) and both sides can depend on the site doing the right thing.


I don't wear a politics hat. I keep a brain where that would be.


"the Washington Examiner is a conservative tabloid founded in 2005 by Philip Anschutz"

That's a bullshit argument ... especially since we aren't likely to get any good dirt from the cheer-leading part of the MSM (almost all of it minus Fox, which at last count Jay Carney is refusing to call on, rather telling, that, and The Wall Street Journal editorial page).

A good argument is from a right winger like me, who from experience says that aside from the columns from major figures like Michael Barone, Washington Examiner is a very low quality rag. To the point that if I see an item linked to on e.g. Drudge, if I notice it's to them, I don't follow it. Just about any major entity on the right is better, save perhaps the Red State web site.

On the other hand, this actually isn't the typical report that entirely depends on unnamed sources. It has several from the inside, speaking with specificity, plus one who's able to go on record who used different methods comes to the same basic conclusion about who's the integrator.


Here's https://news.ycombinator.com/item?id=6572467 a much better/more reliable account from the Right, echoing the CMMS did the integration meme (which apparently first came from the New York Times), and at the end of this long piece the author looks at possible consequences.

Particularly the "danger of a rapid adverse selection spiral", that is, the people who will put up with a very difficult site are the ones who really need healthcare, whereas the "The healthy young man who sees an ad for his state exchange during a baseball game and loads up the site to get coverage—the dream consumer so essential to the design of the exchange system—will not keep trying 25 times over a week if the site is not working."

Also how fixing the upfront/front end problems will exacerbate what look to be some bad back end problems, particularly the calculation of subsidies, which all the state exchanges depend upon. Ah, and according to the 3 insurance types he talked to, they "believe that only Nevada, Colorado, Washington state, and Kentucky have what could reasonably be described as working systems at this point.", although in general they're in better shape than the federal one.


Your quotation from my post doesn't reveal an argument of any kind. It's simply a sourced fact stated plainly.


Surprise, surprise the government is still using waterfall.


Well, read the article. The claim is that requirements were still changing 4 days before the launch. That's not waterfall, that's just no process at all.


I wonder if the requirements had already been finalized and it's just the "final" signed off documentation that was delivered so late. . .yes, still broken/no process.


The fundamental issue here, like many projects of this size, is that it was trying to deliver a solution to a ridiculously complicated process. Moreover the way of procuring that solution is also fundamentally broken. Simplifying the underlying process would take a level of politics that is sadly well beyond even one party, let alone two.


Wait...it was tested?


I find it hard to believe that CMMS actually performed system integration, such as it was. I find it easy to believe that CMMS hired a contractor to perform system integration. And just as the individual contractor employees saw issues, there were probably government employees within CMMS who also saw issues. . .and just enough senior executives - government and contractor - to ignore the "naysayers". . .and insist on having their fingers deeply in the pie of such a high-visibility project.

And, of course system integration was not the only project management/technical issue with this project.


I don't see why you find it so hard to believe. Hubris is a pretty common human flaw, perhaps it was felt this was too important to leave to the professionals.

In this article, we have two sets of sources using two different methods telling us this, and one of them is able to go on record, true name and all. It's also entirely consistent with the observed results (of course, so is having an outside integrator).


Climb down. . .I'm not accusing anyone of lying or fabricating. And yes, I'm well aware of the prevalence of hubris. I find it hard to believe because, having worked on both sides of the fence, I've never known or heard of any agency of any size, even DoD, doing the integration themselves. They almost always have contractors doing it, though they may manage the contractors and call themselves integrators. I believe they may have referred to CMMS as integrators without giving any thought to the contractor who actually performed the integration - at the contracted direction of CMMS.


Errr, I didn't think you were, and am sorry if I gave that impression.

But I've stumbled across even more confirmation, first from a New York Times article on the 12th that I'm amazed we missed (http://www.nytimes.com/2013/10/13/us/politics/from-the-start...), which says:

"One highly unusual decision, reached early in the project, proved critical: the Medicare and Medicaid agency assumed the role of project quarterback, responsible for making sure each separately designed database and piece of software worked with the others, instead of assigning that task to a lead contractor."

And then a National Review Online (flagship publication of the Right, going back to 1955) article by a non-specialist who got information from 5 CMMS officials who also aren't software engineering types (but of course all concerned are learning a lot now, as I'm sure most of us did once upon a time). In http://nationalreview.com/corner/361577/assessing-exchanges-... he says they confirm most everything in the NYT article but the also disputed here by eyewitnesses claim that coding didn't start until February. For integration:

"The people I spoke with did all confirm the importance of one other detail in the Times story: that CMS did not hire a general contractor to manage the exchange project but handled that overall technical management task itself. None of the people I spoke with wanted to get into how this decision was made or at what level, but all of them agreed that it was a very bad idea and was at the core of the disaster they have so far experienced."

The above also might have input from the 3 insurance industry people he talked to.

And it's refreshing to see a lot of people put politics somewhat to the side to figure out just what went wrong in project management and software engineering land. The whole nation is getting an education in this, or as the NRO author put it:

"But [the CMMS officials] are policy and management people, not information-technology experts.

This latter point turns out to be quite important. The reaction of these individuals to what has happened in the last two weeks is the reaction of people who are coming to realize that their expectations and understanding of web development were mistaken. They believed (as I did too, I admit) that whatever technical problems the exchange sites encountered at first could be cleared up quickly and simply once things got going—that the contractors developing the websites could just respond to problems on the fly, as they became apparent. It is now increasingly obvious to them that this is simply not how things work, that building a website like this is a matter of exceedingly complex programming and not “design,” and that the problems that plague the federal exchanges (and some state exchanges) are much more severe and fundamental than anything they imagined possible. That doesn’t mean they can’t be fixed, of course, and perhaps even fixed relatively quickly, but it means that at the very least the opening weeks (and quite possibly months) of the Obamacare exchanges will be very different from what either the administration or its critics expected."

Well, its policy type critics. A lot of us expected the common big project screwups, which are hardly confined to the government.


$500mm web site. 1/2 of a billion. for a website.


> for a website.

No, for a complex web of third-party integrations underlying byzantine business logic. With a web site as the end-user interface.


And with the powers that be constantly changing the requirements.

According to the first anonymous source of in this article, up to the 4th or 6th day before launch.

No matter how good CGI Federal is, if they were being forced to constantly rework stuff that should have been frozen months ago that'll bust the budget and overall decrease quality as other stuff doesn't get the time needed for polishing and testing as well.


Please stop repeating this incorrect number unless you have a source for it.


Here are the most current numbers - $196 Million as of 5/15/2013, and it appears to be well on its way to $292 Million:

http://www.nbcnews.com/video/nightly-news/53309356/#53309356


$94 Million. Still bad, but get it right.


Nope, $196 Million as of 5/15/2013, and it appears to be well on its way to $292 Million:

http://www.nbcnews.com/video/nightly-news/53309356/#53309356


My bad, I hadn't heard that update.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: