
We paid $634 million for the Obamacare sites and all we got was this lousy 404 - Yoms
http://www.digitaltrends.com/opinion/obamacare-healthcare-gov-website-cost/
======
ktavera
For all of you that didn't know, this project was never put out for bid with
an RFP. It was sole-sourced to CGI Federal without any competitive bids.

"Federal health officials have not yet explained why CGI was given the
contract or why it was awarded on a sole-source basis."

[http://www.washingtonpost.com/business/technology/some-
say-h...](http://www.washingtonpost.com/business/technology/some-say-health-
care-sites-problems-highlight-flawed-federal-it-
policies/2013/10/09/d558da42-30fe-11e3-8627-c5d7de0a046b_story_1.html)

[Edit to add source]

~~~
mgkimsal
Are they not also a Canadian company?

[http://en.wikipedia.org/wiki/CGI_Group](http://en.wikipedia.org/wiki/CGI_Group)

Why we'd be committing to using non-US companies to develop US systems... I
guess it must happen sometimes, but on something this ... public?

~~~
alex_anglin
As a Canadian, I don't have any affection for CGI. But I would point out that
this goes both ways: Just yesterday a peace activist in Canada was acquitted
for not participating in the Census because she objected to the use of back-
end systems from Lockheed Martin. (Never mind that the Census is no longer
mandatory...)

Our nations do have a free trade agreement in place, which on the whole I
would say has benefited both sides. Patriotism aside, 'buy American' policies
do not help with these partnerships.

~~~
LammyL
Only the long-form census is voluntary. The short-form census is still
mandatory, and not completing it could land you in court, like what happened
here.

------
ck2
If it really did cost over half a billion dollars for only 50k parallel
connections, that is outrageous.

It is hard to read/understand that spending breakdown, not sure what each
20-50 million dollar allotment is exactly for:

[http://usaspending.gov/explore?tab=By+Prime+Awardee&fiscal_y...](http://usaspending.gov/explore?tab=By+Prime+Awardee&fiscal_year=all&idvpiid=HHSM500200700015I&typeofview=detailsummary)

Given how we can see the problem with this as developers, imagine what we
don't understand about the 10 to 1 or even 100 to 1 overspending on the
military and "defense" budget. Just imagine the TSA and "Homeland Security"
waste.

~~~
JPKab
It is truly an awful scenario. As I stated recently in a comment on this
subject, Federal contracting is a Rent Seeking scenario:
[http://en.wikipedia.org/wiki/Rent_seeking](http://en.wikipedia.org/wiki/Rent_seeking)

I have a weird role: I work at a company which does tons of Federal contracts,
but also has lots of private sector work. I keep my sanity by doing short term
consulting with private clients. There are some sharp people working the
Federal side, but it really is awful.

Winning a contract is EVERYTHING. It is a writing contest, and you can be sure
that the Federal contract officer knows jack shit about the actual
fundamentals of the project. The Contract Officer is a paper pusher whose job
is to look at the line items in the Request for Proposal, and examine which
company's proposal best "hits" each line item. And of course they will select
a low bidder, which often is a bid too low for the company to even profit,
simply meant to get their foot in the door.

The staffing of these Federal contracts is horrific. You get entire teams of
"change management" people. I'm currently assisting on a personnel system for
a major military organization. I recently had to witness an entire discussion
by the change management team on removing hyphens from a particular term to
help the branding. I almost threw up in my mouth.

The people I work with on the Fed contracts, in general, don't touch
technology when they leave work. Therefore the only browser they use is IE8
(according to the customer org, it is more secure......) they do nothing
outside of J2EE/.NET (usually just one of the two) and they HAVE/WILL never
build a website that supports more than 5,000 enterprise users (concurrently).

It is incredibly easy to look like a wizard amongst this bunch, but the
experience of the Federal world has made me avoid these projects like the
plague. The incentives at the business level (and therefore the high salary
jobs) are ALL for winning contracts. Performing well on the work won is just
viewed as another cost center.

~~~
dfc
_There are some sharp people working the Federal side, but it really is
awful._

What are you referring to with "it"? The folks at your company who do federal
IT work? Government IT employees? Federal IT consulting?

~~~
evilduck
Yes.

Been there, done that. Never again.

Federal government consulting firms hire warm bodies at the lowest price
possible. They are the people who employ the college students from Java shops
(because a degree is a must, of course) who barely learn Java and then grow up
to never touch another programming language in their life. Knowing anything
from MS or Oracle is double plus good for your resume. You're totally
replaceable to them and treated accordingly. Anyone competent or specialized
is generally hired as a subcontractor to the main contractor and not by the
government itself.

Government IT employees are nigh-unfireable and they know it. They're about
50% people who've transferred in from other Government positions because they
know it's easy or they're former contractors who join the government so they
have a stable job. The government managers who vet your deliverables and the
upper managers who make contract decisions, well, they're just the previous
technically incompetent people who've stuck around long enough to get promoted
up.

As the individual contractor, contracts are usually 1-3 years so you're
constantly always up for being laid off or moved to a different contract. Your
company is notoriously stingy, but that's mostly because they're not paying
you for competitive skills, they're paying you because butts-in-seats means
they get their lucrative contract rate that's at least 4x more than they're
paying you. Since only behemoths of companies can even afford to try to land
federal contracts, your employer has a large amount of corporate bureaucracy
you have to deal with and that sucks... then you're sent to a government
facility to work, where you get to deal with government bureaucracy on top of
your internal crap too. As a contractor, between both organizations you'll get
to take hours upon hours of totally useless 'computer safety, ethics and
public responsibility trainings' which are mostly written so 4 year olds can
understand them.

The government technology you work with is generally 5-10 years behind the
curve if they even pretend to maintain something. As late as 2010 I was
working on a J2EE app trying to migrate it to _Java 5_. It was stored in
Visual Source Safe 6, and their development shop was totally crippled by what
was essentially "God objects" and file locking. Too many developers, too many
chokepoints in code. I've spent more than 1 day of my life walking around a
building asking if people really needed a certain file locked. Oh, and many
government employees worked 9-40 compressed schedules and are out on Mondays
or Fridays, so you'd better not need to write to a file except on Tues-
Thursdays, even if it is critical to meeting a deliverable deadline.

~~~
evilduck
Extra fun side notes:

While holding a technical discussion about current tasking with a coworker in
a "hallway meeting" our boss's boss and the gov-manager-who-signed-the-
contract walked by. I was later reprimanded for chit chatting too much instead
of working.

Another issue, due to some contract specifics and my life situation I was able
to decline healthcare coverage from my employer and I received that in salary
compensation instead (had coverage through my wife). I was up for a
raise/promotion due to 'time served' but that title meant they needed me on a
different project/contract, so the healthcare compensation the government was
paying my employer changed and trickle down effect meant I didn't get that
anymore. I got a $10k/year 'raise' but lost my [substantial] healthcare
compensation. I lost money per paycheck by getting a promotion. When I showed
my boss he was like "oh, sorry about that but we can't do anything about it",
I turned in my two weeks notice on the spot, thus concluding my life as a
government contractor.

~~~
evilduck
Another one:

We had a government employee who was responsible for building our mess of a
Java project (it was done by hand). It was error prone due to some self-
referential build requirements, and the deployment of projects had to be done
in a specific order. Generally a deploy took a couple tries.

While I was there I skunkworks-implemented Jenkins CI (nee Hudson at the time)
to deflect constantly getting blamed for broken builds and down time when they
occurred. The side effect of CI was that I had automated away this government
employee's entire job and reason for being paid. One day that employee was out
but we still needed to deploy, so in my naive excitement was like _I 've got
it guys, watch this_, pointed the deploy target at our live acceptance testing
server, clicked the build button and ta-da, gov. employee wasn't needed and
nothing was screwed up. Even better, it was completely deployed in a few
minutes instead of several hours.

What do you think the result was?

The CI server VM got turned off.

~~~
JPKab
The average government employee is beyond useless, and sadly the few who are
good get a bad reputation and also a huge workload dumped on them by the
majority.

One of the engagements I get called to once a week involves dealing with some
gov employees. It had the odd property of continuing when the shutdown hit, so
its the contractors and no gov people. Productivity on the project is
completely unaffected. Half of the workforce gone, no change in productivity.
There are certainly less meetings, so that's nice.

------
jljljl
Reread the article, this is complete nonsense. The $634M number is 6 years of
contracts, starting in 2008 (Obamacare was signed into law on March 23, 2010),
which doesn't align with his claim that CGI Federal won the contract in 2011.

The solicitation number he uses to filter the USASpending.gov site is also
tied to a larger PECOS contract, and not to Healthcare.gov like he claims.

We're being had guys. This is poor reporting at best, or deliberate
obfuscation at worst.

~~~
driverdan
Maybe he got the total wrong for healthcare.gov but $634M should be enough to
build and run every government website in the US for more than a decade.

~~~
jonknee
That's a ridiculous statement. $63M a year to build, run, maintain and host
unknown numbers of web sites (thousands!) and services that process trillions
of dollars annually seems absurd.

The total budget for all government websites isn't important (for starters
they aren't paid for out of one bucket), it's just important that individually
each one is managed competently.

------
ScottWhigham
This is what frustrates me about many of our non-USA HN'ers here: every time
the state sponsored health care issue comes up, the non-USA folks can't wait
to talk about how their country has it, or how it costs this little to get
coverage for this or that problem in their country, and then they wonder why
"you Americans" put up with the system you have currently in place (self-
insured, no state plans - pre-Obamacare). I've read my share of comments
(often from European HN'ers) that mock USA citizens' mentality with regard to
state sponsored health care here.

Well here you go - this is the perfect example to share with you as to why
most Americans didn't/don't want "Obamacare". We, the voters, have no
confidence that the current system+administration+Congress (or previous 10 of
each!) could've created a system for "health insurance for all" that worked
and was efficient. It's not anything against Obama - it's that we've seen
administration after administration try to implement some big, sweeping
group/plan for 20+ years and every one of them has turned into an inefficient
holy hell of a mess. The most recent example of a major #$%&-up is Homeland
Security (which I think is the last major agency created).

If the American public believed that the current government was capable of
delivering state sponsored health care in an efficient way, every Democrat and
Republican in this country would've voted for it. So don't think of "those who
are against Obamacare" as anti-Obama, but rather anti-inefficiency (or anti-
bigger government).

~~~
brown9-2
If the argument though is "we can't have nice things because we screw it up",
perhaps we need to look more at why America tends to screw these things up
instead of taking away the lesson that we shouldn't try to have nice things.

~~~
hga
While many of us would argue with the "nice things" concept (ScottWhigham
grossly overestimates the nation's enthusiasm for socialism at the beginning
of this thread), it doesn't take away from his point that it's stupid to try
such grand projects while we have a demonstrated inability to do them.

I mean, _the government is taking control of 1 /6 of the nation's economy_,
why ever did the Obamacare enthusiasts think this would work even vaguely
well???

~~~
brown9-2
To be fair, the government isn't "taking control" of healthcare - they haven't
assumed control over every private health insurance company.

~~~
hga
And how many of those "private" insurance companies can still write "major
medical" high deductible catastrophic coverage polices that so many of us have
used in time past as ... gasp, _insurance_ , as opposed to a tax advantaged
benefit + originally a way to get around WWII wage controls? Etc. etc.

In a system that's structurally rather close to Original Formula Italian
fascism, how "private" nominally non-government companies are when so much of
what they can and cannot do is highly debatable, to the point where I think
your quibble is very very minor.

------
3minus1
Hi, I actually work for CGI Federal on a closely related website to
healthcare.gov. There's some inaccurate speculation going on in this thread
that I want to correct. Employees at CGI Federal are not payed hourly. Also, a
lot of people have mentioned cronyism, which I think is baseless. CGI Federal
already had contracts for Medicare.gov and CMS.gov, so when CMS had to build
healthcare.gov, CGI was an obvious choice.

edit: Also, the projects I work on are Agile and I get to use other
technologies besides .Net framework (e.g. node.js and backbone.js)

~~~
pkfrank
>Also, a lot of people have mentioned cronyism, which I think is baseless. CGI
Federal already had contracts for Medicare.gov and CMS.gov, so when CMS had to
build healthcare.gov, CGI was an obvious choice.

You may be correct, but arguing that "because we generally get these
contracts, we were an obvious choice" does little to defuse assertions of
cronyism.

~~~
3minus1
> "because we generally get these contracts, we were an obvious choice"

There are good reasons a government body might want to give a contract to
someone they worked with before. There's a previous track record of success,
and they know that the company already has expertise/solutions for their
specialized requirements (i.e. section 508 compliance).

~~~
walls
> There's a previous track record of success

Where? Definitely not the Canadian Firearms Registry, or the sites listed
previously.

~~~
genericresponse
Full Disclosure: I'm not certain who 3minus1 is, but I used to work on the
project he/she works on.

There was content at Healthcare.gov prior to 10/1\. The applications that
delivered the content to the site had very similar function to the ACA
application. Intaking consumer information and outputting potential insurance
options. Also, gathering insurer data to do this. Not a full prototype, but
damn close. I like how CGI Federal is taking it in the teeth when there are a
number of contractors who worked on this. QSSI, Experian, Etc.

------
danso
As has been noted on HN at the time, the front-facing part of the site was
released as open-source 3 months ago:

[https://github.com/CMSgov/healthcare.gov](https://github.com/CMSgov/healthcare.gov)

You can deploy it like any other Jekyll site. The code and content has changed
since then but I imagine it still has the same static front facing
architecture, much like the Obama campaign fundraising site, which famously
raised $250M using Jekyll static pages: [http://kylerush.net/blog/meet-the-
obama-campaigns-250-millio...](http://kylerush.net/blog/meet-the-obama-
campaigns-250-million-fundraising-platform/)

So the number of raw visitors may not be the most relevant number, as many of
them may have hit the front page and left, or never got around to the signup
part. But what exactly was the technology in the back-end stack?

------
DanielBMarkham
Political organizations exist for political reasons.

So if you get elected to "do something" about orphans, you'll create an
organization to do something. The goal of this organization is political:
appear to be making progress on the orphan problem. At the very least, do not
appear in the news as an example of government waste.

The current website problem is a political failure -- it looks bad. But that's
just a short term consideration. The long-term bet is that over the next
decades, the ACA will bring great political benefit to the political party
that supported it, no matter what other things it does.

So when we evaluate projects created by political organizations for political
reasons, the success criteria is much different than commercial or non-profit
projects. I don't think this is a failure. Maybe a bump in the road, but it's
nothing that won't work itself out over the next year or so. (And be long
forgotten)

Remember, a lot of government contractors made money building these sites. A
lot of people had jobs. A lot of committees and functionaries are able to add
this to their list of good they've done in the world. Not being able to
actually use the site for a while is a small pittance compared to the real,
measured benefit the sites have created. So far. If it drags on for a long
time, the political math could switch around the other way, but I doubt it.

~~~
001sky
The problem with this logic is that carte balche on an amex is no way to run a
government, even if every dime spent is sucessful in "keeping you in power".
To the extent that one is rationalizing the status quo, it is pointing out the
obvious. Unfortunately, the math no longer works and the system is going to be
reset as _someone else_ has to pick up the tab. It may be republicans or it
may be your children. But whomever gets stuck with the bills is going to be
pissed off, methinks. Part of being a successful politician is not creating
deep-seated enenimities.

~~~
DanielBMarkham
When the U.S. was founded, slavery was a key concern. Jefferson and others
knew that slavery couldn't last, but there was no way to get agreement on it.

So what did they do? They left the problem to future generations to solve. And
the result involved a lot of bloodshed.

In any system of government, there's always going to be a strong desire to
kick problems down the road for somebody else to solve. I think the best we
can hope for is some structures in place to minimize this, but you'll never
get rid of it completely.

What we see now is just 250 years or so of this thinking, with the default
solution of letting somebody else handle it growing more powerful with each
passing year.

~~~
001sky
So this will be solved when we get a hybrid of Abe Lincoln and Steve Jobs
elected president =D ie, someone who see's the big picture and has the
discipline to bring a product to market that doesn't suck. I think being an
optimist is looking at your problems and facing them down. Aknowledging things
are hard only adds to the glory of overcoming the odds.

------
tokenadult
National Public Radio had a good report on 8 October 2013, "Health Exchange
Tech Problems Point To A Thornier Issue,"

[http://www.npr.org/blogs/alltechconsidered/2013/10/08/230424...](http://www.npr.org/blogs/alltechconsidered/2013/10/08/230424841/health-
exchange-tech-problems-point-to-a-thornier-issue)

that discusses the broader issue of United States federal government
contracting for information technology services. Some reforms are suggested in
that story that would help more competent startup companies compete against
the established federal contractors that win most of the big contracts. The
specialized skill that the incumbents bring to the contracting process is not
specialized skill in data-processing or programming for federal agencies, but
rather specialized skill in navigating the federal bureaucracy for bidding on
federal contracts.

~~~
puller
The difference between established federal contractors and startups disappears
if startups get federal contracts. I know a lot of startups would like a
better shot at that cash, but if the problem is in the incentives then it's no
help to get different people doing the same things.

If there are already problems with how federal contracts are awarded, it is
hard to imagine that wholesale deregulation is going to result in a more
selective process. For example, blind removal of lowest-bidder requirements
would make larger-scale corruption even easier.

There are legitimate complaints and federal contracting needs work, but what
kind of work? Let's not underestimate how much worse we can make it. The money
on the table is ample incentive for people to propose innocent-sounding
reforms which really just open up the taps or redirect them to different
parties rather than increasing efficiency.

~~~
mgkimsal
Having a scoring system that takes in to account how many previous govt
projects have gone over budget and over time would help. Yes, small company X
might not have a track record, and might end up folding in 2 years (especially
if they can't get decent sized contracts to stay in business), but is that
necessarily any worse than bigcoXYZ getting contract, then taking 3 years
longer than proposed and going 400% over budget?

Look at the track record of the company, and use that in factoring in contract
awards. Companies that routinely go over budget and over time should be
penalized by having a reduced chance of getting contracts in the first place.
That's one thing that would help level the playing field a bit.

Of course, what would happen is those larger companies would create related
spin-offs that are technically not related, and they'd have 'fresh' stats to
compete with, and the cycle would start over again, probably.

------
zachrose
There are a lot of interesting things to say about a $634M website that didn’t
work when it should have, but what pops out at me here is the description of
“poorly written code.”

This article even links directly to some of the code, which A) looks fine? and
B) is like “Yeah, the problem wasn’t with procurement regulations or clear
requirements or public-private cronyism or managerial competence, it was that
damn person who told the computer what to do and the computer didn’t do the
right thing.”

~~~
joshuahedlund
Here is my personal experience with some of that "poorly written" code:
[http://www.postlibertarian.com/2013/10/yes-the-obamacare-
web...](http://www.postlibertarian.com/2013/10/yes-the-obamacare-website-
really-is-really-bad/)

> the text on the sign-up page, the front-end javascript validation on the
> sign-up page, and the text on the help page _all have different
> requirements_ for valid usernames in the database!

~~~
penguat
That sounds like poor BA/spec work. BA asks: "what do we need from this
username". BA should ask "what do we need from usernames overall"?

It's always going to be hard to be agile on federal contracts - there're too
many customers to really get a clear view from them.

~~~
joshuahedlund
A fair point. Other examples of problems were empty drop-down menus from
presumably non-cached failed DB accesses[0] and forgot password links in
emails leading to an error about no match found for the provided information
[me], which I suppose you could respectively file under 'unexpected heavy
load' and 'bug' rather than 'poorly written code', though at some point no
matter how you classify the problems it makes me nervous to submit my
information because I wonder what security holes are waiting to be discovered
along with the other problems.

[0][http://www.bloomberg.com/news/2013-10-02/untangling-
obamacar...](http://www.bloomberg.com/news/2013-10-02/untangling-obamacare-s-
web-glitches.html)

------
mark_l_watson
I lost my group health plan, which I had for fifteen years, recently so I has
experienced the web site first hand :-)

My latest complaint: I logged in many times last week to fill in information,
a bit at a time before I lost access.

Then starting Sunday morning I could no longer login. On the phone last night
I was told that they had wiped all stored passwords and to follow the "I don't
remember my password" link. Couldn't they have posted that as a huge banner
message and let people know? I had four days of frustration trying to login.

Then, when I got the reset password link, and followed it, one of the profile
verification questions they asked for resetting my password WAS A QUESTION
THAT I HAD NOT BEEN ASKED so I had no way to answer it. I wrote down my
profile questions so I am fairly sure of this. wTF

~~~
nutate
To play devils advocate, have you tried logging into Aetna, Cigna, Blue Cross
Blue Shield, etc recently?

Just as fucked in terms of passwords being screwed, lockouts, other such
nonsense.

~~~
Raphael
No, but I've signed in to Facebook, Google, Reddit, etc. It's a solved
problem.

~~~
nutate
Most cutting edge computer science folks like to make money selling ads not
selling cheap insurance.

------
pkorzeniewski
I'll never understand how it's possible that government web sites cost so
much, yet usually look like shit, work like shit and are easy to hack (at
least in my country). I always get the feeling that companies that win the
contracts are somehow related to people responsible for the decision, it's
like "Hey, it's not our money, so let's milk the budget as much as possible!".

~~~
BlarfWobble
The short answer is that it's not just a website. The website is just the
front of a large, complex project that involves many other things.

I have noticed how shocked people are when they hear how much custom software
really costs. I don't know what you do for a living, but here's an example:

A family member hears you make software, ask you if you can create a website
for his business for a friendly fee. Sure, you think. After reviewing his
requirements, and deducting 50% because he's your uncle and you like him and
want his business to do well, you give him a quote: 3 months and $10,000. He
nearly gets a heart attack. What had he expected then? Well, a copy of windows
costs less than $100. How could it be more than that?

~~~
coldcode
Tell him it's only $100, but he has to buy at least 100 copies. Then see what
MS spent on Windows and how many copies they had to sell to pay for it.

~~~
BlarfWobble
It wasn't a parable about older family members being clueless when it comes to
tech. People really can't imagine how a "new computer system" can cost 3 years
and $90m to build, and are genuinely outraged, because it is so far removed
from what they expect to be a normal or fair amount for a piece of software.

You pay a dollar for an app, $40 for a Sims game, maybe a few hundred for an
OS and office suite, maybe $1500 for a fancy laptop. Sure you can imagine a
'website' might cost a few thousand, but _tens_of_millions_ ?? Surely, that is
a disgraceful waste, right there.

------
jljljl
This seems a little confusing to me. I'm using the site linked in one of the
previous comments:

[http://usaspending.gov/explore?tab=By+Prime+Awardee&fiscal_y...](http://usaspending.gov/explore?tab=By+Prime+Awardee&fiscal_year=all&idvpiid=HHSM500200700015I&typeofview=detailsummary)

This shows the $634M headline number as the amount of contracts paid to CGI
Federal over a ~6 year period. It is filtered based on solicitation number
HHSM500200700015I.

A little googling returns the following page:
[https://www.fbo.gov/index?s=opportunity&mode=form&id=f1522d0...](https://www.fbo.gov/index?s=opportunity&mode=form&id=f1522d0262c63d01b34fab07c6ed5634&tab=core&_cview=0)

    
    
      "The Centers for Medicare & Medicaid Services (CMS) intends to modify the PECOS contract to 
      support the American Recovery and Reinvestment Act of 2009 (ARRA) requiremnts for the 
      Development, Maintenance and Enhancements of HITECH Registration, Attestation and Inquiry Functionalities.  
      This work is already on the contract, the modification will incorporate costs for the option years."
    

So there's not really a lot of detail on what the HHSM500200700015I
solicitation actually entails, and I haven't found a clear description
everywhere else. It looks like it could cover a pretty broad set of work. For
example, the ACA was signed in March 2010, but there's contracts tied to this
solicitation number that go back to 2008-2009.

A few of the news sites linked below imply that the $634M covers the total
number of Medicare and PECOS contracts awarded by HHS to CGI, and that the ACA
website cost only $93M, which makes it seem cheap compared to the private
sites listed in the Digital Trends article:

[http://washingtonexaminer.com/canadian-firm-hired-to-
build-t...](http://washingtonexaminer.com/canadian-firm-hired-to-build-
troubled-obamacare-exchanges/article/2536805)
[http://www.foxnews.com/tech/2013/10/09/3-million-
obamacare-w...](http://www.foxnews.com/tech/2013/10/09/3-million-obamacare-
website-may-face-months-glitches-experts-warn/)

Before we all explode in outrage, are we sure that we are reading the details
of this correctly?

~~~
dmix
> Before we all explode in outrage, are we sure that we are reading the
> details of this correctly?

> the ACA website cost _only_ $93M

What more details do we need? There's much worse going on at the same time
than that ridiculous number. But it is totally sufficient in on it's own to be
outraged.

~~~
jljljl
What is the appropriate cost for a health insurance marketplace that needs to
support 33 states, with the security and privacy requirements such a federal
system entails, and with the need to support millions of customers as soon as
it opens?

Can you provide a more reasonable, informed cost estimate?

------
malaporte
A colleague of mine used to be some kind of manager/director at that company
(in another country though). He once described his former job as "milking the
government".

They hire the lowest quality engineers (e.g. the cheapest), and then bill them
outlandish $/hour to work for government agencies. The project plans are
packed with useless stuff, and are designed to exceed initial estimations.
They get paid by the hour...

------
sailfast
Things not considered by this and other articles that I would urge readers to
consider: 1) RFPs are often written poorly by non-technical people, with
requirements that are not accurate at the time. These requirements then change
a lot to reflect reality which results in a lot of wasted effort and
redirection. (This is probably also the case in large Enterprise
implementations)

2) Compliance with government regulations costs money. Lots of money. This
results in a lot more overhead. It also results in a lot more time to get
people up to speed, on site, and going. This is why government contractors
keep winning bids - compliance costs are huge barriers to entry.

3) Systems you need to integrate with in government (especially legacy
systems) can be a complete pain in the butt. It's more likely you're
integrating with some FORTRAN green screen than a nice JSON API. This makes
large scale systems integration hard.

That said...the app is still very broken and there is obviously a failure
here. Failure to test properly (otherwise poorly written tests), failure to
open to competitive bids judging from another comment in this thread, and many
other issues.

There is a LOT to be done to improve IT acquisition in government, and many
things should have gone right that went wrong for the money spent (Figures I
saw were more like $138 Million in other publications), but readers should
please consider the organizational barriers and difficulties that exist and
then factor it into the cost. It doesn't take the sting away, but it does
lessen it a bit.

------
pingswept
Hmmm. I just created an account. The site loaded quickly. Took maybe 2 minutes
to create an account. The password rules were weird, but I don't see what all
the fuss is about. It seems to me that scaling a site from 0 users to millions
in a couple of weeks is hard.

If it's working now, which is seems to be, what's the problem? (OK, I'll
concede that $634M is totally and completely insane. But I'm not convinced by
the rest of the "bad code" complaints.)

~~~
mgkimsal
Can you actually log in and do anything? I created accounts last week, and now
can't log in with them, or can log in and get a blank white screen.

I've just tried creating yet another count. It's hung up on 'step 3' ('please
wait' \- modal spinner, assuming it's emailing my email address). 3 minutes
spinning so far.

OK - it just got done. Got an email. Clicked the link. Logged in. Pause...
Taken to
[https://www.healthcare.gov/marketplace/auth/userprofile](https://www.healthcare.gov/marketplace/auth/userprofile)
which is just a white screen.

So... still not working for at least some people.

~~~
pingswept
Just clicked the confirmation link that appeared in my email. Had to accept
some terms and conditions. I can't actually get insurance this way because I
already have insurance through my state's exchange (I pretended I lived in
Maine for this test account, since my state doesn't use healthcare.gov). But
my profile page loaded fine. This series of steps also took less than a
minute.

Anyway, sorry it's not working for you, if you're really trying to get
insurance.

~~~
mgkimsal
I'm really trying to get my options. I'm in NC and I think they're limited.
I'd like to get the official word from the horse's mouth, so to speak, and I
can't get in.

------
wikiburner
Scaling on the web is basically a solved issue for sites with such limited
functionality. Even the CIA is using AWS, so I can't believe the regulatory
and security hurdles were prohibitive.

To spend this kind of money on sites with these kinds of results is just
disgraceful:

 _" The site is so busted that, as of a couple days ago, the number of people
that successfully purchased healthcare through it was in the 'single digits,'
according to the Washington Post."_

[http://www.cbsnews.com/8301-505269_162-57606633/obamacare-
we...](http://www.cbsnews.com/8301-505269_162-57606633/obamacare-website-
looks-like-nobody-tested-it-programmer-says/)

~~~
josefresco
Beware of potentially politically motivated garbage quotes. The referenced
Washington Post article quoted an "insurance industry official" who would only
speak anonymously:

 _“Very, very few people that we’re aware of have enrolled in the federal
exchange,” said one insurance industry official, who like many in the
industry, spoke on the condition of anonymity out of concern for possibly
offending the Obama administration. “We are talking single digits.”_

So _one guy_ allegedly in the "insurance industry" who won't give his name has
said enrollment has been in the single digits. Not overall enrollment mind
you, just what this one guy has seen come across his desk.

But that's how this article has been spun. Millions of people! single digit
success rate! as reported by "some guy". A sad state of affairs indeed.

~~~
wikiburner
How about the CBS video linked above? Or the links below? Or the countless
other media accounts of the launch being a "complete disaster"? Or all the
comments on this very page of HNers not being able to use the site?

Not even the administration's biggest defenders are trying to spin this as a
successful launch.

Are you really implying the media as a whole is biased against Obama?

[http://www.cbsnews.com/8301-505269_162-57606633/obamacare-
we...](http://www.cbsnews.com/8301-505269_162-57606633/obamacare-website-
looks-like-nobody-tested-it-programmer-says/)

[http://bigstory.ap.org/article/poll-rollout-health-
exchanges...](http://bigstory.ap.org/article/poll-rollout-health-exchanges-
gets-thumbs-down)

[http://www.washingtonpost.com/business/technology/some-
say-h...](http://www.washingtonpost.com/business/technology/some-say-health-
care-sites-problems-highlight-flawed-federal-it-
policies/2013/10/09/d558da42-30fe-11e3-8627-c5d7de0a046b_story.html)

~~~
spiffyman
I don't think anyone's denying that this launch is a failure. It's clear as
day. josefresco's point, though, seemed to be that _this particular quote_
could easily have been politically motivated. None of the articles you link
restate that 'single digits' claim. One of them even states the number could
be upwards of 20M.

Just saying: take the 'single digits' claim with a grain of salt. You don't
need to buy every ridiculous claim about this to acknowledge that this is a
bad launch.

~~~
wikiburner
Hey spiffyman, reasonable points, but it really looks to me like josefresco is
implying that the launch is getting unduly harsh treatment in the media, but
maybe I misread it.

Also, regarding the 20 million figure, here's the quote:

 _Seven percent of Americans report that somebody in their household has tried
to sign up for insurance through the health care exchanges, according to an
AP-GfK poll. While that 's a small percentage, it could represent more than 20
million people._

As many as 20 million _" tried to"_ sign up, based on their extrapolation of
some polling data.

The last I heard, no media outlet has been able to find a single successful
sign up to interview, and they are definitely looking for one.

The CBS video also claims there have been only "a handful" of sign ups, and
it's getting reported in lots of other places. It could be inaccurate, but
these are mainstream outlets reporting this, not Breitbart or Drudge. The
administration is refusing to release sign up numbers, which certainly doesn't
look good.

~~~
somebehemoth
"As many as 20 million "tried to" sign up, based on their extrapolation of
some polling data."

The quote refers to "single digits" and it is ridiculous. If even close to 20
million people tried to sign up then that means many millions of citizens need
this coverage and will likely end up getting it because having health
insurance is critical to their livelihood. Am I to believe 20 million people
needed health insurance, got an error on the web site and just said, "forget
this, being uninsured is better than dealing with a web site!"?

"The last I heard, no media outlet has been able to find a single successful
sign up to interview, and they are definitely looking for one."

This is a weak argument. It is an anecdote of an anecdote which even if true
would not be persuasive. Are you seriously saying that no one has signed up
despite 20 million people trying (or single digits)? Your evidence is that you
personally haven't found a media outlet that have themselves managed to find
someone to interview? Perhaps we should wait a few weeks to make judgement
after the facts are known and the government is actually running again?

~~~
wikiburner
We seem to be having completely different arguments. At no point did I say
there was no demand for Obamacare. I'm saying the government has shown a
remarkable level of incompetence at running what's essentially a simple lead
gen website. I'm also suggesting that this probably doesn't bode well for the
successful administration of the program.

Regarding your second point, it's become somewhat of a meme in the press that
no reporter has been able to verify or interview a single Obamacare sign up.
That's what I'm referring to - not that I myself haven't by chance come across
such an interview.

~~~
somebehemoth
Okay, I concede you didn't imply a lack of demand and that I misunderstood
your primary point. For what it is worth you are making a lot of assumptions
in order to make your point that Obamacare might not succeed.

My second point stands. Your experience of the media coverage, while valid,
doesn't necessarily represent fact. You have to admit it is possible that a
reporter managed to interview someone who signed up and you and your news
sources were unaware. Even if you proved this point to be true what does it
say? No one has signed up? People don't want to be interviewed? Obamacare will
fail? I don't get it.

------
grej
Eisenhower warned about the military industrial complex. This is just the
latest manifestation of the government-industrial complex. I'd be willing to
bet that CGI Federal (the contractor) has some very well connected cronies to
have captured this boondoggle.

------
Tloewald
> the factors that play into which companies receive government contracts, a
> process called “procurement,” are fundamentally broken

This.

~~~
Osmium
How would you fix it?

I could imagine a competitive process, e.g. pay a number of different
companies to start producing their solution in parallel. And then at the end
of 3 months, pick the one(s) that looks the most promising and pay them to
continue with it. It sounds wasteful, but I just don't see a better way,
because people seem incapable of judging beforehand which companies are
actually capable of providing on their promises.

~~~
mseebach
Generally: get government out of the business of procuring custom software and
shift to procuring outcomes.

Something like these healthcare exchanges should have been put to the market
in a from where the revenue is directly tied to the end goal: The successful
purchase of health insurance by customers (or whatever is the case here, I'm
not intimately familiar with the mechanics). Have providers bid on the full
thing by revenue pr. successful transaction + an SLA that deducts an
appropriate penalty if it isn't met.

Ideally, you don't even need to pick a winner, you can just say that any
provider that successfully operates such a marketplace will get the revenue
per transaction and let the market sort it out. End users will gravitate
towards the sites that are fast and easy to use, "Consumer Reports" will test
them etc. Lean, agile shops will win. Clunky, well-connected, but technically
unsophisticated government contractors will fail.

~~~
d23
Yeah, because you're going to get the highest quality from firms that want to
do completely speculative work. That's why 99 designs has the best designs,
right? If you know you're the best, you'll compete for free!

~~~
mseebach
It's not speculative? If the work achieves the stated end-goal-metric ("A
citizen of the state of South Tennechigan can purchase a health insurance plan
subject to the given constraints.") the operator is awarded the revenue
offered ("$150 per successfully sold health insurance plan").

Think of it as a feed-in tariff. If you successfully purchase, install and
connect a solar panel, and the panel feeds power into the grid, you're paid.
If not, because you screw up any of these steps, you don't get paid. If you
make a substandard installation (say, in the shadow or on a north-facing
roof), you get paid less. The only thing that matters is the desired end goal:
solar power is fed into the grid. You carry the risk, you get paid for the
result.

The idea is that (at least in these cases) it's much, much easier to describe
and document a desired outcome than to accurately describe and anticipate all
the variables that potentially affects the outcome.

~~~
rtkwe
The problem comes in when you factor in the danger of ever changing political
and project climates. In this case if the Republican party had been successful
in revoking the ACA whatever work the contractors were doing was now
completely wasted through no cause or fault of their own.

In your analogy it would be like you're successfully installing the panel and
suddenly there's a massive tree completely shading the solar panels or the
power company suddenly refusing to accept feed-in power from residential
structures.

~~~
mseebach
That's a risk of doing business. AirBnB and Uber are running similar risks
against the political system (probably worse - they don't have ~half of the
political system in their corner). Every other business is always risking that
nobody will buy their product. Having any kind of promise of a guaranteed sale
is a pretty luxurious position for any business.

If the risk is large enough that nobody takes it, then the procurement office
can look at sweetening the deal (perhaps by increasing the payment per
transaction, perhaps by offering some compensation for good faith effort if
the legislation is revoked).

There is enormous overhead in bidding on and winning a government project.
They're the reason the "37signals" of the world isn't doing them even even
though they could probably execute much better on them.

~~~
d23
If a client came to us and said "hey, we have a project for you. You just have
to build it and we can split whatever profit you make off of it", you can
guess where we'd tell them to go. It's very different from basic business
risk.

------
smoyer
Imagine what this community could build with more than half a billion dollars!

~~~
anarchitect
Take a look at what the GDS is doing here in the UK.

* hire really smart people with up-to-date views on technology [1] * build on, and release new open source [2]

[1] [https://www.gov.uk/designprinciples](https://www.gov.uk/designprinciples)
[2] [https://github.com/alphagov](https://github.com/alphagov)

~~~
quickpost
That is really cool. Thanks for posting.

------
dccoolgai
This is absolutely nothing compared to what they waste on technology in DOD...

~~~
devx
I'd be happy if they were actually "wasting it on technology", but this seems
like corrupt money, going into the pockets of big corporations for very little
and poor work. They probably charge at least 10x what it's worth on the
market, and do work that is worse than the average on the market. It helps
that the people signing these contracts are either clueless or don't care,
since it's not their money.

------
mploi123
I have worked with IT consulting for the government (in Brazil), and if the
software needs of the american government are similar to the Brazilian ones,
you can't compare twitter, instagram, etc. with this! They are completely
different kinds of software, governmental softwares have tons of integration
with old COBOL systems that runs millions of sensible data, and tons of
requirements, really, pages and pages of different use case scenarios.
Governmental software require more man-hours than successful start ups, even
when they work and succeed as a project. The "MVP" of a governmental system is
huge. Now, don't get me wrong. the price is absurd, this example is clearly a
total failure. It's just that this article compare apples to oranges.

------
tomasien
The sad part about this is, from what I can tell, there's not going to be any
fixing this any time soon. The company who wrote the code doesn't know what
they're doing, and nobody else is ever going to be able to figure it out NOW.
Sad state of affairs.

------
dkhenry
Good thing we are giving the federal government control over the healthcare
system ( and positioning them to take even more control as we move towards
single payer ). I am sure they won't screw it up like they have the the roll
out of this small portion of the system.

Also I am really looking forwards to the day I can't get my kid seen by a
doctor because there was a budget fight and the government shut down.

~~~
JPKab
Please explain to me how an exchange stood up by the government which enables
you to find a private insurance company to purchase insurance from is
involving the government in your use of the private insurance.

The new system acts as a facilitator for people to purchase insurance on an
open "exchange" market. The government has as much to do with what happens
afterwards as the NYSE has to do with how my 80 shares of Exxon perform. If
the stock goes down, I can surely blame the Exxon board, or an oil spill....
but it certainly has nothing to do with the NYSE.

You should read about a law before you talk about it.

~~~
dkhenry
So the government has a monopoly over dolling out plan s on the exchange. In
addition you need to be "cleared" by their system before you can get
subsidies. Also there are subsidies, so they are directly paying for parts of
healthcare. I mean before this I would call my insurence company and I would
work with them for insurence. Now before I get insurence I or they need to
interface with 30 government agencies to get information about me so I can be
approved for a plan.

Then I can't even choose a plan I want since the goverment mandates levels of
coverage.

Please take your blinders off your enabling your party to do horrible things.

~~~
debian69
Dude here in england where the government provides health care , it works out
that private care is worse in standard (more fuckups personal experience with
private cancer care vs nhs cancer care) and generally the public option runs
trouble free , sure there will be teething problems on the first days of it
opening but in 10 years time , you'll be wondering what kind of idiotic state
ever decided to make it a private only system in the first place and what self
depreciating lies you've had to swallow about freedom and open markets to get
into this state.

------
SmileyKeith
It definitely seems like the amount of money the government spends on projects
like this starts out at something outrageous no matter how complex.

------
gexla
They should have opened the bidding process to Elance, they could have found
someone to do the back-end for like 5K. ;)

ETA:

> And when things still go wrong, they simply throw “more money at the same
> people who caused the problem to fix the problem."

For that much money, I could be a terrible web developer.

------
lnanek2
Haha, yeah, I spend hours across two days on those government health exchange
sites (the US one and NY one) just trying to get a quote. Kept erroring out
and forgetting my history and other problems.

Meanwhile Freelancer's Union site was great:
[http://freelancersunion.org/](http://freelancersunion.org/)

I got a quote easily before I had to do any quizzes. The gov ones were not
like that. There were a few documents that had to be emailed in for
verification, but they took care of that promptly. There was one bug related
to zip+4 not being supported and not being changeable after entering, but they
answered the phone immediately and fixed it in a minute.

------
dc_ploy
As someone who has worked on the front end of a couple of .govs, dealing with
1 mid level IT manager is enough. Having them all "collaborate" is just more
chaos. The culture of the agency (DOD vs HHS) can also be difficult to work
with.

------
kwhite
As more and more legislation starts to be implemented via online apps, it
seems almost inevitable that hacktivists will eventually attempt a filibuster
by DDoS. Given the turbulent launch of the new federal healthcare exchanges,
my confidence that the government would be able to handle such an attack is
rapidly diminishing.

I wonder how the political climate might shift if the threat of a de facto
'Annonymous veto' became something that policy makers actually had to worry
about. On the up shot, it might finally force law makers to gain a responsible
level of internet literacy...

------
medicalquack
Ok here' some other contractor news, Serco in the UK under fraud investigation
home of their corporate offices, one of the exchange contingency contracts
given at the last minute as well as Equifax...

[http://ducknetweb.blogspot.com/2013/07/government-gives-
cont...](http://ducknetweb.blogspot.com/2013/07/government-gives-contract-to-
equifax-to.html)

[http://ducknetweb.blogspot.com/2013/07/obama-
administration-...](http://ducknetweb.blogspot.com/2013/07/obama-
administration-doles-out-yet-one.html)

------
jzwinck
I told [https://www.healthcare.gov/](https://www.healthcare.gov/) that I want
to apply and live in New York. It immediately gave me a link to
[http://nystateofhealth.ny.gov/](http://nystateofhealth.ny.gov/) . I clicked
the link, and it says:

> Access Denied > You don't have permission to access
> "[http://nystateofhealth.ny.gov/"](http://nystateofhealth.ny.gov/") on this
> server.

Why would the top level of such a site have permissions at all?

~~~
cjrandolph
It works now. It was an oversight with their SSL registration. The link given
to the feds was www. Which worked. They just omitted the www.

------
djim
this is an absurd waste of taxpayer money. we shouldn't let the government and
their contractors get away with this sort of robbery. it is no wonder we are
so far in debt as a nation.

------
nhebb
I'm less concerned about the short term scalability issues as I am about the
long term security and privacy issues. The code quality doesn't give me a lot
of hope.

------
jmarin
has anyone here actually attempted to use the site, or just reading about how
it fails? The only commenter I've read was using IE.

I just signed up and had no problems.

~~~
WesleyJohnson
I tried signing up the day it launched and ran into issues with 404s. After 3
or 4 days, I was finally able to create an account. When I went back the
following day to login for the first time to choose a plan, I couldn't login.

I get a message saying my login credentials are incorrect. I hit the forgot
password link and enter in my username. It finds my account and sends me a
password reset email. I click on the link in the password reset email and it
takes me back to healthcare.gov with an error message that my account can't be
found, which obviously makes no sense because they just used my account to
send me an email.

Live Chat has provided zero help, only stating that it's probably a
performance issue and to try back later.

One odd thing is that the registration page says you a need a lower case and
upper case character, a number and at least one symbol from a pre-defined set.
To my knowledge, I didn't use an uppercase character, nor do I use one when do
the password reset, which properly sends me an email. I feel like maybe the
login portion is properly validating the username requirements, but the
registration didn't which is what has left me in this mess.

Sorry, this site isn't worth $600 million.

------
dreamdu5t
A law fining Americans for not buying health insurance is not a "tax", it's a
fine. It's a civil punishment. Double-speak. The sky is still blue even if the
Supreme Court rules its green. The individual mandate is a civil penalty just
like a parking ticket. The supreme court is a joke and the commerce clause
crap has gone beyond the absurd.

------
Yoms
Interestingly CGI is a Canadian company...

~~~
CedarMadness
CGI Federal is run mostly independently to get around the laws preventing
foreign companies from bidding on US government contracts. Source: I used to
work there.

------
arikrak
Why did the government need to build any website? Why couldn't they just
release data, and pass rules that health companies release data? Many websites
would have offered to help bring the data to the users and the Government
could have linked to some officially recognized ones.

------
GrinningFool
I don't see the problem - just look at the site.

It has a nice graphic that takes up two thirds of the page while giving no
significant information.

What could possibly be wrong with that? That's what most modern landing pages
are doing, after all.

------
twrkit
So this is what $634mm gets you?

[https://www.healthcare.gov/marketplace/global/en_US/registra...](https://www.healthcare.gov/marketplace/global/en_US/registration.js)

------
mcguire
Out of curiosity, what is the proportion of non-governmental, large-scale site
launches that have gone well?

Seems like I can remember several very public commercial blow-ups.

------
ashishbharthi
Site just crashed my IE8.

Generally I have had nicer experience with most sites developed under Obama
administration like wh.gov and such.

------
n72
I can't sign up without js enabled. Should this site especially adhere to
strict accessibility standards?

------
vaughan
Should have got Harper Reed and the Obama campaign tech team to knock it up in
Rails on AWS.

------
knodi
Who is in-charge for healthcare.org? What company?

~~~
nitid_name
CGI Federal. Please read the articles.

------
ams6110
Maybe it wasn't ever intended to work.

~~~
swamp40
I'm pretty jaded w/r/t the fed gov, but I don't see where anyone gains here.

People are so outraged that they call up their senator and demand...what?

The only thing people will demand is that the website gets fixed ASAP, because
they were looking for some affordable healthcare options.

------
BlarfWobble
I'm shocked that people seem to think 3 years and $634m is a lot for a project
of this scale.

Are you all offering to design and build it 6 months in PHP for $50k ?

~~~
newsreader
You don't? How many people do you guess were involved in the project? Divide
the $600m by that number of people and you tell me if the cost is reasonable.

