
Contractors See Weeks of Work on Health Site - noahlt
http://www.nytimes.com/2013/10/21/us/insurance-site-seen-needing-weeks-to-fix.html?pagewanted=1&_r=0
======
noonespecial
I suspect that as unbelievable as this sounds, it might actually be true.
There must be a ton of "repeated code" to handle all of the different states
and rules. I'm guessing that they started with one huge bloated "template" and
then wrote automatic code generation systems that just copied _everything_
hundreds or even thousands of times.

They could very well now find themselves in a position where it is humanly
impossible to untangle it all and make even the smallest changes. I'm calling
it failed at this point.

Edit: Title was changed from: _Healthcare.gov "contains about 500 million
lines of software code"_. This comment reflects this fact which seems
unbelievable, but if true, I feel would likely mean that it was mostly created
by code generators that probably copied and ossified original errors and
misconceptions to the point that it is no longer possible to fix. Its a total
failure, the money is gone, but they'll keep throwing even more after it,
until the tiny, disjointed systems they develop in the mean time as "temporary
solutions" eventually become the "real" system.

------
Stratoscope
> On Monday, Mr. Obama will host a Rose Garden event with people who have
> successfully enrolled in the health care exchanges.

It's great that they can all fit in the Rose Garden!

~~~
DigitalJack
I'm pretty sure they'd all fit in my basement.

------
conover
That's a staggering amount of code. Nearly unbelievable. To put that in
perspective, Windows XP was 45 million lines of code[0].

[0]
[https://m.facebook.com/windows/posts/155741344475532](https://m.facebook.com/windows/posts/155741344475532)

~~~
jussij
My guess is they're counting code from existing packages used to create the
site.

So for example, if they used Java, that line count is including the lines of
code found in the JDK.

I see no other way of getting to such a large number.

~~~
joeblau
What was the site written in?

~~~
lukeqsee
Assembly, to protect your privacy.

------
ChuckMcM
Pretty sure that is a direct rectal extraction number not a real number. If
there were 500 million lines of code you would have a prima facie case for
fraud on the part of the contractor since government contractors are required
to use best industry practices and no industry practice supports 10x code
expansion.

~~~
growupkids
That's just not true. Contractors are only required to deliver what they
proposed and the government agreed to.

~~~
ChuckMcM
Is it possible its different for DoD contractors? I ask because Lockheed in
Sunnyvale took some heat (I believe a fine and a restriction on being able to
bid on contracts for a period of time) for failing to use 'established
industry practice' in the testing and delivery of what ever it was they were
delivered. My neighbor at the time worked for them and recounted the gruesome
(or in this case non-standard) details. I never really thought to question
whether or not it was actually written that way, it just made sense that it
would be. When I was with Intel and Intel was building milspec 8080s there
were a list of standards that had to be met and all of the test equipment was
calibrated based on that as well. Seemed like a pretty natural part of the
government contracting business.

So you're saying that Contractors are under no obligation to meet standards?
Or just that this contractor would be exempted? Or perhaps software in general
would be exempted?

( Apparently NIST is involved somewhat :
[http://www.informationweek.com/government/enterprise-
applica...](http://www.informationweek.com/government/enterprise-
applications/nist-updates-software-testing-tool/228200651) )

~~~
rwallace
Contractors can certainly be under an obligation to meet standards,
particularly when this is specified in the contract. However, there is no ISO
standard that says "a program shall not reach five hundred million lines of
code even if you include all the autogenerated XML and suchlike".

------
robomartin
I've said this before. The fact that these people (government and it's
contractors) are going to have and handle our most intimate information AND
that the IRS is also in the loop scares the living daylights out of me. The
level of incompetence being revealed here is almost without description.

To go all the way up to opening day, have these problems and then offer that
five million lines of code need to be redone is, well, I'm at a loss for
words. I am trying to think about how badly one could fuck up five million
lines of, say, Python. That's a lot of code. It's almost like saying that you
set out to write a portal for the ACA and mistakenly wrote an navigation
system for a new spacecraft.

In many ways this is great. I can't think of a better example of how bad big
government can be and often is. It is hard to argue with what is unfolding
before our very eyes: An attempt at centralized control that succeeded on the
legislative front but is failing miserably in execution due to nothing less
than incompetence at all levels. And, to boot, it is costing hundreds of
millions and will probably cost billions if it doesn't implode under it's own
mess.

Look at this in sharp contrast with what I happen to think would have been a
far better approach: Free market decentralized healthcare. All they had to do
is open interstate borders to launch all available companies into competition
and slowly open other gates in order to promote free and open competition.
Yes, judicious regulations would have been required here and there, but always
with the goal of having an ecosystem that is as free as reasonably possible.

In this scenario at worst the government would launch a open-source projects
to develop open and well-debated APIs for publishing and sharing all pertinent
information. Various sites could then connect to any available insurance
company through these standard API's in order to gather, present and sell
insurance to anyone. Instead of having a badly executed centralized attempt to
develop a portal you'd have thousands of entrepreneurs jump into the fray for
an opportunity to offer the best possible services. Very soon the best-in-
breed services would rise to the top and continually evolve better and better
offerings.

Cheaper, faster, better. All three.

Instead we got this shit.

~~~
jjoonathan
> It is hard to argue with what is unfolding before our very eyes

What we are seeing right now (~.5B waste) is a drop in the bucket compared to
waste in the US system as compared to socialized systems. We pay ~2x as much
for comparable health care (per person per mean PPP-adjusted GDP/capita) [1].
That is about 250b/yr we are spending on a (more closely) free market system
that keeps promising to be more efficient than its social counterparts and
keeps failing to deliver, or even deliver in the same ballpark. I realize that
HN is a site about code, but it's important to keep some perspective here.
$500M (itself an inflated figure) is 0.2% of $250b. If it is hard to ague
against $500M of waste, it surely must be nearly impossible to argue against
$250b of yearly, recurring waste, no? How do you ignore these figures?!

If you're going to argue that our system makes up the difference with better
outcomes, I want to see numbers. I happen to know that they don't support this
argument, but I want you to do the search so that you can't dismiss my sources
by assuming they were cherry-picked.

> I happen to think ... Free market decentralized healthcare ... would have
> been a far better approach

Of course, $250b/yr includes the cost of providing care as well as
administration. Perhaps administration of a free market system is more
efficient? Not when you compare the US (31% administrative overhead) to Canada
(16.7%) or the UK (6%) [2], [3]. This isn't a surprising result, either: I
recently listed 7 reasons why exchanges should be more efficient than current
individual plans [4] and you can add the profit margin and advertising costs
to that list to come up with 9 advantages a single-payer system would have
over a free market health care system.

Markets are good at optimizing things that can be quantified, so perhaps a
better comparison would be between medicare and private insurance (not overall
administration, just the insurance part). Again, the government systems come
out ahead: Medicare overhead is 3.6%, Canada's insurance-related overhead is
1.3%, and overhead among private US insurers is 11.7% [2].

> Cheaper, faster, better. All three.

[citation needed]

See [5] for a more complete overview of the pro-single-payer argument and a
truckload of reputably-sourced studies and comparisons (as opposed to hand-
wavey promises) to back it up.

[1]
[http://en.wikipedia.org/wiki/List_of_countries_by_total_heal...](http://en.wikipedia.org/wiki/List_of_countries_by_total_health_expenditure_%28PPP%29_per_capita)

[2]
[http://www.pnhp.org/single_payer_resources/administrative_wa...](http://www.pnhp.org/single_payer_resources/administrative_waste_consumes_31_percent_of_health_spending.php)

[3] [http://nhsrationing.org/tag/administrative-
costs/](http://nhsrationing.org/tag/administrative-costs/)

[4]
[https://news.ycombinator.com/item?id=6582753](https://news.ycombinator.com/item?id=6582753)

[5] [http://www.pnhp.org/facts/single-payer-
resources](http://www.pnhp.org/facts/single-payer-resources)

~~~
robomartin
>> Cheaper, faster, better. All three. > [citation needed]

Lookup the history of laser eye surgery over the last, say, ten years.

Free markets work. That's irrefutable. The problem is we've never had a free
market system when it comes to medicine.

~~~
jjoonathan
You're cherry picking: laser eye surgery has effective competition, elastic
demand, no information asymmetry, and it is priced well within the annual
(even monthly) earning potential of most Americans. It's a service that I
would expect the free market to be able to provide efficiently and it does not
surprise me that it succeeds. The free market is very good at optimizing value
creation under certain circumstances and it's great that laser eye surgery
matches up with those requirements.

Unfortunately, when it comes to policy we do not get to cherry-pick the
conditions under which the rest of the system operates, and the other parts of
the health care market run under conditions that demonstrably throw the free
market for a loop (hence the system-level failures vs socialized systems I
pointed out above). Cancer treatments cannot be paid for out of most people's
savings. "Customers" do not always have the chance to shop around between
providers (w/o insurance) and those that do use insurance experience a 99%
deleveraging effect because only the 1% that have extraordinary costs are in a
position to examine the value of what they bought. Do you have a proposed
mechanism of breaking information asymmetries, re-leveraging
disproportionately expensive consumers, and standardizing the coverage
provided by free market plans? Do you have a way of pricing the 5
externalities I listed into the insurance providers' budgets? Has your
proposal been implemented in a dozen other countries and proven to be
comparatively cost effective for decades?

What is your answer to the figures that show that our system in aggregate is
2-10x less efficient than comparable social systems?

Anybody can cherry pick a counterexample. Anybody can speculatively
extrapolate it out to a convenient conclusion. But the single-payer advocates
can point to a mountain of directly relevant apples-to-apples evidence
regarding the industry as a whole that does not require extrapolating across
markets or cherry picking conditions / good years. To me, it appears to be a
mountain of evidence vs a pebble. Do you really think the the implicit
strength of extrapolation from the (very) special case of laser eye surgery
carries weight against these empirical apples-to-apples comparisons?

~~~
EdSharkey
In abstract terms, I would say single payer systems could (and should) work
well. It sounds nice to me on paper.

I think an issue is that America has a system of government that lacks a long-
term focus. Every 4 to 8 years, we get a new 40-100 year agenda. This leads to
tinkering with these social programs and the original goals can get lost,
muddled.

That, and our government is not structured very well to deliver services with
a largely unaccountable bureaucracy overseen by a divided and paralyzed
government. Our government in the process of failing to deliver a service
right here (with the exchanges at healthcare.gov).

I could see single payer systems working well elsewhere. In the states, I
think it would be implemented badly at 10x the cost. Just my opinion, but I
think we should try developing a competitive market for healthcare and reduce
government regulation in it and see how that goes. Then, when that flops, we
can try out socialized medicine.

------
DigitalSea
Sounds like a load of crap to me. Seriously, 500 million lines of code? I bet
not even mission-critical code written to power NASA spacecraft and satellites
are anywhere near that length.

I propose they give the project to the HN community, divide up the $400
million figure amongst the participants and give us a weekend and I bet we'll
have more to show than that of the so-called developers the government hired.

I am not underestimating the complexity of such a site, but seriously, it's
not like it hasn't been done elsewhere to a degree. if I can do Internet
banking, the a site for providing healthcare can be built a lot easier.
They're both of similar scale and security requirements I would imagine.

This site seems like the greatest scam ever perpetuated. I bet the contractors
are laughing themselves to the bank literally as we speak. Even though it's
bad publicity, money is a higher priority than credibility for some businesses
out there it would seem. Is it any wonder people don't trust the government?
They can't even hire competent people to build them a website. This seems to
be a recurring theme for local councils and governments elsewhere as well, so
it's not just the US this is happening.

People need to get mad about this. That's $400 million dollars that could have
been invested elsewhere like medical research, charities, food programs,
rehabilitation for the homeless... A $400 million mistake has been made, and
another $400 million will probably be spent before it's even ready.

What a waste of money.

~~~
javert
While I completely agree with you in general, a nit pick:

> I bet not even mission-critical code written to power NASA spacecraft and
> satellites are anywhere near that length.

That doesn't seem like a good example. You want to keep the code in those
kinds of things to a minimum, and that's probably not too hard.

~~~
code_duck
It's best to keep the code in everything to the minimum!

~~~
derleth
> It's best to keep the code in everything to the minimum!

Optimizing for readability is usually most important.

Readability is usually more important than reusability, in fact, although
readability very often enables reusability.

But we're talking about mission-critical embedded code, which is a special
case in many respects. Still, making it as readable as possible makes bugs
easier to find.

~~~
javert
Thanks for saying this... it needs to be said far more often.

I wish there were a book about how to READ software. (If anyone knows of one,
please speak up.)

One tip I picked up on my own over time is when you're looking at an
unfamiliar project, don't read the code like a book, read it like a program.
In other words, try to find main() and trace how things work from the bottom
up, rather than trying to understand it based on how the authors have broken
up the code into directories. I don't know yet whether this is truly good
advice.

------
tunesmith
I wonder what languages it focuses on. I haven't heard.

I find some of it amusing - I've worked on some enterprise-level projects with
business systems talking to other business systems, and I can only imagine how
much eye-rolling is going on among the programmers that are actually trying to
do the work. This is a project where some very visible, loud people have a
vested interest in it looking bad even if it were flawless.

I actually think it would be kind of fun to be close enough to the technology
to really identify the core constraints and be in a position to help alleviate
them.

~~~
EdgarVerona
The sad thing is that the site being bad _should be_ really tangential to the
politics. I personally would love to see outright socialized healthcare, where
everyone was mandatorally enrolled in a government created program. I feel
like we'd have the most leverage to bring costs down under a centralized and
non-profit system. However, this system being as broken as it is - and costing
as much as it has - did nothing to help that cause, even though it should be
tangential to it. Very frustrating.

~~~
hga
I guess you can't see how giving the government even more power of life and
death over its subjects can be anything but political?

Certainly the U.K. NHS experience shows the running of such a system is
political. I don't watch it closely, but as I understand it Tony Blair/New
Labour doubled spending, but somehow functionaries and bureaucrats got the new
money, not the front line people and institutions.

You think we can have anything less than a totally political discussion about
euthanasia, most recently known as the Liverpool Care Pathway in the NHS?

------
lifeisstillgood
Again the argument that Government paid for code must be released under an
Open License - all of the speculation and questions here would get wiped out
by a simple git push from the right contractor.

And we could more fruitfully spend our time contributing to the automated test
pack.

------
unreal37
There's no way that's true.

No possible way. For a website. Did they write their own operating system from
scratch? And database?

~~~
adventured
It's definitely a lot worse than that.

500 million lines of code would represent nearly every line of code that has
ever gone into all shipped Microsoft software spanning 37 years.

------
ddoolin
I understand this website is probably more complex than your average website,
but I would put money -- a lot of money -- on it being nowhere near 500
million lines.

~~~
chubot
Yeah it's got to be off by at least 3 orders of magnitude. 500K LOC might be
industry standard bloat, but it should be doable in 50K if you hire good
programmers.

But 500M is not credible, unless you are doing things like counting auto-
generated code and open source libraries. Even then, you would have to be
using 100,000 open source libraries to get near 500M LOC :)

------
strict9
While 500 million sounds really far fetched, this is the point at which
projects explode with bloat and repeated code:

> the separately designed databases and pieces of software from 55 contractors
> work together.

Factor in dealing with the multitude of state laws and companies they are
interacting with, and you have a recipe for problems.

A lot of armchair quarterback technologists, but not a lot of precedent to
compare computerizing a healthcare system in this way to.

------
nostromo
> According to one specialist, the Web site contains about 500 million lines
> of software code.

> One specialist said that as many as five million lines of software code may
> need to be rewritten before the Web site runs properly.

Oh, so only 1%!

Tangentially, why are so few quotes in this article sourced? I see no reason
why they shouldn't be. It's all throughout the article... They keep quoting
"specialists."

~~~
rwallace
In something this loaded with politics, I wouldn't imagine anyone involved
would think it a good career move to have their names attached to bad news
quoted in the press?

~~~
hga
The only people I've seen willing to go on the record have been:

"Stan Z. Soloway, president and CEO of the trade association that represents
hundreds of government service companies" who parsed the contracts and
independently confirmed that there's no prime integrator (i.e. the HHS CMMS is
doing/failing that job,
[https://news.ycombinator.com/item?id=6569042](https://news.ycombinator.com/item?id=6569042)).

One or more big insurance company CEOs; they're the ones who have to deal with
the gubble the backend is delivering to them.

Various mostly bloviating politicians like John McCain.

Even yesterday's HHS blog posting
([http://www.hhs.gov/digitalstrategy/blog/2013/10/making-
healt...](http://www.hhs.gov/digitalstrategy/blog/2013/10/making-healthcare-
gov-better.html?post=yes)) is unsigned.

------
downandout
Let's be clear: This number could not be accurate, even if it included the
line counts of licensed code (SDK's, database engines, etc.). The article
failed to even cite its source, beyond saying that "one specialist" said this.

The NYT is a left-leaning publication. This seems to be a transparent attempt
to try to soften the blow to the Obama administration's image by saying that
implementing healthcare.gov was a Herculean task that no one could have gotten
right on the first try because it required a half billion lines of code. It's
no coincidence that this article emerged the day before Obama himself plans a
press conference on this very issue.

------
lmkg
My own conspiracy theory: XML "code" got included, even the stuff that is
unambiguously data. This may or may not include HTML templates (possibly
machine-generated) and the data files for each individual health care plan.
This is the result either of naive use of a line-counting utility, or a
miscommunication between technical and non-technical folk, or both.

------
spikels
"One specialist said that as many as five million lines of software code may
need to be rewritten before the Web site runs properly."

Yikes! I don't think these "experts" are going to be fixing this anytime soon.

~~~
EdgarVerona
The sad part is that it sounds like they're already being paid to do so - they
were quoted as being Federal Contractors.

We really need a public peer review system for technical contracting of
government projects. This is getting insane. These schmucks fleeced 500
million from our government ( _us_ ) for a barely functioning website, and now
some snake oil contractors are going to come in to "fix it" and fleece us for
even more.

~~~
hga
Given the obviously appalling management "these schmucks" were subjected to,
per the article " _In the last 10 months alone, government documents show,
officials modified hardware and software requirements for the exchange seven
times._ ", plus our hearing from multiple sources that there were getting
change orders through the beginning of the week per launch, can you honestly
say they fleeced us?

I can't imagine any way the contractors could have delivered a working system
under the constraints we've been hearing, including most especially the
unqualified Federal HHS CMMS government bureaucrats taking upon themselves the
role of integrator of the 50+ parts of this system.

For that matter many people at all levels of this have been waving red flags,
which the people running the show have studiously ignored. I mean, you _know_
someone told them "you've got to freeze the requirements and give the
programmers some time to get the job finished", not to mention "it needs more
than one week of testing"....

Some of the remedial steps we're hearing about, like monitors for system
bottlenecks, sound to me like things the integrator should have made part of
the system from the beginning....

~~~
EdgarVerona
You have a good point. Now that I've slept on this article, I feel a bit more
balanced about it. The sticker shock wore off, and I do feel like many of your
points are valid now that I'm feeling more rational about the subject.

I feel like I did underestimate the potential costs of this software. How
much, I couldn't say, but thinking about it this morning I realized that they
probably had to integrate with the systems of (every|many) major existing
healthcare provider in the country to actually make this work as desired. I
can see now where that wouldn't be cheap.

Is it 500 million expensive? I'm not sure I'm qualified to make that
assessment, and now that the initial shock has worn off I'm feeling okay with
that. I'd need to see more data - particularly, the balance sheets for the
project - before I could say for certain whether the cost was justified. I'd
be interested to see that data, but as it stands I'll settle for chilling out
about it unless more information comes out along those lines.

~~~
hga
At a very raw level, given how late in the game it was before the programmers
were able to get started (obviously not their fault when CMS (sic, should be
CMMS but isn't') took on the integrator role), obviously the project is going
to copiously burn money. Pity the other constraints caused it burn quality
(going by the old "pick two of time, money and quality").

" _but thinking about it this morning I realized that they probably had to
integrate with the systems of (every|many) major existing healthcare provider
in the country to actually make this work as desired. I can see now where that
wouldn 't be cheap._"

Actually, that's one part that shouldn't have been expensive for the
government, except in support services. As in they should have standard, sane
input formats for policy details and output for enrollees that any
participating insurer could adapt their systems to. IT _must_ be a core
competency for a successful insurer, since everything? they do aside from
marketing is IT mediated.

There's been some reported troubles with getting the policy details right,
although I think what I heard was about one or more state exchanges---with
guaranteed issue and community rating it's rather more simple than normal, but
then again a customer's FICO score can and I gather generally/always is used
in the pricing. At the other end the insurers aren't reporting trouble in
getting and parsing the data, just in it being self-evidently incorrect, e.g.
some customers being reported as enrolling and canceling several times. Which
they're addressing, while the volume is so low, by calling them up (everyone
wants to complete the circle anyway).

On the other hand, you know it took a while for everyone to hammer out those
formats, especially the input one. The output should be pretty simple, just
needs to have all the particulars correct. E.g it would be a bad sign if an
insurer was getting people enrolled in a policy they didn't offer.

~~~
EdgarVerona
I feel that this integration effort is probably exactly where most of the
expense came from. There's more to it, of course, but I can see where that one
part was probably the large heap of their troubles.

I was not thinking rationally about this whole issue when it first came up and
thus didn't measure my previous experience when I was thinking about the
issue, but I actually have worked with healthcare systems integration before.
Even with all government inefficiency/management aside (we were a private
company), interfacing with healthcare providers is an extremely complicated
and painful business.

For instance, there is a HIPAA standard interface protocol for things like
Eligibility and Benefits and Claim Submission, and most (though still not
all!) health care plans have some sort of system through which you can
interface for that information - though often it's hidden behind miles of red
tape and fundamentally different implementations for various custom fields,
and different interpretations of the meaning of many fields. That alone is
painful, and that part is supposed to be "standardized".

I can't even imagine the state of healthcare provider's policy estimation
systems, particularly when it comes to interfacing with them. It's speculation
on my part, but I imagine that each healthcare provider would likely have to
have their estimation systems tied into. I wonder how many even had it exposed
in such a way at the start of this project that you _could_ tie into it, and
how many of them had a wide variety of custom inputs to take into account in
their estimating. Though you are right that perhaps the new law allowed/forced
them to simplify this estimation process, in which case each of these
providers likely had to modify the ruleset of their ancient systems before the
integrators could actually work with it. Yeah, I could see even a simplified
ruleset for policy pricing and eligibility as being a major factor in slowing
down this process.

I certainly wouldn't have wanted to be a software developer on this project,
thinking more about how bureaucratic and inefficient both the government and
healthcare providers are. I still feel like, even then, 500 million is a
dubious number - and I'd like to see where that money went. (how many people
were employed on the project and for what cost, what were they being charged
for hardware and datacenter(s), etc...) But as a software engineer, I would
certainly not have enjoyed working on such a doomed project regardless of
whether the money was as well spent as possible or if it was lining the
pockets of those further up the chain.

------
eliteraspberrie
Less than 10 million lines of code are required to fly an F-35 -- and that is
C++.

Who is the contractor responsible for this? Perhaps the same people
responsible for the Ontario EHealth scandal:

[http://www.cbc.ca/news/canada/toronto/ehealth-
scandal-a-1b-w...](http://www.cbc.ca/news/canada/toronto/ehealth-
scandal-a-1b-waste-auditor-1.808640)

~~~
bch
> Perhaps the same people responsible for the Ontario EHealth scandal…

I don't know if you were being facetious, but you're not far off:

[http://m.washingtonpost.com/blogs/wonkblog/wp/2013/10/16/mee...](http://m.washingtonpost.com/blogs/wonkblog/wp/2013/10/16/meet-
cgi-federal-the-company-behind-the-botched-launch-of-healthcare-gov/)

------
harold
From March of this year:

 _Chao said that he’d once held high hopes that the exchanges would run
smoothly from the beginning, but that those hopes had been dashed. “The time
for debating about the size of the text on the screen, or the color, or is it
a world-class user experience, that’s what we used to talk about two years
ago,” said Chao. “Let’s just make sure it’s not a third-world experience.”_

[http://www.forbes.com/sites/theapothecary/2013/03/22/cms-
on-...](http://www.forbes.com/sites/theapothecary/2013/03/22/cms-on-
obamacares-health-insurance-exchanges-lets-just-make-sure-its-not-a-third-
world-experience/)

------
EdgarVerona
Yeah, after reading that article, I get the very strong feeling that these
"federal consultants" are in for a second round of greedy-greedy-grabfest.
They're throwing insane sounding numbers out there so that they can justify
egregious compensation. Someone needs to call these people out on their bluff,
there is no way that this terrible site is even close to the LoC numbers
they're throwing out there.

I'm so sick of seeing contractors fleece our government (and by extent, _us_ )
by pulling the wool over the eyes of those who don't know any better and
making sweetheart arrangements.

~~~
hga
Geeze, how many times are you going to make this point in this discussion? See
my first reply here:
[https://news.ycombinator.com/item?id=6586603](https://news.ycombinator.com/item?id=6586603)

I point out there's absolutely no way the contractors could have won with the
insane management they were under, unqualified CMMS government bureaucrats
_doing the integration_ and constantly requiring changes, through the week
before launch, some of which is detailed in this article.

------
mag487
I'm having a really hard time believing that figure.

------
lifeformed
Has anyone actually tried to look for health insurance using the site? It's
indeed as bad as they say. It's probably the buggiest site I've ever used.

I've tried a dozen times over the span of several weeks, and get to a new
impassable failure state each time. I'm literally sent to broken URL's that
have variable names "?id=%applicationID%" in the URL instead of the actual
value. Sometimes clicking "next" just simply refreshed the page with no new
content.

------
chill1
Use mobile version to get around the pay-wall:
[http://mobile.nytimes.com/2013/10/21/us/insurance-site-
seen-...](http://mobile.nytimes.com/2013/10/21/us/insurance-site-seen-needing-
weeks-to-fix.html?pagewanted=1&_r=0)

------
burgreblast
ha. I'll bet it's: LoC * Num_servers = 500M

At least, that sounds like a great way to charge gubbmints by LoC.

------
bluemanshoe2
The entirety of the source code available in the debian repositories as of
lenny is 324 million lines of code:
[source]([http://gsyc.es/~frivas/paper.pdf](http://gsyc.es/~frivas/paper.pdf)).

------
jussij
From the page:

 _One specialist said that as many as five million lines of software code may
need to be rewritten before the Web site runs properly._

It sounds like they've been doing a little too much coding and not enough
designing.

~~~
EdgarVerona
Was that specialist in the running for being paid to fix it, and thus was
overinflating the estimate so he could get paid a few hundred million to fix
it up?

I like the idea of a government healthcare program, I really do. But this
website... man, someone lined their pockets on this scam, big time.

~~~
hga
Not even vaguely a scam, as I reply to you elsewhere in this discussion:
[https://news.ycombinator.com/item?id=6586603](https://news.ycombinator.com/item?id=6586603)

Unless you want to posit that the contracts were accepted by companies that
knew it could not be delivered in the time required. And even then, people are
optimistic, and I strongly suspect they didn't think HHS/CMMS would be so
insane about the number of changes demanded during the process, going all the
way through the week before launch (!!!).

------
jerrya
_One specialist said that as many as five million lines of software code may
need to be rewritten before the Web site runs properly._

In comparison:

    
    
      Windows 95 had 15 million lines of code.
      Windows 98, 18 million
      Windows XP, 35 million
    

[http://www.nytimes.com/2006/03/27/technology/27soft.html?pag...](http://www.nytimes.com/2006/03/27/technology/27soft.html?pagewanted=all)

Luckily they have about 8 weeks.

------
codygman
No wonder it doesn't work. It's _very_ unlikely that healthcare.gov is more
complex than an OS (such as XP noted by conover in this thread).

------
r0h1n
$400M for 500M lines of code. That's roughly $800/KLoC. I wonder how that
compares to any other modern software.

~~~
greenyoda
$800 pays for roughly one day's worth of work for a senior developer in the
US. One thousand lines of working, debugged code (which is presumably what's
being paid for) sounds like an awful lot of code for one person to write in a
day, if the problem domain has any degree of complexity. Some parts of
healthcare.gov could be quite complex, e.g., interfacing to other government
systems whose APIs are not well documented.

And I agree with the people who think that 500M lines of code is unlikely.
Judging by some of the statistics that were cited in other comments (e.g.,
"Windows XP was 45 million lines of code"), this may be more lines of code
than Microsoft has written since the company was founded (many of the lines of
code in XP were originally from NT, etc.).

~~~
r0h1n
It costs ~$300K for a senior web developer in the US currently?

~~~
roel_v
The per-day rate that a company charges is not the same as the salary that
person makes. So yes, it costs 300K to hire a contractor full time. Not only
in the US, but on most parts of Europe too.

I feel a bit dirty for having to explain such simple things. All the armchair
pundits here are declaring 'how can they screw up a simple website'. Nobody
here (except if there are people reading this who actually worked on this
project) knows how hard this project is, and how much integration is required.
Personally, if somebody had asked me to make a quote for a project this size a
year ago, and going on the information we have now, I would have said that
it's impossible to build something this big and this complex within a year.
Can't be done. IT systems of tax administrations of small countries that do
less than this system have cost 10 years to get to a stable, workable state.
How can anyone build this within a year?

------
mattdeboard
I... dunno guys. Multiple disparate systems designed by separate government
contractors who no doubt pumped up their work estimates tutu get even bigger
contracts. Remember this is a $400 mil project. Maybe I spent too long in the
government but this doesn't seem unrealistic to me.

------
fmax30
Who told them to use vanilla php to make everything , 500 million lines of
code is an astounding amount of code . Next step put up a disclosure/FOI
request for the source code of the healthcare.gov so that we can know what
they were writing in there.

~~~
marginalboy
I doubt you'd get it from an FOIA request. I believe there's an exemption that
covers work like this.

~~~
valleyer
Right — no one there knows how to edit the database credentials out of the PHP
source. :)

------
semerda
hahaha what a crock of bull. Probably the same experts that built it were
quoted as saying that. Or maybe it is true... when you cut and paste every
code snippet from the web and mush it up to create this site it does add up to
500m lines of code..

Anyone have detail on language they used and/or frameworks(s)?

Someone should be held liable for this. They just wasted tax payers money on
would be experts and have to spend more tax payers money to fix it. Disaster!

------
ivanbrussik
they might be exaggerating that by including all of the respective libraries
used etc, or just exaggerating in general (like the gvt tends to do)

~~~
jlgreco
Even if they counted PHP's lines of code, that would only throw on less than a
million more lines.

I strongly suspect that somebody is full of shit.

------
kiddz
The USPTO should see this as an opportunity to have a mail-in form which would
create redundancy and also obscure these problems.

------
azifali
Did they do a character count and submitted that as LOC to justify the high
costs?..

