

Healthcare.gov crashing because backend doomed in the requirements stage - blurpin
http://www.forbes.com/sites/anthonykosner/2013/10/21/obamacares-website-is-crashing-because-backend-was-doomed-in-the-requirements-stage/

======
powertower
> In my case, even the most expensive plan for our family in the State of
> Maine, where we live, is half the cost of what we are currently paying,
> exclusive of any subsidies.

The elephant in the room is that this is not the typical case -
[http://www.forbes.com/sites/theapothecary/2013/09/25/double-...](http://www.forbes.com/sites/theapothecary/2013/09/25/double-
down-obamacare-will-increase-avg-individual-market-insurance-premiums-
by-99-for-men-62-for-women/)

~~~
chimeracoder
> The elephant in the room is that this is not the typical case -
> [http://www.forbes.com/sites/theapothecary/2013/09/25/double-...](http://www.forbes.com/sites/theapothecary/2013/09/25/double-..).

It still boggles my mind that people are surprised by this. Mathematically,
including people with pre-existing conditions will mean either that those
people are required to pay for the (expected, not actual) costs of their own
care, or that other people will be required to pay for those costs[0]. This
effect is _much_ larger than the effect of mandating insurance for younger,
healthier people who otherwise elected out of getting insurance[1].

Note: this is _not_ a normative statement about whether this is a "good"
outcome. This is just a a mathematical observation about the way insurance
works, from a mathematical and economic perspective.

[0] Subsidies come from taxpayer money, so they have no net effect on this
accounting - it changes _how_ the money changes hands, but the money still has
to come from somewhere.

[1] I'm waving my hands around the math here, but if people are interested I
can dig up the more detailed blog post draft that I did a while back
explaining this in full detail.

~~~
powertower
> It still boggles my mind that people are surprised by this...

I was surprised because I was told that this new coverage affordability came
from getting insurance companies greed in check.

I didn't know that insurance companies were actually lobying _for_ Obamacare,
not _against_ it.

[http://www.policymic.com/articles/62541/here-s-the-proof-
oba...](http://www.policymic.com/articles/62541/here-s-the-proof-obamacare-is-
for-the-insurance-companies-and-not-for-you)

~~~
EpicEng
Of course they are; this bill feeds them a guaranteed customer base. This is
the largest issue I have with ObamaCare. The sole job of an insurance company
is to assess risk. With a government mandated customer base there is no longer
any risk to asses.

So... Why are the insurance companies involved at all? Why are they still in
business? That's rhetorical, I know the answer, and I don't like it. We're all
paying a middle man to do nothing for us.

~~~
r00fus
> Of course they are; this bill feeds them a guaranteed customer base. This is
> the largest issue I have with ObamaCare. The sole job of an insurance
> company is to assess risk. With a government mandated customer base there is
> no longer any risk to asses.

I'm no fan of insurance companies, but this statement does not compute. Of
course there's still risk involved - the risk that the patient's costs will
exceed premiums. If you set the premium too high, the customer will flee to a
competitor. You also can't raise your premiums too high, because you have the
80% medical loss ratio - you can no longer just eat the rest as profit [1].

Totally agree that government should be the single payer or even the sole
provider of healthcare (though that model isn't as widely implemented (UK NHS,
US Veterans Admin) and has it's challenges as well).

[1] [http://www.cms.gov/CCIIO/Resources/Forms-Reports-and-
Other-R...](http://www.cms.gov/CCIIO/Resources/Forms-Reports-and-Other-
Resources/Downloads/2012-medical-loss-ratio-report.pdf)

------
mbesto
If there is anything that can come of good of my taxes (a ridiculous amount of
it in fact) is that we should educate people on the difficulties of software
development. This would make a fantastic case study on what I presume are the
issues at hand:

\- Maintaining complex code (+1mil LOC)

\- Classic "waterfall" approaches to software development (and the inherent
problem with trying to blueprint requirements up front)

\- The mythical man month[1]

\- Change management

Funny thing is...these are all human problems.

[1] - [http://en.wikipedia.org/wiki/The_Mythical_Man-
Month](http://en.wikipedia.org/wiki/The_Mythical_Man-Month)

------
gry
Thoughtback [[https://thoughtback.com/](https://thoughtback.com/)] sent me a
pertinent reminder this morning. No affiliation, just a great tool.

    
    
        Organizational metrics, which are not related to the code, can predict software
        failure-proneness with a precision and recall of 85 percent. This is a significantly
        higher precision than traditional metrics such as churn, complexity, or coverage 
        that have been used until now to predict failure-proneness. [1]
    

[1] The Influence of Organizational Structure On Software Quality: An
Empirical Case Study

[http://research.microsoft.com/apps/pubs/default.aspx?id=7053...](http://research.microsoft.com/apps/pubs/default.aspx?id=70535)

\--

edit: formatting

~~~
joshdick
"If you have four groups working on a compiler, you'll get a 4-pass compiler"
\-- esr

------
mrharrison
I know the people who worked on this. This is not what happened. Just yet
another propaganda piece by Forbes.

~~~
briandear
Well something happened. You might want to tell those guys that 500 million
lines of code is a bit ridiculous.

~~~
twistedpair
I wonder what those LoC were for? XML parsing remote API calls between various
disparate systems? Sounds like it's well beyond operation system complexity.
Linux kernel is currently 15M LoC.

Then again, kernels and operating systems are quite logical and laid out by
architects. Systems like these are rats nests defined by bureaucrats and
integration nightmares.

------
teyc
I bet you half the problem is the software being specified in a synchronous
and not asynchronous manner.

If a flow chart is expressed synchronously, it immediately bottlenecks around
the slow backend systems. e.g. check the applicant has 4 kids, and if that
checks out, present them with these options.

I attribute some of these problems to inadequate advances in today's
mainstream programming languages.

For instance, if futures/promises were common parlance in programming, some of
these bottlenecks would have been identified earlier.

I once read an interesting essay on what future programming languages could
work on. One of the interesting ideas is a language that allows programmers to
implement undo/redo easily. Recently there was a blog post about fuzzy
computing, where a function is able to return approximate values when it is
busy. The Knight Capital incident teaches us that if we architected our
monitoring systems the same was as intrusion systems are done, we could have
stopped activity when suspicious incidents occur outside the normal operating
range. There was another incident where a Japanese trader accidentally keyed
in the wrong number and promptly lost his employers a lot of money. Languages
that incorporate ideas like validation and expected values as first class
constructs might be able to help here. The Adriane reminded us how badly
humans do when reasoning about systems where multiple agents perform the same
task.

------
wheaties
I can't wait to see what the report of it didn't work actually says. It's
quite possible that one of the services that the website hit couldn't handle
the traffic which in turn caused a bottleneck in other systems.

------
chucknelson
Wow, this particular excerpt from the article is stretching things a bit too
thin:

"Clearly there are problems with the government’s IT procurement processes,
but if the same level of dysfunction had been present in the software behind
Obama’s political campaign (as it was, apparently, behind Romney’s) the
President would have never been reelected."

Really? I doubt the campaign websites were the deciding factor.

Anyway, a lot of speculation in this article...not very insightful or
informative.

~~~
skwirl
They aren't talking about the campaign websites. Both campaigns had complex
internal software systems to help target field worker campaign activities, get
out the vote efforts, monitor and respond to turnout data on election day,
etc.

According to various reports, the Obama campaign was hugely successful in this
endeavor [1] while the Romney campaign's election day voter turnout system was
a disaster [2].

[1] [http://www.drdobbs.com/architecture-and-design/software-
deve...](http://www.drdobbs.com/architecture-and-design/software-development-
in-the-obama-campai/240146307)

[2] [http://www.businessinsider.com/romney-project-orca-
disaster-...](http://www.businessinsider.com/romney-project-orca-
disaster-2012-11)

~~~
twistedpair
But that is a very agile, very incentive aligned project. You don't need to
negotiate compromises and tip toe around various laws to run some stats for
your media predictive analytics. Your vendors want you to continue in the
campaign, so they still have contracts. Everyone is running for the same goal
line.

Health care and insurance entitlement programs are clearly orders of magnitude
more quadmired before a single line of code is written.

------
Plasmoid
Why government software projects go wrong
[[http://www.mrobinson.ca/2012/09/why-government-software-
proj...](http://www.mrobinson.ca/2012/09/why-government-software-projects-
going.html)]

ibid - Part 2 [[http://www.mrobinson.ca/2013/08/government-projects-going-
wr...](http://www.mrobinson.ca/2013/08/government-projects-going-wrong-
part-2.html)]

------
gesman
Could anyone share which platform / framework they used?

~~~
babs474
mnsure.org was throwing springframework exceptions the day it launched

~~~
uxp
healthcare.gov itself was a Ruby Jekyll static site. It _was_ on GitHub[0] for
a while, but mysteriously disappeared. Here[1] is a clone of it, and what
seems to be the currently agreed upon canonical version.

My state appears to be using Joomla based on the meta-generator tag in it's
head tag. [http://www.avenueh.com/](http://www.avenueh.com/)

[0] [https://github.com/CMSgov/HealthCare.gov-Open-Source-
Release](https://github.com/CMSgov/HealthCare.gov-Open-Source-Release)

[1] [https://github.com/blencorp/HealthCare.gov-Open-Source-
Relea...](https://github.com/blencorp/HealthCare.gov-Open-Source-Release)

------
drakaal
I didn't see anything that meant it was doomed from the requirements stage.

This was more of an Obama love fest and defense of ObamaCare than a technical
article describing who the backend was doomed.

For what this cost to build it could have been much better, much simpler, and
handled the scale gracefully.

Yes privacy concerns keep it off the elastic cloud. (not a point mentioned in
the article)

Yes authenticated users limit caching and make scaling harder. (not really
mentioned)

But when you are talking about everyone in the US, you only have so many
combinations. Income in something like 32 ranges, number of kids 1 through 12
(greater than 12 they are all free) a bunch of ages.

We just saw a post that only 15 searches are new to google each day, so the
number of combinations is much less than search which does lots of caching.

The requirements were messed up, the implementation was. Clearly not built by
people who had ever done "Facebook scale" stuff.

~~~
GVIrish
Well, some reports are that the requirements were changing until 1 week before
launch, and that development didn't start until March of this year due to
requirements delays. Just those two factors alone would doom just about any
project of this scale.

