
Healthcare.gov technology stack (infographic) - egarcia9330
http://www.thedoctorweighsin.com/healthcare-dot-gov-how-does-it-work-infographic/
======
ryanobjc
People appear to be missing the point that _the website_ is literally the
smallest part of the project.

While it's easy for people to crap on things like js-minifying, cache control,
etc - the hard reality is the stuff that is the hardest is so hard that most
people have never done anything even remotely close.

Having worked on 10m+ loc codebases, life gets very hard. Dynamic languages
tend to be unworkable. Module control, strong interfaces, and a lot of
division of responsibility becomes the order of the day. And just figuring out
how to build the thing becomes it's own team.

For those who are just thinking "so, just slap together some JSON/web services
and call it a day" \- I invite you to research EDI and then report back here.
Assuming you're still alive.

~~~
daigoba66
Given the complexity of the system, the most shocking thing to me is that how
little time they had to actually build it. And from what's been made public
there was virtually no time for testing (load testing was certainly absent).
It should been "turned on" months before the public go-live to iron out all of
these issues.

~~~
spikels
That's one thing I never understood. The law was passed 3.5 years before they
had to go live. Seems like they should have had plenty of time but it seems
like almost all of the coding was done in the past year with little testing
along the way.

------
patrickg_zill
A friend analyzed the page and noted that they didn't minify or combine
anything.

"One obvious reason is that they're serving unminified and unconcatenated
Javascript and CSS files. A single page load causes 92 requests to the server,
and they're not using a CDN for any of that.

They're also not setting the Cache-Control and Expires headers, so the browser
isn't caching many of those files, including on 80k file full of response
strings.

For the non-technical, this means the site is essentially killing itself. An
optimized and properly tuned production website would load one (1) HTML
document, one (1) CSS document, and one (1!) JS document, they're loading 92.
"

~~~
thejerz
While that is an egregious amount of requests, it's not the reason the site is
broken. Based on my experience as a user and poking around quickly in Firebug,
the issues are largely server-side.

~~~
patrickg_zill
True, it is just an additional data point. You have to wonder, if they can't
minify their JS or even run the pages through the Yahoo YSlow tool, what is
the quality of the backend?

~~~
outside1234
I think we really don't have to wonder at this point.

------
nathas
$293M.... I'm just consistently angry over how much this cost.

There are so many moving pieces, and while I understand that, $293M for one
piece. Half a billion for a website.

Christ.

~~~
dm2
Seriously, that number is staggering.

CGI Federal literally ripped off the government, screwed the American people
badly, and "stole" hundreds of millions. They should never be allowed to work
on another project again and should be required to refund the government. The
project managers should be fired and publicly humiliated.

I STILL cannot even login to the website. The site has said "The system is
down at the moment." for the past two weeks.

The only reasonable explanation that I can imagine is sabotage by someone who
didn't want Obamacare to succeed, the same thing that happened in Congress
recently.

~~~
joering2
> The only reasonable explanation I can imagine is sabotage by someone who
> didn't want Obamacare to succeed

It depends how you define "success". While most people assume success as a
well-oiled healthcare machine that is affordable and cares about you then yes,
they have failed. However, if you have done your homework on Obama and
Democratic party (in biggest part), you will understand that they already
greatly succeed - that is they shoved a huge tax (that over 10 years grow to
10% of your income, starting with 1% next year) in american's throat. While
congress pass the law and Supreme Court upheld it, rest is just a statistic
noise. Who care congress has 11% approval? Who care Obama has vanishing
approval rate and his administration cannot catch up with any "phoney"
scandals they getting into every month it seems? Its all irrelevant. After
all, they did not hire 16,000 new physicians; they hired 16,000 new IRS
workers that will be responsible for collecting what you lawfully owe. That's
all! Its all about the taxes; its never been about you or your health. The
only way Obama ACA would be more successful is that every single american
throw a towel and say "screw it, I will pay the fine, this is too
complicated!". That would be the best scenario for the Obama and the gov.

> the same thing that happened in Congress recently.

I think you mean the shutdown. It was so pathethic that it is funny. Imagine
situation when you drive to work every day. And then every weekend you go out
and drink and drive. The cop pulled you over twice but he let you go with a
warning. You keep drinking and driving. Then one day he pulled you over, boom!
DUI, license revoked. Then you go to your friends and family and you complain
that because of this cop now you cannot do your work, because you drove to the
office every day. If you agree with this reasoning then yes Reps were
responsible for shutting down the Gov and not letting Congress do their work.

------
wil421
The problem isnt the part that you can see, the Front-End. The problem is the
legacy systems on the Back-End that need to talk to each other. And if one of
these requests time out, the whole request fails, doesnt matter if the 5
systems before responded it has to go through the process again.

------
neurotixz
The web part is just a tiy part of a major project. Extracting informations
from 55 databases/unrelated systems in a coherent manner is a nightmare by
definition.

The number of different 3rd party libraries that are required, the type
incompatibilities, matching fields, accounting for changes in the schema, and
talking to legacy systems (I am sure there are many of them in there...), all
in 5 months? Good luck.

Add to this the overhead of talking to IT and management of insurance
companies, the IRS and others, and just the fact that it does answer some
queries is fairly impressive in my book.

Also I am sure that most of those systems are failry deep inside the corporate
networks of these insurers, just getting access to them probably was not an
easy task.

That said the authentication layers failures and lack of testing are far from
impressive.

------
pgrote
Where are the links to IRS and social security admin? During the sign up
process they verify tax return status and ssns of dependents including
children?

Is that what CMS does?

------
guiomie
"I can tell that this system consists of a JBoss Application Server with data
access components and RESTful web services developed using Java. Given my
prior experience with JBoss and Java, although they are great for middleware
development, they’re known to be a bit slow."

I always had this non founded impression that something running on the JVM
would be slow ... this author also. Anyone with data to back this impression ?

~~~
redthrowaway
It has nothing to do with "running on the JVM". The JVM is about as fast as
you'll get any VM offering GC and running an IR like Java bytecode. It's only
slow compared to compiled, native code (and even then nowhere near as
relatively slow as it used to be), which no competing tech stack would be
doing. It's _much_ faster than Ruby, Python, etc.

That said, my experiences with JBoss haven't left me enamoured of it, and it
certainly wouldn't be my first choice.

~~~
curveship
In techempower's web framework benchmarks[1], Java-base frameworks top most
categories. One C++-based framework, CPoll, gets a few percentage points more
in a few tests, but Java servlets are the highest performing across all tests.
(As always with micro-benchmarks, take all this with a grain of salt, but at
least it's some data.)

1 -
[http://www.techempower.com/benchmarks/#section=data-r6](http://www.techempower.com/benchmarks/#section=data-r6)

------
skyraider
I think the interesting lesson here is that even marketplaces should be
competitively constructed - i.e., there should be significant price-
competitive pressure to deliver goods and services; just letting the
government to facilitate marketplaces removes price competition from the
marketplace itself.

I'm not suggesting that the government create a meta-marketplace for health
insurance marketplaces. Just that it isn't as efficient as some people think
to introduce a powerful federal competitor with 'unfair advantages' like
public funding into the health insurance selection market.

~~~
ngcazz
The government should have had specified the APIs and see what vendors created
the best implementations.

------
usaphp
$293Mln...Ok so it's like hiring a team of 100 web developers, pay each of
them $100K a year, and they will work non-stop for 30 years...

Edit: Working non stop for 30 years.

~~~
dm2
Realistically you would need to double that 100k to pay for other expenses
(hardware, benefits, office, management). Even then, CGI Federal screwed the
American people and should be fined for what they've done.

------
hpritch5
Too bad I didn't get this contract...

~~~
talmand
I know what you mean, I could've totally made a failing website for $100
million max. I would have saved them money.

~~~
icn2
For government projects, sometimes they don't want to save money at all. They
are more interested in how money can get in someone's pocket.

~~~
talmand
Fine then, my budget for the failing website is now $500 million. Savings be
damned!

Oh crap, sorry, inflation. Price tag's now $750 million.

------
icn2
At least they didn't use windows/.net stack this time.

~~~
sourc3
What makes you say that? Having built systems in this space with very large
number of users using the .net platform, I wonder why you stated this.

I am a supporter of open and non-MS software but I don't think we can blame it
on the technology this time.

In my experience so far a well engineered system, will perform regardless of
the technology stack picked. Picking technology X or Y will not compensate for
bad architecture and engineering practices.

~~~
newsreader
Couldn’t agree with you more.

