
Programming Language Breakdown for the HealthCare.gov Website - yawz
http://www.randalolson.com/2014/05/22/programming-language-breakdown-for-the-healthcare-gov-website/
======
LVB
At least this pile of code turned into _something_ , unlike my local Cover
Oregon which has been killed following a $250 million outlay and still no end
in sight [0].

Funny that oft-mentioned Oracle used their work with Oregon to tout how their
enterprise prowess "plays an essential role in establishing Oregon as a leader
in healthcare reform", but the link to the source piece is now dead [1]. I
did, however, save a copy of this PR fluff when it came out in 2012, just to
see how it would all pan out... [2].

[0]: [http://www.latimes.com/nation/nationnow/la-na-us-
attorneys-o...](http://www.latimes.com/nation/nationnow/la-na-us-attorneys-
office-probes-oregons-health-insurance-exchange-20140521-story.html)

[1]: [http://www.oracle.com/technetwork/articles/entarch/oeea-
home...](http://www.oracle.com/technetwork/articles/entarch/oeea-
home-280423.html)

[2]: [https://www.dropbox.com/s/jnt15vqdc15aopj/state-of-
oregon-18...](https://www.dropbox.com/s/jnt15vqdc15aopj/state-of-
oregon-1864645.pdf)

~~~
vhost-
Cover Oregon was a disaster. That work should have been done by an agency here
in the state. I don't understand the logic of government sometimes.

~~~
selimthegrim
I once went out on a Tinder date with one of the IT managers for Cover Oregon
(in December/January). She was a music major at Denison and apparently Oracle
was vacuuming up everyone with a pulse and a BS in DC and the East Coast to
manage their contractors. I guess she was doing pretty well too since she
could afford to go to Sun Valley once a month from Portland. What I wonder is
why couldn't Oracle find managers with more experience?

~~~
esbranson
Incompetence?

~~~
dwd
I've met a lot of Tier 1 contracting consultants. Smart people, excellent
communicators, well presented and great at generating ideas, but with zero
problem solving experience they were no better than a fresh graduate.

You need a few seasoned professionals who have seen it all to keep the
children in order. Reminds me of my year 12 chemistry teacher who was out of
the class for a minute and on returning had the class evacuated within
seconds. He recognised the smell of mercury vapour from a burst thermometer
the second he walked in.

------
aaronem
I've recently been solicited by a couple of recruiters, on behalf of
contractors who've got some of that work. One of them in particular amused me
by trying to hard-sell a six-month contract job with a firm which organized a
"career fair" an hour south of here on two days' notice. When I showed
disinterest, he "sweetened the pot" with an offer of a $25 gift card. That was
when I told him not to bother contacting me again. (It's not even so much that
he tried to bribe me, as that he tried to bribe me so _low_. I mean, really --
$25? What kind of naïf does this jackass usually deal with?)

If that's the at all representative of recruitment efforts around
healthcare.gov, then I'm not sure exactly what it says about the quality of
the staff working on the project, but I can't imagine it saying anything good;
the best of them are probably quite bright, painfully inexperienced, and not
yet equipped to understand how awful this project is going to look good on
their résumés -- and I'd tend to suspect that the best of them are a small
fraction of the whole. Between that, and the inevitable bloat and quality
issues of any federal technology project with this many fingers in the pie,
I'm really not surprised the whole thing has turned out to be such a rolling
catastrophe.

~~~
us0r
Based on the last notice they sent out - around 400 people are working on this
mess. I got a feeling they are more concerned with just filling the positions
then hiring (and paying) experienced developers.

~~~
watwut
Based on what I seen in hacker news few month ago, a guy trying to hire people
on healthcare.gov described it somewhat like this: we are saving the project
and are looking for people. We hire them for few months stints (3-6 months),
they work super hard that time and then leave. He mentioned occasional
sleeping under desk, it did not sounded like a joke.

If true, they have perpetual death march with constantly changing workforce
and everyone being essentially newbies on the project. How many experienced
people can you get with that sell?

~~~
Crito
"Saving the project" has become a massive red flag for me.

In my experience, when anything about "saving" is mentioned, it means a
handful of people started the project, made a hash of it, then after a few
short weeks passed it off to another team who were tasked with "saving it".
This second team blames all their failures on the first, thinks that they are
making the best of a bad situation, then after a few months passes it off
again. The new people comfort themselves with the same "saving the project"
narrative for a while more until it again comes time for somebody else to save
it.

Each team after the first lasts only as long as they can plausibly believe
that everything is the fault of the people that came before them. As soon as
they realize that _their_ shit stinks too, they bounce.

In the end, 90%+ of the man hours in a project were put in by people who
thought they were saving it.

------
dpeck
So I have a theory:

[http://www.ohloh.net/p/openjdk/analyses/latest/languages_sum...](http://www.ohloh.net/p/openjdk/analyses/latest/languages_summary)

The numbers even lineup fairly well in the 2010-2012 time frame which is
around when the project would have started.

------
outside1234
I still don't understand how they have that much code. 3.5M lines is still
crazy - what exactly is all that goop doing?

~~~
watwut
Maybe they counted all dependencies or libraries. If you add all that to any
project, size will grow ridiculously big. I suspect so because the coding part
of the project too only half a year or so, so they could not possibly write
that many line by hand.

~~~
Cowen
> Maybe they counted all dependencies or libraries.

This is really the only way I can imagine they got to that number.

Spring, a beastly enterprise Java codebase if ever there was one, is still
"only" 460K lines of Java code according to cloc.

------
ineedtosleep
With all the healthcare.gov hate, I've been wondering why there's barely any
scrutiny towards coveredca.com -- at least I haven't seen any.

~~~
brikis98
I just tried to go through the coveredca.com application process and it was a
nightmare. It took me 5 hours, spread over two days, 3 tech support calls (I
_never_ call tech support!), and 3 different web browsers to get through it.

1\. Fill out the very first part of the application - just basic name and
address - click "continue". Error page. WAT.

2\. Notice they have an outline view of the application. Use that to get to
get past the error page. Now it's asking for my "Certificate of Citizenship"
number. WAT. No form I have ever filled out - tax returns, financial aid
forms, employment forms, DMV forms - nothing has ever asked me for this. Spend
a day searching for it.

3\. Finally find the number, fill it in, click "continue". Error page. WAT.

4\. Use the outline again to get to the income section. It lists me twice, as
if there are two separate brikis98's in my household. It even asks me "how is
brikis98 related to brikis98"? WAT.

5\. Call to tech support #1. After 45 minutes of messing with the application,
clearing cookies, logging out/in, we couldn't figure out how to remove the
extra "me". There is no way to completely restart an application. Solution
from tech support: sign up for a completely new account - new user name,
password, etc - but don't enter your SSN too early or it'll detect the
duplicate account. WAT.

6\. Sign up for a new account. Start filling out the application from scratch.
Error page. WAT.

7\. Use the outline trick again. Go to income section. No duplicate "me" this
time - yay! However, now it asks for "monthly income". I just left my job, so
my monthly income is zero... but I obviously had a monthly income the first 5
months of this year. Call to tech support #2. They tell me to take all the
money I've made this year, divide it by 12, and put that as a monthly income.
WAT.

8\. Finish filling out the application. Get to a page that has a button to
"See Eligible Plans". Click the button. Error page. WAT. There is no outline
view to get past this one.

9\. Call to tech support #3. After half an hour more, the suggestion is to try
another browser. I switch to Firefox. It tells me coveredca.com does not have
a valid SSL cert. WAT.

10\. Switch to Safari. No complaints about the cert. Log in, click the "See
Eligible Plans" button. The page loads - yay! However, the UI is broken, with
strange scrollbar-in-a-scrollbar setup that makes it very tough to browse the
plans. Download PDFs of each one for easier reading.

After all that, I found that the plans were all way overpriced for the
comically bad benefits they offered.

~~~
ericd
I ended up signing up for one, and I've been pretty happy with it, not too
expensive, and decent when I've tested it. I also experienced a bunch of
errors on the way to getting there.

The worst part, though, was after going through the 500 error gauntlet,
hearing the person in charge of CoveredCA go on NPR and just pat himself all
over the back for how great a job they had done, and much better the result
was than Healthcare.gov. After spending only a few hundred million to create
that hellish mess. One of the few times I've ever yelled at my radio.

------
bjohnso5
In what way is Maven a programming language? Is it simply to disambiguate
other XML configuration from the build declarations?

~~~
rnd33
How is 50k lines of build declarations even possible?

~~~
fournm
That's not actually surprising--it's really easy to have a 1500+ line long POM
with how verbose it is. Depending on how many maven modules there are, I could
see 50k pretty easily.

------
dang
Url changed from [http://www.randalolson.com/2014/05/22/programming-
language-b...](http://www.randalolson.com/2014/05/22/programming-language-
breakdown-for-the-healthcare-gov-website/), which points to this.

~~~
rflrob
I would argue that the graphical breakdown on the original link is more
helpful to grok the breakdown of languages than the original reddit post.

~~~
dang
Ok, we changed it back.

------
joshwa
from [http://www.enterprisetech.com/2014/04/04/hyperscale-
lessons-...](http://www.enterprisetech.com/2014/04/04/hyperscale-lessons-
healthcare-gov/):

“The way that a lot of the code was developed for the application was using
automated modeling tools” he says. “They configured their modelers to
[generate middle-tier Java objects] in a three-tier style architecture.
Essentially, they created a layer of abstraction that didn’t need to be
there.”

------
draker
I've got to work on my submission titles.

> Healthcare.gov developer provides estimate in response to "500M lines of
> code" _5 hours ago_

The chart from the submitted blog post does provide a nice visual perspective
of the breakdown that you don't get from the source alone.

[http://www.randalolson.com/wp-content/uploads/healthcare-
gov...](http://www.randalolson.com/wp-content/uploads/healthcare-gov-code-
count.png)

------
Phlarp
What could you possibly need that many individual yet mostly similar languages
to accomplish? Even if we grouped things like XSD, XSLT into the XML category
or Groovy and JSP into the Java category it feels silly to need this many
languages in a single app.

I can only imagine what kind of things all the long tail files are doing. A
smattering of Perl and Python, then a lonely VB and ASP.NET page?

~~~
pnathan
I've used cloc a fair bit... cloc is a bit sketchy in recognizing code
occasionally. I wouldn't trust the "tail" of its output, it's frequently wrong
or misleading down there.

~~~
glassx
Yep. In the same Reddit thread, someone ran cloc on the Linux Kernel, and it
even reported one ASP.NET file in there too (just like it did with the
Healthcare.gov source):

[http://www.reddit.com/r/dataisbeautiful/comments/265yns/mill...](http://www.reddit.com/r/dataisbeautiful/comments/265yns/million_lines_of_code/chogad9)

------
jaunkst
Seems like a result of alot engineers and an inhuman level of understanding
the systems architecture. Reminds me of the Lsd usage for IBM. There must be
an outrageous number of bugs and exploits in the code. Google Chrome is
something like 0.06 per thousand lines of code.

------
gulbrandr
At the recent dotScale conference in Paris, Robert Kennedy explained how he
managed to fix the site.

See:
[https://twitter.com/search?q=%23dotScale%20Robert%20Kennedy](https://twitter.com/search?q=%23dotScale%20Robert%20Kennedy)

------
turar
So the user-facing portion of healthcare.gov is written with JSF then?

------
dukerutledge
Well there's 2.4 million lines of Java that will have to be maintained for the
next 100 years.

------
ChrisArchitect
wasn't this floating around 6 months ago? do we really need to discuss this
again?

