
Health Exchange Tech Problems Point To A Thornier Issue - jessaustin
http://www.npr.org/blogs/alltechconsidered/2013/10/08/230424841/health-exchange-tech-problems-point-to-a-thornier-issue
======
vinhboy
I have never worked on a high traffic site before, so I am just mostly talking
out of my ass. But when I looked at the html source for the coveredca website,
it had a lot of of basic problems: there were 3 DOCTYPES on one page,
developer comments were still visible (you can actually see the names of all
the developers who worked on the project), Java error pages were visible, they
disabled the back button completely by enforcing javascript refreshing, they
did not minify any of their assets (which would also remove the developer
comments) AND they were actually loading many copies of the same assets
multiple times because, like I said earlier, their site had 3-DOCTYPES in one
html page.

As someone who has never worked on a huge site, I don't want to naively say
that it could have been done better. But I feel like those are basic issues
that could have easily saved them a lot of bandwidth.

Also, it seems to me that if they had so many issues on the front-end, it's a
hint that maybe they also had many similar issues in the backend as well.

~~~
ericd
Yeah, the CA version
([https://www.coveredca.com/](https://www.coveredca.com/)) had so many visual
glitches and various problems (404s and 500s), I threw up in my mouth a little
when I found out that Accenture charged the state over 180M to build it and
another 174M to operate it for four years. I don't know how that's possible.

~~~
frozenport
I had a job interview with Accenture that had 2 parts a dinner and a technical
chat.

1\. At the dinner their representative spoke about how life at Accenture was a
breeze and how as a undergraduate he spent most of this time drinking. He
drank on the weekdays but his favorite time to drink was on Sundays. He said
he used to do some SQL but now he had moved onto the business side of things.
One of the candidates told us how his hobby of skydiving let him see the big
picture and another told us that his desi background would enabled to come up
culture specific features for software services. I probed this question
further, and he said that in India showing your muscles was considered hubris,
while in America it made you look strong. On his facebook page he wouldn't
show his bare chest... Nobody at the table `looked` like they could do any
programming work.

2\. During the technical interview I was asked what `clubs` I was in. I told
them I wasn't in many clubs and instead spent my time doing research work
(which by that time had yield publications) and administrating websites. I
told them my hobby was programming. I wanted to tell them how I dealt with EC2
sight outages, worked with clients and their clients to write WordPress
plugins. I wanted to talk about my experience engineering our production and
outsourcing workflow. I wanted to summarize the techniques I learned
overseeing our production staff. They didn't comprehend and asked he what
`clubs` I was in.

I was scheduled to have 3 interviews, I walked out after the first. Don't work
for Accenture, they are a joke because their employees are a joke, which is
much in agreement with this article.

They also have 275,000 employees.

~~~
tedsanders
I often wonder if this attitude makes the problem worse. If good programmers
don't work for Accenture because Accenture is full of bad programmers, then
how can Accenture ever become better? How can it evolve to provide better
contracting services? If you were in charge of Accenture hiring strategy, what
would you change?

~~~
frozenport
Easy, rebrand and say they are gathering people for a new team. Maybe call it
Accenture+. Maybe don't even use the Accenture name. Start a lean technical
consulting service company that is 100% owned by Accenture. With 275,000
employee it is clear they are looking for volume and not talent.

~~~
ericd
Good point, working for Walmart Labs is actually a much more appealing idea
than working for Walmart.

~~~
frozenport
Yes, and if they weren't retarded they would call themselves Walton Research,
or perhaps be publicly traded with Walmart owning 80% of the stock and being
the sole client?

~~~
ericd
Heh it could be better, but the important part is that it's separate and
mostly independent, so hopefully less mired in bureaucracy.

------
downandout
The government IT procurement process puts people that don't know what they
are doing in charge of multi-million dollar budgets. The primary determinant
of whether or not a company gets hired is its ability to navigate the
procurement process - not their ability to actually do the job. In fact, many
companies specialize in winning government contracts and then subcontract the
work out to others.

It's a sad, broken system that will almost certainly never change - in part
because we don't vote for the people in charge of these things. It really
doesn't matter who is in the White House or in control of Congress. The
fiefdoms within the government that control these issues have maintained a
culture of mediocrity for decades, and it will likely remain that way.

~~~
dnautics
how do you propose reconfiguring the procurement process? Remember, it's got
to be fair. You have to figure out how to charge a fair price. Everything has
to be documented, so that it can be audited by the bureaucrat in a few months
and given a second rubber stamp. Also we have to score the service providers,
don't forget to ask them if they are minority-, veteran-, or disabled- owned,
because each of those gives them +20 points.

~~~
lisper
> it's got to be fair

Why? It's not fair now.

------
drewpc
The article mentioned only one side of the problem: the acquisition process.
The government must follow the Federal Acquisition Regulations (FAR -
[http://www.acquisition.gov/far/](http://www.acquisition.gov/far/)) when
contracting purchases of equipment and/or services. The FAR is not agile and
is so strict that you CANNOT already know what you want, you must make the
contract as generic as possible. Especially on large contracts (over $5M). For
example, you can't say that you need laptops with Intel Core i5 processors
because that would unfairly exclude other processor vendors from competing.

The other side of the problem, arguably the bigger side, is that most of the
people in the acquisition process do not know how to write a contract that is
flexible and really gets the government what it needs. For such a large
project as the health care website, the people at Health and Human Services
most certainly did not know how to articulate what they wanted. They could've
hired a company to help them determine their requirements and write a
proposal, but that's a significant (and long) undertaking itself.

In short, government systems will never be user friendly, scalable, or secure
because we don't know how to acquire such a system.

~~~
MichaelGG
I thought you can get around excluding vendors just by specifying generic
requirements that only one vendor happens to fulfill. Like "must have inter-
processor bandwidth of at least Xgbps" or whatever generic-yet-specific thing
you can find.

------
niels_olson
I work at a DoD hospital. Trying to do some research. I have been told

1) Python is not allowed

2) Open source software is insecure

3 Perl is allowed

4) there is a list of approved software.

5) I can't see it

6) Enthought is on at least one version of the list

7) I can verify there are several python projects on forge.mil

~~~
staunch
#3 solves all your problems then :-)

~~~
niels_olson
Are you serious? Is there a perl implementation of python? Did I mention I am
having to formally present to the executive steering committee to get
permission to run perl?

~~~
staunch
Hah. Sounds terribly bureaucratic. But yes, I'm serious. It may not be trendy,
but there's nothing Python can do that Perl can not. The differences are
fairly cosmetic.

The key is to write good quality Perl, so it's fun:

•
[http://modernperlbooks.com/books/modern_perl/chapter_00.html](http://modernperlbooks.com/books/modern_perl/chapter_00.html)

•
[http://my.safaribooksonline.com/book/programming/perl/059600...](http://my.safaribooksonline.com/book/programming/perl/0596001738)

------
balloot
It should be noted that healthcare.gov might be the highest traffic website
launch in the history of the internet. With millions of visitors on day one, I
certainly can't think of one that would even compare.

They are dealing with scaling issues that are almost completely unprecedented.
Every other e-commerce site that deals with many millions of visitors a day
has had months, if not years, to figure out the touchy parts of their
infrastructure and make adjustments while the traffic ramps up.

Given this, I can't help but think that 90% of the scorn heaped upon
healthcare.gov is politically motivated. When Twitter spent YEARS going down
on a regular basis, or Simcity just completely shut down on launch day (with a
fraction of the users of Obamacare), nobody demanded the companies that
running them be shut down. Shit happens when you build a website that gets
incredible amounts of traffic. It will work out just fine.

~~~
wdr1
> It should be noted that healthcare.gov might be the highest traffic website
> launch in the history of the internet. With millions of visitors on day one,
> I certainly can't think of one that would even compare.

I can think of two that I've worked on personally:

\- Yahoo!'s tribute site for the first anniversary of 9/11 \- Ticketing for
the 2008 Beijing Olympics

Yes, in both cases, the companies had experience & infrastructure for building
high-scale website, but that should have been a requirement for the entities
building healthcare.gov as well.

In both cases I've cited there were a lot of _significant_ changes to the
technical stack (e.g., it was first time we used PHP for a large scale project
at Yahoo!).

TL;DR: This isn't really excusable.

------
PaulHoule
More generally the problem is a lack of standards in technology management.

Malpractice is endemic in the IT space and nobody, either technologists or
managers, are held accountable.

For instance, the issues @vinhboy talks about are "bad smells" that reek of
unprofessionalism -- like the primary keys that aren't unique, the predictable
race conditions, and all of the problems that recur again and again.

------
justinsb
Apparently integrating with "legacy" systems was a big bottleneck (e.g. to
verify your SSN). I don't understand why this verification had to be done
"live" though; the obvious alternative would be to collect all the data,
verify it asynchronously/overnight, and then flag up those people that had
issues. You enter your birthday, SSN, etc and then the 1% of people that got
their details wrong get an email the next day saying "we weren't able to
verify your details, please log in and fix it".

------
siliconc0w
Just saying there were like 20 articles about the all star obama campaign tech
team that built a super scalable cloud infrastructure and ab-tested, email
spammed, and narwhal'ed their way to victory. Couldn't get any of them for the
actual governing?

~~~
TWAndrews
No, quite literally not. The campaign team is a private organization, not
subject to rules for hiring federal employees or spending tax money.

This is precisely the sort of shit that make people believe that the
government is incompetent. It really is for certain undertakings that require
some degree of operational flexibility.

~~~
smsm42
Wait, isn't the government the one who makes the rules? So if they made the
rules that makes them incapable of doing things that they are supposed to do,
while the people, technologies and ability to do the same thing is readily
available and the same thing can be and is done by others - isn't that a very
definition of incompetence? Not doing what is perfectly doable because of
self-inflicted inhibitions?

It doesn't "make people believe", it _is_ incompetent. And, unfortunately, the
very capable and competent people are working hard at making it worse by
electing exactly the people who would perpetuate and expand the problem.

------
jaggederest
Ha, sounds a lot like most big companies. Why get and train competent staff to
build a software development process when you can pay 10x as much to a vendor
for software that doesn't really work.

~~~
maaku
Because in many cases hiring competent staff (aka civil servants) requires an
act of congress.

------
nostromo
The jaw-dropper here is how much this website cost: $634,320,919.

[http://www.digitaltrends.com/opinion/obamacare-healthcare-
go...](http://www.digitaltrends.com/opinion/obamacare-healthcare-gov-website-
cost/)

With all the state exchanges included, the websites for this program must be
over a billion dollars spent. Amazing.

~~~
ericd
Yeah, CA alone brings it almost to $1B (another $330M). It's pretty
unbelievable how wasteful the procurement process apparently is.

------
ars
"that contractors built a system that could handle 50,000 people using the
site at once"

There are over 100 million families in the US and they build for 50,000???

Is that number at least 50,000 pages / second? (Which would be reasonable.)

~~~
dangrossman
Even if they could theoretically make a site that can handle every person in
the country using it at exactly the same time, it'd still be bottlenecked by
the much older systems it relies on at the IRS, SS, VA, DOD, DHS, etc. As the
article alluded to, the Healthcare.gov site checks your family size, AGI,
social security benefit status, citizenship status, incarceration status,
veteran status and enrollment in other government health programs during the
signup process to determine what you're eligible for.

~~~
justinsb
I posted this in another comment, but do you have any ideas as to why that
information has to be verified in real time?

~~~
URSpider94
Because it's giving the user real-time access to which plans they are eligible
for.

The only other two options I can think of would be:

\-- Pre-verify all of the potential applicants, many of whom may already have
coverage and never come to the exchange. \-- Ask users a bunch of questions
that they may or may not know the answers to.

~~~
justinsb
I really like the suggestion of doing the calculation for everyone in advance.
We could even have sent a letter to everyone that would be better off if they
enrolled.

------
robomartin
The real scary part is that we entrust these people with our most private
information as well as decision making matrices that might very well determine
what, when and if you receive the appropriate care.

To me these incidents, the broken procurement systems, the mediocrity (or
worst) that permeates government and many other factors only serve to
reinforce the idea that we need massively less government reach into our
lives, not more. Frankly, the combination of healthcare and the IRS scares the
living daylights out of me.

------
Spooky23
Blaming IT is an easy scapegoat. The notion that the Feds and various states
should be operating marketplaces in the first place is absurd on its face.
California and New York aren't subject to federal procurement guidelines, and
their systems are basically non functional as well.

Government is successful with complex public facing IT in areas where the
processes are well understood. The DMV, tax collection, and unemployment are
good examples in most states. Medicare billing is a success story in the
federal space. Most other things are not very successful unless they are
limited in scope.

The whole system is a screwed up farce. "Obamacare" is a flawed system to
begin with, to the point that nobody really understood what was needed for
months. So the Feds and states were forced to roll with some highly generic
requirements and specifics were made up as they went along. That scenario is a
dream for Accenture (Cali) or CSC (ny) to churn hundreds of millions of
dollars.

~~~
threeseed
Hilarious considering that Medicare had far more serious problems throughout
its inception and you consider the billing component to be successful.

Like all website launch problems in the months and years from now people are
going to forget they ever happened. And instead will find having an
independent marketplace to be incredibly useful. The states that have
exchanges have already seen insurance premiums drop significantly.

~~~
itbeho
I keep reading anecdotal stories on the Internet about _premiums dropping
significantly_ but I, some in my family and a friend have recently experienced
the opposite.

------
interstitial
I actually know someone who has worked for the VA in medical coding contracts.
He still has Cobol programmers on his speed dial because some of the live code
is still written in it.

~~~
trimbo
They might have COBOL around, lots of banks and government agencies do. But
the VA's critical system, VistA, is written in MUMPS[1].

It's NoSQL!

[1] - [http://en.wikipedia.org/wiki/MUMPS](http://en.wikipedia.org/wiki/MUMPS)

------
rurounijones
Good luck changing the procurement process. The UK Gov has been trying to get
more SMBs involved in UK Gov procurement[1] in hardware and software is
failing pretty miserably at the moment.

[1] [http://www.informationweek.com/smb/services/uk-pushes-
govt-i...](http://www.informationweek.com/smb/services/uk-pushes-govt-it-to-
use-smb-suppliers/240149701)

[2] Nuts, cannot find the link. Was an article on theregister.co.uk for what
it is worth

------
perfunctory
> In order for the exchange marketplace software to check on which coverage
> and subsidies you're eligible for, it has to ping requests to the ...
> Homeland Security data

Really?

------
jes
To what degree is the sheer complexity of the underlying government
legislation responsible for cost, delays and complexity in the IT systems
government pays to have built?

No wonder we are going broke. It's nothing but special cases, all the way
down.

------
joshuaheard
Start at 1:00

[http://www.nbc.com/saturday-night-live/video/weekend-
update-...](http://www.nbc.com/saturday-night-live/video/weekend-update-
winnerslosers/n41636)

------
philipashlock
For those wanting deeper technical analysis around the problems and building
off of some of the arguments Clay made about procurement, check out
[https://news.ycombinator.com/item?id=6518739](https://news.ycombinator.com/item?id=6518739)

------
pasbesoin
Make it work. _Then_ make it pretty.

You missed a step.

------
tantalor
This is the same company that build MapBox?

~~~
quannah
No. But the company that built the _working_ sections of healthcare.gov - that
is, everything except /marketplace* - and open-sourced it
([https://github.com/CMSgov/healthcare.gov](https://github.com/CMSgov/healthcare.gov))
is Development Seed. They are indeed the makers of MapBox.

