
The Obamacare software mess - csa
http://www.economist.com/news/united-states/21587816-signing-up-obamacare-still-ridiculously-hard-obamacare-software-mess
======
DanielBMarkham
I work with a lot of technology teams, in a lot of different situations. I'm
also a programmer.

I think the tendency in these Monday-morning quarterback sessions is to blame
the people or the tech. _If they only used X,Y, and Z!_ people will say, or
_Why can 't they find qualified people who can do this, it's not that hard!_

When that fails, we start blaming _groups_ of people. _Those damn contractors.
Always screwing us over!_ or _Dang government bureaucrats! Lazy bums!_

These patterns of analysis are almost always mistaken. You don't drop a
billion bucks and get screwed because you hired dumb people, because you're
lazy, or because they decided to use WhizBang 1.0 instead of 2.0 You get
screwed because you have smart, motivated people working inside of a system of
delivery which is hosed. Incentives are wrong, feedback loops are loose (or
non-existent), and so on.

So while this is a nice political topic, and sure to get everyone's emotions
roiling, be aware that we see this same stuff all of the time. (I would make
the case that government is far worse, for exactly the reasons I mentioned
above, but I won't because it's only going to push many of your buttons).

It's not the simple stuff that kills projects. It's certainly not tech. It's
particular configurations of people that choose bad tech, or that refuse to
change bad tech once it's identified. Most all of technology development
problems boil down to _social_ problems, not the other stuff. If you've got a
good social structure with ten qualified people, you can rule the world. If
you have a bad one with a thousand geniuses, you'll be lucky to get your shoes
tied in the morning.

~~~
jacques_chester
Like Weinberg said in _Secrets of Consulting_ :

    
    
       No matter how it looks at first, it's always a people problem.

------
jzelinskie
I had someone visit one of my classes at PSU and describe the backend system
as it is being implemented in Pennsylvania. The system that was described acts
as one big middle man and interops multiple legacy systems mostly written in
COBOL. I was told they have the time constraint of 7 seconds to process
approvals, which to me seems like a lot, but I'm sure the older systems are
probably the real constraint. If these legacy systems supported modern APIs
and data transfer formats, this could probably all fit cleanly into any web
framework and you wouldn't need a backend dedicated to doing tricky data
cleansing. I hope the plan is to use this system as a shim while they improve
the legacy systems.

~~~
XorNot
Those systems are legacy systems for a reason. They're _never_ being upgraded.

Also, before anyone gets all "hurr durr government!" on this - this is the
exact same situation at any major bank or company founded before about 1990.

~~~
triton
Spot on this is normal. I work in the financial services sector and stuff
still arrives on dialup and ftp from mainframes regularly in all sorts of odd
formats. SOAP is what is in the executive bathrooms and JSON is the cleaner's
name and that is it :)

~~~
mkr-hn
I ordered a part at the local franchise of a major auto parts place (I forget
which one). The guy acted a little embarrassed as we stood there listening to
the dial-up modem connect to place the order with the corporate system. That
was in 2012 when even the small corner bookstore down the road had free wifi.

------
pasbesoin
This situation has reminded me of the Perl Success Stories booklet that
O'Reilly used to spread around. And how, for example, the Bank of Scotland or
some such institution ended up keeping the Perl prototype for production, it
worked so well.

I can help thinking that a few good Perl programmers, and a few high-level
bureaucrats to force open the maws of the respective agencies, could have had
this sorted in much shorter, and more cost-effective, order.

Also, as I commented elsewhere before, _first_ make it work. _Then_ make it
pretty.

I guess I'm being a bit snarky. But I'm also being more than a bit serious.

~~~
dangero
The goal of a contractor is to extract as much money as possible from their
client, not necissarily to do efficient quality work. In some cases it makes
sense for a contractor to do quality efficient work, meaning that is the best
path to profits, and sometimes it is not.

~~~
shitgoose
If contractor is IBM or Kumaran, then yes, their only goal is to extract
profit. They are not concerned about product they make it look like they are
building. If contractor is Joe Blow Perl coder, then most likely his goal will
be to deliver a good system he can take pride in.

~~~
AJ007
Which will also mean further contracts and recommendations from others.

Contractors who do shitty work are putting most of their effort in to landing
new clients to replace the ones who will never repeat.

------
pbreit
I think a lot of this "gov is incompetent" is overblown. Complex systems like
this rarely work exactly right on launch (and rarely launch on time). People
have another 2.5 months to register before any coverage actually kicks in.
This whole episode was just a layup for right wingers to torch government.

~~~
dm2
I'm not a right-winger but am disappointed that the healthcare.gov site isn't
working at all.

It's 2013 and there are tons of technologies that allow this type of system to
be set up fast and scalable. The fact that I can't even login (with a
registration I made a month ago) and that it doesn't even give me an error
message, and that support (phone and chat) just say "it's experiencing
problems right now" is inexcusable when you consider that they had half a
billion dollars to work with.

Is a secure registration and login system that at least gets people started on
the site too much to ask?

~~~
hga
" _Is a secure registration and login system that at least gets people started
on the site too much to ask?_ "

Evidently so, even given 3 and a half years after passage of the bill that's
this Administration's signature accomplishment to date. While it's hardly
unknown in other sectors, I've read one of the problems was HHS/Sebelius
making important decisions way late in the process, and I think I recall the
usual required changes as well.

But unless the signup process hooks into outside live databases for
verification or what have you there's as you note no excuse, especially since
disallowing window shopping is one of the biggest modern UI hindrances,
something pointed out to the people doing this ( _some_ information gathering
is needed to offer the right plans, but others and I have compared it to the
medicare.gov Part D sub-site, which collects that but doesn't require giving
all your information prior to you officially selecting a plan, after which the
site informs the chosen provider).

Anyway, I _am_ a "right-winger" who's upfront about wanting to ... eliminate
many parts of government, and just from memory I can point out several massive
and notorious IT procurement failures that would kill or cripple most private
concerns. E.g. last time I checked (which is some years ago):

The FAA is still using their _1940s_ (sic) techniques, and every attempt to
really modernize the initial system architecture that automated it has failed.
(They famously hired a couple of retired IBM engineers to make microcode
patches for some of their old machines for Y2K). This one has really serious
economic consequences.

Every comprehensive attempt to upgrade the IRS following their very first
system has failed. They're still using code for the 1959 IBM 1401
([https://en.wikipedia.org/wiki/IBM_1401](https://en.wikipedia.org/wiki/IBM_1401))
in their front end processes.

The FBI is legendary for their failures to ditch paper. I remember at least
one failed attempt pre-9/11, there's one acknowledged failure after it, and I
don't know if they're really using the Sentinel system as they claim as of
2012, after 7 (!) years of troubled development.

These are the really big ones that come to mind. Which of course is not to say
the government is always incompetent. The Census Bureau was famously cutting
edge, using punched cards in _1890_ (sic), buying a UNIVAC I in 1951 (sic),
and when I talked to some of their people in the mid-90s I got a sense of
serious competence.

As noted in previous discussions, the same contractor operations CMS.gov and
Medicare.gov, and I can attest that the latter site, while a bit clunky, works
just fine. CMS's backends seem to work as well.

Off the top of my head, the CDC and FDA do well in getting out timely info.
The DoJ's statistics sections are good, the NIH's sites are stellar. Heck, the
BATF has a nice web page where you can type in some of the numbers of an FFL
(firearms dealer, gunsmith, etc.) and confirm it is legitimate and still
valid. Etc.

------
dangrossman
They're making progress at least. When the site first opened I couldn't get
in, but I had no trouble today. It works just like eHealthInsurance.com -- the
site I used to buy my own insurance 3 years ago.

[http://i.imgur.com/GwXTkdc.png](http://i.imgur.com/GwXTkdc.png)

------
Encosia
> _Officials say that too many people tried to log on at once; within three
> days, 8.6m had visited healthcare.gov._

That is an order of magnitude less traffic than I expected. 33 visits per
second isn't child's play, but my _WordPress blog hosted on a 1GB Linode VPS_
has easily handled bursts of that kind of traffic in the past.

That's totally an apples and oranges comparison, of course, but it's more and
more difficult to understand how this project has failed so miserably. I half
bought the excuse that it was one of the highest traffic site launches in
history, but apparently that was more speculation than truth.

~~~
jasonlotito
Except your making the classic mistake of assuming that it's 33 visits per
second because you are looking at the 8.6m visitors happening evenly over the
course of 3 days. You're also making the mistake of confusing the number of
visitors with the number of hits on the site. 20 requests to the server for
information is still one visit. And yes, 33 hits per second on a static site,
cached at various points is far, far easier to develop.

> but it's more and more difficult to understand how this project has failed
> so miserably.

So, imagine someone estimated 8.6m visitors over three days, and decided to
load test the site for 33 visits per second... =)

Then it becomes really easy to see why it could fall over.

Disclaimer: I've been doing load testing the last few weeks, and have been
beating up on other peoples server side applications. It's a lot of fun when
it's not your code getting hammered. =)

~~~
jacques_chester
Agreed. Capacity planning is not about the average case; it's about the 99th
percentile case and whether you can afford it.

------
tunesmith
Sort of short on details or perspective.

I do remember reading articles six months ago or so that the federal
government was disappointed that they would be managing the exchanges for so
many of the states. They were originally hoping that they'd only be managing
the exchanges for a handful of states, rather than the 36 they're having to
shoulder the load for. I can see how that level of political uncertainty adds
an extra challenge for estimating load.

~~~
TazeTSchnitzel
I think it's mostly red states that aren't running their own ones.

~~~
hga
Idaho is the only solidly Red state that's doing its own.

A number of Blue or Purple states are also punting to the federal exchange, or
doing a "partnership" where the federal exchange is doing the heavy lifting:

New Hampshire, New Mexico for the moment for individuals, Utah permanently for
individuals, Arkansas, Iowa, Michigan, Illinois.

In many of these cases of partnership, whatever the governor of whatever party
wanted, the legislature didn't buy off. New York is the only case I know of
where the governor told the legislature to jump in a lake and set up one for
the state through an executive order using the temporary money the Feds are
donating early on.

And there's Missouri, who's people outlawed a state run exchange, and New
Jersey's governor vetoed 2 bills to set up one.

~~~
hga
Further reading my source ([http://www.healthinsurance.org/idaho-state-health-
insurance-...](http://www.healthinsurance.org/idaho-state-health-insurance-
exchange/)) after seeing the map in a DailyKos item linked to by
shavenwarthog2, Idaho is depending on the Feds for 2014 while it is working on
its own full exchange. Still, the exception that proves the rule.

------
jsumrall
I read that CGI Federal developed the site, but did they get the contract to
maintain it? I can imagine that, compared to a non-gov't contract, they have
to go through a lot of red tape to fix the issues and build up capacity.

~~~
Maxious
> The contract has an estimated total contract value of US$93.7 million over a
> two-year base with three, one-year option periods. The US$55.7 million base
> contract has been awarded.

[http://www.cgi.com/en/CGI-selected-build-US-wide-
competitive...](http://www.cgi.com/en/CGI-selected-build-US-wide-competitive-
health-insurance-exchange)

------
pingswept
The evidence in this article is:

    
    
      1. Jason Lahoz couldn't sign up.
      2. @Ginnyproffitt was frustrated.
    

I tried signing up a couple of days ago. The site loaded quickly and worked
fine. The only quirk I noticed was that I received two registration
confirmation emails instead of one.

We should not be so quick to jump on the government without evidence that
there is a problem, especially when you can just go look at the site yourself
and see if it works.

~~~
pjmorris
I tried signing up on Thursday. I filled out the account information,
submitted, the site spun for a few seconds and replied 'Important: Your
account couldnt be created at this time. The system is unavailable.' (Note no
apostrophe in couldn't, verbatim from site.) Twenty minutes later I received
an email 'Your Marketplace account has been created.', with a link to verify
my email address. I read the email an hour later, clicked the link, and was
informed that the link had expired and that I would need to re-enter my
account information.

I realize that the plural of anecdote is not data, but there's got to be some
multiplier of plural that is data.

~~~
pingswept
Agreed, but I think it's a pretty big multiplier.

------
smartician
I was surprised that they had _anything_ ready on October 1st. I expected at
least a six month delay. Not because of _hurr durr government can 't do
anything right_, but that's the nature of large scale IT projects. They're
extremely hard to manage, especially when there are a lot of third party
systems involved.

------
shadesandcolour
I wonder if this would still be the case if fewer states had punted the work
to the federal government. The fact is that the federal government is running
36 exchanges was never the plan.

------
curlyquote
When will we finally hold contractors that build shit products for the public
accountable for wasting a huge amount of tax dollars?

~~~
DanielBMarkham
They are being held accountable. The press prints a story, you get all angry,
hopefully you vote somebody into office, then they create new rules, and in
ten or twenty years we can start making generalizations about how well those
rules are working (or not).

How else would you do it? Public executions of any public-funded program that
looks bad? We have an accountability system -- it just might not be designed
to optimize the things we would like for it to optimize.

~~~
curlyquote
Why not have the federal government have an actual organization of software
developers (with possibly a standards committee) that do it themselves instead
of contracting it out to private companies?

Something like a NASA for non-space software development?

~~~
mcguire
You do realize that NASA contracts everything out to private companies? All of
the NASA employees I know are either project or program management (the two
sides of the contracting coin).

~~~
hga
From an obit, [http://www.washingtonpost.com/wp-
dyn/content/article/2007/05...](http://www.washingtonpost.com/wp-
dyn/content/article/2007/05/03/AR2007050300879.html)

 _Walter Schirra; Fifth Astronaut in Space

By Patricia Sullivan

Washington Post Staff Writer

Walter M. Schirra Jr., 84, one of the original seven astronauts and the only
man to fly in the Mercury, Gemini and Apollo space programs....

[...]

[His second] mission was delayed a second time when the Titan II engines
beneath the space capsule ignited at countdown and then shut down.

For several heart-stopping minutes, Capt. Schirra and astronaut Tom Stafford,
sitting atop a highly explosive mass of rocket fuel, chose not to pull the
ejection handle, which would have scrapped the mission. It was a calculated
risk. Capt. Schirra trusted that the booster rocket would not explode and that
the first attempt to rendezvous with another spacecraft, Gemini 7, could still
occur. The risk paid off, and three days later, the launch was successful.

Asked later what he thought while sitting on the launchpad, Capt. Schirra
replied, "This was all put together by the lowest bidder."_

E.g. the story of the reasearch and developemtn of the legendary Apollo Saturn
V first stage engines by Rocketdyne, first for the Air Force and then for
NASA, is also legendary:
[https://en.wikipedia.org/wiki/F-1_(rocket_engine)#History](https://en.wikipedia.org/wiki/F-1_\(rocket_engine\)#History).
They used explosives to debug it! That's got to count as a hack....

------
nudetayne
The WA state one is full of XSS and SQL injection vulnerabilities. It's quite
beautiful.

