
Inside the race to rescue a health site, and Obama - kanamekun
http://www.nytimes.com/2013/12/01/us/politics/inside-the-race-to-rescue-a-health-site-and-obama.html?pagewanted=all
======
dpapathanasiou
Does anyone know what the author meant by this?

" _Software that assigned identities to enrollees and ensured that they saw
only their own personal data, known internally as the EIdM, was being quickly
overwhelmed. Customers could not log in to create accounts._ "

" _Experts disagree on what went wrong. But several said that errors in the
software code written to stitch the Oracle product into the online system and
improperly configured hardware trapped users in endless technological loops.
It would take eight days to resolve just that one bottleneck._ "

~~~
DougWebb
The first part sounds like they created a user management system: normally a
single table with fields like username, password-hash, real name, address,
etc, and sometimes a couple of other tables for groups/roles and cross-ref
tables for relationships. In this case the user table would also probably have
ids/references to the various government backend systems that the site used to
verify your identity, income, etc as you signed up.

The rest of it sounds like they botched making this system scalable, to handle
both the number of possible users (every adult in the US) and the rate at
which they'd be signing up. "Endless technological loops" might refer to
either a circular reference in an overly-complex schema, or to normal
dependency checking in the schema which takes too long to evaluate within the
database when the database is under enormous load.

~~~
VladRussian2
no offense, man, it is just sound like you've never developed or deployed
Enterprise Identity and Access Management. I envy you :) The keyword here is
"Enterprise" \- insanely complex [lets just don't touch whether such
complexity is really necessary, it is just the fact of a design to meet any
possible RFP in the space :) ] and [in many aspects as a result of the
complexity] slow non-scalable products.

>The rest of it sounds like they botched making this system scalable

these systems just don't scale. In a BigCo with local replication to regional
sites it takes noticeable time for such systems to perform. Trying to build
such a website - well, it is a government procurement, though i don't think
this is a worse one around (Halliburton's and the likes' cost+ in
Iraq/Afganistan definitely bigger :).

>normally a single table with fields like

It is "normal" from another world :) The simple way you described would be
done by some startup, and they would horizontally partition it when time come
to scale. It is not the way the "Enterprise" things are done.

~~~
makmanalp
This is just fascinating. This must be why I've seen many a store clerk / govt
employee / postal office worker / bank worker / etc. sit there want wait for
10 minutes while the "system is processing"?

> "Software that assigned identities to enrollees and ensured that they saw
> only their own personal data ..."

I guess the simplest EIDM is a where clause in your SQL query :P Just kidding
- I do wonder what this thing does more than just a regular authentication /
authorization system though. Maybe sandboxes data so that it's impossible to
even query for data that doesn't belong to you?

~~~
genericresponse
In part because it handles a separate authentication questionnaire with
Experian. That's a PII sensitive integration that needs dynamic content.

------
rdhyee
The folks at MarkLogic must be unhappy with being singled out in the following
paragraph:

 _Some of the companies building the system opposed an early decision by the
Medicare agency to use database software from a company called MarkLogic,
which handles data differently from systems by companies like IBM and Oracle.
Some suggest that its unfamiliar nature slowed their work. By mid-November,
more than six weeks after the rollout, the MarkLogic database -- essentially
the website 's virtual filing cabinet and index -- continued to perform below
expectations, according to one person who works in the command center._

The company's response is recorded as:

 _In interviews, MarkLogic’s executives faulted inadequate computing power and
instability at the site’s data center, as well as the failure to properly
integrate their product, problems repeatedly cited by other website vendors._

Any insight on what is meant by how MarkLogic "handles data differently"?

~~~
arosenbaum
Not really a slam at all. Customers choose MarkLogic to replace Oracle and DB2
specifically because it "handles data differently".

"Schemaless" aka "Schema on read" is the biggest difference between Only "SQL"
and "Not Only SQL" systems. Scale-out on commodity hw vs. scale-up on shared
storage is another. Customers often choose MarkLogic instead of other NoSQL
systems because of it's ACID capabilities, ability to run multi-statement
transactions between multiple databases (including Oracle and MarkLogic in the
same transaction) and a multitude of enterprise capabilities from government
grade security to point-in-time restore.

Read more: [http://developer.marklogic.com/pubs/architecture/inside-
mark...](http://developer.marklogic.com/pubs/architecture/inside-marklogic-
server-r7.pdf)

~~~
realmandan
Totally, not a slam at all. As long as you don't think a company name and
"perform below expectations" sitting next to each other in a NYT long-form
piece is a BAD thing...

~~~
arosenbaum
I'll repeat what our CEO said in the NYTimes article "He said MarkLogic is
performing up to standard, but “the network and the storage systems are not
properly sized and not properly run.”

Put _any_ database on top of the resources that were provided and I think
performance "below expectations" would have been likely.

------
eliteraspberrie
As a Canadian I wonder, with all the American technology giants, why was this
project outsourced to Canada? If the idea was that outsourcing would lower
cost, why were the contracts no-bid?

~~~
bane
CGI Federal is a big-ish Fed contractor in the U.S. and, though a subsidiary
of a Canadian firm, for various legal reasons is owned and operated as an
American firm. It performs quite a bit of tech work for the U.S. government,
work won in generally open competition. Because the U.S. Fed space is such a
big market, it's not unusual to have locally owned subsidiaries in the mix
e.g. BAE, Quinetiq, etc. all have similar groups.

My understanding is that CGI was already handling various health care related
work and was selected to do the work without a proper competitive bidding
process. But having seen these processes, they definitely wouldn't have
resolved the kinds of widescale technical issues.

~~~
hga
Addressing you and growupkids, CGI Federal is responsible for the Medicare.gov
site, which I can attest to being a bit clunky but otherwise working well for
claims and Part D selection and application (the prescription drug benefit
program).

They were on a list of already vetted vendors, plus if "a proper competitive
bidding process" had been done it could easily be hung up in the courts by
losers lawsuits.

As you note, and as I've been noting, given the structure and processes of the
project, especially it being run by political and bureaucratic types from the
White House on down to CMS (the integrator prior to getting replaced in late
October), there's no way they and the other contractors could have possibly
succeeded, especially with change orders continuing though the week before
launch. Heck, the integration tests CMS ran in the last week or two before the
launch told them it couldn't possibly work, but it was launched anyway....

~~~
hga
Per this article [http://www.nytimes.com/2013/11/23/us/politics/tension-and-
wo...](http://www.nytimes.com/2013/11/23/us/politics/tension-and-woes-before-
health-website-crash.html?hpw&rref=) there were three other bidders: IBM, QSSI
and Computer Sciences Corporation.

All I can say is thank god CSC wasn't chosen! And to the extent CGI Federal
retained some of the qualities of the civilian part of AMS the company bought,
again, not an obvious bad move (I worked for a short time in a national
security part of AMS, which went to the American firm CACI because foreigners
couldn't own it).

------
sylvinus
Amazingly, the website appears to be down again!
[https://www.healthcare.gov/](https://www.healthcare.gov/)

Looks like they had planned that downtime, but I wonder how messed up this
whole project must be that they can't even manage to have a static maintenance
page up!

~~~
dkhenry
It up for me. I was just able to register an account. After looking at all the
steps to do that

1\. This is already too complicated. ( why a username when you know I am going
to need to use my SSN ) 2\. This is not a complicated web app. I am now fully
convinced that this could have been done with better results by anyone other
then the government.

~~~
markdown
> I am now fully convinced that this could have been done with better results
> by anyone other then the government.

But it wasn't built by the government. Do you think Obama asked for a username
field?

It was built by a private contractor ie. your usual capitalist, profit-
oriented enterprise IT company.

~~~
growupkids
You've never built a system for the government I take it? You don't build
these things in a vacuum, they approve and co-design the system, drive the
technology decisions, require you to use "approved" technologies, ban the use
of icky technologies because they aren't on the interoperability list (which
locks you into a few vendors, the usual suspects, Microsoft, oracle, etc.),
slap on all kinds of additional requirements, force you to use hardware from a
cut rate vendor so they can make their 8A quota, and so on.

It's nothing like a private company building a system. It's more like being a
peasant to a lord. Sure you can do what you want, as long it's exactly what
they want to you, when, where and how they want you to do it.

------
rathbun
Data In. Data Out. I have seen very little of exactly which databases are
being integrated. I saw a reference a few days ago that there were many
arguments about using SSN in the system. Does that mean that some of systems
being pinged for data don't use it? Very hard with SSN, I can't imagine trying
to wing a match with similar name, similar address, similar birthday.

------
United857
Anyone know what tech stack they're using? Linux or some other *nix or
Microsoft?

~~~
toomuchtodo
[http://medcitynews.com/wp-
content/uploads/healthcaredotgovin...](http://medcitynews.com/wp-
content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg)

Mirror:
[http://web.archive.org/web/20131201071252/http://medcitynews...](http://web.archive.org/web/20131201071252/http://medcitynews.com/wp-
content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg)

------
knowitall
I'm not from the US. I wonder, isn't this overly dramatic? So it takes a while
to fix the web site. People have presumably been waiting for health insurance
for decades, what is one more month? Also, really, people get worked up about
government processes that are difficult to use?

~~~
kevingadd
If your previous policy was cancelled due to ACA and you can't get a new one
because the enrollment processes for the exchanges are broken, you get to
enjoy a gap in coverage. That is never a good thing.

~~~
d23
That's nonsense. All of the previous ways to sign up for health insurance are
still around, and the policies that were cancelled basically didn't give any
coverage anyway (that is, they didn't cover medication or hospital visits and
allowed insurance companies to drop the owners of those policies at any time).

~~~
zaroth
What's nonsense is to claim "the policies that were cancelled basically didn't
give any coverage anyway." The ACA added several minimum requirements to
private insurance plans, such as maternity and prescription drug coverage,
which people could reasonably choose to forgo in exchange for the lower
premiums.

FYI, not having "prescription drug coverage" does not mean insurance does not
cover any medication. It is a _pharmacy_ benefit, in other words, over-the-
counter medication. In my personal experience, I pay significantly _less_
without the coverage than with it. I also don't have to worry about
formularies, prior authorization, or generic swap-outs.

To give you a specific example, in 2013 I paid $107/mo for a private
individual policy with a $2,700 deductible, $5,250 max out-of-pocket
(inclusive of the deductible) and unlimited lifetime benefits. Next year under
ACA I'll be paying $283/mo for a policy with a $4,500 deductible and $6,350
max out-of-pocket (inclusive of the deductible) and unlimited lifetime
benefits.

The most important clause in the ACA is limiting the rating factor due to age
to a maximum of 3x. That clause alone makes it the single largest tax increase
ever passed on young middle class families with children. By my estimates, it
will cost families like mine about $50,000 in increased insurance premiums
over the next 10 years, because the _least_ we can now pay is 1/3rd the rate
of unhealthy 64 year-olds.

~~~
quesera
> FYI, not having "prescription drug coverage" does not mean insurance does
> not cover any medication. It is a pharmacy benefit, in other words, over-
> the-counter medication.

Am I parsing that incorrectly?

"Prescription drug coverage" has always covered prescription drugs, in my
experience. I've never had a policy that would (to my knowledge) cover OTC
meds.

~~~
zaroth
Too late to edit my original post, but as you suspect, I'm trying to
differentiate between prescription drugs like birth control, anti-depressants,
statins, etc. which you buy at a pharmacy with a prescription, and medication
administered in an inpatient/outpatient setting, which is also _prescribed_ of
course, but covered by the medical benefit not the pharmacy benefit.

