
“The real problems are with the back end of the software” - wwilson
http://marginalrevolution.com/marginalrevolution/2013/10/from-the-comments-dan-hanson-on-aca.html
======
jroseattle
"The front end technology is not the problem here."

Let me fix that statement: "The front end technology is not the _worst_
problem here."

Looking at the resources loaded for the sign-in page, I counted 58 separate
Javascript files. Including one that implied by name was minified, which on
inspection clearly was not. I didn't bother counting CSS or image resources. I
returned to the page two days ago, which indicated it is down for scheduled
maintenance. It remains in this state.

CGI obviously borked this project. The government deserves its own special
classification of criticism, but poor planning, change management, etc. from
the government is no excuse for CGI not building an architecturally sound web
site.

The contract was $350 million? Good grief, they overpaid. Nonetheless, if we
could go back in time AND assuming we needed to spend this budget, here's what
I would have done:

1\. We make investments of $15 million in 20 different startups, and tell them
to implement the initial phase -- let's say we call it the "minimum viable
product" or MVP. Each startup has the same deadline for delivery.

2\. On the delivery date, all companies meet with us to review their MVP. We
call it a "demo day" and view all 20 demos.

3\. Through some set of criteria, we create a short list of five companies
from the 20 demos. Those five companies receive an additional $5 million
investment, and another delivery deadline.

4\. The companies iterate on their MVP and come back for another demo, this
time with a deep dive.

5\. We pick a winner from those five. The winner gets another $25 million
investment and is responsible for any additional work to be completed.

TechStars for government, essentially.

~~~
GVIrish
I don't see how giving 20 startups $15 million each would have led to success
instead of failure. Even if you found a better company to do the
implementation than CGI Federal, there were several enormous problems that no
company would have had any control over.

1\. Requirements were delayed so much that development didn't start until
March of this year. That spells doom for a system with this kind of
complexity, regardless of who the implementor is. You need several months of
functional, load, and integration testing so that effectively means you would
only have had 4 to 5 months to code healthcare.gov. And that's assuming there
weren't any big requirements changes.

2\. The people responsible for integration (Center for Medicare/Medicaid
Services) had no large IT project integration experience. These are people
that thought 1 week of full integration testing would be enough.

3\. Healthcare.gov had to integrate with legacy systems from the IRS,
Medicare/Medicaid, Social Security, in addition to the various state
exchanges. Any number of those systems could have serious flaws that would
make it extremely difficult to interface with. On any project a poorly
implemented legacy system can dramatically affect the effort needed to be
successful. Again, even the best companies would've had a sizable challenge
dealing with that.

4\. One of the biggest challenges in government IT is the customer. The
government decision makers often don't know enough about software engineering
to make sensible decisions on requirements, timelines, testing, you name it.
In this case there was the added political pressure of, "This cannot fail,"
even though it should've been clear at least a year ago that there's no way
they were going to make the deadline. But you get people who think that you
can just deploy and fix it as you go along. Or you get people that think you
can just add developers to make up for lost time.

5\. And what does being a "startup" have to do with it anyway? Either a
software company can do the work, or they can't. Whether they are a startup or
an established entity really has nothing to do with it.

~~~
integraton
It looks like you and jroseattle are demonstrating the main two competing
views about how to approach new software projects: 1) deterministic,
controlled, strong planning, avoidance of failure vs 2) nondeterministic,
flexible, embracing uncertainty, expect and handle failure gracefully. There
are many different versions of this in technology: one big expensive server vs
many commodity servers, waterfall vs agile, ACID vs BASE, BigCorp in-house R&D
vs distributing risk across startups.

Large parts of the technology world operate according to the latter model, and
they do so for a variety of very valid reasons. Obviously, the government and
government contractors do not.

jroseattle's comment presents a speculative model for how to apply a
distributed, fault-tolerant model to this kind of technology project.

~~~
GVIrish
I'm not saying that you can't or shouldn't distribute risk across many small
projects, I'm saying that in this case doing so would not have helped you at
all. If you had a competition to build a MVP of healthcare.gov where the
requirements didn't get delivered until 48 hours before it was due, then the
requirements changed substantially 30 minutes before the cutoff time, none of
the teams would have succeeded.

------
waterside81
Hyperbole aside ("... an act which would border on criminal negligence if it
was done in the private sector and someone was harmed ..." \- what does that
even mean? So all of us who have shipped buggy software for our customers are
borderline criminals?) - this doesn't surprise me in having dealt with the VA.
They have legacy upon legacy upon legacy, with all sorts of fun limitations
like not being able to have a "\t" in your content because that'll screw up
their backend which relies on tab-delimited data. Health care in the US is
playing catch up technology wise to almost every industry. And not for lack of
technology, but for lack of political will power.

My favourite example of this was trying to deploy an app within the VA that
was written in Django. I was told "Python is not on the list of acceptable
languages." So we came back to them and said, "Good news everyone, we ported
it to Java." Of course, it was just Jython, but that's the sort of stuff you
encounter.

Multiply this by the complexity involved in trying to herd all these cats into
one backend like healthcare.gov and it was doomed to fail.

~~~
Spearchucker
Such projects are doomed to fail because Big.corp, as well as USA.gov doesn't
get that they're not using the right _people_.

High profile project? Want it to work? Hire the right people. Want the right
people? Pay whatever it takes.

I've been involved in more such projects than I care to remember, and the
problem is _always_ the same. A project manager with rudimentary delivery
process knowledge owns a large technology project. What's needed is a
technically astute lead that knows how to abstract away from delicate backend
dependencies, knows that some projects _need_ big design up front, and knows
people that have the specializations he or she doesn't.

.Gov projects unfortunately are turf wars, where people scramble for a piece
of the cake because money smells good, and success is someone else's problem

~~~
dasil003
Even without being cynical about the motivation of the players: it's the
unknown unknowns that eat them alive, even if they have the best of
intentions.

It's also a tough problem to solve because very competent people don't want to
work on a boondoggle, and any large government IT project has boondoggle
written all over it before it even goes out for bid.

I wonder if Obama asked for any advice from some of his campaign people. If he
didn't before I'm sure he is now.

~~~
Spearchucker
I don't buy the unknown unknowns. Big system integration problems follow
patterns, and an experienced tech will recognise them. The problem with any
half-assed strategy is that people gloss over the knowns, which is fatal.
First thing anyone should do on a large (actually any) project is to assess
the current situation[1]. When you discover a big nasty piece of legacy you
need to integrate with, you look at how it works. The rabbit hole must be
followed to it's end, or something will bite you. Blaming unknown unknowns in
IT is the same as telling the teacher that the dog ate your homework.

[https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef...](https://www.wittenburg.co.uk/Entry.aspx?id=46870dcd-70cb-4ef8-98ed-91c94714fe03)

~~~
dasil003
It was a poor choice of words on my part. I meant to suggest that they were
knowable unknowns, but just that the people involved didn't have a clue.

------
ams6110
Slow, legacy backend systems are not an intractable problem. You can do things
such as copy the data to a faster cache, or you use some kind of queuing
system so that queries are processed only as fast as the backend can handle
(of course the frontend needs to be able to "check back later" for the
results).

This does support the widely held disbelief that this system will be fixed
anytime soon. Clearly the management of the project and the design of the
architecture are/were fundamentally flawed, and its very unlikely that it can
be fixed in 30 days or whatever at this point.

~~~
vinhboy
Regardless of how hard it was to make a functioning "data hub", they should
have, at the very least, built a functioning registration system that is
independent of all other fail points. This way, at the very least, they could
have the names and contact information of everyone who wanted to fill out a
form. Then they can go back and process the forms at a later date.

~~~
hga
Supposedly the problem there was a very late change from somewhere in the
White House to CMS, in August or September, to preclude the possibility of
window shopping and require registration with a lot of verification up front.

E.g. we know Experian is part of that process, said to be for identification
and income verification. It's been reported that people who don't have credit
records (e.g. those sufficiently ill they've stayed under the care of their
families) get told by Healthcare.gov to contact Experian, who then upon
discovering they don't have a record of them tells them they have to get the
Healthcare.gov people to create a “Manual Process Identity Proof” ... which
few if any of them know how to do, or whom to send someone to.

~~~
lnanek2
That's what I hated most about healthcare.gov. Other sites I could get a quote
quite easily and decide to go with them or not. healthcare.gov had a bunch of
forms that didn't work so I couldn't get a quote after hours of trying. Even
if I let long loading pages sit there for long periods of time, some failed
out and other times my data was lost.

------
critium
I've worked in and out of the public sector for the last 10 years and
unfortunately, this actually _PAR FOR THE COURSE_.

This is not the contractors fault. Its the government. Before I left to work
with a startup, I was abhorred by the lack of ownership on the client's side.
Everybody is looking to shuffle responsibility, keep the lowest profile, and
do the least amount of work.

It doesnt matter who's writing the code, unless they find somebody competent
and passionate on the government side, large projects are destined to fail and
better left off to be written by the public sector. This is government waste
at its best.

I'm neither republican or democrat but just to add, if my rinky dink app I was
working on for the Dept. Of Commerce gets shown to the president when its in
'ALPHA' state, there is no way the most informed person in the world didnt
know that the site was going to fail from the get-go.

~~~
Iftheshoefits
It's the contractors' fault, too. Both parties are playing the same game,
here: corporate welfare for the contractors, who usually are owned or operated
by people with connections to the government, and long-term job security for
the government project management officials who oversee these things.

I don't know if you've ever worked for a contractor, but I have, and I
guarantee you the same responsibility shuffling, profile munging, least-
amount-of-work attitude exists there. Without it, these contractors wouldn't
be able to keep feeding at the trough with the rest of their corporate welfare
recipient friends (while they bitch to each other about how evil liberals are,
how disgusting entitlements are, etc.).

~~~
critium
The argument i'm making is that no matter who comes in, so long as they dont
have a dedicated and passionate lead, it will be all for nothing.

I dont argue that the contractors are/can be part of the problem but from my
experience, the government _IS_ the limiting factor, especially when they say
that they're the systems integrator, not the contractor as is in this case.

I did my damnest to write/architect/design the best system that I could but if
the government forces you to use RH JBoss 2 versions behind the latest, just
because they have a support license then refuses to buy JRebel because it was
too expensive after paying tens of thousands of dollars on said JBoss, and
then changes the system requirements every 6 months you're doomed. Worst of
all we were _NOT_ allowed to talk to end users. We can only talk through our
Task Order Managers and supposed Technical Leads (that have little or
extremely dated technical experience).

------
greenyoda
" _There are no easy fixes for the fact that a 30 year old mainframe can not
handle thousands of simultaneous queries. And upgrading all the back-end
systems is a bigger job than the web site itself. Some of those systems are
still there because attempts to upgrade them failed in the past. Too much
legacy software, too many other co-reliant systems, etc._ "

30 year old (1983) mainframes and databases were designed to handle large
transaction loads. For example, airline reservation systems and banking
systems were built on them.

And upgrading a mainframe (at least an IBM mainframe) to a faster mainframe
isn't such a daunting task, since all the code from 30 years ago (or even from
the 1960s) is still object-code compatible with the new machines - you can
make it run even if you've lost your source code. There's still lots of 30
year old (and older) Cobol code running on mainframes today.

I agree that re-writing the 30 year old software would be hard, but simply
getting it to run faster could probably be done just by spending money on the
latest mainframes and disk drives. But if nobody ever did a load test on the
site, they wouldn't have known that they had to do this. They probably just
thought: "Oh, we have to write a web site that talks to a bunch of databases,
how hard could that be?" (By the way, they could have written test code to do
a load test on those legacy systems without even having a web site running. In
retrospect, that's the _first_ thing they should have done, and it would have
shown them that their critical path wasn't the user interface.)

~~~
GVIrish
In theory you could at least improve the hardware the legacy systems is
running on. In practice that may not have been anywhere near enough to ensure
success.

1\. In this case you would've had to start benchmarking the performance of the
legacy systems early, far earlier than when they started development in March.
If you determine that hardware upgrades are needed, then you'd need to
initiate procurement and upgrade projects at one or more of these other
agencies. Projects like that may not necessarily be quick to implement.

Maybe the physical space can't accommodate new hardware. Maybe there isn't
enough budget to do an upgrade like that. Maybe there aren't enough personnel
resources to plan and implement an upgrade of that scale quickly. Maybe those
organizations are just barely keeping their heads above water with the way
those systems are currently functioning. Maybe they don't even have a handle
on what their hardware configuration is. I know someone who worked at an
agency where they had to start unplugging stuff to figure out what server did
what.

2\. This is all making the assumption that the data in those systems is
correct and well-formed, and the business logic in those systems is free of
bugs. Maybe you get the database schema and find out that A. It's out of date,
B. There's no data dictionary, and C. There's 250k lines of business logic
tied up in undocumented triggers. Good luck.

Load testing might just be the tip of the iceberg in situations like this. But
bottom line is, if the people leading your project don't even think to start
looking into this kind of stuff very early on, you might be screwed before you
even started.

------
digikata
"Failure isn’t rare for government IT projects – it’s the norm. Over 90% of
them fail to deliver on time" Is this really much different than the success
rate of startup culture where VC's count themselves successful if 10% of their
investments yield a return? The startup environment has the "success rate
advantage" that if the venture really isn't getting traction, you can walk
away from it, or change directions a do something related, but not your
original objective.

Government projects like the healthcare exchange don't have that degree of
freedom - if they go down the wrong track, the only choice is put in more
resources until it's back on track. Giving up or changing objectives isn't a
decision under the control of the project - it's a legislative or budgetary
question.

------
JulianMorrison
The answer is that you can't structure the transaction as a realtime query.
You have to structure it as something that's sent and gives you a ticket, and
the reply associated with that ticket will come back in its own time.

Stick the processing pipeline in Twitter Storm (which can retry any step until
the whole pipeline is done) and structure the requests as nearly-idempotent
(so a repeated reply is harmless, and the first arrival associated with the
ticket wins). Finally, you have an "inbox" where people can wait for and see
their answer, with optional SMS and email notification.

~~~
hga
Interesting, but at some point people have to be shown a selection of
offerings. That selection must allow seeing if your doctors are in the
network.

Your suggestion would allow for that, but not I sense in the "instant
gratification" way we're used to dealing with web sites. I.e. "thanks for your
input, wait for SMS/email/N hours till you log in again for the next step".

BTW, I've read it's a 30 step process. Not all will require "take a ticket and
wait to be called", but more than 1 or 2 I suspect.

~~~
JulianMorrison
Well, they aren't getting instant gratification now, are they?

I suspect a lot could be cached - that is you take a list of questions up
front, what comes back from the "ticket reply" contains any further questions
you'd need to answer and enough canned data to process the replies
immediately. So you wouldn't need to take a ticket more than once.

But even if you did need to, at least it would be something people could fill
in and walk away from. They wouldn't be kept around, drumming their fingers
and worrying about being late to work.

~~~
hga
" _Well, they aren 't getting instant gratification now, are they?_"

Indeed!

But I was referring to the initial architecture of the site. E.g. the March
objective, “ _Let’s just make sure it’s not a third-world experience._ ”
expressed by Henry Chao, CMS CIO, who in theory should have been in the front
lines. Sigh, I bet he was, knew what was coming but didn't have the authority
to make it work, or as you implicitly suggest, down-grade the
architecture/expectations into something that might possibly work in the time
left.

And then there's the last minute (well, month) change to requiring a heavy
duty, validated up the wazoo registration before doing anything. Doubt they
could have made that work in less than 2 months ... maybe. This was needed,
but at lower transaction rates, e.g. otherwise window shopping would allow a
lot of people to see they couldn't get a better deal through the exchange
(e.g. no subsidies for you, which only the federal system is allowed to
calculate).

Suppose prior to that they arranged for "X rate of idempotent reads" with e.g.
the IRS, and now they need a 10/100/1000X rate, meaning they should have
cached everything that doesn't change quickly. Almost certainly too late to do
it by then.

My, my, years from now this is going to be one of the most significant case
studies in project development.

------
bhauer
I have not followed the development of this news closely, but skimming these
updates has been amusing. I do have a couple very basic questions. If these
are stupid, I apologize ahead of time.

My understanding from previous coverage is that some of the state exchange
sites, such as California's, are performing acceptably. If that is true, do
those state sites also connect to and query the same legacy systems as the
federal site? If so, why doesn't the federal government simply ask for or take
that code? Surely it's been made available to them? If not, are the legal
requirements for the states' exchanges somehow different than the federal
site? That seems unlikely since my understanding is the federal site is simply
standing in for states that elected to not create exchange sites. I don't see
why it would be subject to extra requirements.

What am I missing here?

~~~
hga
First thing is only 4 state sites are reliably reported to be working well,
and California's isn't one of them, although their's might be fixed by now. I
haven't heard of any that didn't really do integration testing, and Hawaii and
Oregon declined to launch their not ready sites Oct 1st.

The obvious answer is that until late last week, the grossly incompetent CMS
(and some people in the White House and perhaps others in HHS) were running
the show. Now that it's under new management, maybe that possibility will be
investigated. On the other hand, with a 7 week deadline for those who _must_
get new insurance for 2014, like the 16 million with now outlawed, not
grandfathered individual policies ... there's not much time to do something
that disruptive. But an idea worth checking out in the next week or so.

------
taternuts
> Amazingly, none of this was tested until a week or two before the rollout,
> and the tests failed.

This is absolutely incredible.... two weeks?! Dealing with these legacy
systems should have been the absolute first thing tested, is it not the most
likely point of failure/bottleneck? Someone on the team had to have been
screaming about this and ignored, all the while shitting their pants waiting
for go live for the whole thing to crumble.

------
patja
I'm wondering why states were allowed to build their own systems and opt out
of the federal site. From the Washington state site we get passwords emailed
in clear text, a failure to even allow people to enter all components of their
income (resulting in inflated tax credit decisions), using monthly income
figures where annual ones should be used (again, more incorrectly inflated tax
credits). In Oregon they say they can't even log in or get through the
application. Each of these state-specific sites cost tens of millions, each
resulting in their own unique set of defects on launch, to implement a federal
program.

The press seems very focused on the obvious availability and performance
problems as well as the errors that come up within the sites that prevent
someone from completing their application. There are a whole slew of second-
order defects that make it appear your application was successful and correct
but were based on incorrect calculations, incomplete data, or other bugs that
are not obvious to the user at the time they complete the process.

~~~
jaibot
The original intention was for all the states to build their own exchanges.
The more liberal democrats wanted a single federal exchange, but compromised
to state-level exchanges to get enough votes for passage. Then the
conservative states declined to make their own exchanges so the federal
government had to do it anyway, because the American political system makes no
sense.

~~~
hga
Minor correction that proves the rule: "Then all conservative states except
Idaho declined to make their own exchanges...."

I'd correct your latter statement to "most recently, the American political
system makes no sense". Prior to Obamacare, _every_ major entitlement program,
from FDR's Social Security to G. W. Bush's Medicare Part D prescription
program was passed with large bipartisan majorities, strongly suggesting they
had wide majority support.

Obamacare famously did not, heck, it only passed within the Democrats by a
whisker, and a lot of them found themselves spending more time with their
families after their next election.

So any sane plan by Team Obama to implement it should have factored in very
strong resistance in a lot of places. Heck, I along with 71% of the people in
my _Purple_ , very much not Red, home state of Missouri voted in 2010 to
outlaw mandates
([http://ballotpedia.org/wiki/index.php/Missouri_Health_Care_F...](http://ballotpedia.org/wiki/index.php/Missouri_Health_Care_Freedom,_Proposition_C_\(August_2010\))
) and then in 2012 62% voted to outlaw the creation of a state exchange unless
authorized by the people or the legislature
([http://ballotpedia.org/wiki/index.php/Missouri_Health_Care_E...](http://ballotpedia.org/wiki/index.php/Missouri_Health_Care_Exchange_Question,_Proposition_E_\(2012\))
).

~~~
tunesmith
Curious why you voted against the mandate. System-wise, a vote against a
mandate is basically a vote for allowing rejection of pre-existing conditions.

~~~
hga
Note, you're having labeled me as dishonest, that ... frees me a bit from my
usual constraints (not that that's costing you upvotes from me for posts like
this one:
[https://news.ycombinator.com/item?id=6623010](https://news.ycombinator.com/item?id=6623010)):

How about, "Because I have a fucking clue and could see that the ACA would
make things at net a lot worse, and very possibly result in my premature
death?"

Did you even note the vote was almost certainly symbolic? Not that making the
most firmest statement (almost 2/3rds of Missouri voters) was pointless.

Anyway, you're focusing way too much on the micro-details of political
promises. I prefer macro reality.

ADDED: ah hah, I now see the ACA is in theory going to save you 20% over your
previous costs. No wonder you're defending it. I just hope for your sake you
don't e.g. experience a shift from access to healthcare to access to a waiting
line, or as Obama himself said, Liverpool Care Pathway style, you don't get a
pain pill instead of treatment that will save your life.

~~~
tunesmith
You're asserting that my defense of the ACA is _because_ my premium went down.
That's a pretty uncharitable interpretation of my views. In that same comment
I wrote that the chief goal of the ACA was not to lower our premiums. To make
it explicit, I would also support the ACA if my premiums had gone up.

It sounds like the rest of your views are based off of your beliefs of how
things will play out, which is of course difficult to argue about.

~~~
hga
In the context of your labeling me as dishonest, I think it's a fair counter
jab. More seriously, not that you can be bought for so few pieces of silver,
but let's suppose your costs tripled as have so many others?

These particular long term views are relevant in the context of my likely (at
the time) symbolic Proposition C vote; it's indeed difficult to impossible to
argue about them, but I hope I've at least demonstrated a basis for my
"Obamacare belongs in the ash heap of history" mid-2010 opinion.

------
fauigerzigerk
Focusing on the performance or scalability of these ancient backend systems is
beside the point. It's simply not a great idea to connect a significant number
of backend systems run by different organizations in one synchronous online
transaction. The overall probability of failure may simply be too high,
irrespective of any scalability issues.

------
snowwrestler
I think the easiest fix at this point is to simply design around the known
delay in synchronizing all the 3rd party data calls.

Have people enter their info, then show them a screen that says "your quote
will be emailed to you in 24 hours." Then the integration system has 24 hours
to retry any failed data pulls, match up all the data, and generate a quote.

~~~
hga
Problems:

An insistence on a heavy (throughtly validated) registration process.

People want to have choices. Maybe they don't want to get insurance from
company A. There are also monthly payment vs. deductible tradeoffs in the
Bronze vs. Silver etc. plans. And some demand cost sharing after you've used
up your deductible (I wouldn't really care if the plan had a 10 million dollar
limit if I was expected to pay 20-30% of that...).

ADDED: plus you must be able to see if your doctors are in a plan's network.

Despite the "one size fits all" new minimum gold plated plan, there's still a
lot of tradeoffs ... and then email isn't necessarily reliable enough for the
response. I can't see them avoiding a "or check back on the site tomorrow..."
option.

------
snorkel
I don't know how much of this is true, but I bet the truth is no less
hilarious. It wouldn't surprise me if this system has no concept of usability
and offline processing queues. No matter how complex it is to process an
application it's common sense to just give the user immediate feedback "Thank
you for your order. We'll contact you by email within N days to followup and
report your application status." Do these people expect Amazon to process
orders in realtime and fling physical goods at their door in minutes? Should
buying health coverage be zero conf one click instantaneous?

~~~
hga
Well, I've heard there was a "7 second" response time metric that was part of
the plan.

But, yes, Amazon's "eventually consistent" system takes its time, e.g. while
it's never happened to me, I know that somewhat delayed confirmation email is
only made when the whole system has reserved a book for me, even if there's
only one copy in stock and someone else made an order for it at about the same
time. Etc.

------
ape4
The frontend assumed the backend was fast enough. That's the problem. If the
frontend was made to handle really slow responses from the backend it would
look different. It would not make people wait while transactions occurred. Or
if might have a page that that displayed your progress: in other to do this
for you we need to contact 10 databases - here is the progress of each:

    
    
        Database One:   [=======----------]
        Database Two:   [============-----]
        Database Three: [==---------------]

~~~
hga
Heh.

But I would half joke that that would result in a bunch of angry people with
torches and pitchforks showing up at sites of the owners of Database Two....

Which might be totally unfair if it's not really their fault.

~~~
ape4
My ASCII graphics were trying to show Database Two as fast and Database Three
as slow. That's where the pitchfork toting mobs should be.

~~~
hga
Oh, yeah, progress toward completion, which would obvious to most everyone
using the site.

But I'm sure _some_ would get it wrong even then!

Visions of a family gathered around the monitor:

"Go IRS!"

"Social Security, we KNOW you can do it!"

Etc.

(I do _really_ like your idea, it just seems to be inevitably politically
impossible.)

------
seivan
I don't believe in this anymore

"Everyone outsources large portions of their IT, and they should. It’s called
specialization and division of labor. If FedEx’s core competence is not in IT,
they should outsource their IT to people who know what they are doing."

These days I believe each department of government that needs an iPhone
application would do better to hire an iOS developer full time to maintain and
polish the fuck out of it, continually.

~~~
Dwolb
It's interesting that a core competence of government isn't IT. How much does
the government depend on IT to run its day-to-day business?

If the answer is 'not much,' I can understand the opinion that work should be
outsourced.

If the answer is 'a lot,' then IT should be a core competence and the
government should invest in acquiring in-house IT expertise.

I'd guess the answer is 'a lot'.

~~~
GVIrish
Currently government IT is operating under this idiotic idea that almost all
IT should contracted out because :

"We can fire contractors if they don't do a good job"

"Contractors stay up to date on technology"

"Contractors are cheaper because we don't have to pay for their government
benefits"

So for one, contracting companies that do a bad job may lose out on future
work. But the reputation of the big companies is never hurt enough by this to
get forced to improve. Even then, the fundamental problem is that government
employees are very difficult to fire or even hold accountable so people think
it's easier to hire contractors. When really the problem that needs to be
solved is accountability in government.

Secondly, even if developers who work for contractors stay more up to date,
it's really irrelevant because the government is extremely slow to upgrade on
anything. We're still running Windows XP in a lot of places.

Third, if the bill rate for a developer is about 3-6 times what you'd pay a
full time government developer, you're not really saving money on benefits.
The thinking is that you only pay a contractor for a few years while they
develop something for you, but IT is a continuing need. If your operation
relies on software (which almost all of them do), you're going to be paying
3-5 times the cost per developer for the foreseeable future. Over time that
certainly costs more than government employees.

------
malandrew
Strangely, based on the title I thought this was going to be about future
startup trends. i.e. For the last 6 years or so we've seen a revolution in
interface design as a competitive advantage when creating a new startup, but
as the low hanging fruit opportunities are used up, as lot of the really meaty
opportunities are going to be in software where there is a significant backend
component performing a lot of heavy lifting and magic.

I'm not in the least bit surprised to see that a lot of the work and resulting
problems with healthcare.gov are on the backend.

I just wish the government realized that we have all these amazing developers
over in the Bay Area that can do a better job than the majority of those
developers currently writing software for government contracts. I'm shocked no
one in government has said to themselves "What do we have to do to make our
software problems accessible to the types of engineers working at the Googles
and Dropboxes of the world.

~~~
GVIrish
Having the best developers in the world is useless if you have incompetent
management.

In this case, requirements were delivered extremely late, and they were still
in flux up to a week before deployment. The system integrator went forward
with only ONE WEEK of integration testing. There were 55 companies involved in
this, not to mention all the state exchanges and other federal software
systems that had to be integrated.

Better developers would have made a positive impact, but nowhere near enough
to make this project a success.

~~~
malandrew
If the managers are not developers then they are by definition incompetent to
manage a software project like this. I have met a lot of competent managers
that were not technical, but I have not met one that was competent enough to
manage a project involving legacy systems with poorly written code and little
to no testing or automation. If you are not a developer, you simply cannot
understand these types of problems enough to manage it.

~~~
hga
We can infer from their mistakes, which GVIrish succinctly lists, that the
managers in CMS on up to the White House weren't developers, otherwise they
would have instinctively known that e.g. requirements had to be frozen for
some time to give the developers time to finish implementing them.

------
natural219
"Unless it is enjoyable or educational in and of itself, interaction is an
essentially negative aspect of information software. There is a net positive
benefit if it significantly expands the range of questions the user can ask,
or improves the ease of locating answers, but there may be other roads to that
benefit. As suggested by the above redesigns of the train timetable,
bookstore, and movie listings, many questions can be answered simply through
clever, information-rich graphic design. Interaction should be used
judiciously and sparingly, only when the environment and history provide
insufficient context to construct an acceptable graphic."

"Interaction considered harmful", by Bret Victor
[http://worrydream.com/MagicInk/](http://worrydream.com/MagicInk/)

------
USNetizen
The issue I see here is the author of this article has marginal experience
with federal contracting. "The people who wrote the code for these systems are
long gone...they are prone to transaction timeouts" ... wrong, wrong and
wrong. There are plenty of coders still around maintaining these systems, even
with the obscure technologies like MUMPS and all and they are running on
rather robust hardware in huge datacenters.

Second of all, the government should NEVER outsource integration - the systems
integrator requires an authority to manage other contractors that only the
government is capable of holding.

~~~
hga
Are you really sure about your first point? I have a state government
counterexample here:
[https://news.ycombinator.com/item?id=6622223](https://news.ycombinator.com/item?id=6622223)

As for the second, whatever might be right, we've been told that only the
Pentagon has retained that ability for "medium sized" weapons projects. Anyone
know anything to the contrary?

------
erichocean
As an aside, when I hear about the problems they're having understanding the
legacy data formats, it makes me wonder how far you could get with a high-
powered, big data NLP system to "parse" the data. Sort of like how Google
translate works.

There _are_ rules, after all, they're just not written down. Why not let the
computer figure them out, with continuous training from people until the
computer's accuracy is high enough?

I suspect instead they tried to write parsers and trusted "the spec", which
was never even right the day it was written down. :)

------
colomon
Can anyone verify this info? I was a bit surprised to see it wasn't better
sourced than the comments of a previous Marginal Revolution post. (Nothing
against MR, but this seems like huge news if true.)

~~~
hga
Well, it's been established all these external data sources are being used.

So its either on-line, maybe with caching (maybe someday), or using a
periodically run off off-line copy inside of Healthcare.gov, right?

I'll bet the House hearings made clear which is happening.

------
whistlerbrk
"Or IBM, which has become little more than an IT service provider to other
companies?" Come now, that's absurd. Incredible things happen at IBM research.

------
chernevik
Yes, and the problems are probably still worse than this.

Because integration means integrating _requirements_, leading to determination
and priority of requirements. The current organizational structure doesn't
seem to have anyone responsible for even coordinating that. But even if there
were, they would need terrific knowledge of each agency's internal systems and
legal requirements to determine what is and isn't necessary. And enormous
authority, meaning both credibility and power to dictate, to get their
determinations to stick.

Absent someone looking over the process, each agency will just "require"
everything they might need or want. Leaving something out is risky, unless you
know a lot about what you are doing and what will happen next and trust your
management. Even if they had all those latter characteristics, bureaucracies
don't do risk.

We all know how complexity grows exponentially. I bet the requirements
document for this thing doesn't exist, and if it did it would be a clusterfuck
of epic proportions.

Here is my wild theory: The possibility this could succeed died the day Tom
Daschle withdrew his nomination for Secretary of HHS. Not that Daschle himself
is special, though he is pretty bright. But he was slated for an unusual joint
role, running HHS and a White House appointment running the health care
effort. A position like that might have had access to the specialized
knowledge to know what needed doing and the Presidential delegation of power
to get it done. If IRS says "we must have X" and Daschle KNOWS they don't
because a real expert knows they don't, he can get them in line or they can
explain the problem to the President's chief of staff.

Here is the wild part. Daschle was canned, inexplicably, over a truly stupid
tax issue (didn't declare a car service as income), while others had far more
serious issues waived (Geithner lied about CASH income despite instruction to
declare it). Why? I speculate, precisely because the role he designed for
himself was remarkably powerful, and effectively outside any review because of
the complexity and specialization of its task. Wouldn't the President want
someone with the power and knowledge to implement his most important policy?
Yes, but not someone beyond his control. Politicians are about power. JFK
didn't use the legislative skill of Johnson because he feared Johnson would
serve Johnson's interest, not Kennedy's. Once Obama and his people realized
that Daschle could become effective President, and Obama something of a
titular head of state, they shivved him.

It's all speculation. But it is all plausible enough to suggest why government
doesn't work. Massively complicated projects like Google work because its
people are, by and large, working for a common purpose on tasks that are
commonly understood under common accountability. Government and bureaucracy
are fundamentally divided in purpose and understanding. The components can be
united by power and knowledge, but by its very nature the system resists
establishment of such power and knowledge.

~~~
hga
" _I bet the requirements document for this thing doesn 't exist...._"

The NYT reported that in the last 10 months CMS on up made 7 major
requirements changes. We've heard that the one requiring up front thoroughly
verified registration was made rather late, August or September. And we've
heard from multiple sources changes were made *through the week before
launch".

Someone in CMS probably drafted one ... well, maybe, they aren't experienced
in their former role as integrator. I'll bet it has less of a relation to
reality than any requirements document I read in my quarter century career as
a programmer....

Very interesting theory about Daschle. At the time, I thought it was "one
scandal too many for an appointee" ... but as you note, the administration did
allow others through, with Geithner as the head of the department that
collects taxes being intergalacticly beyond the pale. So, yeah, why drop
Daschle? And I can't think of anyone on the Cabinet with as much juice as he,
excepting perhaps Holder, an old Clinton Justice Department hand, and the
special case of holding over Gates for the DoD, which is not unheard of when
you're in the middle of a war (while it was intraparty, Truman and LBJ did
it).

------
vpeters25
"... for some inexplicable reason the administration decided to make the
Center for Medicare and Medicaid services the integration lead for a massive
IT project despite the fact that CMS has no experience managing large IT
projects"

Time Magazine's "Bitter Pill" article stated Medicare had an IT system that
made them more efficient than private health insurance providers. Isn't such a
system large enough?

~~~
hga
Doesn't mean CMS was the integrator for any of the systems it uses, or any
really big ones.

E.g. I don't rag on CGI Federal as much as some others, because they're
responsible for CMS.gov and Medicare.gov, and the latter is more than "good
enough for government work" ^_^.

------
devx
I just hope that when this whole mess of a project goes online, and is hacked
to death, NSA & friends won't be using this an opportunity to tell us that
"see, this is why you need to give us bigger funds and spy on everyon - to
protect you against those hackers (offensively)!" \- even though the whole
issue would be the bad programming and security of the system.

------
robomartin
...and the website is not the worst part of the ACA ride we are now on.

Part of me has been ignoring a lot of the chatter around the ACA as potential
right wing fabricated drama. Too much noise and bilateral bullshit being
thrown about these days.

That was until a few days ago, when I would learn our insurance has both more
than doubled in cost and is also scheduled for cancellation. Doubled and
cancelled. All as a direct result of the ACA. Brilliant! To say this was
shocking is an understatement. Our annual cost will go well past $15K.

There's a tragedy of unintended consequences, side effects and direct effects,
being played out in the background that hasn't completely come to the surface
yet. We certainly can't be the last family to get news of this kind. That
means in the coming months it is likely hundreds of thousands, if not
millions, of additional individuals and families are going to receive these
dreaded letters. Apparently hundreds of thousands already have. Last week was
our turn.

At one point this and other issues will be difficult to ignore. And they will
dwarf the IT issues. The website, as much of a disaster as it is, is likely to
pale in comparison to all of the other, non IT, issues.

Some of what's happening is related to the incredible disconnect between
Washington and technology. All you need to do is listen to some of these folks
talk about the website issue to see how little they understand. I heard one
senator say something akin to "they just have to re-enter a list of five
million codes". In other words, the term "code" to some of these guys means
"numbers" and that someone made a data entry error in copying "codes" into the
website.

BSS (Balaji Srinivasan) covered some of this in his excellent Startup School
talk:

[http://www.youtube.com/watch?v=cOubCHLXT6A](http://www.youtube.com/watch?v=cOubCHLXT6A)

A talk which, he comments, has been mutated into something far different from
what he said by the modern equivalent of the "broken telephone" game.

[https://news.ycombinator.com/item?id=6619068](https://news.ycombinator.com/item?id=6619068)

I agree very much with his suggestion that an "exit" is required. Not meaning
that we ought to pull-up roots and go, but rather that the tech community
ought to almost ignore the dinosaurs and go ahead and evolve a society more
aligned to modern realities. In his talk he gives examples of various US
cities that have been "exited" to some extent through technologies developed
in the free market.

To some extent, it's an Innovator's Dilemma kind of a problem.

[http://www.amazon.com/The-Innovators-Dilemma-
Revolutionary-B...](http://www.amazon.com/The-Innovators-Dilemma-
Revolutionary-Business/dp/0062060244)

The only way to make step changes is to do it well outside of the organization
looking after the status quo, because that's all they know and that's all they
can focus on.

~~~
tunesmith
This is a huge story that is really difficult to get a handle on, because the
premiums changes are affecting everyone very differently, and the patterns
behind them are very tough to tease out. I myself saw my premium go down by
over 20%, with better benefits. I've also seen a lot of accusations of lying
on either side, which isn't helpful either.

Some factors to keep in mind:

Health insurance plans have to be ACA-compliant. Plans that aren't have to be
canceled and replaced.

Just because your new plan is ACA-compliant doesn't mean it's on the exchange.
Check the exchange, too, as the price differences could be extreme.

Some companies (Humana in particular) have been sending out inaccurate
cancellation letters, with inaccurate "new premium" amounts. They got in
trouble a few weeks ago in Kentucky for implying that people _had_ to switch
to the replacement plan even before the Kentucky exchange was up. Humana also
recently got in trouble in Colorado and had to send out a second letter
retracting their first one.

It's possible that some companies may be sending out inflated premium amounts
to lower risk, while knowing they will have to send out refunds in a year.

Your old plan may not have been ACA-compliant. Specifically, it may have had
annual or lifetime maximums (where the insurance company says, "Sorry, we have
spent enough money on you; you are on your own now.") These are now
prohibited. Even though this raises premiums, this is a _good_ thing.

Some states have crappy competition - Wyoming for example. There's no one else
there to encourage an expensive hospital to lower prices, which means
expensive premiums. That's a failure of the free market, not a failure of the
ACA. (I say that because if the solution to an ACA's woes would mean MORE ACA
instead of less - say a public plan - then it doesn't really fit the narrative
of the ACA causing these problems.)

And most importantly, the chief goal of the ACA was not to lower your premium!
It was to lower health care costs over time (not compared to now, but compared
to what they would otherwise be). It was to raise society's health over time
(compared to now). And it was to protect people from going bankrupt from
health care costs. (This is already successful thanks to the elimination of
pre-existing condition rejections, and the elimination of annual/lifetime
maximums.)

~~~
mynameishere
_It was to lower health care costs over time_

I'm not sure how this is possible. The other goals you mention are feasible,
but only to the extent that they prevent cost saving.

 _the chief goal of the ACA was not to lower your premium_

Well, it was intended to lower some, and raise others. People with preexisting
conditions cannot be insured. They can only be subsidized with something
falsely called "insurance". The losers in ACA are pretty much the same as with
single payer: The young, healthy, and productive. Call it fair or unfair, the
situation is much different than with single payer in that ACA is going to let
people see just how much _they themselves_ are getting boned.

~~~
tunesmith
I think the CBO has already projected a lowering of the future "cost curve". I
guess we'll have to see how it actually plays out.

------
narrator
Only supports 200 simultaneous transactions ay? Well just put a big ol' queue
on the front of it with a hard thread limit and tell people they'll get an
email when it's ready.

~~~
hga
But it crashed at 200 in the testing the week before launch....

For all we know it crashed at 2 "truly simultaneous" attempts. I've watched
more than one project run beautifully until they attempted a second
simultaneous logged in user, which is not quite the same thing, but....

------
pessimizer
The real problem is with the weird indirection of the US government providing
payment to doctors/hospitals/pharma through subsidies, tax breaks, and tax
penalties granted to or extracted from citizens that can only be spent at
private insurance companies or mitigated by spending at private insurance
companies who then, in turn, pay for your healthcare.

The sheer complexity of this rent-seeking indirection makes keeping track of
the millions of distinct participant-instances that can play out in hundreds
of different ways, involving integrating tens of massive legacy systems with
new, flexible business logic (for a law in flux), impractical.

With single-payer, they could have scrapped the vast majority of this
complexity.

------
dreamdu5t
The problem is government contracting. CGI will continue to get contracts.

The problem is a system where if you don't deliver you get paid millions of
dollars and still get jobs.

------
microcolonel
Since when have they even been good enough at this to critique?

They're going to continue to suck royally, as royalty does.

