
How Healthcare.gov Changed the Software Testing Conversation - Baustin
http://blog.smartbear.com/testing/how-healthcare-gov-changed-the-conversation-about-software-testing/
======
powertower
I tried to get coverage on the 22nd. The process is very long and tedious.

I spent a solid 5 hours going through the entire process about 6 or 7 times,
from start to finish, only to get told each time afterwards that it's not
complete for some odd reason, and to start over again on step #1!

I tried multiple things, deleting the application, using Chrome instead of IE,
etc.

On the 23rd you could not even log in - they took that option away.

What bothered me the most was, besides your social security number and a slew
of personal information, they even demand that if you are a naturalized or
derived citizen, you locate your naturalization or citizenship certificate and
enter numbers (Alien # and Cert #) from it that -- get this - their Javascript
refuses to validate (it kind of looked like their rule match is off by a
digit). And you get stuck on that step.

I also noticed that if you enter your income as below $10,000, the system
tells you that you qualify for no benefits. But the moment you enter $11,500,
you get a $260 tax credit. Go figure that one out.

~~~
bmelton
> I also noticed that if you enter your income as below $10,000, the system
> tells you that you qualify for no benefits!

I know that the common plight there is that some are poor enough that they
qualify for Medicare vs. Obamacare, though I admittedly don't know what the
distinction is. If you need insurance, and Healthcare.gov is telling you
you're too poor, it might simply be failing to tell you to go check with
Medicare and see if you're eligible there.

~~~
powertower
No. It was rather -

Enter $10,000 - you get no credit.

Enter $10,500 - you get big credit.

Enter $11,000 - you get no credit.

Enter $11,500 - you get big credit.

Etc.

~~~
bmelton
Ah. I misread then. That's... odd, to say the least. About in line with the
expectation though, I suppose.

------
jeffdavis
With so many problem's, I doubt it's just an issue of testing. They probably
have serious design issues and other engineering problems.

~~~
yggdrasil
Exactly. The cardinal rule of competent software development is that you don't
test quality into the software. The root causes were requirements churn, split
development, and no project coordinator. Those would be nearly insurmountable
on an average size project but were hopeless on this project. It's a marvel
they were able to stabilize it to the point it's at.

------
diminoten
I chuckled when we had the president of the United States of America
announcing software delays on national television.

~~~
larrys
Anything that is done by a public official at that level (not just the
President) is well thought out and implemented after much discussion to
achieve a particular goal. You can reverse engineer it and draw your
conclusions as to why they said what they said.

In this case my interpretation is that it's to shift the blame a bit and also
to appear to be out in front of the issue and let people know that "things are
being handled help is on the way and we're on it".

An analogy I would use is this.

Let's say you are hosted somewhere (say Rackspace or AWS) and you lose
connectivity and your site is down.

Better for RS or AWS to "appear to be on top of it" (whether they are or not)
and to be "doing something" than to have this black box of "hey exactly what
is going on and when will it be fixed". The anxiety is what kills you.

Once you know that at least (using another example) "a spare airplane is on
the way but it is currently delayed by a snow storm that should clear in 2
hours" you are feeling much better than if you don't have any answers at all.

------
larrys
"School children and grandparents alike are now worrying about whether their
passwords are being passed in the clear now. Imagine that."

Even if they are "worrying" (and I really doubt that they are actually) they
would only be worrying the same way they do about flying right after a major
plane accident. That anxiety tends to go away very quickly.

~~~
lstamour
Yep. With every password breach the word gets out that you should have unique,
strong passwords for each site. And yet with every breach we learn of more
shared, common passwords. Not to mention how little attention most pay to
entering a password on a site without SSL. Credit cards they'll worry about
but passwords, less so. And imagine explaining session cookie hijacking as we
tried to when Firesheep launched? ;-) Once the big players launched with SSL,
problem "solved". That said, most do now secure their WiFi, so the public can
learn, and I think has learned from healthcare.gov.

~~~
larrys
"That said, most do now secure their WiFi"

Then there is the linksys wifi camera which you can access over the net but
doesn't have https. So in order to get to it securely I have to ssh in to a
machine on the network with a tunnel and connect by proxy. A real pain.

~~~
lstamour
Or the devices you don't check. In my case, I've yet to intercept traffic from
a Dropcam to see if it's SSL'd correctly everywhere. Let's not start on the
alert screenshots it emails me....

------
a3n
Interesting post, probably true enough, but I don't think testing is the major
problem.

The law itself was rushed through, to the point that the President and members
who voted either way didn't really know what was in it. It was huge, and
rushed because they had to beat the clock of the potential coming of a hostile
Republican House. Even champions of the bill in Congress admitted that it
would take years to fully understand what the bill does. The mind reels.

The law has many hostile stakeholders, and many turf grabbers, and the real
requirements were not managed by a single entity who wanted to get it done the
best and most economical way possible. Members of Congress (keepers of the
purse) were and are fighting it at every turn. Government agencies had to have
a piece of it, providing data for deciding customer eligibility and policy
level; law enforcement and immigration concerns play a part. The requirements
were a result of compromise (in the political sausage making sense) and turf
dominance, some of which are part of any project, but at a hyper level when
government agencies are involved. There was probably no technical adult in the
room telling political people that this and that just can't be done in any
economical and working way.

The government does not normally provide web sites to manage and coordinate
anything between government, citizens and corporations on this scale. The most
you'll usually see, from the outside as a citizen/consumer, is listing of
information in text or pdf format, possibly filling in a form to make an
appointment or communicate with a representative. They don't know what they're
doing, in a profound sense and on a grand scale.

The government just doesn't have experience managing and running such a site
with such requirements. They're trying to do something like Facebook in its
current incarnation right from the start. It's like never having gone to the
moon and strapping some astronauts on the top of an ICBM and hoping for the
best, which is a formual to "fail fast" in the worst sense.

Because they didn't know what they were doing, and because it was probably
thought of as just another IT contract effort, they followed standard
procurement practices and used standard ("we know them") vendors. And that
failed miserably.

And because of all of the above, and general inexperience for such a project,
and relatively unlimited funds to get it as wrong as possible, they started
actual specification and implementation way, way late, with the added
complication that the possibility of a time overrun was just not there,
because of a hard political deadline. It's no wonder that testing was done so
poorly and ineptly, they just had no time at all.

~~~
jeffdavis
I don't see how Congress can be responsible for most of the reported problems.
If the law was just too complicated, then I could see how portions would be
inaccessible.

But what we see here are major UI failures, availability problems, losing
applications partway through the process and forcing people to start over, and
inexplicably inconsistent results and behaviors. These are failures of
execution.

~~~
a3n
> These are failures of execution.

They are, but in my opinion they happened because they tried to do a full on
web site in a rush with outdated federal procurement and hostile stakeholders.
The implementers and implementation were doomed.

There's a lot more to a web site fronting a huge federal program beyond mere
technology.

------
danso
Software developers should better appreciate the uniqueness of their testing
situation: like any other field, writing, maintaining, and executing takes
time. But in no other field that I can think of is testing so relatively cost-
free to continue doing..."regression" testing is either not possible or
feasible elsewhere. The ability to automate and near effortlessly run tests
with every code change has a fundamental impact.

To give a non-programming example...journalists make silly mistakes all the
time about dates and misspelled names. It's not always, or even usually,
because the reporter is incompetent. Sometimes they hurriedly typed in a fact
from the top of their head and forgot to put a "TODO" near it (some in-house
CMSes do not make meta-comments easy). Or, just as frequently, something got
changed as the text moved from one editor's desk to another...there's pretty
much no such thing as diff software, which is fundamentally different than
keeping revisions.

Spellcheck can help detect and auto-fix some problems. But generally, you need
to manually proof-read things to verify them, and at some point, you just
assume that no one is going to change what you've verified, and then you press
"Publish". It's not that continuous-proofing isn't possible, it's just not
feasible.

It can be frustrating working with people who think testing software is to
keep "proofreading" it over and over again...user testing is vital, obviously,
but I'm talking about people who test the wrong, already-verified things, and
then sap their energy not checking for other variations, and this is
understandable and very human, of course.

I guess the big picture to understand is that screwups are frequent in every
field, all the time. The best surgeons forget to wash their hands...not
because they're idiots, but because emergency surgery will cause all kinds of
things to go haywire, including basic procedure.

And obviously, very basic mistakes can occur in production code. But as
programmers, we uniquely benefit from what we can do to prevent that. And this
superpower of ours is something that I wish was more conveyable to the greater
world.

~~~
jeffdavis
As a counterpoint, software development arguably suffers most from the
combinatorial explosion of behaviors. In other words, testing software is
easier but less effective (or at least less complete).

------
dannyrosen
It should be noted that testing consists of such a wide array of tasks that
it's often as hard to find the right people than it is to identify the right
type of testing they should perform.

While performance and load testing on Healthcare.gov should be obvious tasks
we get into murky territory when asking questions like: Does the UI need to be
tested more than the APIs that power it? Will the value of automated end-to-
end testing be deprecated after a UI refresh takes place? (Will there be a UI
refresh?) How much time should be put into scalable UI tests? Are automated
smoke tests enough?

I could go on, but the question I'd like answered is what did the actual
testing plan consist of?

------
undoware
"Hey Frank, you test this?"

"Naw, Larry, government contract."

"Shit, I think this one's important though."

"We'll fix it later. It's easier to beg forgiveness than permission to change
spec."

------
evolve2k
Anyone have a link to the github repo she mentions? Also was there an
announcement of the open sourcing of code that I missed (been travelling and
off the grid for a while).

------
robomartin
Thinking that this is about TDD or software management is to grotesquely miss
reality.

This whole fiasco has demonstrated, to an incredible level of clarity, why
we've reached a point where government is too big to work. This has nothing
whatsoever to do with who's President or which party controls what. This is a
simple case of an entity that has grown so large, complex, ignorant and
bureaucratic that it simply can't figure out how to produce anything useful,
from laws to websites.

The difference this time around is that this has been very public. This is a
problem that forces people to pay attention and get involved. Healthcare
affects everyone directly and people care about it.

Most of us who have understood the devolution of government over the years
have recognized the incompetence of the organization as a whole for quite some
time. Some have been more vocal than others in trying to highlight the issue.

It's a difficult position to hold because it is relentlessly attack by those
who, through indoctrination or religious-like following, stick to their
respective parties and simply won't even admit they are being screwed despite
mounting evidence to the contrary. The President lies about healthcare to the
country and the world dozens of times and Liberal media contorts itself to try
to figure out a way to spin it into some alternate reality that makes sense.
We had exactly the same kind of thing happen with Bush and his wars, but this
is about the ACA. We let them lie. Some of us see it and call them on it while
others shoot us down based on party loyalty and continue to support the
effectively criminal behavior.

The difficulty in gaining mass awareness for these kinds of problems has been
in that most people, at the end of the day, couldn't care less about what's
going on inside the sausage-making factory. They are too busy trying to earn a
living and going through their daily lives. There are things in most people's
lives that are far more interesting to them than what government does and how
it works. True to this I think it is fair to say that most US voters are
utterly uninformed and get their opinions (and voting decisions) from the
media --mostly TV.

Now things are different. This is something that is important to everyone.
Health, food and housing are top-level concerns for everyone. This
ACA/Obamacare mess is achieving something no government critic could possibly
buy for any amount of money: Bring to the forefront the incompetence, waste
and mismanagement that has become part and parcel of what our government has
been about for years.

Everything in our government is done this way. Everything. You just don't see
it or don't care to dig into it for other issues. Healthcare just made it
first page news for everyone.

Everyone now sees how the sausages are being made. There is no way to hide it.
Everyone can now balance the equation of what politicians said and promised
them against what they've actually delivered. A family who's healthcare
premiums doubled, who's deductible skyrocketed and who lost the ability to see
their doctors at their hospital has no way to satisfy a promise of lower
premiums, better coverage and keeping what you like.

I still remember Los Angeles Mayor Villaraigosa deciding to change the outcome
of a vote in a very public way right in front of the cameras while a woman
came over to him and said "Let them do what they are going to do", all of it
picked-up by the microphones and cameras. There is no shame or respect any
more. Politicians know they can get away with murder because there are no
consequences for such things as publicly and visibly changing a vote or
spewing out lies.

Hopefully people are starting to think about just how ridiculous it is to give
a blank check or a pass to anyone in government. I also hope they are starting
to become more convinced that if we don't make those who lie to us responsible
for their lies we are never going to improve government.

If it had not been for Snowden who in government would have told us what was
going on with the NSA? They've been lying to us for years and they would have
continued to do so had it not been for him. They lied knowing there are no
measurable consequences for their actions. At worst they move elsewhere in
government and live goes on. It's a joke.

Imagine if you were involved in a contract negotiation for you business,
signing the papers for your home purchase, auto purchase or a lease rental
agreement and the other party said something akin to "Don't worry. We have to
sign the contract to see what's in it". How fast would you run from that one?
You would, right?

Then, why is it that we let those in government play such games? By extension
they --the governing class-- over time, start feeling they can do anything, to
anyone at any time. I would not be surprised if this is the sort of thing that
leads to mass surveillance programs not being given a second thought within
that community. If you can get away with murder you keep murdering, plain and
simple.

The solution, I am afraid, is not technological, it's political. I've said
this before: If you are in tech you need to be smart about politics because
bad government and bad policy can be more destructive to your efforts than bad
code. The NSA fiasco alone will probably cost US companies billions. Trust is
a very difficult thing to regain, whether it is with your girlfriend,
boyfriend, wife, husband, company or country. Once trust is broken it takes
orders of magnitude more work to regain it. If ever.

So, there you are, becoming a highly skilled technologist thinking that
government and political issues are of no importance to your mission, only to
find out that your international trust and market potential keeps getting
boned by ignorant fools in government who really have no clue what they are
doing. Government is almost the only entity that can completely destroy your
startup with the stroke of an ignorant pen. Look at the startups fighting
against the transportation and housing unions, rules and regs. How different
would things be in terms of competition and progress if those barriers where
not there?

Bottom line: You have to be informed and get involved. You have to fight
against indoctrination to see what reality looks like. Stop taking other's
opinions as your own. Question everything, research, critically observe,
measure and use your CS and mathematical skills to model. In other words, be
the high information citizen.

The devil isn't in the details, it's in politics.

------
Datsundere
I'm not sure what people that earn below the required threshold do if their
state isn't expanding on medicaid. How do they apply for medicaid?

------
yeukhon
I am just an undergraduate, so what the heck do I know about agile in big
project. Please feel free to critique. What do guys from big projects have to
say?

Here is how I imagine people arguing about agile vs water fall.

WF: It probably will take the same amount of hours and people to hash system
together in agile, making sure all components are integrated properly without
a strong and detail system requirements.

AG: But you don't wait until the end to test your software and wait until the
end to discover shit and changes. You can iteratively change stuff.

WF: Ah. Fine. I will just tell my co-workers to test more often. Now, you
agile people just want everything to be short-term. How do you ensure people
don't block other teams from completing works? you see when blockers appear,
the time requires for completion is the same as everything done at once.

AG: But if you finish 80% while waiting on the 20% to block forever, you still
have 80% done. Plus, in agile you have some short term and long term goals.
You order them by importance and blocker. When the whole sprint is blocked,
just do more testing and making sure existing systems are working as said.

WF: Fine. Now what about people going monkey on their own? The point of water
fall is to ensure no one is outside the design and minimize risk. Now everyone
goes agile and shit going to come because some teams are not playing nice.

AG: This is a tough problem. But the only way out is by scrum, well, at least
meeting daily or weekly. The benefit of agile plus regular meeting is that
everyone knows what the heck is going on and changes can be made. Once you
made a specification forever, it's hard to change.

WF: But it is the same in agile that a chance can be tough to make because
people didn't make the right decision at one of the early sprints.

AG: Yeah. That certainly is. I won't lie agile people don't make that easier.
But I argue it is easier if you just work on smart parts.

WF: Are you sure it will work for multiple-billion dollars project? With a lot
of legal blockers and a lot of other software to work with? You need to do a
thorough analysis. You need to conduct research how existing healthcare
registration works, find out the pros and cons. That's a lot of time. Now if
you go agile, you will just keep doing research.

AG: ....

YEUKHON: I will fill in the blanks, but hey what the heck do I know about big
project software engineering. I think agile doesn't require people to start
right away. Agile just ensure nothing blocks forever and that goals are more
or less short-terms. it is always required one to have good knowledge of how
existing solutions works. So if you need to find out how to integrate the new
healtcare.gov with the rest of the solutions out there, do that research
first. Obviously you can't just develop a site like a baby punching keyboard,
hoping the baby compose some legendary music. It takes real talent to do that.
So I say myth buster, agile still requires professionals to do priori
research, have lengthy discussions about how to go about implementing the
system, give a pretty good sketch about how things supposed to work, and then
go off and start implementing something. But don't come up with the full
solution so detail that you ought to obey that contract.

WF: Yeah, but your client is spending 100 millions and this is impacting tens
of millions of people. Any example out there like healthcare.gov did finish on
time and running successfully using agile? I know WF sucks, but real examples
of agile out there? You know, people tend to do a lot of talking and sketching
when they are working with so many people and so many things involved. "Oh Bob
and his teams are doing research on existing solutions works and my team is
working on consumer APIs? Okay, let's talk about what fields we want, what
schema we want in the database, etc"

