

This Obamacare contractor doesn’t take security seriously - RougeFemme
http://www.washingtonpost.com/blogs/the-switch/wp/2013/10/25/this-obamacare-contractor-doesnt-take-security-seriously-that-needs-to-change/

======
eigenvector
> In Slavitt's defense, data security may not have been an explicit feature of
> QSSI's federal contract

Coming from the construction industry, there isn't a contractor in the world
that cares about things which aren't in the contract. Implementing features
which the client has not asked for, in the design-bid-build world of public-
sector contracting, just means you will lose the bid to a lower bidder who
won't do those extras.

~~~
greenyoda
" _Coming from the construction industry, there isn 't a contractor in the
world that cares about things which aren't in the contract._"

In the construction industry, a contract might not explicitly say that the
building has to comply with local building, fire and electrical codes - that's
taken as a given. To construct a web site that deals with sensitive
information without taking security into account is like building a skyscraper
with no fire exits - not something that any honest or competent builder would
do.

~~~
lmkg
Yes, but those are actual codes. In fact, _that 's why_ they're codes.
Contractors won't do more than they're legally required to (nor should they be
expected to), so safety features were made legal requirements.

Relying on the honesty or competency of your contractors go above & beyond
their requirements is an inefficient and unsustainable solution in a free
market. This is a clear case of Moral Hazard: In the same way that bad money
drives out good, corner-cutting organizations out-compete honest ones.

~~~
benmanns
What about reputation? I regularly do more than the legal minimum for my
clients and there are companies who do the same for me. Meeting the legal
minimums only makes sense for commodities, e.g. this salt is 99.5% pure. This
software is 99.5% "secure"? This software is 99.5% "complete"?

~~~
pessimizer
I don't think that most RFPs require or make many allowances for "reputation."

~~~
eigenvector
True, but quantifiable aspects of reputation (like having successfully
completed similar projects in the past) can be part of a pre-bid qualification
process. This is common for large or complex projects.

------
david_shaw
Most organizations don't take security seriously.

I work in the information security industry--primarily focused on healthcare
security--and you'd be amazed and appalled at the lengths that CIOs will go to
say that they "don't need to be HIPAA compliant."

More and more, though--especially as high-profile incidents become more widely
reported--security is starting to become an accepted norm. There's still not
much budget for it, and it's still considered an inconvenience, but at least
people recognize that spending some _preventative_ dollars can save a
significant amount in _reactive_ responses (DFIR, legal fees, etc.).

The HN crowd might be interested to know that the _worst_ security offenders
I've seen (even worse than old legacy systems) are _software startups._ Why?
Slogans like "Ship First," or agile development cycles that don't include
QA/security testing as part of a meaningful SDLC. What good is shipping
something every two weeks, if one of those releases loses your customer
database?

My latest drive is to help organizations realize that security is a _feature,_
not just an inconvenience. Once that idea is more commonly accepted, we'll all
be better off.

------
UnoriginalGuy
I might be horribly misreading this, but it reads like they're talking about
no specific threat or problem but rather arguing if that was part of the
contract or not?

If that's the case then shouldn't lawmakers, being lawmakers, pass laws so
that future government projects include a clause in the contractual
arrangement? Isn't that just common sense?

Seems like this is a trend with the US government, they try to find others to
blame instead of just fixing the damn issue. They had this massive set of
hearings about people abusing military disabilities (e.g. claiming for
disabilities obtained before joining the US military) but didn't actually, you
know, fix the freaking issue -- or even try to. They just berated the people
being advantaged from it.

~~~
dragonwriter
> If that's the case then shouldn't lawmakers, being lawmakers, pass laws so
> that future government projects include a clause in the contractual
> arrangement?

They already passed a law creating mandatory security requirements for Federal
IT systems, whether contracted out or built in house or any combination. [1]

[1]
[http://en.wikipedia.org/wiki/Federal_Information_Security_Ma...](http://en.wikipedia.org/wiki/Federal_Information_Security_Management_Act_of_2002)

------
uptown
So Rep. Mike Rogers is now suddenly concerned with protecting the security of
data? The same guy that's been defending everything the NSA has been exposed
to have done?

~~~
cobrausn
So, I know we are looking for hypocrisy here because _gasp_ hypocrisy, but
there really isn't any. You were providing your information to the government,
making sure nobody other than the government sees it isn't at conflict with
defending the NSA's actions if you see this kind of activity as legitimate.

~~~
uptown
On one hand he's alarmed that enrollment data may not be properly protected.
On the other hand, he's protecting an agency that's requiring technology
companies to create back-doors and weaken encryption algorithms for the NSA's
convenience. How are those two objectives not in conflict?

~~~
shortstuffsushi
> How are those two objectives not in conflict?

A. The government wants all your data available to them. B. The government
does not want all your data available to the public.

A != B

Edit: Not to come off as rude, I just tend to agree.

If you want to say that in that sense, the government is being hypocritical,
perhaps that's true. At the same time, though, their goal here is (hopefully)
to keep your data safe from outsiders, whereas we can "trust" them to be good
with the data they backdoor from companies ;)

------
mrweasel
I hope (naively) that one day, software developer will collectively tell the
governments that they work for "Sorry, no amount of money will deliver what
you're asking for".

Pretty much every government IT projects I've seen fail, that's most of them
here in Denmark, has done so because of politics, laws and regulations. The
thing is: the big contractors like CGI, aren't going to turn down the money.
They are only to happy to deliver a poor product because they are paid
millions and never asked to account for the failures of these projects.

It would be better for the industry as a whole to say NO to projects we know
will fail. I don't for a second believe that CGI software developers thought
that they could make healthcare.gov a massive success, or even just plain
decent. If they couldn't see that, then they shouldn't be doing software
development in the first place.

Doing a postmortem this way is refreshing, except you'll never get the right
answer: "With the time available (not the money) this was never going to work.
The integration with legacy system and the shear amount of these integration
points makes this an almost impossible project". In reality will get a lot of
blame allocation and CGI won't be made accountable for accepting a project
that should have be rejected by anyone with half a brain.

~~~
hga
Do you think that when CGI signed the contract they realized the integrator,
the government's CMS, plus those above, HHS and the White House, would be
constantly demanding major changes, _right into the week before launch_???
Also check the date of the "you must register first, no window shopping"
decision, it was quite late and had a very major impact.

That CMS, responsible for integration testing, would delay that until 1-2
weeks before launch (see above change orders problem), see the tests failed,
and launched anyway.

Well, I'm actually sure CGI wasn't surprised by the latter, but, still, I
can't blame them for any of the above. The White House -> HHS -> CMS, all
insufficiently experienced in running a major development project, turned out
to be a customer from hell. Heck, given how important this site is, a key part
of Obama's legacy, I and many others assumed the Administration would treat it
with appropriate seriousness. We were wrong.

~~~
mrweasel
>Do you think that when CGI signed the contract they realized the integrator,
the government's CMS, plus those above, HHS and the White House, would be
constantly demanding major changes, right into the week before launch???

When the second or third of the changes happens, CGI should have told them
"NO, this cannot be done professionally" and hand them their money back.

~~~
hga
Well, that's also a decision to get out of the business of Federal
contracting, and to lose a lot more of their contracting business.

Did you notice how obsequious CGI's VP Cheryl Campbell was when asked the
usual leading questions about the bureaucrats and political appointees running
the show? This bit from the hearings is really telling
([http://www.washingtonpost.com/politics/house-panel-grills-
co...](http://www.washingtonpost.com/politics/house-panel-grills-contractors-
on-troubled-health-insurance-web-
site/2013/10/24/8f42c748-3ca7-11e3-b7ba-503fb5822c3e_story.html?hpid=z1)):

" _Later, Rep. Leonard Lance (R-N.J.) asked the contractors whether they could
conceive of “a more incompetent administrator” than CMS.

“I have no opinion on that,” Campbell replied._"

Might be a lie, might be she's been jaded by so many customers from hell she
really can't precisely rate CMS....

I'd also note that it wasn't necessairly the 2nd or 3rd change that were
fatal, while the late ones clearly were. I'd further note that a lot of us are
optimists, believing we can perserve in the face of tough situations including
clients from hell, and we can take that too far when things become manifestly
impossible.

It's also possible that CGI et. al. believed, certainly hoped, that when the
first integration tests in the last 1-2 weeks before the scheduled launch
resulted in total failures the administration would admit reality and delay
opening a site that they _knew_ was going to fail hard. I mean, this steady
dribble of bad news is a zillion times worse than a one time hit in a program
that's had a zillion delays and waivers already.

I guess I'm saying I'm not sure CGI could accurately gauge the depths of
stupidity of this client prior to the most insane decision to turn on the
known failing system.

------
jcutrell
If anyone saw this briefing, it was mostly a lot of really technologically
illiterate people asking pointed questions about a gigantic set of systems
that can't easily be answered to technologically illiterate people. At one
point, a congressman, with the tone of a disappointed parent, said
(paraphrased) "I am very disappointed in you all. have yet to hear any one of
you say sorry to the American people."

Someone in the room retorted with "we didn't hear an apology when the
government shut down either."

(You'd expect someone to get timeout for this kind of pettiness.)

With that said, one of the obvious problems was blame shift, as this article
points out. The guy was (I don't remember this particular interchange) most
likely outlining the fact that the question being asked of him wasn't relevant
to "his responsibilities" in the contract. However, when responsibilities of a
project are distributed, things fall through the gap.

This article is a bit too short to give enough space for the subject, but I
think it does point out the most important issue - no one really had full
control of the project.

~~~
monsterix
Totally agree. One of the thoughts that came to my mind is that a public
project like this one sh(c)ould be _open source_ right from the beginning.
That would solve a lot of quality issues on one hand, and probably the pricing
will also be better for the Government/taxpayers.

While I am pretty sure that most congressmen do _not_ get it, but there are
some very qualified and awesome technical people who work in the Government.
Probably, someone has to have the balls and call it open source one fine day.

~~~
Osiris
My opinion is that anything that the government funds should be open source.
Patents that come from government funded research, software projects,
electronic voting, should all be open source. If we, the people, paid for it,
we should be able to have access to it.

~~~
monsterix
> anything that the government funds should be open source.

Generalization to open source everything that the Government does, or
taxpayers fund, might not be such a great idea. There are some projects which
are supposed to be secret (Like a NSA that's focused only on anti-socials)
because they give our Government an edge over others.

Probably a competent agency with an oversight to decide on open/closed aspect
can help, but the subject needs much longer and detailed debate. IMO,
healthcare, education, civil investigations are perfect examples where we can
go completely _open source_. Or get disrupted the valley way.

~~~
swalsh
To be fair, I'd imagine the technology running PRISM is really incredible...
i'd love if they could open source any of that.

------
mberning
In this latest witch hunt I am sure there is plenty of blame to go around.
This is not the first 1.0 system to have serious problems. It will get better.
You would think they crashed the Space Shuttle and killed some people given
the level of grilling they have received.

~~~
mgkimsal
Ummm.... many people are now legally required to buy insurance, and this
marketplace is the only option for many of those people. Saying "use the
system or pay a fine, oh and by the way the system doesn't work" is not the
same comparison to many other "1.0" systems which are optional to use.

~~~
mberning
My understanding is that it IS possible to register rather quickly during non-
peak times. You are also not required to go through the marketplace to acquire
insurance. And finally, open enrollment ends in March 2014, so there is plenty
of time to get this sorted.

The media coverage and congressional circus is blowing this way out of
proportion.

~~~
mgkimsal
Not entirely.

They've completely hosed up existing data.

Took me 2 weeks to get in, finally got in. Application created for my wife and
I. System broke. Logged in a couple days later. My wife's data is gone. No
ability to add it back in. No ability to delete current application. No
ability to start a new application. It's a mess.

Registering and using vastly different. Given that it took them years to get
this far, I don't have a whole lot of hope that this will be usable for
everyone, with data problems sorted, by March. _Possible_ , but not likely.
Especially because the attitude of "hey, just go somewhere else to buy" is so
quick to be used. If that's really the case, why did we bother to even put
this together, if it's not at all necessary or required?

There's going to be massive loads of hidden bugs that will plague this for
months, if not years. Having such a tight deadline on it is crazy because it's
not at all how any large insurance agency works, nor is it how the federal
government works.

~~~
MartinCron
I think it's pretty obvious that if the bugs aren't sufficiently fixed within
the next few weeks, the individual mandate penalty will be waived or at least
delayed.

~~~
abarringer
I don't give a flying rip about the individual mandate penalty. But..

If you need insurance and you just received a cancel notice from BCBS that
your policy is gone as of Jan 1st it becomes imperative that this exchange
work within the next few weeks.

This isn't a political game real people are going to be royally hozed and soon
if this doesn't get fixed.

~~~
MartinCron
You've got more than nine weeks, I have faith that even the sub-par government
contractors working on this will get it sorted out by then.

Feel free to come back and tell me how wrong I am if my faith turns out to be
misguided.

~~~
hga
No, 9 weeks would require everyone who _must_ get a new policy by Jan 1 to get
all the way through all the systems, exchanges -> insurance companies, in the
last three days of December.

Even if several paths are kludged for these most desperate millions of people
losing their current coverage Jan 1, I can't see how any more time than 7
weeks is feasible, leaving 2 and a fraction to process them. And to know the
system is really working, large numbers, 10s of thousands, had better be
getting successfully through the systems before then.

------
sailfast
The most useful part about this article for me was to discover that Northrop
Grumman's CISO is phishing its own employees. Brilliant. Wish I had thought of
it. Great way to keep people on their toes and aware of what's going on in the
space.

~~~
gohrt
This is standard practice for all IT organizations, and any org that isn't
testing its security is failing at security.

~~~
ihsw
Same for backups. If you have six months of backups but you haven't tested any
of them, and they broke four months ago, then it's going to be very
unpleasant.

------
lightblade
Can we talk about software engineer certification yet? I tought things like
this suppose to push on the movement.

~~~
fiatmoney
I guarantee that the contractors involved were loaded with a significant
number of Java & Microsoft certified developers. Furthermore, I'd guarantee
that the Big Dig, et al, were loaded with certified "real" engineers.
Certification (even some fancy "project management" certification) doesn't get
you any guarantees of project quality as a whole.

If anything, I'd expect a focus on certification to decrease quality. Those
things are expensive & time-consuming and the only ones who invest in them are
the people who expect to be in that consulting ecosystem for a long time.

There are processes, not certifications, that result in generally high-quality
projects (see NASA's processes) at the cost of being incredibly slow and
expensive.

~~~
lightblade
Sorry, perhaps I didn't make this clear enough. I wasn't talking about those
Java/MS certs. I was talking about this: [http://cdn1.ncees.co/wp-
content/uploads/2012/11/Exam-specifi...](http://cdn1.ncees.co/wp-
content/uploads/2012/11/Exam-specifications_PE-Software-Apr-2013.pdf)

~~~
swalsh
Is there like an example test online? I've been in this game for 7 years, but
I still learn new things all the time. I'd be curious how well i'd do :D

------
gxela
Regardless, government site or not, who currently does care about security?

Most code I have seen widely in production is using old sql injection
vulnerable code. No sanitization and no proper session handling.

You would think that people would care about something like security with all
the privacy concerns.

I wish I had better words to say this, but I had just woke up and out the door
to breakfast.

~~~
viraptor
I keep thinking whether it would be a good idea or not to hold companies
responsible for security issues affecting their users. If you fail to secure
passwords, why shouldn't there be a fast-track way to sue? Almost any physical
service provider I can think of already is responsible for damages. Your
plumber fails - in most cases the company is insured and will fix at no cost
and will be responsible for damages. Your mechanic fails - there's likely a
clause in their agreement that allows you claim for the damage caused. Even if
your payment card is misused, the bank is expected to reverse the charges. Why
are the internet service providers excluded from this treatment?

Sure, it's possible that this kind of danger would be then aimed at separate
developers working in those companies too... but I'm not sure it's a bad
thing. It would be easier to resist stupid requirements (I won't do it,
because it would bring legal issues on me) and even the comments on SO would
be different too ("if you do this, you will be sued", rather than "this is not
a secure way to do it").

------
beauzero
It was a 3 year project with requirements changing up until the last 4 months
where end-to-end was managed under waterfall (some pieces/contractors used
agile like methodology)...and there was 2 weeks of final end-to-end manual
QA...what the hell else did they expect to happen?

~~~
hga
It was worse, the "no window shopping" decision was made less than 4 months
before launch (August to September as I recall), and changes were made through
the week before launch. The NYT reports 7 major changes in less than 10
months.

And it didn't matter what the QA reported, the site was hell or high water
launch Oct 1st whatever they revealed.

As always, those with political power (reported to be the White House and CMS,
maybe some of HHS) trump those who have to make it work in the real world.

------
joshuahedlund
I don't think the one line quote from the hearing necessarily supports the
headline, but I certainly don't have high hopes that the site is secure; the
number of bugs and inconsistencies I found just trying to create an account
dissolved any hope of that.

------
patrickg_zill
Blame the contractors?

Instead of the ridiculous way the gov't handled it, with constantly changing
requirements and a "change the car tire while we are driving" attitude?

Sounds like the WashPost is trying to do some damage control.

~~~
hga
I'm seeing enough of this trope, not to include the BBC initially, to wonder
if this didn't hit the current incarnation of the Journolist.

Unfortunately this disaster is so huge that's not adequate to tamp it down.
But it sure looks like some are trying ... hmmm, what is the broadcast part of
the MSM aside from Fox saying?

------
VMG
Or else what?

------
gohrt
The overall project needs an _independent_ security evaluation by a separate
organization. Without an audit and tests before launch (and ongoing after),
any problems discovered in production are the primary contractor/procurors's
responsibility.

