
Equifax Breach Caused by Lone Employee’s Error, Former CEO Says - DLay
https://www.nytimes.com/2017/10/03/business/equifax-congress-data-breach.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news
======
sp821543
[http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...](http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf)

7) Post-accident attribution accident to a ‘root cause’ is fundamentally
wrong. Because overt failure requires multiple faults, there is no isolated
‘cause’ of an accident. There are multiple contributors to accidents. Each of
these is necessary insufficient in itself to create an accident. Only jointly
are these causes sufficient to create an accident. Indeed, it is the linking
of these causes together that creates the circumstances required for the
accident. Thus, no isolation of the ‘root cause’ of an accident is possible.
The evaluations based on such reasoning as ‘root cause’ do not reflect a
technical understanding of the nature of failure but rather the social,
cultural need to blame specific, localized forces or events for outcomes.

~~~
shub
That quote sounds good, but I don't think it's necessarily applicable to this
situation. The author seems to be talking about complex systems that are
designed and operated to be robust against failure, like the space shuttle.
Saying that Challenger blew up because of an O-ring is technically correct but
also horribly wrong, as an example. Equifax IT does not appear to be operating
at a level to prevent a single failure from causing terrible damage all on its
own.

That aside, it's hardly true that one person can bear all the blame for not
patching their systems, even if they did successfully prevent patches from
happening. For one thing, how the hell did they keep their job after doing
that? Unless it was the CEO (well, now that they have a new CEO maybe they'd
like to put all the blame on him), there was someone up the chain who could
insist that the patch get applied. I think you definitely could apply root
cause analysis techniques here, and I strongly suspect that such analysis
would uncover numerous serious deficiencies in Equifax's IT operations. Of
course, guessing that a large boring corporation has terrible IT practices is
similar to guessing that a given duck quacks and has wings, so there's that.

~~~
zemo
> Equifax IT does not appear to be operating at a level to prevent a single
> failure from causing terrible damage all on its own.

they're operating at a level where over a hundred and thirty million people
could have their ability to get a mortgage, open a bank account, or start a
business harmed. If you think that such responsibility does not mandate the
highest requirements for data safety, you should not work in this industry.

------
MBCook
Good to know.

And what about the person who’s job was to make sure that one guy did his job?

And the guy who was in charge of that person?

And the department who’s job was makin sure nothing was insecure?

And the guy managing them?

Yep. All one guys fault. Poor guy, ruining the American credit monitoring
system for the rest of us.

~~~
blubb-fish
As somebody with full IT administration access I am just too aware of how easy
it is to entirely destroy an entire company with a single inadvertent command.
And the risk of this happening increases swiftly with stress levels and work
piling up and "just get this done quickly!", "why aren't you done yet?", "we
need this tomorrow!" etc. And that work environment is the norm - not the
exception.

~~~
MBCook
I’ve been there, and I’ve seen someone come really close to ruining a company
with that one mistake.

But even in a small company there were others who could patch things. There
were people above me who kept an eye on if patches were applied (or at least
reported to be applied).

It wasn’t just ‘we told guy X to patch and never followed up’.

~~~
blubb-fish
it is also worth noting that the experience of being "responsible" for a
company being destroyed could cause suicidal risks for the respective person.
I once accidentally didn't close the door to the office with a key - the door
had a defect which caused it to not lock sometimes - this made me realize how
heavy of a burden it would have been if the company would have gotten robbed
or something - which could have been very well its end. That was a very tough
experience - luckily nothing happened.

------
rgbrenner
$3B/yr in revenue; 9500 employees

One person responsible for the security of the enterprise.

If there is truly one person for a company this large, then he was setup to
fail from the beginning. The management is negligent and incompetent for not
creating a system for this. That's his job.

I think more likely, the CEO is full of shit and they're scape goating some
poor person. But even if that's not the case, this is a terrible thing for him
to admit. If he's really that incompetent, he has no business in management.
Hopefully he never works in management again. Kiddo needs to go back to
school, he's clearly forgotten all of his training.

~~~
sillysaurus3
Well, this is what happens when people call for jail time. The moment someone
goes to jail for something like this, it will change security issues forever:
People will stop reporting breaches, and developers themselves will be at risk
of going to jail.

Also, if you try to kill Equifax, companies will stop reporting breaches.

I don't know what the ultimate outcome of all of this will be, but it's
important to keep perspective. People are out for blood, and it's both scary
to watch and unsettling to think of the precedents it might set.

~~~
toomuchtodo
> Also, if you try to kill Equifax, companies will stop reporting breaches.

The US government killed Arthur Andersen. Financial fraud is still reported.
Equifax is not too big to dissolve.

[https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal](https://en.wikipedia.org/wiki/Arthur_Andersen#Enron_scandal)

~~~
rrdharan
Many reasonable people seem to believe that the backlash and horror inm
response to the US government killing of Arthur Andersen and the subsequent
job losses were what led to the later toothless reactions by the DoJ to
subsequent corporate scandals:

[http://www.npr.org/2017/07/11/536642560/is-the-justice-
depar...](http://www.npr.org/2017/07/11/536642560/is-the-justice-department-
shying-away-from-to-prosecuting-corporations)

[http://www.slate.com/articles/podcasts/slate_money/2017/07/t...](http://www.slate.com/articles/podcasts/slate_money/2017/07/the_chickenshit_club_sexual_harassment_in_silicon_valley_and_hobby_lobby.html)

[https://www.amazon.com/Chickenshit-Club-Justice-
Department-C...](https://www.amazon.com/Chickenshit-Club-Justice-Department-
Criminals/dp/1501121367/?tag=slatmaga-20)

~~~
toomuchtodo
I mean this entirely seriously: perhaps its time we be less reasonable people.

------
bogomipz
>"The company sent out an internal email requesting that its technical staff
fix the software, but “an individual did not ensure communication got to the
right person to manually patch the application,” Mr. Smith told the
subcommittee."

So someone forgot to forward an email? What else does ensuring email
communication got to the right person mean?

When the security or hundreds of millions of people's data relies on a process
of selective email forwarding, the "lone individual" in question is the CEO.

~~~
uptime
Relying on email alone for this is negligence.

But a ticketing system at least should have been used. How were they planning
to check compliance with that email? Obviously there was no audit to check
that the email was followed.

------
didgeoridoo
This was clearly a failure on the CSO’s part, for which the CEO should take
responsibility (after all, he hired her).

One thing I don’t get, though. How did the CSO get hired? It seems obvious
that she had no qualifications or skills whatsoever for the job. How do I get
a seven-figure gig like that? (I’m kind of serious — how do these positions
get filled by people who are so fundamentally incompetent, when many, many
individuals could do a better job?)

------
noncoml
That's even worst. In most cases there are failures in multiple levels to
reach to such a catastrophic event. If this hack is because of the error of a
single employee, it means the have no safeguards or procedures setup to
prevent such failures of happening. In other words he and his CTO have failed
miserably at their job. A company should never depend on a single employee for
anything.

Also we should expecting to be see more issues in the future.

------
UnoriginalGuy
The DESIGN of their whole infrastructure was terrible for years.

I work at a school district. If someone broke into our public web server
they'd realise the entire webapp points at an WebAPI interface that will still
only let you make requests as a logged in user. Meaning it does the same thing
as the GUI, nothing less, nothing more. To get "full access" they have two
different layers they have to break through.

But even worse for the attacker, in this case full access doesn't even get you
full access. Our credit card processing, employee SSNs, and accounting system
isn't part of our main database/WebAPI system, and has IP restrictions. In
order to log into that you need username/password and 2F provided by SMS.

A completely flat design where a single breakin gives you the keys to the
kingdom is unacceptable for any organisation that holds sensitive information.
The school district's system was only improved after an external security
audit flagged our flat design as dangerous, and they were correct.

No, a single employee was definitely not responsible. This is a systemic issue
likely starting at the top. A CEO who thinks a single employee COULD even be
responsible is ignorant.

------
FLUX-YOU
>The company sent out an internal email requesting that its technical staff
fix the software, but “an individual did not ensure communication got to the
right person to manually patch the application,” Mr. Smith told the
subcommittee.

Why.. why did you just not send this person the email instead of having
someone send the email to this person? This sounds like BS.

(as an aside, managers are now going to start constantly asking "did you get
that email, bob?" to cover their asses)

------
partycoder
That's not a valid excuse.

The job of a leader is in part to identify and mitigate risks, or hire a
competent person do it for you, while still being responsible for it.

The fact that risks of these magnitude were being mitigated by a single lone
guy is a leadership issue.

Then, the problem was not only in the risk mitigation but also in the handling
of the incident as well. That's again on the leadership.

Then, the exfiltrated information is not secondary to Equifax's business. It's
the core of their business. It's not that they were Target, for instance,
where the core of their business is retail... the proper handling of that
information was Equifax's only goddamn job.

------
toomanybeersies
No it's not.

It may be the actions of a single employee that finally caused the breach to
occur, but there was a series of failures that lead up to this point. There
should have been no way that it was possible for a single employee's error to
cause such a massive failure.

------
marcell
Lone employee, meet bus.

Talk about a failure to take responsibility. Maybe it's the CEO's error to
allow a single employee to oversee a catastrophic security breach.

------
rietta
Absolute rubbish! For a company that needs to protect sensitive data, the data
breach could be traced back to the decision to put that much data on a
publicly accessible web application without any defense in depth. I stand by
what I wrote the in the week after the breach
[https://rietta.com/blog/2017/09/18/equifax-defense-in-
depth](https://rietta.com/blog/2017/09/18/equifax-defense-in-depth).

------
jamesmishra
Equifax is going to turn into a business school case study on what not to
do... and what not to say.

You could read any PDF or Kindle eBook on leadership to realize that this
headline will play very badly.

On a more technical note, how is it possible for a single person to ignore
that they needed to upgrade Apache Struts and nobody else notices or cares?

------
andreimackenzie
The CEO may blame a lone underling, and congress may blame a lone CEO, but
congress shares in the overall blame. Regulation and stiffer penalties are
needed to balance incentives so that corporations are motivated to invest in
security. As things stand, executives seem to rationalize skimping on security
as a smart business decision.

------
AdmiralAsshat
Thanks alot, Bob. Over half the country's information got stolen and it's
entirely and solely _your fault_.

~~~
cft
actually, if you only count adults with credit history, it more like 80%+

------
m8urn
Yeah, that's pretty bad blaming one employee when a single security hole on a
single server resulted in the loss of personal information for 146 million
people.

------
DiabloD3
To summarize what is actually going on (and pretty much what has been
repeatedly said in here): a lone employee's error did cause this, and that
lone employee is the CEO himself.

------
zitterbewegung
Ok even if you accept that it fell on one person that caused the error why did
disclosure take two Months ? Why were incentives or policies not in place to
correct the mistake ?

~~~
thunderrabbit
Disclosure took two months because one employee forgot to send an email about
the incident.

/s

------
kw71
Error, Singular?

One single employee developed the requirements for these errors, implemented
these errors, tested the errors, documented the errors, and signed off to ship
the errors.

What a piece of scum.

------
whoisthemachine
If you really think your security is dependent on the practices of _one
person_ , then _that is the problem._

------
egocodedinsol
What's the best way to a) structure the credit system so this doesn't happen
and b) incentives to ensure compliance with that structure?

I suspect most people here find the CEO's explanation lacking (and most people
who read the NYT, hence the headline): it's no use venting here.

I'm more curious about how to move forward, but I'm not a security expert.
Let's assume credit bureaus are here to stay: we, as a society, have decided
to lower the price of loans by reducing risk for lenders via easily available
credit histories (with all the benefits and drawbacks).

How have some companies and agencies have managed to keep data secure, and how
can we encourage other companies and agencies to do so, via carrot and/or
stick?

------
jacknews
That lone employee would be the CEO...

~~~
olliej
right?

------
geebee
I would like this CEO to tell me this one employee's pay grade. How is this
worker high enough on the org chart to be capable of this sort of impact?

A chief surgeon doesn't blame the lab tech when a patient dies, lead council
doesn't blame the paralegal for botching a death penalty case. They would
consider it a public humiliation to blame an underling, especially a
paraprofessional.

------
mathattack
_On multiple occasions, Mr. Smith referred to an “individual” in Equifax’s
technology department who had failed to heed security warnings and did not
ensure the implementation of software fixes that would have prevented the
breach._

If an employee isn't heeding a significant warnings _(plural!)_ then it sounds
like a management problem too.

------
ChrisBland
Well I mean that one person must have been paid $40 million + a year right? If
they were the sole source of protecting a multi billion dollar enterprise
surly their worth is more than that of the CEO...oh wait they didn't get paid
that...oh..never mind.

------
ungamed
Yeah it was one guy, the CEO.

------
Jayakumark
What this Guy is going to say about not encrypting DOB , SSN , name and
address etc and storing everything in plain text. is it also a single
employees fault ?

~~~
jmcgough
Equifax clearly never cared about security or protecting people, because we
aren't their clients. But yeah, some of this is just laughably bad.

------
patrickg_zill
Culture of the the company, not this one guy, is to blame.

------
tehwebguy
If this falls on one employee's head, well, the CEO might not be stoked to
when he realizes who is _actually_ responsible.

~~~
greglindahl
Former CEO. Who had to quit.

~~~
bdcravens
"Retired". With full pension, etc.

~~~
greglindahl
Firing someone for cause is difficult. Everyone likes to complain about
uncontested golden parachutes, but the lawsuits that result in the few cases
that are contested are never pretty.

------
westmeal
This reminds me of the story about the intern that wiped the DB with a single
command except it's worse.

------
bdwalter
The individual responsible is him.

------
Hasz
Pin the tail on the scapegoat, House subcommittee edition

------
plandis
With great power comes no responsibility, apparently.

------
chronid
"human error" is not a root cause.

------
cratermoon
Sure, throw that one guy under the bus, CEO.

------
tanilama
The one person is the CEO, right? Right?

------
danols
Yeah that would be the CEO then!

------
Khaine
Hiring this guy as CEO?

------
c0smic
All of these revelations read like xkcd comics. I mean really? One guy in a
corp of 9000+? Critical email threads? This situation is a joke.

