
Equifax Releases Details on Cybersecurity Incident, Announces Personnel Changes - diggernet
https://investor.equifax.com/news-and-events/news/2017/09-15-2017-224018832
======
azurezyq
If a hacker can use just one CVE to break into your system and do a database
dump (or equivalent), the system is architecturally wrongly designed and only
being protected by a single layer of security. Which means, any one from the
inside can access pretty much the info as the hacker, which is horrible.

For example, are the items in DB encrypted? Are database backups encrypted?
Are different items encrypted using different keys? I don't think EFX did it
right.

~~~
sillysaurus3
_If a hacker can use just one CVE to break into your system and do a database
dump (or equivalent), the system is architecturally wrongly designed and only
being protected by a single layer of security. Which means, any one from the
inside can access pretty much the info as the hacker, which is horrible._

This is how it works at almost every company. Call it horrible if you want,
but everyone stops short of describing _in detail_ the exact architectural
replacement. And when someone does, it inevitably has flaws that make it
impossible to meet business goals.

It's too easy to handwave this away as "well, get a better architecture."
Everyone knows that; the hard part is, what architecture? How does your
architecture work in practice? Encrypting items won't help when the keys need
to be stored in the same place to access the data. The attackers will just go
after that box instead of the DB.

You can guard it more carefully, but the point is that _some_ box _somewhere_
needs access to a substantial amount of the data at any given time. It's the
nature of a credit bureau.

~~~
js2
In multiple threads now I've seen you express this sentiment. I just don't buy
it. Vanguard hasn't had a leak like this. USAA hasn't had a leak like this.
ETrade, Fidelity, etc, no leaks like this of which I'm aware (correct me if
I'm wrong). These are big plodding institutions too. No leaks like this from
Google nor Facebook nor Amazon. Maybe you're arguing it's just a matter of
time, but I disagreee.

Equifax had a trivially easy architecture to break.

Minimally, the data should be on a separate set of servers protected by an API
with rate limiting, and that API should be implemented on top of a different
framework than the front-end servers to minimize the chance that a single flaw
would open up the entire set of data. This isn't rocket science. Equifax is a
multi-billion dollar company. They could and should have done better.

Please stop defending them?

Disclosure: I work for Yahoo (er, Verizon) since late 2013. I don't know the
details of Yahoo's data leaks, but it was not as trivial as getting RCE on
Yahoo's public facing servers.

~~~
sillysaurus3
When two people are clearly intelligent but disagree, the reason is almost
always a different set of experiences. My experience differs from yours; if I
could Vulcan my brain to yours, you'd see why I feel this way. You might still
disagree, but it would be far less clear cut.

Now, I've seen nearly a hundred codebases and they all looked pretty much like
Equifax. I don't even need to see Equifax to know the kind of disaster their
codebase facilitated.

Equifax's codebase is your code, and my code, and Bob's code. The only way to
make this situation better is to talk openly about it. And the best way to do
that is to actively defend the unpopular side in order to call attention to
uncomfortable truths. It's a calculated strategy.

Just because you worked at a place with good security hygiene, it makes no
difference to the world at large. Everybody else looks like Equifax. We need
to (a) acknowledge that and (b) actively seek out solutions. That's what's
happening here.

Ask yourself: "Why do I disagree when I've seen one or two enterprise
codebases?" Those big, plodding institutions you mention are similarly
vulnerable. Everyone is.

Why aren't data breaches happening almost daily? Because it's illegal. It's no
coincidence that companies with public bug bounties are among the hardest
targets to hit. When you make it not-illegal to attack you, surprise: you find
the vulns.

A good question is, "is it illegal for a white hat to attack my business?" If
the answer is yes, seek to change that. It's a low effort thing that will
increase your security posture. But it's no substitute for $40k pentests.

~~~
js2
> Everybody else looks like Equifax.

I don't doubt that there are other Equifaxes out there, nor do I doubt that
there will be more leaks. But this wasn't a simple matter of a horrible code
base. Heck, the leak wasn't even due to Equifax code, it was due to third
party code. Other companies of Equifax's size apparently do better. Equifax
should have done better, and I'll bet if the penalties for losing this data
were more severe, would have done better.

~~~
sillysaurus3
(replying to your "What if $40k for a pentest becomes a rubber stamp?"
question, which was an important observation)

Mm, it can become that. It's important to get a pentest from a reputable firm.
At the one I worked at, it was a point of pride to come up with clever
attacks. We'd make a sport of it and try to one-up each other. But it's easy
to imagine some other pentest firm just going through the motions, running
Nessus and calling it a day.

Unfortunately, that's where money really does make a difference. $40k is at
the low end of how absurdly expensive pentests are. Yet they make all the
difference -- one app that I pentested was at a trading firm. I took that app
from "typing ' leads to SQLi in their Node backend," and "there was SQLi in
their Java backend if you were clever" to "you'd have to be more clever than
me if you want to breach this app."

 _Most_ attacks come from unsophisticated adversaries. This implies that
raising the bar higher than these guys gets you 80% of the way home. The other
20% is much harder, but it's doable if your company has a culture of caring
about security.

~~~
js2
Thanks for replying and sorry for moving my comment.

~~~
sillysaurus3
It's all good. Just happy the world is starting to care about security.

------
yebyen
Wow, MarketWatch is right. They are trying to erase Susan Mauldin (the former
CISO), she is not even mentioned in this article. Does anyone with Google
juice have a way to recover the interviews that are referenced in [1]?

I watched them before they came down on Sept 10, and they were eye opening. I
can say with certainty that the transcripts are not complete, because I
remember "resistance to cloud is futile" and other such gems which are nowhere
to be found in the partial transcripts that you can still find on the linked
archive.is pages.

[1]: [https://hollywoodlanews.com/equifax-chief-security-
officer/](https://hollywoodlanews.com/equifax-chief-security-officer/)

~~~
hn_throwaway_99
This article, [http://www.marketwatch.com/story/equifax-ceo-hired-a-
music-m...](http://www.marketwatch.com/story/equifax-ceo-hired-a-music-major-
as-the-companys-chief-security-officer-2017-09-15) , is brutally ridiculous
and unfair. The major point seems to be "Hah, they hired a stupid music major
to be their Chief Security Officer!"

Some of the best software developers I've known were music majors. I know
nothing about Susan Mauldin, including any of her other qualifications or lack
thereof, but implying someone is only qualified to do what their college major
was tells me this author is a complete idiot.

~~~
yebyen
That is not my purpose. I have no idea what her qualifications are, I saw the
LinkedIn page and it contained almost no information. That is hardly relevant.

I personally believe that she was backed into a corner, based on the
interviews I watched from some time before the breach (that I can't show you,
and I won't link to the transcripts because I know they are incomplete and by
far less impactful.)

She sounded to me, exactly like a person who was given a budget that was
effectively no budget, and then put into this role because people with a
larger stake were sure that she would comply when they said "this is your
budget, and not a penny more." I know exactly what that is like, because I
have been made CIO and put in that position before. (I'm sure she was better
paid than I was...)

There is absolutely 100% a coverup going on. I am so bummed that I did not
save the interviews when I first watched them. You would agree too, if I could
show you.

Edit:

Here is the transcript, but with the disclaimer that it is 100% not the full
transcript. Although it is in her own words. Make your own judgement about her
qualifications please.

[https://archive.is/6M8mg](https://archive.is/6M8mg)

~~~
jlgaddis
I found it for you... gimme a few minutes...

\-----

 _Edit_ : Well, I found _one_ video (11m30s), were there more?

\-----

 _Edit_ :

From a quick check, this seems to be the same interview that was transcribed,
though I'm not sure if it's the _entire_ interview or even the one you were
looking for:

80MB, M4V:
[http://evilrouters.net/media/susan_mauldin_cazena_interview_...](http://evilrouters.net/media/susan_mauldin_cazena_interview_201603.m4v)

I'm uploading it to YouTube as well but I wouldn't be surprised if it gets
taken down at some point.

~~~
yebyen
There was a second interview with Cazena that looked a lot like the first
interview, but the topic was different. It looked like they were possibly
recorded on the same day.

Edit: Bless you for doing this, however you found it.

~~~
jlgaddis
You're very welcome. Is that the video you were wanting?

~~~
yebyen
Someone has let Hollywood LA News know that you found it, and it's been
updated and reposted to their site now.

They also gave you credit for finding it! ;)

~~~
jlgaddis
Yeah, I found their article from links here so I reached out to them via email
right after posting it here. According to a later email, they independently
found my HN above post a bit later.

Looks like the embedded version on their page is missing about 1m30s of the
interview, though.

~~~
yebyen
Srsly?

------
guelo
The CVE and the patch came out on March 7. Exploits were already being
detected in the wild at the time by perimeter security vendors. See
[http://blog.talosintelligence.com/2017/03/apache-0-day-
explo...](http://blog.talosintelligence.com/2017/03/apache-0-day-
exploited.html)

Equifax "believes" that the hackers got in on May 13. They had some kind of
intrusion detection system that finally detected the intrusion on July 29. 5
months after the "Critical" CVE alert went out. During that time security
vendors were adding firewall rules to stop the attack. But apparently Equifax
didn't have any other security in front of the Struts server.

That just seems like unconscionable incompetence and malpractice for such a
high value target.

~~~
Flammy
> They had some kind of intrusion detection system that finally detected the
> intrusion on July 29.

If they had some sort of intrusion detection system, I gotta think they would
have picked up on this in days, not more than 2 months later.

I feel like this is one of those situations where a dev is saying something
like "where is all of the CPU on this database going?" or "Why is the network
connection so slow?" and digs into it and finds some strange behavior...

------
noncoml
\- In addition, credit card numbers for approximately 209,000 U.S. consumers,
and certain dispute documents with personal identifying information for
approximately 182,000 U.S. consumers, were accessed.

This looks really bad. I don't know what's the point of letting them operate
anymore. They utterly failed on the single thing they were supposed to do.

~~~
harryh
The business they are in is supplying information to creditors about the
credit-worthiness of potential customers. AFAIK they have succeeded at this.

I don't want to downplay their security failure. It's obviously really bad,
but saying that security was the "single thing they were supposed to do" is
false.

~~~
noncoml
You are right, it was a hyperbole, but I still don’t see why they should be
allowed to continue operating. Would you let a bank continue to operate after
they have lost all their money?

Never mind, I guess too big to fail applies here as well.

------
rmrfrmrf
Actually looking at the CVE made my stomach drop. That is a horrible horrible
bug. Getting access to the shell while under the web app user's environment
potentially means that all secrets were available either in the app server
administration, the user environment, or in a readable file on the system.
Effing yikes.

Also knowing how JNDI usually gets configured on app servers (sometimes with
credentials and all) would have made recon ridiculously easy.

I think some have alluded to it, but what some people in the comments don't
seem to understand is that RCE is an "all bets are off" type of situation.
DEFCON 1 to be sure. Prevention is really the only good answer.

------
simpfai
Failure to apply a patch for a two month old bug led to this entire nightmare
scenario. What are some best practices to ensure that ones dependencies are
always up to date?

-asking as a relatively inexperienced dev

~~~
ccrush
Don't hire someone who majored in Music Composition as your Chief Security
Officer is key.

~~~
jsemrau
Based on her work experience she seemed qualified.

~~~
DonHopkins
You mean her "Head of Security when Equifax was Hacked" work experience?

~~~
AlexCoventry
I heard she was in a similar role at First Data, prior to the Equifax gig.

------
niij
They were able to use a subdomain for their announcement, but not for their
Breach service which asks for your SSN.

------
qaq
For everyone patting themselves on the back for how much better they are at
securing their data realize that for way over 80% of incidents the attack
vector is email and social engineering. Look at Red team exercises of
competent teams and try to honestly answer the question would have they
succeed with that tactic at your company. So yes having much better practices
vs what we see here is very important but will not really help much if you are
being targeted by a competent adversary.

------
blacksqr
Gotta admire the artful way they gave the appearance of disclosure while
avoiding answering the most damning question: why did it take so long for them
to patch Struts?

"The particular vulnerability in Apache Struts was identified and disclosed by
U.S. CERT in early March 2017."

"Equifax's Security organization was aware of this vulnerability at that time,
and took efforts to identify and to patch any vulnerable systems in the
company's IT infrastructure."

?????????!!!!!!??????

"While Equifax fully understands the intense focus on patching efforts, the
company's review of the facts is still ongoing."

~~~
freeone3000
It's struts, and the CVE fix was pushed to a newer version than the one they
were using. That's an intense app rebuild, akin to moving mongo versions.

~~~
krirken
> That's an intense app rebuild

That is not even close to an excuse. A remote code execution vulnerability has
the potential to destroy your whole company.

~~~
dannyw
I hope Equifax will be learning from this, but can you tell your CEO that your
core business must be shut down for 3 weeks as you upgrade and rebuild the
system?

~~~
krirken
Yes, the risk is much higher than the cost. From the article:

> The company's internal review of the incident continued. Upon discovering a
> vulnerability in the Apache Struts web application framework as the initial
> attack vector, Equifax patched the affected web application before bringing
> it back online.

That bullet point lies between the "July 30th" and "August 2nd" bullet points.
Based on that timeline, the vulnerability took days to patch.

------
hyperbole
Having a remote execution exploit shouldn't mean keys to the kingdom. I find
it hard to believe that this company whose whole business is electronic didn't
adapt it's technology stack to remedy this type of attack limiting the scope
of a data leak. I wouldn't be surprised if struts has another exploit of
similar magnitude, what then?

They might as well be running their business on a cluster of tomcat servers
sitting atop sqlite.

Hopefully they don't recover from this - they should not have the data they
posses if they cannot mitigate risk.

------
heisenbit
They are really in deep trouble and need to manage the story carefully. If
people on a broad basis start freezing their credit history (one of the most
effective means to protect against abuse) their and their competitors ability
to sell data to other parties will suffer (see also
[https://wolfstreet.com/2017/09/15/equifax-
sacks-2-executives...](https://wolfstreet.com/2017/09/15/equifax-
sacks-2-executives-gets-devious-to-stop-you-from-getting-a-credit-freeze/) )

------
21
What about storing sensitive data in something like a HSM, which rate-limits
access rate, so you could only lets say access 10000 records per day.

Yes, developing against such a system might be annoying (think about updating
a new piece of data for all records).

But it feels to me that we need a way to rate-limit access to sensitive data,
to prevent wholesale dump in a short time. But you still need other systems in
place, to prevent a hacker lurking around for months until it gets all the
data.

~~~
dannyw
Or, you know, monitoring systems.

"Hey... Splunk tells me there's been a 34x increase in record queries. Page
page"

------
leesalminen
I just had a scamming debt collector call me about a debt I already paid off
to another collection company. They had all the right information. I suspect
it has to do with the hack. Watch out for First Equity Alliance.

------
dingo_bat
Do they plan to change their Chief Security Officer who is a MA in Music
Composition?

