
US city rejects $5.3M ransom demand and restores encrypted files from backup - GiulioS
https://secalerts.co/article/city-knocks-back-ransom-demand-and-restores-files-from-backup/c785f0f3
======
rsync
ZFS snapshots are immutable (read-only) for normal users.

So, if you had your data stored on a cloud storage platform that created and
maintained ZFS snapshots, Mallory could gain _all of the credentials_ and
still not be able to touch your daily/weekly/monthly snapshots.

Now, if only there were a cloud backup platform that included zfs snapshots
...

~~~
VectorLock
I wonder when ransomware will get smart enough to detect backups and insinuate
themselves into them to foil the "just restore from backups" strategy.

~~~
emmp
Yes. They are often targeting IT providers of SMBs by phishing/otherwise
compromising their credentials, and deleting backups before encrypting. It
works sometimes, 2FA hygiene and secondary site redundancy are usually good
enough to protect you.

~~~
ev0lv
2FA is so overrated. You can so easily do a sim swap to get access to a
particular phone number and bypass 2FA.

~~~
parliament32
TOTP and HOTP 2FA are unbreakable and supported by literally everyone. Who
still does SMS 2FA anymore?

~~~
out_of_protocol
Lots, unfortunately. With no way to opt out

------
hartator
> "I decided to make a counter-offer using insurance proceeds in the amount of
> $400,000, which I determined to be consistent with ransoms recently paid by
> other municipalities,"

They didn't say no. It's odd there is so little recourse against things like
this.

~~~
cardamomo
You're right, they didn't outright give a hard "no," but the story gets more
interesting after New Bedford's counter offer. By attempting to negotiate, the
city bought valuable time that they used to harden and restore their systems.

~~~
solidasparagus
That didn't make sense to me. What did negotiating actually buy them? I don't
see any indication that the attackers paused the attack.

~~~
rdc12
Little to gain in restoring from backup if the same attack vector is still
open. So it was really just a bit of misdirection.

------
wmf
I don't understand what I'm reading here. Why did they offer to pay $400K to
recover ~100 PCs if they had backups? Was it so expensive to restore those
PCs?

I guess the good news is that we can now sell backups as "anti-ransomware"
cybersecurity.

~~~
jonknee
> "I decided to make a counter-offer using insurance proceeds in the amount of
> $400,000, which I determined to be consistent with ransoms recently paid by
> other municipalities," Mayor Mitchell said during a press conference (image
> above). "The attacker declined to make a counter-offer, rejecting the city's
> position outright."

Because that's what they had an insurance policy for.

~~~
secabeen
Just because they made a counter-offer doesn't mean they intended to actually
send the money to a criminal.

------
quickthrower2
Why not make it a crime to pay the ransom demand for cryptolocker attacks?
This way the flow of money to these gangsters will stop.

If you get hit with a cryptolocker and you have no backup you simply lose that
data. Or you can pay the ransom, get your data back and go to jail.

While this might seem unfair, it might stop 1000 other attacks because there
will be no money in it. It would be for the greater good.

~~~
slg
Why don't we make getting mugged illegal while we are prosecuting victims. If
you give your wallet to a mugger, you go to jail. Then no one mugs anyone
anymore, right? /s

People pay ransoms because they want back whatever is being ransomed. Making
it illegal to pay the ransom isn't going to stop that. It will just push
everything underground and make it harder to catch the ransomers.

~~~
arcticbull
Mugging involves threat of physical harm. Getting ransomeware'd doesn't, and
if cities stopped paying then this kind of attack would stop. It's more like
outlawing giving muggers your wallet when you're dressed like Iron Man.

~~~
tyree731
I'm not sure why the distinction between physical and virtual harm is
relevant. Physical harm is valuable for a person to stop because their life
has value to them, and virtual harm is valuable for an organization or person
to stop because the virtual assets being harmed have value to the organization
or person.

We could get into the monetary value of a life and whether a sufficient amount
of virtual harm, especially when that virtual harm might reasonably translate
into a life (emergency systems, insurance payouts, etc.), but I don't think we
even need to.

~~~
istjohn
The difference is the mugger gets your wallet one way or the other. The only
difference is if you survive to tell the story. The ransomware attacker is
only rewarded if the victim capitulates.

Edit: On the other hand, if the counter-factual was a mugger at an ATM
demanding you enter your PIN, that distinction doesn't exist. So maybe you're
right.

Maybe the right answer is to outlaw paying the ransom except to save human
life, but also create a federal fund to compensate victims of these attacks.

------
pilif
Do I read this right? They offered to pay $400K even though they had backups
to restore their data from? Does this mean that the restore operation cost
more than 400K or was this an incredible sample of laziness?

They'd rather pay the ransom to a criminal than invest the time it takes to
restore backups?

~~~
bluescrn
Even with good backups and a prompt recovery, you could be looking at losing a
day or so of data/work across the affected organisation. If the system had
been backing up encrypted files for days/weeks then there’s a much higher cost
to restoring from older backups

~~~
SahAssar
Making sure the systems are actually clean after getting the files decrypted
costs too though. The comparison would be (due diligence to make sure systems
are clean + ransom) vs. (restoring backups + lost work).

------
jumelles
"Restore from backup" really ought to be the default response to ransomware.
That will probably never happen though...

~~~
TallGuyShort
It's very liberating to know you can do this. You can sleep at night knowing
your confidential data is safe. From a business continuity standpoint, worst
case scenario you get a new machine, restore, and you're back in business.
When I trade-in my work laptop for a new one I love the feeling of handing
them my old one, picking up my new one, and not looking back. Works just as
well for ransomware, other viruses, ruined hard-drives, etc.

~~~
reustle
> It's very liberating to know you can do this. You can sleep at night knowing
> your confidential data is safe

If ransomware was able to infect the machine, then I doubt the data was safe
either

~~~
hobs
Exfiltrating is insanely more difficult especially of "the useful stuff" \-
and even then you have to sell it or use it to embarrass your target, it's
pretty difficult unless you target people ahead of time - easier to just
interrupt business.

------
H8crilA
See also Matt Levine's take on ransoms (scroll down):

[https://www.bloomberg.com/opinion/articles/2019-08-27/the-
li...](https://www.bloomberg.com/opinion/articles/2019-08-27/the-libor-change-
is-coming)

And the linked article:

[https://www.propublica.org/article/the-extortion-economy-
how...](https://www.propublica.org/article/the-extortion-economy-how-
insurance-companies-are-fueling-a-rise-in-ransomware-attacks)

Tl;dr: perverse incentives, paying some ransoms is in the interest of cyber
crime insurers, as it expands the cyber crime insurance market. Also there's
more ransomware crime now that the word is out that insurers do pay out
ransoms.

------
Gustomaximus
I wonder if a law that you can't pay ransom money for files should be passed.

This should work to take any government and larger companies off the target of
these groups as they are likely to obey these laws. Kinda like the we dont
negotiate with terrorists approach.

~~~
jasonhansel
Yep. I'm actually surprised it isn't already illegal, given that the money is
likely being used to fund organized crime, rogue states, etc.

~~~
blueadept111
Not only that, there's money to be made by insiders leaking (or even creating)
security holes for the hackers to exploit and then getting a kickback.

------
avivdeg
NetApp has a solution for all of that: Cloud Volumes ONTAP is a virtualized
storage platform running on AWS, Azure, and Google,it consumes native cloud
resources and it provides NFS/CIFS/iSCSI.

Cloud Volumes ONTAP has the best snapshots out there, immutable and without
any resource penalty. you can take any size snapshots (or restore) in seconds.
you can also create clones out of these snapshots, so you can check if that
data been affected or not, again in seconds. Adding to that Cloud Manager's
Ransomeware protection that blocks known Ransomware files.

In short- this is the best solution out there for any hybrid/ cloud and it can
actually be cheaper than free, if you have enough capacity, due to all of its
storage efficiencies like dedup, compressions and compaction, with auto-
tiering of unused blocks to the checper object storage,.

------
iandinwoodie
So they offered $400,000 for restoring what was most likely a week of lost
work for 158 employees. That works out to $2,531.65 per employee for that
week. Is the average salary of the compromised employee $131,645.57
($2,531.65*52)? It sounds like the town was just willing to throw a lump of
cash at the attacker based on what other victims had paid with no regard to
what the lost work was actually worth. I know the attacker did not accept and
it allowed them to strengthen their security and etc., but it bothers me that
the attacker could have just gotten a $400,000 payout. It’s almost as if the
moral of the story is “even if a town government had backup system in place,
you can still pull in a few years of income with a ransomware attack!”

~~~
Zenbit_UX
You would lose more than a week of work if your entire team is no longer doing
this week's work to make up for last week's.

Let's say they instead of doing this week's work, they do last week's, next
week they're still a week behind and you have to start paying 100+ government
employees overtime in order to regain the catch up on the remaining week. This
could take a while to catch up and be incredibly expensive.

The $400,000 wasn't taxpayer money it was their insurance companies offer to
make this go away.

~~~
dsfyu404ed
Local government work is mostly cyclical and the "customer" is captive. It's
not the end of the world if they bill someone a day late or have to re-add the
new librarian to the payroll system. It's not like people can up and switch to
a new government because they don't like the quality of service.

Based on having grown up in the region and having, um, "connections" to state
and local government I think it is highly likely that their desire to not
throw out work was based around reasoning around how their public image would
be affected if a batch of fines/taxes/fees/bills had to be waived. Cutting
people (collectively, waiving something on a case by case basis is fine) a
break because the government screwed up is kind of a non-starter because of
the possibility of setting a precedent.

------
tastroder
Could somebody contextualize how targeted this particular attack really was?

According to random Google result #1: [https://www.2-spyware.com/remove-ryuk-
ransomware.html#qm-h2-...](https://www.2-spyware.com/remove-ryuk-
ransomware.html#qm-h2-4) the specific malware distribution is unclear but
likely involves email attachments and/or vulnerable and exposed RDP.

While reporting on ransomware cases often sounds like targeted APTs, more
often than not the details in these stories read like "we didn't bother to pay
enough admins to actually patch and secure our systems" and "we didn't train
our users not to click on every random attachment".

------
msmerberry
It sounds like they played every card right, and that's probably not by
accident. I'd be fascinated to see a case study made out of this - especially
with a breakdown of how they performed against their CSIRP - because there's
no way they didn't tabletop that plan at some point, and it paid off in the
end. Sort of like insurance, eh?

------
shameshame
The city was extremely lucky the attackers played their hand too early. If
servers had been encrypted it could have a lot worse.

------
peter_retief
Restore from backup isn't as easy as it sounds, well done to a disaster
recovery plan that actually works

------
olliej
Honestly this success might encourage city governments to make sure their
backup systems work better.

The counter offer of 400k was presumably the break even point for the cost of
losing X days of work (depending on what was lost that could involve manually
recovering things like tax payments, billing, tickets, etc)

------
auslander
Great read: [https://www.wired.com/story/notpetya-cyberattack-ukraine-
rus...](https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-
crashed-the-world/)

------
zelly
The real solution is for them to hire out for their ops. No reason these rag
tag city governments should store their stuff on-prem. Until then these okie
dokie city bureaucrats will keep getting pwned.

~~~
olliej
Ok, but city governments are frequently very large, which means bringing in an
outside contractor will be very expensive. Remember that every level of
contractor adds additional overhead in the form of city funds being shifted
into company profits.

For a small city, then yeah outsourcing might make sense. For a large one?
Probably not.

The maths for cost is very simple: every person the contractor brings in is
being paid for by the government, but now in addition to paying those
employees, you also have to pay for a separate set of managers, and
executives, and finally the business markup for profit.

If your org is large enough there is no way outsourcing is cheaper - it may be
“easier”, but it inherently must cost more.

~~~
dsfyu404ed
New Bedford city government has no reason to be that large though. Boston and
some of its most urban suburbs maybe, but certainly not New Bedford or any of
the other small cities in MA.

~~~
jmclnx
I have to hand it to New Bedford, it is a poor City and they know how to run
an IT dept, esp. compared to what larger cities do. So they should give their
IT dept a nice bonus and maybe educate other gov. entities.

------
CryptoPunk
Cryptocurrency is hardening IT infrastructure.

------
ashelmire
Paying hundreds of thousands to restore 158 backed up machines would be
absurd. Surely a week of work for those 158 employees isn't worth that.
They've gotta put a financial analyst on these issues, and not leave it up to
panicking executives.

And look at how greedy the attacker was. Missed out on a 400k payday.

~~~
raxxorrax
I would like to see your calculation for that.

My current company is much smaller in size and had 2 ransom attacks that I saw
happening (in ~2 years). We use nimble as a backup and it is quite expensive
(hundreds of thousands actually, depending on infrastructure) but worked
flawlessly to restore all our systems in a short time. You will loose a few
hours of work though. Attack vectors are the usual: mail with infected
attachment and users with too many rights on documents. The attackers even
know names of people working at the company and use correct mail signatures
(not actual mail signatures, just the colorful stuff you put at the end of
every business mail).

If production stopped for a week, the contractual penalties alone would
probably eat up sums like that.

------
hello_tyler
Why is a backup system so hard for people (especially businesses) to
understand????

~~~
stanski
Only other people need it.

------
aitchnyu
> Because the attack happened at night, most of the city's systems were turned
> off and the ransomware was unable to spread.

I used to complain Indian banks used to allow fund transfers only during
working days and office hours. Now I see why it was useful.

------
elchin
Why isn't it mandatory for government orgs to use cloud?

~~~
Shivetya
I do not agree they should use "the cloud" but what is all too common is
local, city, state, and even federal governments, love to enforce regulations
and laws upon others they do not obey themselves.

the first rule of laws and regulations should be, government must adhere to
the letter if they are to be enforced upon others seeking penalties if
compliance is not met.

it is really difficult to hold government agencies responsible, it doesn't
even have to be computer security, just look at the number of cities in the US
who violate lead levels in water. even better, in the US, a lot of what
happens can be protected from suits by sovereign immunity.

So keep it simple, what is required of private organizations and people is the
minimum standard that a government agency must meet

------
gesman
Key: backup

Do you ... backup?

------
mxuribe
Kudos to the city of New Bedford!

------
auslander
No Windows no cry :) Seriously, how can one compromised machine (inevitable)
infect all others? Isn't there host firewalls on every machine?

~~~
SmellyGeekBoy
If you believe that ransomware doesn't exist for macOS and Linux I'm afraid
you've been misinformed.

~~~
auslander
Well, there is no SMB v1 on either, main propagation bug. Next propagation
tools are SSH bruteforce, subnet scan for vuln apps, Domain admin creds stored
locally and sniffing NTLM hashes from network interface. Again only SSH and
app vulns are viable in non-windows.

I was asked once to do reference design of Windows on AWS. After I learned how
many ports has to be open for every machine and all of them had to be in same
network as Domain Controller, I quit my job.

> If you believe that ransomware doesn't exist for macOS

Example? I have not seen any.

~~~
lgl
Absence of evidence != Evidence of absence

~~~
auslander
but is a pretty good guideline :) If Windows got ransomed every time and MacOS
is never, that is interesting, right?

Windows holds _perceived_ monopoly on company-wide identity control. There are
Mac solutions: [https://www.jamf.com/products/jamf-
pro/deployment/](https://www.jamf.com/products/jamf-pro/deployment/)

