
Baltimore Blew Off Ransomware Demand Only to Find Data Had Never Been Backed Up - miles
https://www.techdirt.com/articles/20191004/19564743128/city-baltimore-blew-off-76000-ransomware-demand-only-to-find-out-bunch-data-had-never-been-backed-up.shtml
======
blakesterz
I remember hearing lots of back and forth on whether they actually used
EternalBlue or not, I'm still not sure if that's settled or not, is it?

"Given the fact that $6 million has already been pulled from parks and public
utilities funds to "harden" city systems, the $76,000 demand now seems like a
bargain."

That doesn't seem fair at all. They'd still have to harden everything and it
would still likely cost millions. From the looks of it they'd at least have
their stuff back, probably, but they'd still need to put all the same
time/money/work in, wouldn't they?

~~~
joncrane
First of all, it's a terrible article, it's full of typos; why aren't we
linking to the source article on ArsTecnica?

Secondly, Later on in the article is the more relevant

>The city figures it will cost $18 million to recover from a rejected $76,000
ransom demand. I guess if you're going to play chicken with extortionists, you
might want to make sure your backup plans at least meet min spec.

Either way, it's a terrible article and I'm on the fence about flagging it.

~~~
totaldude87
on the contrary, i kinda liked this sarcastic writing..

>>The person in charge of the city's systems was Frank Johnson, who went on
leave (presumably permanently) after a post-attack audit found the IT director
hadn't done much IT directing.

------
jrochkind1
The techdirt article author suggest they should have just paid the ransom.

Even knowing that paying the ransom is no _guarantee_ you actually recover,
and that you have to then harden your system anyway after that (which you
should have been doing all along)... I tend to agree.

I am a Baltimore resident, and I've also had that opinion since the day after
the ransomware attack was announced -- just pay the ransom. (In part because
from what I experience of Baltimore City government, i was pretty sure the
backups and recovery/continuity plans were going to be basically nonexistent).

But on previous HN threads, it was a _very_ unpopular opinion, in the articles
right after the attacks were announced, very few in the comments threads
seemed to think it was acceptable or a good idea to pay the ransom.

I'm curious if that remains true?

~~~
herenorthere
I work with a cryto exchange, and fwiw, can tell you that most businesses
would agree with you. They pay the ransom. And they use common exchanges like
Coinbase, Binance, and Bittrex to do so.

The only way not-paying the ransom is beneficial (aside from having proper
backup systems in place obviously) is if there was a large public sentiment
shift promoting NOT paying ransoms. If society at large came together and
decided the majority of entities are NOT going to pay, then ransomware viruses
would be less profitable and thus not as favorable projects for
hackers/scammers.

But that would take a lot of effort, organization, and favorable circumstances
to have everyone do that simultaneously going forward. And there would be
casualties at the beginning before the public sentiment cemented itself in the
collective conscious.

But yeah, the only way ransomware will stop is if it stops becoming
profitable: which means companies either need to have proper OpSec and backups
(so they have no need of paying), or collectively agree that no one will pay
ransomware attacks.

Seems like a pipe dream that we'd ever get to that point though. So I imagine
companies will continue to fork over the ransoms.

Edit: this got me thinking, say for example, the US government outlawed paying
the ransom to hackers. And they could somehow enforce this law effectively.
Wouldn't that pretty much stop ransomware attacks in the US? or where ever a
law like that could be effectively enforced?

~~~
jrochkind1
I think society has come together and decided we're all going to pretend we're
not paying and talk publicly about how nobody should ever pay, while mostly
everyone is paying.

Society also seems to have come together and decided we'd rather save money
than spend it on effective backups and security.

------
whack
You never know who's swimming naked until the tide goes out.

Would be great if every organization conducted targeted attacks in order to
probe the reliability of their assumed safeguards. If your system hasn't been
tested by someone who has a real incentive to break it, you have no idea if it
is really as secure as you think it is.

~~~
dangerboysteve
We do this. Every place I have worked at I've had the senior admin explain and
show the backup plans for servers and desktops and demonstrate various restore
requests. I have caught a few with their pants down.

Another important thing is fine granular permissions on network shares.

~~~
wil421
We have our own pen testing team in addition to the 3rd parties who also do
it. There’s a big DR test this weekend for our 2 largest data centers as well
to make sure backups and fail overs happen. Plus multiple networks for
different products that’s completely different from the regular employee
network. You even have to sign in if you plug into our Ethernet.

------
duxup
For a while I worked with some products that were popular with municipalities,
and school districts in a previous career. The scale of under funding, and
incompetent leadership in those roles is staggering, and sadly not surprising
to me.

Internal politics are always an issue of course, especially at schools where a
handful of luddite teachers / administrators can kill good idea, but that's
also an issue of IT leadership too.

I was glad to move away from those products.

~~~
covercash
I briefly worked for a school district that had 8000 Macs to support. They
fired the entire technical team (for reasons unknown) except for me and then
proceeded to hire people who had never used a Mac in their life. The woman in
charge of the department was like Lorraine Bracco’s character in Hackers -
extremely overpaid, could barely figure out how to work a computer, and used
God as her password. I’m still disgusted when I think about how corrupt and
incompetent that district was and likely still is. A few of the fired guys
wound up with a nice settlement, I’m sure the webcam recordings of the
superintendent exchanging teaching positions for “favors” probably worked to
their advantage.

~~~
listenallyall
Reasons unknown? For starters, they were conspiring to covertly spy on their
employer.

~~~
covercash
Nope, the district had no idea they were doing that until after the firing. I
did hear the actual reason was because they were considering joining a union,
but I didn’t stick around long enough to dig any deeper.

~~~
listenallyall
Conspiring criminals tend to do lots of conspiracy-ish and criminal-ly things.
If they secretly installed webcams, they likely also were reading users' email
and viewing confidential documents like performance reviews, budgets, salaries
and student test scores. Their firing was likely highly justified.

Since you seem to know all kinds of facts that discredit the school district,
the superintendent, and the IT supervisor, yet "didn't stick around" to absorb
info that might look bad for the group of employees, makes you a narrator who
is difficult to believe is being objective and non-biased.

------
fredley
If you don't test your backups, they're not backups. But I guess in this case
if you don't have backups at all, you also don't have backups... 1 is 0, 2 is
1, etc...

~~~
decasia
Question:

OK, so I know how to test — manually — that a few randomly chosen files are
correctly backed up in my backup systems.

But what if there are larger classes of systemic error in the process I
haven't thought of testing? What if some particular file type or directory
tree has vanished from the backups, but since it isn't in my manual testing
process, I never catch it? Are there best practices for validating that whole
directory trees are correctly backed up across the board? Or any form of
automated testing for backups (but, presumably, separate from the software
that does the backup process)?

~~~
rhinoceraptor
I would just use ZFS snapshots and replication. They're read-only, and they
only use as much space as the difference between the current state and the
snapshot.

Plus, you'll have solid data integrity instead of relying on buggy firmware in
your RAID controller.

~~~
tracker1
And there's no way to compress/eliminate snapshots?

Also, redundancy and snapshots are _not_ a backup.

------
DoofusOfDeath
I have to wonder if, in macro economic terms, we'd be better off spending
whatever is necessary to successfully investigate and prosecute
scammers/telemarketers/ransomers.

~~~
bluGill
We are best off not paying them regardless: if there is no payout the bad
actors will give up the attack. If you are in a group that pays out target
them. It doesn't pay to target someone who has good backups (though it might
not be worth checking if infecting everyone is easy)

~~~
crankylinuxuser
>if there is no payout the bad actors will give up the attack

I'd argue that isn't true. Ransomware is a form of terror in the form of
"losing everything". And that threat of impending loss combined with 'backups
are hard wahhhh' and the fact that no Corp I know backs up user machines...

As a numbers game, it pays. Well. And even if it didn't, some still hack for
the "lulz".

~~~
wpietri
Definitely. Look at all the people pursuing "make money fast" scams. Some
people will happily do anything they think might make them a buck. Not paying
out will help, of course, but it doesn't eliminate the problem.

~~~
bluGill
When there is no money in it nobody will be developing the exploits that make
it easy. There is a large difference between buying a recent vulnerability and
exploiting it vs finding a vulnerability.

------
Ancalagon
The irony in this is absolutely hilarious. I remember we were all praising the
city for having such forethought and for doing their due diligence (unlike
most of these cases) and keeping backups of their data.

~~~
coldcode
I wonder how the IT director was hired and who monitored what the department
was doing. I worked at a place that hired a MS system support person but no on
one knew anything about it and he seemed to know more than they did. He was
fired a couple months later as he actually knew nothing and spent all his time
making support calls trying to get someone to tell him what to do. He was on
the phone for hours every day. Then when he was fired no one changed passwords
so he logged into the DNS server and randomized all the IPs, leaving the whole
company without networking. Hopefully not the same person...

~~~
paul7986
Baltimore has been through a good amount of IT Directors who were caught up in
scandals/corruption. Not surprising as in the past ten years we have had two
disgraced/corrupt mayors abruptly leave office.

~~~
electriclove
What is the end state for Baltimore?

~~~
maximente
as a city? probably status quo: continued grift and unaccountability as a once
proud manufacturing town decays.

it's somewhat attractive price wise simply because it's a very affordable
urban east coast city. there's a vibrant/edgy art scene (to include drama,
etc.) that's attractive to younger artists, sort of like what detroit is going
through (as i understand it)

------
totaldude87
>>The person in charge of the city's systems was Frank Johnson, who went on
leave (presumably permanently) after a post-attack audit found the IT director
hadn't done much IT directing.

love this writing man :) :)

Dont know why would a city's chief digital officer go backup free?!!! even at
worst , backblaze would have helped :|

------
rsync
I guess a failure to do backups of any kind is beyond any solution we could
offer, but if one could take even a _single step forward beyond nothing_ , a
read-only (immutable) offsite destination is well worth your consideration.[1]

Most end users looking for "ransomware protection" probably just want to drag
and drop some files, like Dropbox, which is why a simple SFTP (filezilla)
solution is nice, but of course you could point any old thing[2] if you were
more sophisticated ...

[1]
[https://www.rsync.net/products/ransomware.html](https://www.rsync.net/products/ransomware.html)

[2] borg, restic, rclone, git-annex ... rsync ...

~~~
vultour
Surprised someone would use or recommend FileZilla after their numerous adware
issues.

~~~
rsync
I'm just thinking of simple SFTP clients ... WinSCP ? psftp.exe ?

Whatever works for you.

------
ping_pong
To be fair, the CIO was only on the job for 1.5 years when the ransomware hit.
It's not known whether or not he had the budget for reorganizing all of the
data, etc. Maybe it was on his list of priorities, but it didn't have the
funding, etc, and it was pushed off until the next year. I'm not absolving him
of this because backups are a first order object when you're dealing with IT,
but it could be part of the reason why it wasn't implemented yet. This is the
reality of working in a bureaucracy like city government.

------
swebs
How do we increase IT competence in the public sector? There's no reason
important servers should be running on Windows machines with "backups"
consisting of making a copy of a file on the same hard drive as the original.

~~~
sailfast
Apply for a job. Run towards the fire. Help fix it.

~~~
brianlweiner
I applied for the Baltimore City director of digital operations position. I
interviewed with Frank Johnson and a variety of other city department heads.

I got the impression they were a group of people that broadly understood their
problems and were finding it very difficult to steer the city towards good
solutions.

Although Mr. Johnson was the City's Director of IT, realistically he had very
nominal oversight over many of the city's actual IT departments - which were
spread across a series of departments with their own employees, budgets, and
resources. This was something he had been actively trying to improve but with
limited progress.

I don't know what the best solution is - their hiring process is flawed and
it's difficult to remove or replace problem employees. The budget is about 60%
of what it probably needs to be and there's no path towards improving it.

~~~
sailfast
That sounds right. Good for you for going out for it. Hiring / firing in
government is certainly not helping the situation, and budget fights are
always a struggle.

------
mdip
"That can't be real?"

Really? I'll never forget the description given to me by a close friend who
had left a "Government IT" job for the private sector: (1) You have standards
and practices like everywhere, but forget one and you're explaining yourself
to a judge instead of a boss and (2) there's never enough money for doing all
of the things required to meet regulations let alone make things better which
is why you'll see silly things you haven't seen on banking websites in a
decade still prominent as "security features" on state treasury websites.

I found the story about "important data on a desktop drive", and the shock it
caused, surprising. Maybe my past life (a decade ago in infrastructure) was
unique, but I specifically recall an incident where I was called down to make
an _old_ IBM NetVista (mind you, Lenovo had owned that line for a while at
this point) boot up[0]. I noticed some numbers on a printed label and a boot
error about the CMOS battery, realized it was drive telemetry, realized that
nobody who was looking at the problem had ever heard of plugging in those
values (or probably had touched an IDE controller -- server guys -- and it was
ancient technology).

The rest of the story I might not have completely correct as parts of it are
assembled third-hand, but this desktop was located in our data center, hooked
up to a modem (2400bps) and it handled submitting charges to another carrier
to the tune of "a few layoffs" for every week it wasn't functional.

How does this happen? Well, the company went bankrupt and emerged, then was
purchased by another company. During that time, a large part of our operations
was moved from one state to another, hardware and all (but mostly not the
people). This predates all of that, of course. At some point, a NetVista was
put in place to test setting up an automated process for billing this carrier
-- something carrier imposed (must have been one of the big guys). The
developer who set up the test system was successful ... on his final week of
work before being laid off. A few months later, the carrier continued working
the migration plan and switched things over, and after a short delay,
accounting rang the alarm bells. A busy developer stepped in, found the
offending system was connected to test and re-configured it to point to prod.
Everyone went on their day. And hey, when desktop migrated everyone to Windows
XP, they put UPSes at every desk so it literally ran in a cubicle until the
Data Center migration (where it failed the first time and the label was
printed). Rather than figuring out what, on earth, it would take to fix it,
they put it on a shelf and plugged it into the UPS in the rack until I was
called several years later.

[0] I was one of two people who were called when everything else was tried.
This sort of incident happened to me in very similar ways at least 4 times
(once with an old Thinkpad Laptop).

------
jiveturkey
It makes perfect sense!

The same kind of people that would say no to a pocket change ransom, are the
same kind of people that would use taxpayer money to not just do a poor job,
but to not do it at all!

The kind of people whose shit don't stink.

~~~
bluGill
Never pay a ransom: it just encourages the bad guys to try harder. Soon they
will figure out how to corrupt your backups as well (if there is anyway to do
that - now we are in an arms race). Better to write off the loss now and
ensure the bad guys don't continue to think of new attacks.

There is one exception: if the ransom is paid in such a way that the FBI (or
equivalent) can track where the money goes and thus arrest the criminals.

~~~
dangerboysteve
Well when the FYI recommends paying what options do you have?

[https://www.nytimes.com/2019/08/14/opinion/ransomware.html](https://www.nytimes.com/2019/08/14/opinion/ransomware.html)

------
duckqlz
TLDR; if your company / city has millions invested in a cyber infrastructure
get cyber insurance. :facepalm:

Also $76,000 is oddly low... like low enough to be the price of a well
established firm to do a pen test on your network...

~~~
davinic
True, but managers also usually fail to account for the time and cost of
remediation after a pentest.

Had the city engaged in the pentest, would there have been an appetite to
spend 10-20x that amount on remediation?

~~~
NullPrefix
Paying the ransom doesn't cover the need to spend money on fixing the
problems. Having a ransom payer mark on the organization raises the security
requirement, doesn't it?

------
rednerrus
This headline gives me anxiety.

