
Hit by Ransomware Attack, Florida City Agrees to Pay Hackers $600k - waynesoftware
https://www.nytimes.com/2019/06/19/us/florida-riviera-beach-hacking-ransom.html
======
jrochkind1
> A similar breach recently cost Baltimore $18 million to repair damages.

No. $18 million was an _estimate_ somebody gave once, who knows where it came
from.

In fact, damages have _not_ been repaired in Baltimore. 6 weeks later, most
city services are still down. You can't pay a parking ticket or a water bill
online. (You can send a check in; I am not sure where they _record_ that you
paid when they cash your check, and am not particularly confident they'll
actually have a record I paid).

We in fact do not know how much money they've spent thus far, there have been
no press briefings on this. Estimates of how much they will spend before it's
over (will it ever be over?)... we all know how IT estimates work.

I think it will probably be quite a bit more than $18 million. And then
there's estimating "damage to the economy." (There were two weeks when real
estate transfers were frozen, because there was no way to check city liens.
They can be done now, using a paper-based system that actually has those
involved in the transaction sign an unusual contract agreeing to take on
liability for unknown liens in unusual ways (I'm being vague cause I don't
totally understand it), that some but not all title companies are willing to
use).

The Baltimore ransomers only wanted ~$100K. If I were the mayor, yeah I'd pay
it.

</Baltimore resident>

~~~
albertgoeswoof
How can you separate necessary security upgrades out from the $18m? $100k
wouldn't be enough to maintain a single critial app securely for more than a
couple of years.

If you're running a bunch of critical services you need to secure them, and
that takes millions every year, it's unavoidable. A single hack that exposes
this is irrelevant

~~~
jrochkind1
It is true that the cause of the pain is that they haven't been running their
IT in a secure fashion, and this is the real problem.

I don't know how much of the money/time that will be spent (the "$18 million"
figure is entirely imaginary) to instead try to recover their data/systems to
functional... but I don't think it's a great use of money.

But indeed, actually running secure systems is what they got to be focusing
on, and will be expensive. I think many organizations are going to find that
they can't actually afford to do what's needed to reliably run the systems
they already rely on, having not been running them reliably or securely.

I hope that the 'recovery' efforts, done in an emergency fashion, don't
distract them from getting to that point and figuring out what they're going
to do about it.

------
mysterypie
Could they have avoided the ransom by having daily (or hourly) backups to non-
rewritable (write once) media? So the malware won’t encrypt the backups,
obviously.

I think that the last day’s work (or last hour’s work) will be lost or will
require a lot of manual fixing regardless of whether they pay the ransom. If
they pay, they’ll still have to fix partial database transactions, corrupt
files, etc., for the attack date. If they don’t pay, they can recover from
earlier good backups and reconstruct that one day’s worth. My reasoning is
that the attack date’s data is going to be corrupt and untrustworthy in either
case, and it’ll be equal work either way. (Or at least it’ll be less than
$600,000 of work to fix that one day.)

I imagine that they either weren’t doing backups at all, or their backups were
directly accessible and writable by the malware.

~~~
jjav
A very good solution here is ZFS. Office file server runs ZFS and makes
regular ZFS snapshots (hourly, or more often). Snapshots are immutable so even
if the employee machines get infected, nothing can be lost (except the last N
minutes since the last snapshot). Obviously, still have to do backups (off of
those snapshots).

~~~
eeZah7Ux
> Snapshots are immutable so even if the employee machines get infected

Not really. You can easily extract data and overwrite the disk surface.

~~~
jjav
Not from the machines which typically get infected, which are the employee
desktops accessing the file server.

If the malware manages to get itself running on the file server itself (and
with root privileges), then sure. That's not a common case though.

------
blotter_paper
>On Monday, Councilwoman KaShamba Miller-Anderson, the chairwoman of the
board, asked Justin Williams, the interim information technology manager, for
something seemingly simple. Could the elected officials’ new email addresses
be posted online for the public to get in touch with them?

>Underscoring the enormity of the city’s troubles, Mr. Williams explained that
the webmaster hoped to get to that soon.

>“He’s been working very feverishly to get that done,” Mr. Williams said.

...the webmaster is working feverishly to post a static piece of text to a
website? I guess it really is hard to fire government workers.

~~~
scarejunba
It's the government. He probably needs to find long lost keys to SSH in, file
four forms (each in triplicate), and receive approval from each person to get
it out.

~~~
jlgaddis
It's municipal government. There almost certainly were no SSH keys.

The problem is that the credentials that the webmaster needs likely only exist
in an Excel spreadsheet that was saved on the desktop of the "Administrator"
user account on one of the machines that got hit.

------
nyolfen
maybe state governments should preempt this and make it illegal for
municipalities or state agencies to pay ransoms, so they are less attractive
targets

~~~
throwaway13337
"We never pay any-one Dane-geld, No matter how trifling the cost; For the end
of that game is oppression and shame, And the nation that plays it is lost!"

But the problem is that those that don't pay hurt even more.

The US makes it illegal to pay kidnapper ransom and, as a result, US citizens
have much worse outcomes (often murdered) when they are ransomed abroad.

The theory that it makes US citizens less attractive targets is confounded by
the fact that some families/friends of the victims can and do pay anyway
(illegally).

Planet money did a podcast on it.

[https://www.npr.org/sections/money/2017/09/01/548032302/epis...](https://www.npr.org/sections/money/2017/09/01/548032302/episode-792-the-
ransom-problem)

~~~
dagw
Another interesting podcast on the topic of kidnapping:
[http://www.econtalk.org/anja-shortland-on-
kidnap/](http://www.econtalk.org/anja-shortland-on-kidnap/)

Goes into the details of kidnapping as a business venture from the kidnappers
perspective, and how a price equilibrium is found between 'buyers' (ransom
payers) and 'sellers' (kidnappers).

On interesting story was when the partner of a small business owner got
kidnapped as punishment for failing to pay some protection money. When the
business owner went to negotiate the ransom, the kidnappers had had an
accountant already go through the businesses fiances so they knew exactly how
much they could ask for, without it bankrupting the business owner (so that
the owners company could keep thriving and thus could keep paying protection
money).

------
technion
Note the specific detail here:

    
    
        the City Council unanimously agreed to have its insurance carrier pay

~~~
Macross8299
Seems like a classic case of moral hazard here.

I'm surprised cybersecurity insurance doesn't mandate best-practice auditable
backups as part of the process to grant a policy.

~~~
mruts
I mean, do health insurance companies mandate best health practices in order
to get health insurance? Of course not, they just charge premiums commensurate
with the risk they're taking on.

~~~
noahl
Interestingly, basing insurance on healthiness seems to be a new trend
happening right now.

I'm a runner, and recently I've seen a lot of ads for a company called
HealthIQ (I think) that offers cheap life insurance, but only for people who
can run a 9 minute mile.

I think breaking into health insurance would be much harder because a) the
administration is way more complicated and b) most people get health insurance
through their employers, and normal employers won't be able to guarantee that
_every_ employee can pass a healthiness test, but I imagine they're working on
getting around these problems right now.

------
dbg31415
Look, virtually nowhere in the public sector is security taken seriously. And
nowhere in local government is security taken seriously. City governments
might as well be pinatas... the way their budgets work, they'd never be able
to replace large systems that were compromised. Without legislation banning
them from paying, paying the ransom is likely really appealing to them.
Security should be bumped up, but let's face it... that's not going to happen
given how nobody who knows anything about tech would be caught dead working
for local government. So many things have to change.

~~~
hanniabu
It really doesn't make sense to me that software isn't created at the
government level for cities and states to use. This way it's easier to make
sure everything is functioning properly and the cost is only paid once.

~~~
dragonwriter
> It really doesn't make sense to me that software isn't created at the
> government level for cities and states to use.

What government level? The federal government?

> This way it's easier to make sure everything is functioning properly and the
> cost is only paid once.

The federal government is no paragon of software virtue, nor is it likely to
produce software adapted all that well to all of the needs of various states
and cities, so what you'd end up with is software less fit for purpose, not
particularly free from vulnerability, and where all of the vulnerabilities
expose every state and local government in the country rather than just one
jurisdiction.

And that's still assuming good intent, but in many cases the state and federal
government have adversarial relations on particular issues, which might lead
to the federal government actively designing software in a way to frustrate
the needs of particular states.

------
iliketosleep
The laxness of infosec in government continues to astonish me. It's not like
these types of attacks are new either. I can only assume that the people in
charge of infosec in such situations are bureaucrats without much technical
knowledge.

~~~
Dirlewanger
It's a failing of the American condition. The country was founded by radical
conspiracy theorist farmers that didn't want to pay taxes. Distrusting
government is in our national ethos. It pervades to this day in the form of
governments generally being staffed with people too incompetent for private
sectors. The pay sucks. It's hard to get raises. It's hard to do _anything_
because Americans hate taxes; they'll help their neighbors, but they won't
help those they can't see beyond their porch. There's little personal
incentive to work in local/state governments. And that's partly how we end up
with events like this.

~~~
enraged_camel
> _It 's a failing of the American condition. The country was founded by
> radical conspiracy theorist farmers that didn't want to pay taxes.
> Distrusting government is in our national ethos._

Not true at all. Distrust of government is a relatively new phenomenon in
American politics. It can be traced back to Reagan's infamous "The most
terrifying words in the English language are: I'm from the government and I'm
here to help" quote. This is because Reagan strictly believed in small
government, and wanted to limit government interference in most things.

Before Reagan, Americans had no issues trusting government to solve big
problems or accomplish major goals. See the Space Race, and Roosevelt's New
Deal policies two decades before that. American people were largely optimistic
about those endeavors because they trusted their government.

~~~
magduf
The Vietnam War did a _lot_ to destroy Americans' trust in their government.

------
xupybd
Great, now they have encouragement to do it again.

------
billpg
Will the attackers restore the attacked machines once the payment is made?
They are just as likely to take the ransom and run.

~~~
Consultant32452
If they want the next city to pay, they will restore the attacked machines.
Gotta remember, this is a business for the attackers.

~~~
billpg
I think that time has passed. I've read about too many people who pay up but
don't get their files back.

------
dredmorbius
I would very much like to hear from the insurance carrier here, and know what
the post mortem and preventive countermeasures will be.

Update: The servicer appears to be Gallagher Basset based on the 2018-19
budget and legal cases cited online.

City records (CC agendas, minutes) are painful if not impossible to navigate.

------
dredmorbius
Curiously, no open IT / security positions listed:

[https://rivierabch.applicantpro.com/jobs/](https://rivierabch.applicantpro.com/jobs/)

~~~
otakucode
The first thing I do when I read a news article that includes any company or
organization saying that they are 'serious about security' is go to their
website and check their job listings. Sometimes they have openings for
security-related positions, but what is actually telling is their software
openings. They NEVER so much as mention a single thing about security or
knowing how to create secure applications. I've literally not seen a single
exception, personally. Most companies do seem to have reached a level of
'caring about security' but it amounts to hiring some people to play Patch
Patrol and nothing more. I guess that's better than nothing, but it won't
actually help much.

------
nwmcsween
I've noticed outside software or technical companies IT is basically 100% turn
key with off the shelf mostly junkware (even 'enterprise') software being
used. I attribute this to the mismatch between HR and the position being hired
for and what higher education teaches w.r.t IT. Honestly what needs to happen
is interviews need to be farmed out to places that understand the respective
industry and not just certifications and higher education.

------
sjreese
Let's look at this a bit differently - Ransomware is a type of malicious
software designed to block access to a computer system or computer files until
a sum of money is paid. Most ransomware variants encrypt the files on the
affected computer, making them inaccessible, and demand a payment to restore
access.

Ransomware is a type of malicious software designed to block access to a
computer system or computer files until a sum of money is paid. Most
ransomware variants encrypt the files on the affected computer, making them
inaccessible, and demand a ransom payment to restore access.

Ransomware is rarely individually targeted, but rather a “shotgun” approach
where the attackers (Clue I) acquire lists of emails or compromised websites
and blast out ransomware.

Microsoft used a method to install software giving it superuser rights without
a login. (Clue II) Most ransomware is based on this same install job. It is
lightweight but identifiable.

Ransomware is a tripartite intruder and is based on what's already there on
Windows (mscexe) in your compute and a substitution of legit program (outlook
encrypt) Once the 3 parts are there your system is theirs and only a windows
product key method "EFHST-G6ERT-VXWMT-FF8MB-MYERR" can free it - all thanks to
Microsoft's product key methodology.

Oh and "backups" & PCmatic won't help and because Microsoft uses the same
method to stop you from sharing software. You have seen the screen yourself =>
you have entered an invalid the product key!

Ransomware can be shipped with a NSA crack( EternalBlue ) forced onto the city
of Baltimore (Clue III ) but the same code to create a superuser is open to
the public is the end to all protection - because it hides using Microsoft's
hidden directory method.

Well what to do now, pay the BTC? Yes and NO Yes buy BTC and NO this is where
we create a pigeon drop for out NSA connected friends - we don't accept the
face price and try to keep our BTC keys and Encrypt theirs.

For the FBI and NSA the profit from robbing Venezuela, Iran, Russia, Ukraine
and Switzerland has been too great for them to stop. As witnessed with
Venezuelan money gone and power outage.

That said, demand that Microsoft be held liable for product defects and to
make all actions visible to the end user community ( no hidden files or
directories ).

------
tjpnz
So the US is willing to pay ransoms - just not for people.

~~~
mulmen
> So the US is willing to pay ransoms - just not for people

A city government in Florida is not the US federal government.

~~~
PlasticTank
Well now the various groups beheading people on YouTube know where to ask.

~~~
vonmoltke
Those people care about creating a spectacle, not getting paid.

------
EvanAnderson
I will continue to smugly assert that backup must include an offline
component. Given that total data loss is a non-zero possibility (and,
increasingly, more and more likely) the argument that having even a simple
offline component (say, some encrypted USB disks for a small business, tape or
such for a larger business) is too expensive or cumbersome doesn't make sense
to me.

------
PlasticTank
What evidence do they have that the hackers will actually send the keys? seems
like a pretty big gamble on trusting proven criminals.

~~~
AnIdiotOnTheNet
Ransomware relies on trust, because if you don't actually hold up your end and
decrypt the data then no one will bother to pay you in the future. The entire
"market" is best served by playing "fair", in so far as that can be applied
here.

That doesn't mean some criminals won't just take the money, but it does mean
that most of them wont and that the larger players have a vested interest in
keeping that behavior to a minimum.

------
Circuits
Lady opens a random email (most likely in her junk folder) from someone she
doesn't know and end's up costing the company hundreds of thousands of
dollars? In 2019? Something is rotten in the state of Denmark.

------
knorker
Apparently some people do negotiate with terrorists.

------
AnaniasAnanas
Shouldn't the one responsible personally have to pay for it rather than the
city and its taxpayers?

~~~
tremon
You mean the one who wrote the attacking code? Or the one who wrote the
vulnerable code? Why do we even assume there is a "one" here?

~~~
AnaniasAnanas
Whoever made the decision not to take backups for example. The ones who will
have to pay for their mistakes will be the taxpayers otherwise.

~~~
albertgoeswoof
This is a public service, aren't the voters responsible? They could have voted
in competent leaders.

~~~
magduf
This is it exactly. The voters are the ones who are ultimately responsible,
and they'll be the ones to ultimately pay, just as it should be. They should
be voting for competent leaders, and for sufficient taxes to pay decent
salaries to attract good IT talent, but they don't, so this is what they get.

Every nation gets the government it deserves. - Joseph de Maistre

~~~
AnaniasAnanas
The voters are not one person. Sadly democracy ends up being the fascism of
the many.

