
Update on IT Security Incident at UCSF - 1024core
https://www.ucsf.edu/news/2020/06/417911/update-it-security-incident-ucsf
======
schoolornot
CISO: [https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-
br...](https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-bruin-now-
ciso-at-ucsf/)

I can't think of any reason not to use a cloud hosted service for backup
today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you
versioning amongst a million other security features. AWS even has offerings
that let you take periodic snapshots of on-premise volumes.

Point in time recoveries for the entire account would be nice add too but not
having to fork over a million dollars in exchange for a few clicks sounds like
a bargain. Hell we were setting up write-only S3 buckets for critical data
stores 5+ years ago.

Gross incompetence, I'd fire everyone. A district-wide outage for a week,
fine. A 1.4 million dollar check to get data that should have been archived
somewhere GTFO.

~~~
AnthonyMouse
There are reasons not to use cloud services for backups, not trusting them
with your data being a major one. But then you should still be using some kind
of internal backup system.

There is no excuse for not having backups.

~~~
nindalf
“No, we can’t use leading cloud providers to store this data. What if someone
unauthorised looks at it?”

This attitude leaves the victim paying millions in ransom. Look, Microsoft and
Amazon probably have a better handle on security than your IT dept which is
two people who also have to fix any issues like the WiFi not working or
software not updating.

~~~
vinay_ys
There is no reason to believe clouds don't have security issues or software
bugs that cause inadvertent data loss.

Just because the cloud provider is a big company doesn't mean the security is
better. At the end of the day, they are a team of software engineers with all
the usual people and org problems and can have usual human mistakes.

But cloud solutions are better in practice due to totally non-technical
reasons. Here's how -

If UCSF can afford to pay millions in ransom, they can definitely afford to
hire the right experts to do their IT systems properly.

It is likely that they hire the right people, but then the most commonly
recommended security policies were considered but not implemented due to
resistence from powerful non-security team members inside the organization.
(Usually, the highly paid CISO is playing politics with peers to keep the job
by ignoring those security experts).

If it also possible that the IT+security team is incompetent and is only good
for deploying expensive but useless vendor solutions that do a lot of security
theater but do very little to improve actual security. Also possible that the
vendor solutions are susceptible to be misconfigured easily to become
insecure.

Public cloud solutions can help overcome these above shortcomings of in-house
security+IT teams in a org politics friendly manner.

~~~
AnthonyMouse
> Public cloud solutions can help overcome these above shortcomings of in-
> house security+IT teams in a org politics friendly manner.

The problem there being that the choice of "public cloud" is subject to all
the same forces, so then the vendor with the best corporate marketing team
gets the business even if their security is crap.

------
hoomank3
The paid ransom, will unfortunately embolden the criminals to strike again in
search of the next big payday. If it worked once, it could work again.

~~~
mcny
If the data is worth paying a million dollar ransom to unlock, it is worth
setting up proper backups. I for one am grateful to people who commit these
crimes in which they "lock" data in place rather than sell it to the highest
bidder.

Proper data hygiene isn't brain surgery. There is zero excuse for this event.
I don't blame the criminals. I blame the university system. Shame!

~~~
brippalcharrid

        ... "lock" data in place rather than sell it to the highest bidder.
    

Why not both? And once the rightful owner of the data has paid a fat ransom,
surely that's got to provide some kind of proof of its market value. The
University did say that

    
    
        The attackers obtained some data as proof of their action
    

so unless they're logging their outbound traffic, who's to say they didn't
exfiltrate all of it? It's the kind of thing that the University would remain
tight-lipped about unless they were either sure that it hadn't happened
(doubtful, seeing as they aren't running a tight ship) or had some kind of
mandatory reporting obligation for the data.

~~~
akiselev
The data is worth that much to the university because they're critical to
grant continuity - it'll be hard or impossible for their researchers to keep
the money flowing without it. It's pretty much useless in everyone else's
hands because those grants also depend on individual reputation and research
history.

------
robbrown451
Maybe there should be a law that if you pay a ransom, you are required to pay
the same amount as a fine. Because paying these ransoms is funding the
criminals.... how about you have to also fund law enforcement to combat those
criminals?

(also, this should reduce the amount that actually goes to the bad guys, since
the amount of ransom would have greater downward pressure, i.e. if they'd
probably not be able to collect more than $0.57M because that would cost UCSF
$1.14M)

~~~
ukulele
This would make the payers far less likely to report it, and ultimately make
it much harder to track

~~~
robbrown451
Well if it was illegal to not report it, they'd be putting themselves at a lot
of risk. I can't see a university or corporation making a large payment like
that illegally.

Also, it may not make the costs to them more. Remember the amount of ransom is
based on what the ransomers think the ransomees are willing to pay. Today, the
reason it was 1.4 million instead of 2.8 million, is that they didn't think
UCSF would pay the latter amount. So if they knew UCSF would have to pay
double the amount of the ransom, they'd have to only ask for half as much.

------
throwaway87634
I've worked in an institution for which I rewrote a public facing database to
replace their old and clunky system. I had numerous emails with them to
transition smoothly between the old and the new system, and the last step was
for them to give me the latest snapshot of the MSSQL db that would allow me to
import the last two weeks of data entry. Scripts were ready, so the down time
would be 10 min which was totally acceptable for that tool. That's when I
realize the backup was two weeks old, that they had already deleted the
machine as soon as the transition started (against our plan), and that the
automated backups at the scale of the university had not been working for
months (they realized it because of this incident)...

------
locusofself
What kind of data is worth at least a million dollars and isn't properly
backed up? Unbelievable. Some heads should roll .

~~~
duxup
It's a university, university IT is often ultra political and those who win
the various battles make the rules, regardless of competence... and often
without IT's sign off.

Some universities generally have done better about such things and are making
progress... but generally there is a push and pull for IT dollars by unversity
departments who want to spend that money as they wish for their given programs
and then that money comes FROM IT ... who down the road are then tasked with
the costs related to maintaining it and the terrible decisions a department
made in the meantime... or in the worst of cases tasked with securing that
data and / or making it work at all.

It's the same story for IT in the private sector to some extent, but it is way
worse at many universities. Imagine if your HR director got to pick the PCs to
support, networking equipment, software, backup methods (if any) all on their
own and wanted zero input. That's kinda how it is at many universities.

I spent months helping a large university dig out from a program where they
hooked up some super special microscopes worth millions of dollars ... to low
grade network switches and storage. I got to try to explain why you can't put
10,000 pounds of data into a borderline consumer grade network ... in all of a
couple milliseconds.

~~~
throw0101a
> _It 's a university, university IT is often ultra political_ […]

While you're not necessarily wrong, another option is budget.

If this is academic- / research-generated data, then it could have been paid
for by grant money, and most of the cash goes to paying grad students and
perhaps some computer equipment.

IT may have chargebacks (they have bills / cost centres to pay too after all),
and no one wants to "waste" grant money. Often these things are 'shadow IT'
run in an _ad hoc_ fashion by just throwing together some PCs.

If the group's expertise is in medicine / biology, how many members want to
give up their day-light research hours to run the computer infrastructure?

I've spent about half my IT career in the academic sphere, and cheap solutions
can be a fight to implement even if they solve the problem; even free (open
source) ones can be an effort if they take time or slow down the workflow.

And these people aren't stupid: they 'know' they should do some of these
things. But people 'know' they should get exercise, and how many folks do
that?

------
dangero
What I find crazy about this -- no guarantee that the ransom payment would
unlock the machines -- did they send 1.14M in one go or was it a smaller
amount for the first machine, then an additional fee for each additional
machine?

Also would be interested to know -- was it Bitcoin or some other
cryptocurrency that was used?

~~~
recursivecaveat
Apparently crypto-ransom people are actually pretty trustworthy about
unlocking the machines. It doesn't really cost them anything (0% chance you
were gonna send a 2nd payment if they didn't unlock), and their reputation as
'fair' is very important for securing future ransoms.

~~~
closeparen
Someone's gotta be thinking about doing a ransomware operation that doesn't
unlock the data in order to poison the well.

~~~
Animats
It's been done. There was one "ransomware" attack that just erased everything.

~~~
LilBytes
NotPetra

------
ed25519FUUU
> _The data that was encrypted is important to some of the academic work we
> pursue as a university serving the public good. We therefore made the
> difficult decision to pay some portion of the ransom, approximately $1.14
> million_

Once you’ve paid the Danegeld, you’ll never get rid of the Dane.

------
adrianN
How did they not have backups of important data?

~~~
trhway
there was mentioned a threat of publishing of some secret data, and that i
think may be a reason why UCSF was willing to pay

[https://www.trialsitenews.com/hacking-group-launches-
success...](https://www.trialsitenews.com/hacking-group-launches-successful-
ransomware-attacks-against-ucsf-experts-think-covid-19-connection/)

~~~
eigenvalue
This seems much more likely than the scenario where they simply didn’t have
any backups of the data.

------
Veserv
What a cheesy attack. It only cost them $1.14M to get their data back. I can
not tell if the attack hit something inconsequential, the criminals are
stupid, or they just do not understand finances for them to ask for such a
tiny sum.

UCSF received $1.43B in grants and contracts during 2017-2018 [1]. Assuming
they are generating an equivalent amount of value in knowledge evenly
distributed over time, the loss of one day of research would be ~$3.9M. So, if
the the last whole organization backup was one day ago and the attackers were
only able to stop access to the last day of work since they did not think to
corrupt the backups before they went out, then the ROI of paying off the
ransom would be ~3.43. If they were able to affect the entire organization for
an entire week, then the cost would be ~$27.3M with an ransom ROI of ~24.

So, assuming they did any damage of consequence, asking for ~$1.14M seems like
robbing a person at gunpoint for their pocket lint.

[1] [https://www.ucsf.edu/news/2019/02/413396/ucsf-top-public-
rec...](https://www.ucsf.edu/news/2019/02/413396/ucsf-top-public-recipient-
nih-funds-pushing-unbroken-streak)

~~~
WrtCdEvrydy
Doesn't matter, it's the perfect crime...

The encryption is done on the user's machine using their processing power, and
there's virtually no downside for you (either they pay and you decrypt or they
don't and you just dissapear).

Ransomware will be an issue for years to come.

~~~
Veserv
I am confused by your response since it seems to agree with my post but you
used words that indicate you think that they do not agree. Did you mean to
reply to someone else?

I am saying that that the ransom is hilariously small compared to what they
would probably be able to get. So, when they realize they can actually ask for
a number that is not a rounding error they will probably increase the
number/outcome of attacks. They are criminals robbing people at gunpoint and
only asking for their pocket lint. Once they all realize they can ask for a
wallet and get it, I think the number is going to go way up.

~~~
rovr138
On the other hand, the more they ask for, the less likely they’re going to get
paid for it.

If they keep asking for $1M from organizations that will pay because that’s
pocket change, the more organizations they can attack.

If they start asking for too much, they’ll see they won’t pay as quickly,
other people will start selling services to protect against you that seem more
attractive, etc.

------
eitland
I recently listened to Mikko Hyppönen as the company I work for had invited
him to give a talk. It was very interesting.

One thing I hadn't realized before was that ransomware criminals has developed
their own backup strategy:

\- in addition to encrypting the data they will also exfiltrate it and
threaten to publish it ob the internet for everyone to see.

That way it doesn't necessarily help an organization just because they have
multiple layers of offsite read-only backups.

------
janwillemb
The article is inconsistent about the severity, which lowers its
trustworthiness. It states that everything is under control, a top super
leading security guru expert is on top of it, no important data was stolen or
exposed, everything will be fine soon. And... oh yeah, BTW, we'll be paying an
astronomic sum of money to the criminals to get our data back.

------
justinclift
Wonder if the cost of backups for the affected servers would have been less
than the US$1.1M they paid? ;)

At a guess, they could probably have put in a some fairly high quality storage
+ backups for that price, and still had money left over for all expenses paid
staff vacations. :)

------
Gatsky
Is there a way to restrict encryption at a hardware level? The conditions
where you would like to voluntarily encrypt data are usually quite rarefied.
Allowing any sort of encryption activity on your system seems like a hazard
these days.

~~~
justinclift
Storage systems will just see a lot of data being changed.

You _could_ have some sort of alert in place, but if the malware doesn't write
fast enough to trigger it, you'd still miss the problem.

Generally, the approach places take is having backups (eg to tape, off site,
etc), and/or having storage that makes a snapshot every few hours and retains
them for days/weeks/months.

Snapshotx are generally very low cost and easy these days, as it's just a
pointer manipulation thing on (say) ZFS rather than a complete copy of the
entire data set.

~~~
Gatsky
Backups are the default solution. But my understanding of this UCSF case is
that their system was compromised and then ransomwared. In this situation, the
attacker could interfere with your backup system for several weeks or even
spoil existing backups.

It would still be nice to restrict encryption on a system that you control. I
suspect, but don't know, that encryption has a particular pattern of memory
and CPU activity that could be recognised. Or if there are a few commonly used
libraries you could do it that way, although an attacker could roll their own
encryption.

------
rubatuga
To understand why they didn't back up: a) they are academics, which means
funding is lacking and b) it's in the field of medicine, known for its lack of
tech adoption.

------
Isinlor
Here is Dutch University of Maastricht paying 0.2 million euro ransom last
year in December.

Seems like business is blooming.

[https://www.reuters.com/article/us-cybercrime-netherlands-
un...](https://www.reuters.com/article/us-cybercrime-netherlands-
university/university-of-maastricht-says-it-paid-hackers-200000-euro-ransom-
idUSKBN1ZZ2HH)

------
surfpel
Don’t they have insurance for these things? A small college near me had an
attack like this but paid via insurance.

~~~
gorgoiler
Would you want the insurance policy to pay out though?

At some level of recklessness, insurance becomes void. I think a lack of
infrastructure to restore a hacked server — with data valued at over $1M — is
negligent enough to not be covered.

But maybe UCSF are on MegaCo’s _YOLO_ tier of server insurance, which is so
expensive and isolated it has no impact on my MegaCo pet insurance premiums?

~~~
closeparen
I'll venture a guess that UCSF was nominally in compliance with some relevant
bureaucratic regime (ISO 27001, SOC 2, etc), and that was good enough for a
stodgy insurance company that's not very sophisticated about "cyber risk" (in
case use of the term "cyber" isn't a giveaway...)

~~~
gorgoiler
The CISO’s top 5 “security tips” PR piece didn’t mention off-system backups.
Policy voided.

------
noodlesUK
This just illustrates how incredibly important solid backup strategies can be.
A big university should be able to figure out how to make WORM (write once
read many) backups of their data. A million bucks buys a shitload of cloud
storage or physical airgapped tapes...

~~~
TwoBit
The same weakness that resulted in lack of backups may be related to how the
intruders were able to get onto the machines in the first place.

------
ppiuser
How incredibly negligent and incompetent of the ucsf IT staff to not have this
data sufficiently backed up, not to mention protected. Does this university
offer IT courses? If so their reputation has been seriously tarnished.

------
Gatsky
I have to assume things were backed up, and the attackers were deep enough in
the system to find or delete the backups.

Anyway, I wonder how this payment was made...

------
speedgoose
I'm surprised that paying the ransom is a thing.

This is a very selfish thing to do, as you encourage the authors of such
attacks.

------
chmaynard
Ouch. No backup?

~~~
craftyguy
It's pretty obvious that the answer is 'no backup'...

~~~
scottwb
It’s not that obvious. It could be assurance that the data would be deleted
and not sold/shared.

~~~
craftyguy
> We therefore made the difficult decision to pay some portion of the ransom,
> approximately $1.14 million, to the individuals behind the malware attack
> _in exchange for a tool to unlock the encrypted data_ and the return of the
> data they obtained.

I assume you read that though before you replied, right?

~~~
rovr138
> and the return of the data they obtained

~~~
hanche
… as if you can’t “return” the data and keep a copy at the same time …

~~~
rovr138
Might require ‘best effort’ to keep data private.

------
rkho
As a reminder, in 2017 UCSF offshored all of its IT staff to HCL Technologies
and forced their then-employees to train their replacements before laying them
off.

They brought the replacements into the Bay Area on H1B temporarily while they
were trained by their soon-to-be-laid-off counterparts and then sent back
overseas to continue their roles once training was complete.

[https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-
worke...](https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-workers-have-
last-day-on-job-after-outsourcing/)

~~~
ahi
How the heck did they qualify for H1Bs when they were replacing already
employed Americans? Not only did the H1Bs unemploy citizens, we didn't even
get the long term benefits of trained workers living and paying taxes in the
States.

~~~
trhway
using H1B for 4 months stint is very strange. It is too valuable a visa slot
(significantly oversubscribed lottery usually, at least in the recent years)
to be wasted that way. Usually it is done using something like B1 for such a
short trips, especially if it is training, etc. I wonder whether the
journalists and others did mistake one visa for another, especially given that
they naturally wouldn't be privy to such internal details of a 3rd party
company like the HCL in this case.

~~~
rkho
I looked a little more and found this subsequent article talking about it:

[https://www.latimes.com/business/hiltzik/la-fi-hiltzik-uc-
vi...](https://www.latimes.com/business/hiltzik/la-fi-hiltzik-uc-
visas-20170108-story.html)

