A lot of this sounds like they were under-resourced and the business increasingly adopted new technology with no ongoing support for their IT infrastructure.
> These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.
> There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security.
> Our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack.
A lot of lines like the following, also indicate to me IT was increasingly were involved in fighting fires and maintining operational systems ("keeping the lights on") rather than deploying new infrastructure and automation, updating software etc.
> Some of our older applications rely substantially on manual extract (...) which in a modern data management and reporting infrastructure would be encapsulated in secure, automated end-to end workflows.
Modern business is IT, I know that I am preaching to the chior but this sounds a lot like their IT was seen as a cost.
> However, the first detected unauthorised access to our network was identified at the Terminal Services server. This terminal server had been installed in February 2020 to facilitate efficient access for trusted external partners and internal IT administrators, as a replacement for the previous remote access system, which had been assessed as being insufficiently secure. Remote usage
expanded during the subsequent Covid-19 pandemic because of the greatly increased requirement for remote working and the range of IT projects being undertaken with third party support.
While I'm certain they are underfunded and overworked, this sounds like they had an internet accessible terminal server. I'd like to imagine IT screaming this is a bad idea but a suit somewhere saying they needed easy access for partners. I can only imagine how insecure the solution they replaced with this one was.
I think it's part of a general trend where UK govt institutions have notoriously poor IT, usually consisting of semi-obsolete infrastructure, multiple legacy systems, sticking-plaster upgrades, one or two new state-of-the-art bits where budget is available, etc. Consider the NHS, the MOD, DVLA, etc.
I would be fully supportive of the GDS (https://www.gov.uk/government/organisations/government-digit...) taking on additional responsibilities and providing support and assistance to other government agencies. gov.uk is almost universally praised by the general public and tech people.
Agree, but they can't really do very much about the massive number of legacy systems in departments that can't or won't spend money to modernise. My favourite example to hate is the Driver and Vehicle Licensing Agency which tracks different things in multiple systems, and still requires snail mail interactions (!!!) for some services, such as reclaiming a license after a medical suspension (personal experience). To DVLA, people like me are a pure cost, as are the systems that record my data.
Having experienced both the DVLA and (California) DMV, the DVLA feels miles ahead, like it's living in the future.
Things like finding out the status of a renewal involved finding a fax machine, everything but the most trivial renewal (say, renewing if you're on a work visa) seems to be done in person with handwritten paperwork, and the amount of busywork that seems to be done by hand by the DMV agent is quite easy to blame for the impressive wait times, multiple hours even if you have an appointment.
My DVLA renewal was trivial comparatively, they could even use my passport for an updated ID photo. But maybe if you're not a UK citizen they also make you jump through weird hoops?
I'm not saying that the DVLA is good, just that it could be even worse.
> I'm not saying that the DVLA is good, just that it could be even worse
Some things they do reasonably well, yes. But edge cases like mine are the pits. To get a licence back after a medical suspension involves DVLA and the NHS posting physical letters to each other! It took 5 months for this purely admin process to complete, after I was medically fit. Grrrr. That is a long time to be denied the right to drive.
I've known people who have worked in IT in national museum settings, and from what I heard it sounded like a mix of traditional IT support—ensuring the lights stayed on, printers could print, emails and phones worked, and a very simple website stayed online.
Some aspects sounded quite interesting, but these weren't places pushing the envelope in any aspect of technology. I'm sure they were running outdated software and configurations on everything, but IT was closing their tickets and meeting their SLAs. And with no disrespect, these people weren't necessarily disruptors looking to shake up and modernize the museums' infrastructure and take it into the future either, they just did their job to the best of their ability and went home at the end of the day.
To generalize I find that this usually holds true in a lot of non-tech industries, and IT is generally seen as a burdensome cost as opposed to enabler of business.
Good report. Well written incident summary useful for cyber-students
to follow and learn.
> The Library utilises numerous trusted partners for software
development, IT maintenance, and other forms of consultancy
> increasing complexity of managing their access was flagged as a
risk.
> first detected unauthorised access to our network was identified at
the Terminal Services server. This terminal server had been
installed in February 2020 to facilitate efficient access for
trusted external partners
Sadly their response seems to be using more cloud infrastructure and
outsourcing more.
trusted != trustworthy
The essential lesson - that good IT and security people within your
company cost money. It is worth paying for vigilance, loyalty and care
- has not been heeded.
I happened to be there while this attack was in progress (ocotober 23). And all there systems were really offline, POS didnt work, wifi didn’t work, literally anything connected to a computer didnt work.
What’s unfortunate is that they flagged this vulnerability in 2022 and planned to review it in 2024 ???
Does it usually take this long to identify impact of users? They mentioned they paid for identity protection for their staff & ex-staff as well.
I work in a related field (cyber insurance response) - typically takes a few months to identify exfiltrated data and then analyse it to understand what is in it. This might seem simple but there are usually in the region of hundreds of thousands to millions of files, and that may contain spreadsheets with tens of thousands of rows. This all has to be analysed, filtered and reduced to the point you have a list of PII which has been impacted, and can decide on what to do.
Credit monitoring is usually offered as standard when a breach occurs, the UK is much less litigation friendly than the US so in the absence of any actual harm, that would discharge most of their obligations to protect you following an incident.
Who decided credit monitoring was an adequate remedy for these breeches? I think I've accumulated three or four lifetimes of it by now, but it's never done anything but spew false alarms.
> The increasing use of third-party providers within our network, some of which has been due to capacity and capability constraints within Technology and elsewhere in the Library, was noted by the Library’s Corporate Information Governance Group (CIGG) in late 2022, and the increasing complexity of managing their access was flagged as a risk. A review of security provisions relating to
the management of third parties was planned for 2024; and the tightening of access provisions that would be enabled by improvements to underlying computer and storage infrastructure and the migration of storage to the cloud, which is currently being implemented. Unfortunately, the attack occurred before these necessary pre-requisites for this work were completed.
Price of everything and value of nothing. Outsource everything, underfund everything from systems renewal to staff salaries.
So Tom, Dick and Harry all have Terminal rdp access into the core infrastructure and they slept well knowing that they had - what was it? Ah, yes, - prevented clipboard copying as a hardening measure. That'll stop them pirates in their tracks. Nicely written post mortem. Though I can't help but notice the amount of committees and acronyms. Is it a British thing?
I have to applaud the library for releasing this report. In Canada, the most likely response to cyberattacks is mealy mouthed platitudes like "Please be assured that we take your privacy very seriously and are doing everything possible to recover the data and ensure that something like this does not happen again." and on and on.
"Our major software systems cannot be brought back in their pre-attack form, either because they are no longer supported by the vendor or because they will not function on the new secure infrastructure that is currently being rolled out."
I see a few comments indicating that connecting Microsoft (? not mentioned anywhere in the report??) t Terminal Services to the internet was a wholly bad idea.
Aside: is the report using "Terminal Services" generically, or do they mean that the server hasn't been updated since before 2009 (? when it seems Terminal Services became Remote Desktop Services (RDS))?
Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure, or ...? RDP is default enabled on MS Windows installs (I always disable it), is that more of a problem than one might imagine?
Do they say anywhere where the access was from (maybe only GCHQ know that). Presumably the firewall would only allow known connections - did they report on analysis of all the remote clients?
> Is there something inherently insecure about remote desktops, or is MS software here known to be particularly insecure...
Exposing RDP to the Internet directly has been frowned-upon because of the attack surface being presented, there's no two factor "story" out-of-the-box, and you're opened up to brute force attempts on cruddy user passwords.
Older versions of the Microsoft Remote Desktop Protocol had a much larger attack surface than current versions. The current versions with Network Level Authentication (starting in Windows Vista/Server 2008) present a smaller attacks surface. Older versions used "homegrown" Microsoft crypto, whereas current versions use TLS.
Disclosure: I made a FLOSS fail2ban-like tool for RDP many years ago[0]. I had a situation where I was forced to expose RDP to the Internet and I didn't like having it open w/o some protection against brute force attacks. This tool happens to still work in Server 2022 and will slow the velocity of brute force attacks. I still highly recommend not exposing RDP directly to the Internet anyway.
(The ts_block tool is missing some fairly essential functionality that I never got around to implementing. It works fine and is really easy to install but some things are sub-optimal.)
The term has become a bit generic these days and people will use it in place of a range of things. Citrix or vmware are often just called "terminal server" by aome people.
There is a huge difference between a port forward on port 3389, and publishing the gateway behind azure app proxy - the latter supporting mfa, account lockouts, and not actually requiring any open port to the internet. Much of the discussion online treats these as equal.
"The Library utilises numerous trusted partners for software development, IT maintenance, and other forms of consultancy"
...
"this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA)"
Occasionally malware groups do patch vulnerabilities to maintain exclusive control over the victim machines. But that wouldn't be my default expectation, so relying on virus software to provide security does not seem like a great idea.
> This paper provides an overview of the cyber-attack on the British Library that took place in October 2023 and examines its implications for the Library’s operations, future infrastructure, risk assessment and lessons learned.
For a report from British--and a library, no less--the lack of Oxford comma cocnerns me.
Despite its name, use of the Oxford comma is more frequently promoted in the USA than it is in Britain. As a British person myself, I generally avoid it. N=1, but I wouldn't expect the London-based British Library to use a construction named after an Oxford University Press style guide.
Nice job on publishing this detailed report, I wish after every attack all organizations disclosed in such detail so we can create future defence and counter measures in an open source way.
> In common with other on-premise servers, this terminal server was protected by firewalls and virus software, but access was not subject to Multi-Factor Authentication (MFA).
> When alerted by the Library following discovery of the attack, Jisc (who provide the Library’s internet access and monitor movement of data across their networks) identified that an unusually high volume of data traffic (440GB) had left the Library’s estate at 1.30am on 28 October.
"Jisc is the UK digital, data and technology agency focused on tertiary education, research and innovation."
State-owned quango asleep at the wheel. Unsurprising.
> State-owned quango asleep at the qwheel. Unsurprising.
This used to be what we called JANET. Back in the day this was top
banana and prestigious to work for like GCHQ etc.
I expect they've died from a thousand cuts under the Tories. Every
university I've been in the past 10 years have their ICT run by
Microsoft, and is absolute rubbish.
Because the state (eh) of State-owned or state-adjacent anything, in modern Britain, is simply terrible. The dominant Thatcherite ideology ensures that state-provided services are almost invariably second-rate, thanks to systemic under-funding.
In this case, it looks like Jisc was basically turned into a charity in 2011, so technically they're not even state-owned anymore.
No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence.
They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical.
Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.
> they refuse to pay the ransom and have no other way to tell what was stolen. Farcical.
It's bad that they don't know what was taken, but as for paying the ransom, I wouldn't do it either: first, because it's danegeld; second, because you're just exposing yourself to even further risk by accepting files from criminals; third, because as others said, it would be UK tax money.
I suspect they don't have the forensic evidence to determine the root cause. Chances are there are probably too many ways it could have happened, and the evidence was encrypted or simply wasn't being captured.
At least they seem to have a plan moving forward that seems considered, though I think a lot of what they want to do is easier said than done effectively. I wish them the best of luck.
> everybody within their IT team should be fired immediately for gross negligence.
That may be true, but by that standard about 90% of every sysadmin, IT
managers and even CISOs would be out of a job next week.
Most companies are just "getting by" and hoping it won't be them next.
We have a multi-national cybersecurity crisis due to decades of
kicking the can down the road, excusing poor software engineering to
allow unfettered commercial development, and destroying our education
and training sectors.
If 90% of them qualify as grossly negligent, then they should be fired. That is kind of what grossly negligent means.
You do not really worry about what would happen if all the grossly negligent doctors get fired. Who will do those procedures with a total disregard for safety, said no one ever.
> You do not really worry about what would happen if all the grossly
negligent...
But I do. I care about them as people. People who have families and
need a job. I'd rather help them to not be grossly negligent than
see them fired (and probably worse idiots take their place since we
are in a major skills crisis right now).
The world is getting complex faster than anyone can track. Tomorrow it
could be you, or I who is getting called on gross negligence because
we can't follow it. So I choose to be a teacher even though telling
people the truth is getting REALLY F**ING HARD these days - cos no
one wants to hear it.
No, they should not continue to be in a position where they can continue committing grossly negligent actions and harm others.
You can train them once they are removed and reinstate them when they can do the job right, but supporting their continued harm of others so they can “support themselves” is detrimental, counterproductive, misguided, and extremely selfish.
You are literally better off paying them to do nothing. Please at least do that instead of paying for harm.
"should" is doing a lot of work there. Im so many ways we're in
agreement. But I do this in the real world, and experience has shown
me we must deal with the world as it is and not merely as we wish it
to be.
Good analogy. It is. People's livelihoods and even people's lives are
at risk.
But we've utterly normalised digital ignorance and built what Edward
Snowden very rightly calls an "Insecurity Industry".
I'd go further, we've turned a celebration of ignorance around
cybersecurity and dismissive attitudes into virtuous slogans.
"Don't make me think" - Krug
"Move fast and break things" - Mark Zuckerberg
"If you've nothing to hide you've nothing to fear" - J Random Idiot
And those who are charged with advising and protecting are deeply
conflicted - because they want backdoor access or at least insecure
products.
What it boils down to is that presently there's more money and power
in insecurity than there is in security. Our industry has multiple
principal agent, Shirky Principle and Pournelle's Law problems, see
[0].
We allow ransomware and stalkerware companies, and outfits like NSO
(which I only mention because they are most well recognised) to
operate as legitimate.
We flood markets with defective IoT crap and reduce consumers
expectations to the level of accepting vendor malware and backdoors
installed out of the box.
And then we turn around and complain that "stuff ain't secure".
> I'd go further, we've turned a celebration of ignorance around cybersecurity and dismissive attitudes into virtuous slogans.
> "Don't make me think" - Krug
That quote has nothing to do with cybersecurity, it's the title of a book by Steve Krug about web usability.
I am unfortunately old enough to have read that book when it first came out, and it's exclusively around how to design front-end UIs on websites to reduce user complexity. There is no mention of infrastructure or security at all.
You're making a quote around how we should make websites more usable and understandable to users - so they can use them without thinking - into something it isn't.
I know exactly what the book is and I read it. It's actually an
excellent book on UX and I expect Steve Krug picked the title because
it sounds cool.
No disrespect to that author intended, but it (maybe unwittingly)
expresses a sentiment that has grave implications about the position
of technology in human affairs. To understand why, please look deeper
into what we used to call Human Computer Interaction (HCI) or
"Cognitive Ergonomics".
I think I recently mentioned it in this online chat [0]
Explicit cognition is the "thinking slow" part of our brains that uses
so-called left-brain linear reasoning and logic. It sits high in the
cognitive stack. But as people use devices today, in what McLuhan [4] or
Innes [5] would call an "acoustic" (nothing much to do with actual sound)
way, we drop down a cognitive level to a faster, visual-haptic loop
that bypasses explicit reasoning.
Designing applications that bypass this has major effects on security.
The work of B J Fogg will show you more about this [1].
Tristan Harris also has lots on it [2,3].
One of the disastrous effects of this "distracted" level of HCI is
that people use more emotional cues, rote, colour, word association,
implicit trust and other models that make them easy prey for phishing
and other kinds magic and trickery.
If you're interested in a much broader understanding of cybersecurity
I give you a sincere invitation to check us out here [6].
Coming from these sort of businesses, I usually read these sort of comments as "they should be fired because when they recommended mfa they management said no".
> No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence.
The IT team most likely begged for years for funds to upgrade their infrastructure, but did not receive any of it. Public institutions are already short on money, but education has it even worse.
If anyone is to blame, it is the last British governments, who have focused their attention on Brexit and Ruanda crap instead of providing services for the citizens.
It's a government with huge civil service infrastructure. The people involved with Brexit and Rwanda miles away from this stuff. Willing to bet that in your counterfactual world lacking Brexit and Rwanda (and let's throw in, say, a Labour government), this would still not have been financed.
If i was user/staff, I would sure prefer if they paid the ransom…
Since I dont trust the library to actually assess my impact, or track records of companies getting hacked often drag their feet making it up to victims. (equifax)
@everfrustrated: There is nothing in your piece that can be refuted. Therefore it must be modded into invisibility.
> This report is a joke.
> No root cause. On other forums it is understood they were running very old and unpatched VMware os. Which is simply embarrassing and everybody within their IT team should be fired immediately for gross negligence.
> They can't inform people whos data has been compromised because they refuse to pay the ransom and have no other way to tell what was stolen. Farcical.
> Their ability to rebuild in a timely manner was hampered by not having any spare servers and presumably because all their server hardware was compromised and couldnt be used for restore.
> These legacy systems will in many cases need to be migrated to new versions, substantially modified, or even rebuilt from the ground up, either because they are unsupported and therefore cannot be repurchased or restored, or because they simply will not operate on modern servers or with modern security controls.
> There is a clear lesson in ensuring the attack vector is reduced as much as possible by keeping infrastructure and applications current, with increased levels of lifecycle investment in technology infrastructure and security.
> Our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack.
A lot of lines like the following, also indicate to me IT was increasingly were involved in fighting fires and maintining operational systems ("keeping the lights on") rather than deploying new infrastructure and automation, updating software etc.
> Some of our older applications rely substantially on manual extract (...) which in a modern data management and reporting infrastructure would be encapsulated in secure, automated end-to end workflows.
Modern business is IT, I know that I am preaching to the chior but this sounds a lot like their IT was seen as a cost.