Hacker News new | past | comments | ask | show | jobs | submit login
US companies hit by 'colossal' cyber-attack (bbc.com)
611 points by sedeki on July 3, 2021 | hide | past | favorite | 486 comments

Honestly, I think this should be the death knell of these "remote monitoring and management" tools that have extreme low-level access to networks and systems, but just like the SolarWinds attack, it feels like these are run by companies with extremely poor security culture.

I mean, I'd be willing to trust security to Microsoft or Apple (I mean, at some level, you've got to trust the OS). But giving the keys to the castle to some mid-tier company is just a recipe for disaster, and the bad guys know how extremely lucrative these targets are.

A lot of these companies are actually huge enterprises with dozens if not hundred(s) of cybersecurity consultants and engineers. All of them are CISSPs and GICSPs(I do put my CISSP in the signature when working in those places too though).

I go through security reviews all the time with them, they have so many security processes that you get dizzy and on paper everything looks fine. They create security zones with massive risk reviews, but for some reason those security zones then share subnets with the entire LAN.

They also have a default configuration which makes everything access the standard intranet directory once its deemed secure. Enterprise security tools like Cyberark are deemed more secure than say yubikey HSMs, which may result in root ssh being enabled in a lot of settings. They have system configurations that are done with massive Excel sheets. Their cloud VPCs basically only have one risk profile and once its deemed secure it gets access to things in the intranet. They also vehemently refuse to do threat modelling when designing anything.

These people can tell you so much about the theory of security by heart that it will make you dizzy but then won't actually understand the underlying problems.

And the offenders are always the same, advised by Accenture, Infosys etc.

>These people can tell you so much about the theory of security by heart that it will make you dizzy but then won't actually understand the underlying problems.

I've thought greatest failure of many professionals in this field is in the "protect the network" perspective rather than "protect the data". While many of them fess up to "we can make it difficult but not impossible" to breach the network, that is not evinced by the protections instituted.

If companies actually understood that they WILL be hacked, the focus would turn to protecting the data. Actual resting data protection would allow a "I don't care if I'm hacked," posture. Either behind encryption, VM's, segregation, or architectures, preferably all, if data is actually protected, then a hack can be weathered. It's still a pain if the computer-touchers have to rebuild and reload, but that's what you pay them for. If the data is protected, a hack is just a painful exercise rather than a newsworthy event.

I do understand that segregating (through protection and architecture) data is difficult, but I do not understand why it is not the focus.

I always thought that when thinking security a compromised system HAS to be rebuilt. I have never seen that happen in an enterprise though. They never ever rebuild compromised systems they just try to improve perimeter protection.

Rebuilding the entire IT deployment is prohibitively expensive.

Imagine if the solution to Covid was "the virus can spread to anyone, we need to replace all humans"

When a machine/server/laptop/PC is hacked, it's considered compromised. No amount of pruning that system can restore trust in it. It needs to be reset.

It's only prohibitively expensive because most of these enterprise tools and servers are actually very much focused on manual setup.

>> Actual resting data protection would allow a "I don't care if I'm hacked," posture.

That's quite interesting. Where can I read more about that ?

I think something along these lines is called “zero trust” and especially Google has been aggressively implementing it. But I could not find have any high quality articles on the concept.


Zero trust protects the servers, not the network, and not the data.

Protecting the data means encrypting it (which Google also does) such that the client needs a key to decode it.

But if the client is hacked, the hacker can get their keys! so encrypting the data isn't a magical cure-all.

If hacker hack your systems, they get the decryption keys too. They just might need to hit two separate targets with two attacks or a attack that both are vulnerable to.

In many cases you don't want to let an hostile party read confidential data, and if he takes control of the machine aren't all bets off?

Moreover what if a hostile party constantly hammers-disrupts your IT, letting you teams "rebuild and reload" 24 hours/day (in other words you don't have any information system anymore)?

I think the idea is that after you get compromised, you do forensics to address the vulnerability that exposed you, then after addressing the fixes in your perimeter, you reset the compromised systems. The attacker wouldn't be able to hammer you again without having another vulnerability in that case

I have never understood this; the whole Enterprise™ security business talks about all these things where half the time I literally don't even know what they're on about. They all seem to take it very serious; great! And at the same time they miss basic stuff like, I don't know, subscribing to Apache struct release mailing list. Or not keeping employee credentials around on public servers used to file credit disputes. Or in the case of Solarwinds not using "solarwinds123" as a password (probably not used in the hack, but still).

None of this is rocket science and these people probably aren't stupid, so somehow, somewhere, something is going horribly systemically wrong (incentives? Training? Organisation? I don't know).

Seems like kind of a corporate/organizational culture thing. Imperfectly distributed knowledge, hierarchical decision-making in groups with misaligned incentives, the limitations of communication and the capacities of individuals... These and more make it hard to operate a large enterprise intelligently and cohesively, and oversights will happen. Corporations can certainly seem to act dumb or just learn slowly as a whole, regardless of who they're made up of. If you go bigger and look at nation-scale, the same problems are present on a greater level.

Humans can't get this stuff right every time either.

It's not like Colin Percival or Theo de Raadt , perfect as they are, could just audit and secure all of the Fortune 500.

ISC² has done so much damage to the industry via enabling the fallacy of appeal to false authority it is mind-blowing. The cissp is such a terrible proof of whether someone knows anything, everyone knows it, but for some reason people keep falling for it.

I view it as a shared level of baseline knowledge that helps with conversation. If I see someone has it, it at least tells me they understand the words I’m using and have a basic knowledge of the concepts we are discussing (or should, at least). It also tells me they are good at taking tests.

It doesn’t tell me whether they understand how it all works together, or if they understand the organization’s environment, or if they are a good worker.

I don’t hold it against people who fail (I’ve seen good people fail the test) or who don’t have it - I just have to ask a few more probing questions to ensure they know the tech I’m discussing. But I don’t outright ask if someone is a CISSP, so typically I ask the clarifying questions anyway so our understanding of the problem is accurate and aligned.

And cert or not, I’m still more interested in whether you know what you’re doing than what you put on your resume.

I wish that were true (genuinely, a shared vocab would be super useful). I've heard so many nonsensical things from CISSPs, I should really start a parody Twitter account. Did you know, for instance, that SSL is an important control for preventing SQL injection? How about that salting is not effective against rainbow tables because of the birthday paradox (yes that's actually what they said).

It's just a cram-and-forget vocab test, it doesn't mean anything other than that they could afford the training and the test.

Sounds like it's a test in Security Fluency, not Solution Providing.

Fun fact: the word “security” comes from the Latin word for carelessness - “securitas.” se = without, curitas = care.

That actually makes sense to me. If I feel secure, I feel carefree. Maybe there should be a different word when providing a secure environment from the word where people enjoy that secure environment.

I wonder if the Swedish security company "Securitas" knew this when changing name.


I’ve often wondered this. I see them around a lot, and I’m like “someone doesn’t know their Latin.”

"Centralized hosting and management (SaaS, PaaS model) has the advantage of security at scale."

it follows that

"Centralized hosting and management (SaaS, PaaS model) has the advantage of insecurity at scale."

Can you explain more about the ‘root ash’ issue please?

By using some of these tools they are under the false assumption that things that are otherwise considered security threats are somehow okay because for example the tool rotates passwords for you. It gives a false sense of security and allows you to do things that would otherwise be considered security threats.

It's as if someone sells you a laser that shoots intruders and tells you, you can leave the front door open from now on, but that laser only works 1 in 3 times.

I see so if people have cyberark they might feel like it’s safe to enable root logins over ssh? That does sound like the sort of thing that would happen.

One could hope but I doubt it. CFO's gonna CFO and it's "cheaper" to outsource IT. I had one of these vendors really pushing me to "take a call" or "let them show me how they could cut costs". It was ALL about the costs. And I eventually called the CEO and said we would consider it if the company would take out a $100M bond that we could call on to repair any damage that occurred as a result of their managing our IT systems. He thought that was ridiculous of course and thought poorly of me. Since that time at least two of his customers have been the victims of breaches that IT either directly facilitated or indirectly made possible by providing additional attack surface that was required for their business to work.

But not every person who has executive oversight of operations thinks like I do, and all of them are represented in the company's finances as a 'cost center' that is second only to Payroll in terms of how juicy a 'cost reduction' target it presents.

So when the going gets tough, the company cuts back its IT budget.

I keep reading over and over again indignant comments about "cost centers" on Hacker News and I think it's not a good term to use because I looked up the definitions and the only logical consensus I could find is that everything which isn't shareholder profit is a cost center. It's just rhetoric.

I don't think it is - I think it's cultural and organisational. The CFO and Finance in general see businesses as capital flows, they don't see value being added - just opportunities for leverage and cash management. The description of a cost center is a labelling denoting a target for removal and reduction - the destruction of value that occurs (typically 12 -24 months after the exercise) is seen as disconnected and irrelevant.

Eye of the beholder topics don't generalize. If your CFO and Finance team is doing things like laying off all the information security people since they thought Axa would pay the ransom gangs, then state the name of the company. Otherwise it's just venting handwavy frustration about people whose job requires taking risk mitigation seriously.

I'm not going to name companies as I don't fancy the blowback, but the fact is that CFO's aren't doing risk minimisation, they're doing bonus optimization.

There is a common misconception that CFO's fiduciary duty to their shareholders determines that they should protect the long term stability of the company, but now most shareholders are in the company for 6mths tops. The duration of a CFO's fiduciary duty is arguably about 6mths out. The devastation of large companies in the economies of the west since 1980 is a testament to this.

But everything is a cost center , even product development and sales. A salesperson is a cost center except for the singular moments when they sign a deal.

It comes down to the balance

Fake Numbers: sales "generates" 100,000,000 in revenue and "costs" 1,000,000 however IT "generates" 0 in revenue and "costs" 2,000,000

So to an bean counter, clearly the "investment" in sales is better because they make money for the company, where IT just costs money...

Thus IT is often treated like a utility service, say something like custodial services, where they want to find the cheapest way to keep the bathrooms not disgusting... not clean mind you, just not disgusting

I always recommend that engineers who aspire to manage at the executive or "C" level take some classes or read up on how business school teaches business leaders to analyze the health of their company. Those are the classes where 'gross profit margin', 'marginal costs', and 'operational efficiency' are discussed and explained.

If you are looking at US curriculum, my experience is that you will see the discussion in terms of dollars and their "flow" through the firm from the customer and perhaps ultimately to a bank account (in the case of having positive cash flow) or how much 'short' the company is when it comes to a negative cash flow situation.

Understanding the cash flow dynamic for a company is critical to the company's success. If a company does not understand how they make money and how they spend money, they will not be able to manage themselves to a sustainable level.

As with engineering, it is a simplification to group "like" costs, and "like" revenues together. So for example all the money made by extended warranties and charging for repairs might be grouped as "service revenue." Similarly, all the money spent on leasing office space might be grouped of "real estate costs."

Every accounting program I have seen (and it isn't exhaustive of course, just consistent in my view), facilitates this grouping of costs into larger and larger groups. Depending on the size of the enterprise, the manager at a particular layer who had "profit and loss" responsibility could see a small number of these groups (which I have only ever heard referred to as either "revenue sources" or "cost centers") and they could get an idea of the health of their part of the business by seeing if their margin target (total_revenue - cost) / (total_revenue) was being met.

And at the managerial level, they typically would split their activities into ones that "improve revenue" or "cut costs." Doing either increases the gross margin which is what they are measured on by their manager, whether it is another person at the company or the board of directors. Because these are fundamentally an accounting thing, increasing money coming in by say raising the price of the product or restructuring pricing plans is called "growing top line revenue" because that usually the top line of a financial report. And when they cut costs or improve efficiencies so that they can make more product for less money, that is called "growing bottom line revenue" because the amount that gets subtracted from the top line is reduced and so the number at the bottom of the page gets bigger.

Finally, nobody is an expert on everything. And the larger the enterprise the wider the expertise needed to understand the costs and expenses of that enterprise. What is worse, is that sometimes the people in that role were experts at one time but the area where they developed their expertise has moved on and so they believe they know what is the right answer and don't bother to check. And sometimes they don't know the right answer but don't want to "look stupid" and they buy all the reasons the sales guy gives them for using their product as pass that along as justification without knowing the risks.

It adds up to a bad choice. And when that choice is to move to open offices (for example) the impact of losing productivity in people who cannot deal with that environment isn't readily apparent. And when it leads to outsourcing something which wouldn't be outsourced, the error might only become apparent when you're suffering a ransomware attack.

Meanwhile, best practices are slow to reach the curriculum and so there is a lag between people doing things poorly and it being taught as a bad thing in business school.

Thanks for explaining it to me!

The problem here is that when, P&L in IT/Security is L, the answer is always "reduce costs", not "get more credit for enabling and protecting the revenue of other cost centers". IT undercharges for the value they are expected to provide. Often times (hello GDPR and ad tech /spyware) skimping on IT is way to cover up the fact that the business is fundamentally not sustainable or sustained only by breaking the law or negative externalies.

The CEO and CFO are right because this notion of reducing "attack surface" is not static. It changes from day to day and no technologist can guarantee what changes they add today makes any difference tomorrow. The promise is False. Therefore the principle of least action is justified.

“I will gotta leave open the windows and the door, and won’t even finish building the gates because I’m planning on extending the building and I will have to tore down a part either way”

> But giving the keys to the castle to some mid-tier company is just a recipe for disaster

It sucks, because I know my company is quite small but we take security extremely seriously (we have 9 people, 4 are security engineers, and the other 5 have varying degrees of experience in security). I think people might worry that, because of our size, we won't be as secure as a larger company. But the irony is that larger companies are often far less secure than us, because we've done shit right from day 1.

There's just not a lot of ways to prove it. Compliance is meaningless. You could get a pentest report, but it really comes down to who's doing the pentest, and so if your pentest becomes a public doc the incentive is to have them go easy on you - not to mention that lots of reports contain "findings" that are nonsense but a casual reader might misunderstand.

We plan to give talks and blog about how companies at our stage can do things that would make companies 100x our size jealous, because that's kinda the only thing we can do to really explain that it's possible.

I think it's totally criminal that companies ask for RCE on all of your devices and then push out some closed source C++ app that's probably parsing all sorts of random shit, reading poorly authorized commands from some C2, etc.

This really gets at the issue. These are compliance tools mandated by auditors and accountants that are intended to provide centralized config/control of everything. Ultimately, they make companies insecure as they have tentacles into everything important. They are used against the companies (by hackers). That should not surprise anyone.

You can be compliant and buy insurance or you can be secure. Pick one.

> We plan to give talks and blog about how companies at our stage can do things that would make companies 100x our size jealous

Sounds interesting and rather extraordinary, would be great to read more on your thoughts on that. Do you refer to the Grapl blog?

Yeah, I'd say watch the blog, I have some draft posts written up.

Much easier to do it Ina small company, very hard to get it right in a company "100x your size".

I wasn't trying to say otherwise - it's a huge advantage to be this size, with regards to security. It would have taken me years at Dropbox to accomplish things that take a weekend now.

I wouldn't trust a company that implements security critical projects on the weekend...

I think you're probably misunderstanding. As an example, rolling out a policy at a company with thousands of people has to be done slowly and incrementally, with buy-in across teams, etc.

To do the same is trivial at a small company. What would take years and lots of effort becomes something you can do in spare time.

Of course, we put considerably time into security, it's not just something that one does once in a while with spare time. The point is that we can go much much faster.

Agreed. Companies that are great at selling to governments and massive enterprises tend to be great at security theatre and security certifications, but that’s not the same as being great at security. Their tech tends to be bloated spaghetti full of tech debt, with a huge surface area for attacks, and systems like that are nearly impossible to secure in a truly robust way.

Embedding this kind of software deep in your internal networks/systems, with access to basically everything, is a recipe for disaster. I expect these sorts of supply chain attacks to get more and more common, they’re excellent back doors into basically every government agency and megacorp.

The fix, known all over industry, it insurance coverage that pays out for externality incidents. Then actuaries figure out what real security is.

Would you mind briefly explaining the concept of "tech debt" to a layperson?

First we have to ask, "why does programming get harder as the project goes on?"

Let's say you are designing a system - any kind of system - with the philosophy that everything should be connected to everything else. Your first part goes in quick with no connections. Your second part goes in quick and has one connection. Your third part has to be connected in two places for it to work right, but that's not a problem. Your hundredth part has to be connected in a ninety nine places for it to work right, and now you're spending more time wiring than you are on making parts.

Then we ask, "what can we do when that happens?"

You have to put effort into the design of the system, reassigning duties and studying the nature of the problem it's solving, so that you lay down the connections along the true contours of the map, and not between every single component. Afterwards the next component you add has to be connected only to the three other things it's actually related to and you're back in business. This results in a period of time with no new features or even bugfixes, but afterwards you move faster.

Then we ask, "why do people call it debt?"

Because you pay interest on it when you have it, you run it up when you're short, and you better have a plan to pay it down or else you will go out of business.

I intend to add it to my quotes collection.

Should I attribute you or someone else? :-)

Edit: added as a private bookmark to pinboard with tags:

  technical_debt quotes by:whatshisface

Two ways, I think they're easy to understand but I have no experience in teaching:

Technical debt is like not cleaning your house to save a bit of time everyday. When you actually have to clean it, it's going to take longer than the time you saved. And until it's not clean, everything you do will be a bit worse because the house isn't clean.

"Remember when you were a student and didn't do the dishes, and then when you finally did them everything was dry and sticky and stinky, and it took you a lot of time to wash everything and you felt terrible? That's dishes debt. Technical debt is the same. When you make a change, you produce dirt in the codebase, and if you don't or can't take the time to clean every time, dirt accumulates."

Worse than mere accumulation, it grows toxic mold.

You know how you're working on a project, and everything mostly works but some stuff isn't quite up to spec, and you swear you'll fix it later because you have a lot of stuff to do? This is that, compounded over a few decades.

You're patching over problems with short term solutions instead of investing the time and effort to fix it "the right way".

Like when you need to fix all the support columns in your building, but instead of spending millions to take them down one at a time and replace the corroding rebar inside, you just patch over the exterior cracks. They will look fine from the outside and get the job done on a day to day basis, but they hide structural problems and one day that debt will come due. Most of the time it's in the form of a giant project to finally fix everything, but sometimes it's catastrophic failure.

It’s a pretty broad term, but I’d define it as properties of a software system that make it hard to modify/maintain safely and easily. And it’s fixable, but takes an investment of time/effort/money to fix. The debt metaphor is that it can make sense to have a bit of this, but too much becomes crippling.

Often the most maintainable solution is simple and elegant, but it takes a lot of refactoring to implement, so a hacky, complex solution is implemented instead, because it’s faster/easier to implement. Such solutions tend to either contain bugs, or lead to bugs when built upon, and a lot of security vulnerabilities are basically bugs in hairy parts of systems that are hard to understand.

> Technical debt (also known as design debt or code debt, but can be also related to other technical endeavors) is a concept in software development that reflects the implied cost of additional rework caused by choosing an easy (limited) solution now instead of using a better approach that would take longer.


I don't think it's just rework, it's also any other future risks or difficulties implied by taking the easy way for now.

I should just edit the wikipedia page, but they won't accept edits from my current IP address.

This is it. It's not just "stuff that's not perfect". The term technical debt refers to things the business accepts as debt to be repaid later for more as the price of getting a feature out of the door sooner.

It's a quick fix that will take more time to fix later than it would to do it properly now but typically gets done to meet a deadline because time is of the essence currently for some reason.

There is significant cognitive load in understanding the code and what it does and why it does it that way etc. Keeping all the important bits in one mind is challenging and a lot of the little fixes can lose sight of the big picture in a way that comes at a cost and, over time, this can really add up.

Over time, different people may work on the code and have different reasons why they made different choices and at some point it may all stop playing well together. Then there comes a point where someone needs to try to reconcile all the different bits and understand what needs to happen and why and rebuilding the entire catalog of goals, features, etc. in one mind at one time so someone actually understands it all and gets it right is a substantial future cost that only grows as you keep delaying that step.

(From one lay person to another -- I do write code, mostly html, and run some web projects, mostly blogs and Reddits, and spend too much time on HN. So technical debt isn't alien to my experience though I'm not really a programmer.)

"Code other people who are not me have written and frameworks I didn't pick"

(I jest. A bit cynical but that's often how it comes out in practice).

There's 3 people.

1. Knows the wider requirements but isn't involved in the implementation. They can't fully specify what's needed without doing the actual implementation; the map is not the territory.

2. Is told the broad requirements but probably can't grasp the things they aren't told in the imperfect spec. So under time pressure, in good faith, they do the simplest workaround possible.

3. Is given the next set of requirements. Instead of re-engineering the original design, under time pressure and in good faith, they add a workaround for the workaround.

Each new workaround is "tech debt".

When you add the next feature you now have to deal with multiple levels of complexity not in the original spec.

Understanding the actual implementation now takes more time than expected. The chances are that no one fully does, which leads to further mistakes and workarounds. So more tech debt.

Either you pay the debt down and re-engineer or you pay the compounding interest forever.

...and so on. Each new level of complexity gets harder and harder to understand and debug because no one really knows how the real design, held in the actual works.

When you cook, you make food, that's the primary result. You also produce waste, dirty dishes and general disorder. This secondary result is tech debt.

You can cook a bunch of times ignoring these secondary results but over time cooking will be slower and of worse quality due to the mess and at some point it will be impossible (too dirty, no usable pots and pans, etc).

its like not combing or washing your hair to save time but it ends up turning into dreadlocks and then it still kinda serves as hair but is much harder to work with and untangle into other hairstyles.

"Softhwair", if you will :D.

I was looking for a definition a few weeks ago and found the wikipedia article succinct and accurate (it met my needs anyway): https://en.wikipedia.org/wiki/Technical_debt

"tech debt" is when you decide that you will cut a corner to build something.

If you did it when building a plane, and it killed people, you would go to prison, but in the technology industry, this is acceptable as "the cost of being first"

Also the cost of understanding the problem more fully. And the cost of discovering a better way to do it.

I presume you were not trying to be ironic with this request (given how you chose the easy option of inconveniencing others rather than Google/Wikipedia)

Anyway, here's a good introduction: https://en.m.wikipedia.org/wiki/Technical_debt

Or, maybe they thought someone on HN would be able to explain it better than the Wikipedia article. It did not seem like an unreasonable request to me at all.

The interesting part about last year's incidents of solarwinds, fireeye and fortinet is that there's a switch away from actually targeting the hosts after the first line of defense.

Redteams / hackers now target the infastructure, because it's way easier and they're more outdated in regards of code, stability and used libraries.

Most enterprise-grade VPN solutions still use OpenSSL from decades ago, and most of their fixes (even if they react to CVEs) are always too late.

As SOCs need VPN access because they are usually not on-site, especially at larger corporations...the result is when you exploit the VPN gateway, you are the new administrator because you have a large time window until the SOC team arrives on-site. These couple hours are usually everything you need as a time window to raid the place, install and run ransomware, and clean up afterwards.

From a cybersec perspective I cannot even begin to write how stupid it is to put literally all your company's value in the hands of a single security company - which is legally not responsible for anything by contract. Security through obscurity never worked, why should it do in this case?

Last year showed that we desperately need an open source OpenVPN based graphical and scalable alternative that uses a standard TOTP based token generation mechanism and not some proprietary crap for authentication.

WireGuard is a simple and secure new protocol that most VPN companies are moving to. It doesn't do the key rotation or TOTP authentication part however.

Most of the attacks the last years were also targeting the enterprise auth apps that were heavily outdated (e.g. vasco token apps like "Auth ES" or "Enterprise Auth" etc). Lots of the breaches could've been prevented by just using a standardized (maintained) TOTP token generator that doesn't have an RCE backdoor with its included analytics scripts.

Using a token generator with embedded analytics was just wrong in the first place, but...yeah.

Personally I'd love to see better Wireguard support and adoption outside the Linux world.

I specifically have experience with Kaseya. I kicked and screamed to get us off of it, the IT people insisted it was top notch.

So when I became CFO I fired them (outside company), not just for this, but it didn’t help.

It’s bad software. 24/7 full low level access is exactly what it is. We had an add on that stored admin credentials in a JSON… so looking back on that, it seems this should have happened sooner.

Additional information about the Kaseya angle:


Did you think it was an unintentional technical liability, or did you think it was intentional?

Unintentional I think. If it was intentional I think things would have looked better on the surface.

The craziest part to me was their pushing of the vPro integrations.

I dumped Intel servers for Epyc and Kaseya entirely. So, I’m looking pretty good with those decisions.

Just a month or so after the attacks, one of our large government clients signed up to no less than three such vendors and deployed their products to almost all of their production servers.

I discussed this with their security team leads, and they answered with a straight face that it's okay because they had to spend their budget before the end of the financial year.

This entire "we need to spend our budget"-attitude is something I will never understand. So what if you get less money next year? Firstly, it's not going in your pocket, and secondly clearly you can get by just fine with a lower budget.

And everyone knows this is how it works too – so the Powers That Be keep setting the wrong incentives too.

This is why I never worked for a large Enterprise company or government agency. I'd go crazy.

You do not have to "trust" the OS at some level. Use Linux or BSD, demand open hardware. You only feel like you "have to trust" shitty closed-source OSes because the orgs behind those OSes have been able to abuse market-dominant positions to stifle competition.

Security by obscurity is laughable nonsense. We should all be demanding transparency in hardware and software from our vendors. I'd pay handsomely for it.

I think that in this context the meaning of the term trust is different.

Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0)

Most companies are not Linux contributors, they are trusting the kernel developers to write bug free, secure code.

Minimizing the TCB and opting for an auditable open source TCB are really useful concepts in security.

But the cause of these breaches is much more trivial than what you are worrying about: these companies are basically installing whichever piece of software can decrease their costs without thinking about what they’re doing.

I agree with you.

>Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0)

It's way nicer to be able to look at the code running in Ring 0 =)

I used to work for an MSP and we had used Kaseya.

There was an AV integration, and then Kaseya changed to Kaspersky. I don’t remember what the prior AV software was.

I always thought it bizarre we were actively installing AV software from Russia on banking and medical office PCs.

That has been a consideration in the AV software I recommend to friends, family, and professionally as an informal part of my threat assessment model.

I viewed it as safer to buy products from anywhere other than someone that has ANY potential at all to go to war with the government of the country I live and work in. I really hope it never happens, but 'cold war' tensions might be waged with little cyber attacks and that software came to mind as a risk.

On the other hand… Kaspersky software isn’t shit. And I expect they catch a few things other US based companies might be incentivized or politely asked to look the other way on.

I wouldn’t run K, but I know from experience it’s actually effective.

Two more things to consider:

- Can you articulate specific reasons to buy anything beyond the default windows defender?

- If anyone went to an actual war with the US, would the source of your antivirus software get even close to top 5000 things you care about at that point...

As for default Windows Defender, there isn't really good reporting tools related to it. There are reporting tools for Defender, but those are paid license add-ons.

And yeah there's a decent chance if the US went to war with another country it might not impact the majority of US businesses very directly especially in the short term IRT their IT plans. McDonald's kept selling burgers when we invaded Iraq (multiple times). Ford was still producing vehicles during WWII. There have been lots of military engagements the US has been involved in where things in the mainland US weren't massively affected in day to day operations. Who knows what some potential future war with Russia would look like. Would it be a true head to head war with tanks rolling, fighter jets scrambling, cities bombed? Would it be more skirmishes testing how far the other would really go? Would it just be escalation of supply chain attacks and attacks on infrastructure to weaken the other? Of course this greatly varies based on the specifics on what that potential future war looks like, it would be naïve to think wars will always look like WWII, Korea, Vietnam, Iraq, etc from a US mainland perspective.

The post above was saying "at all to go to war with the government of the country I live and work in." US going to war somewhere is one thing. Another country going to war with the US would be something different.

RMM is absolutely vital to securing systems. This is as ridiculous as suggesting we should just get rid of firewalls because there are vulnerabilities found in them. RMMs are how enterprise scale networks close off every other security hole on a network.

That being said, RMM tools have plenty of examples that they need to beef up their security practices or get replaced.

No, RMM is the magic beans someone wants you to trade your cows for. All OS and networking vendors have better tools, but people pay for RMMs because they make a lot of promises and charge less money. People who use them will invariably get burned.

This is... laughably and demonstrably false. Neither Microsoft nor Apple tools even pass muster for large-scale IT management. Generally speaking, you'll pay more for a worse product direct from the OS developers. In most cases, your Microsoft-based solutions are less secure (RDP) than pretty much anything else. Apple is pretty much going to give you the minimum glue necessary to use a third party tool (see Apple Business Manager).

RDP is not a systems administration tool. Windows ADK is table stakes. It and its associated tools can handle any scenario from two-person startup to tens of thousands of employees. This is why Windows Commercial alone is a multi-billion dollar business, and these third-party RMM tools are pilot fish.

Actually, I think that the rapid blossoming of firewalls everywhere in the late 90s was the first big step toward security theatre that I noticed in my career, and illustrate how we have a cultural problem. Blocking access according to port number is no substitute for lack of control and visibility over what services are spawned where.

I think that the problem is these companies are publicly-traded. Chasing YoY returns and never having a down quarter are antithetical to building a lasting security model.

Microsoft, Apple, and Google seem to be doing ok.

That's true, but when have FAANG unicorns ever had the same "laws of physics" that other companies have?

Statistically EVERYONE has extremely poor security culture.

It's been wallpapered over as just cutting unnecessary expense for too long.

It is almost proof we can't collectively think statistically.

I get it at a pretty deep level individually but even knowing this I make enormous mistakes.

What does "statistically EVERYONE..." mean to you?

It sounds to me like if it means anything, it's denying that any probability depends on the exact dimensions of your ignorance.

There is no statistic that applies to everyone, unless that person is a completely generic person with no known qualities.

It made my life as an MSP easier for sure and allowed us to support more clients and bigger businesses and get more done. But I fear you might be right, and it just isn’t worth the risk of a hole like this. Now I’m going to be up for days restoring servers, and data on any workstations that wasn’t backed up is gone. I think we’ll be filing a claim for this one.

Unfortunately, these tools are so damn useful that people almost feel compelled to buy and use them. Eventually, as these attacks becomes more and more commonplace, I think companies will start looking for two things:

1) How secure is the software? Where are the audits?

2) If your software compromises my business, how much of the losses I incur as a result will you cover?

What if Apple and Microsoft out sorces to some mid-tier company? That's what is happening in my business, the big telco/consultancy business. We're always out sourcing things and there have been several scandals related to these small to mid-tier companies that get only a small part of the contract.

Agreed. I can't imagine outsourcing monitoring/metrics/etc, despite the mild hassle of maintaining our server of one of the popular options. It requires attention every now and then, like once a year or two, but can be integrated easily with LDAP and our SSO provider.

Yes, and there is a reason why big IT providers like Accenture are preferred enterprise vendors. They have the financial power to mitigate such risks. There are usually vendor risk checks which include potential damage costs.

You know it is ironic to say you trust MS when they are in the middle of a major security incident themselves i.e "PrinterNightmare" for which they still have not patched

I know we're still pretty close to the Ubiquiti breach, but since then, they've added 2FA.

Is your opinion of their products the same?

Ubiquity introduced new vulns while fixing that fiasco from last year: https://www.zerodayinitiative.com/blog/2021/5/24/cve-2021-22...

On the other hand, all of the other networking HW sucks just as much. E.g. here are Netgear vulnerabilities published just this week: https://www.microsoft.com/security/blog/2021/06/30/microsoft...

Some things, like updating firmware automatically, are ahead of their competitors.

IMHO, the worrying things about Ubiquiti at the moment are:

1. Their handling of the security breach/downplaying/whistle blowing fiasco which came to light some months ago. Check our Troy Hunts podcast from around that time.

2. Requiring a cloud account to manage your local device. Everyone seems to do that these days. It's not impossible to remove the cloud account management but it is an extra post install PITA step to work-around. And has some consequences if you do.

I'd like to see if they've learnt their lesson from at least the first point and become less opaque security-wise going forwards. Not sure their security is passing the smell test at the moment.

> I'd be willing to trust security to Microsoft

Well then I guess we will have a lot threats from hackers for days to come.

And I am curious to know how many such vendors there are.

> this should be the death knell of these "remote monitoring and management" tools

Yeah, sure. We should have a person on each of hundreds of sites whose only job is to check manually every router, switch, and vending machine. Maybe in the best HN traditions you will train the necessary workforce in a weekend?

Your quote cuts off before the salient part, and you seem to be attacking an imaginary argument.

> tools that have extreme low-level access to networks and systems

The emphasis being on the low level access. The solution is not having hundreds of people checking things by hand (though I'm sure that could contribute to security). The solution is more privilege separation; so that when the "remote monitoring tool" is compromised, not every part of your infrastructure is also compromised by default.

I'd agree partially with you. However, once a remote agent is compromised, it will be chained with some privilege escalation vulnerability and this same argument will be repeated with the twist that now every foreign executable with remote connection is an attack surface.

Having hundreds of people in each location whose only task is to do a boring monitoring an very occasional management tasks is a waste of your resources and their intelligence. We do automation to escape doing stuff that we can but which are to mind numbing. The illusion that every one of those hundreds of people will do their job to the necessary level of quality and without lapses of diligence is optimistic to say the least. Doing automation badly is not a reason not to do automation at all.

IMO, a big issue is conflating monitoring with management.

Management is always going to have access, so maybe you should not enable remote management access of everything to a centralized system? Make it lean and secure, possibly segmented, dual-factor, use HSM etc.

Monitoring - there is no good reason why it should have access to anything. Make it ingest only (use firewalls and reasonable protocols), and you've cut out most of the "monitoring and management" vulnerabilities.

There are trade offs there as well. Now you've decreased the attack surface, but still every foreign agent is a legalized rce. Observe that the case of Kaseya is not direct hacking of the agent, but a compromised update where firewall rules won't help. As I said, the next level of the argument is that this rce is dangerous and what it lacks is a privilege escalation.

At the same time, you don't have a way to solve a problem that your monitor has alerted you for. Every solution proposed includes either a person at the location who does the job manually or a way to connect to the network from the outside which is vulnerable to similar attacks as before with added costs and possibilities to mismanage keys and passwords.

Security vs convenience is a well-known dilemma that people very often love to solve in the most absolutist way.

If the software used a one-way protocol, then unless you updated the closed-source agents, which are the parts I have the biggest issue with, there wouldn't be RCE.

As for remote management. I'm saying you should wisely choose what needs to be remotely managed, what doesn't, what are the foundations for your security and then balance it with reasonable methods to secure access. Which would probably not be "Kaseya VSA Remote Monitoring and Management" for all your systems and devices.

Yes, make sure you don't need on-site personnel to restart your web server, but maybe also don't expose management of your switch that you never reconfigure to your monitoring SW, and maybe use separate HSMs¹ or at least HSMs instead of the enterprise management system for the most important parts.

¹ e.g. FIDO2 ed25519 for ssh

So, now you have one hardware key that you have to manage either for all networks and therefore is both a single point of failure and constraint on availability, or you need to manage multiple keys with the ensuing chaos. Kaseya was hacked through a patch so the type of protocol does not matter and you are trading convenience for management overhead that you have to deal with because all of your clients and likely many of your employees are not in your HQ.

I have the bad feeling that this achieves security through unavailability.

What are you actually arguing for?

For the record, I think none of your points apply.

One of Sweden's biggest grocery stores / supermarkets, Coop [1], is keeping all their 800 physical stores closed today, since their payment system is not working because of an IT-attack somewhere in their supply chain [2]. Connected to this attack?

[1] https://www.coop.se/ [2] https://sverigesradio.se/artikel/coop-butiker-haller-stangt-...

Most definitely, googling "Coop" "Kaseya" gives a few articles showing they've implemented it for parts of their organizaiton since at least 2009.

Patients in Region Skåne were also unable to access their journals on Friday afternoon (possibly unrelated) and Coop's competitor ICA's apothecary company Apoteket Hjärtat seems to be affected by Kaseya/REvil attack also.

A cashless society is scary. Cash should always be an option and the inventory system should be disconnected from the internet.

You can pay with cash at Coop. Most if not all cash registers allow cash payments. Most customers don't pay with cash.

Yes but how does the cashier know what the price is?

How does the cashier open the safety box to reach the money?

Or open the cash registry?

I don't even think it is legal to accept money without a working cash registry for tax registration reasons.

But that was my point exactly. Sure, you can live in Sweden without even knowing how cash looks like so it is cashless in a way. But, if the cash register is not functioning then you are done* with or without cash in your pocket.

* I'd wager that if you know the prices and keep track of what you sell, you'd be fine recording the transactions after the fact.

I wouldn’t be surprised if it’s illegal to accept payment without offering a receipt with all of the correct info, which among a bunch of things include a unique incrementing receipt number.

It is more about being able to produce the receipt to the tax authorities. Even street hawkers have to have a certified machine in sweden now.


> Yes but how does the cashier know what the price is?

My understanding was that it was just payment processing that was affected, not the point of sale systems. The scanners and things probably work fine, and I think they could accept cash payments without issue. It’s just not worth it when almost no customer pays with cash.

I'm quite sure that it was the whole POS. The Coop next to where I live accespts Swish (QR/mobile payment) and it was closed as well.

In my country (Switzerland), while they have massively invested in cashless solutions, a lot of places are still accepting cash, and I think it is a good thing. One of the big retailer (Coop) has self-checkout machines that accept and give back cash (you can insert 200CHF~216USD at a time if you want).

Jordan (head of SNB) is not going to let cash go and even kept the CHF 1000 bill under EU pressure. Thank God.

What urks me is the obvious "never let a crisis go to waste" where we have visa etc marketing that cash might spread Corona.

Yes, I've put CHF 200 in coop register before. Funny, they don't care but if I scan a tiny bottle of alcohol I need to wait for someone to approve it...

The thing is I was in Coop yesterday when the attack started and they had at least two payment methods working fine. Swish and cash.

They likely closed to avoid issues with rejecting customers who didn't get the message. Or perhaps just to be on the safe side because they didn't know who the attack was aimed at.

Gee, cashless is such a great idea.

In related news, I saved money by replacing all my house's circuit breakers with old pennies.

I dont think this is necessarily due to 'cashless' as much as general computerization. Stuff like prices, article numbers and inventory are likely all digitized nowadays, so even if people could pay with cash I imagine they'd still be keeping closed.

Why isn't the local shop's systems autonomous - the should sync to the company central, sure, but they shouldn't need constant connection to lookup prices.

I think that this is the case, from the reporting it seems like it’s just their payment infrastructure that is affected. Likely they could handle cash transactions just fine. It’s just that the vast, vast majority of Swedish customers don’t use cash anymore, so it’s not worth it to keep the stores open until it’s fixed.

That sounds so wrong, they should try to use cash if they can.

Going cashless is extremely common for customers in Sweden. They would get so few customers (everyone would just go to the next grocery store), and the aggravation it would cause from customers who haven't heard the news and can't pay probably just makes it not worth it to have them open. Take the loss, fix the issue, reopen all the stores when it's done.

The Microsoft team at a company I used to work for tried to push this very software out onto all staff machines.

Our Platform Engineering team managed to push back on it based on the grounds that it was a serious security concern and is essentially an "enterprise" backdoor.

The following year the bulk of our team decided to resign move on to other employment - I was told Kaseya was rolled out to all machines shortly after.

Companies need to ensure that risks raised by senior engineering teams are taken into account before deploying company wide software.

What software are you referring to? The article only mentions "VSA tool", and that does not ddg well.

The article links to https://us-cert.cisa.gov/ncas/current-activity/2021/07/02/ka... , which says it was Kaseya VSA and links to their advisory.

VSA appears to be a proprietary name. They are usually referred to as RMM tools (remote monitoring and management).

They essentially are enterprise level back doors with good intentions.

Think firewall/antivirus/backup software suite run by a remote team.

Really good thread here:


When these things happen, I feel like there's a predictable response. A few smaller vendors (above, Huntress Labs) provide a great running commentary. Then two weeks later, the dust has settled, everyone's patched, and I'll start receiving sales calls from Enterprise Vendor X wanting to talk about how they were all over it.

Following this, suspicious write up here:


> we were already running a broad investigation into backup and system administration tooling and their vulnerabilities. One of the products we have been investigating is Kaseya VSA. We discovered severe vulnerabilities in Kaseya VSA and reported them to Kaseya, with whom we have been in regular contact since then. Additionally, we have, in confidence, also reported these vulnerabilities to our trusted partners.

It is a sad say when Reddit has higher quality details than HN.

Why is that sad?


| We received an emergency call from our Kaseya rep to shut down our onprem VSA

I never quite understood why these ransom-ware attackers restrict themselves to a small subset of the MSP's clients. E.g.: The SolarWinds attack affected only something like 1% of their customers, when it could easily have been 50% or more!

If you're evil and out for money, wouldn't you want to cast the widest net possible? Similarly, by encrypting a huge number of corporations concurrently, you'd "exhaust" the ability of a country to respond. There's only so many recovery specialists and IT contractors available to respond in an emergency. Encrypt only a few hundred targets and they can all recover. But if you encrypt a few hundred thousand, then there wouldn't be enough warm bodies available!

Thinking about it, I wonder if these attackers have set up permanent operations, with staff, payroll, and everything. Maybe they just to fly under the radar and collect a nice steady income instead of a risky but potentially huge one-time payoff...

Give it time, these are start-ups bootstrapping themselves. They don't have the support infrastructure in place yet to scale to beyond a few hundred companies. As it is, there are going to be a lot of over-worked people at REvil doing crunch time, missing family dinners and their kids' recitals and soccer games managing the logistics of this hack.

No worries though, the ransom from this round should serve nicely as a Series B round of financing & enable rapid scaling of the post-hack ransom extraction process.

I wonder how much of a human element is involved in each individual hack. I would have thought the sticky note, encryption, payment & decryption was all automated.

As I understand it there is often a lot of discourse that takes place between the hacker and the hacked - agreeing prices, haggling, proof of files etc.

Yes much can be automated but there is usually a human element to these deals and that costs the hackers money.

They also want to be careful to limit their hacks to companies their handlers are happy for them to hack. Go too wide and you risk hitting a company directly or indirectly linked to your state/handler/patron.

You are correct. I have had the "pleasure" of going through the negotiation process before. There are even companies that specialise in it, and have DBs on who is a "trusted" threat actor (the industry term) who will actually honor the terms of the transaction or not.

There are thousands, if not tens of thousands, of such deals done every year.

You'd think a GPT3 / GAN could be created to handle much of that. It's a percentages game anyway.

Why would you want GPT to handle multi million dollar negotiations?

Sorta playing into stereotypes about engineers here.


Perhaps not the largest groups, but the smaller ones, posssibly.

That stuff is automated.

What's not is managing big sums of money, turning crypto in to a more traditional currency/assets. That side of the operation probably has more people doing leg work than you'd think.

We really need someone from REvil to do an AMA on HN for this sort of detail. How did they get their first paying "customer"? What's their churn rate? Do they appreciate strong security measures, rendering each lost "sale" somewhat bittersweet? How are they handling the transition from developer-driven startup to a more mature organization?

More importantly, how long are they on Dogecoin? (Funny post, btw. The whole thing is totally absurd.)

I know, and yet I'm only half joking because they probably face some of the same issues as any legitimate tech business. There's plenty of extra issues on top that go with any organized crime-- money laundering, worrying about law enforcement, loyalty of their members and brutal enforcement of it. I really am fascinated by what the structure of this would look like from the inside. Of course much of it depends on the degree to which it may actually be state-sponsored, or just lightly assisted or politely ignored. Now with the added prospect of a powerful country with a vendetta against them.

It's even conceivable that if they go too far and political pressure in the US builds high enough, and Russia &/or their countries of residence are also put under pressure, that they could find themselves on the wrong end of a drone strike or no-knock flash-bank assisted rapid entry to homes and business locations. All they have to do is pick the wrong target that directly leads to deaths-- hospitals the most obvious, but industrial accidents or "rapid unplanned disassembly" of something like a chemical plant...

I was shocked at the pipeline attack, followed by one on the US's food supply. These rise to the level of terrorism, and when fear & anger become dominant motivating factors the event horizon for any ability to predict what happens will become significantly shorter and less certain.

And in the middle of all of that will be a team of techies and support staff struggling to cope with day to day realities of running a thriving organization. There's an IT Crowd satire show somewhere in there that Netflix should consider.

Unsure why everyone is acting like this is a new phenomena. These organizations have been getting multi-million payments for the better part of a decade, it is just only being covered by the media now.

Why couldn't they have bootstrapped years ago? I suspect the real reason is they actually want to avoid extensive media coverage.

The new differences are:

1) Scale of the attacks. Taking large portions of a country's petro-chemical/energy pipeline is far above the threat level presented by most prior hacks. The same goes for shutting down ~20% of the nation's pork & beef food supply. And now hundreds of companies impacted as a result of a single breach. Ransomware isn't new, but it is in hockey-stick growth mode.

2) Increased market for crypto currencies. Criminal activity may not, by far, be the dominant activity, but the more legitimate transactions there are, the easier it is to hide criminal transaction.

3) Bootstrapping this type of thing takes time because it's not just about capital in this case. It's also about accumulating vulnerabilities and compromising systems long enough that backups-- for example a week or month old-- are still useless (also encrypted). And going back to earlier backups will lose the company too much essential data.

4) And as you said, avoiding media coverage that will bring too much attention, and with it the potential for a crackdown. The slow burn on increasing ransomware over the years has acclimate people to it in a way that makes even the most recent massive attacks a little more normalized, especially when they pay the ransom & get back up & running in a few days. That limits the amount of public pressure to fight this head on with mandated increased security and massive resources thrown at pro-actively going after these hackers.

5) In 2016 ransomware wasn't quite as mature. 2020 is different, and the political landscape is different: I'm not making a partisan comment here. I'm not saying the previous US administration prevented these things better or the current administration dropped the ball. What I'm saying is that when there's any new administration, there are threat actors that will test the waters, see how far they can go. I definitely thing that's a factor here, especially so close after the Biden administration delivered its list of 16 untouchable sectors to Russia & Putin. There's going to be a lot of adversarial interest in just how firm those limits are, and what the response will be.

Otherwise, from a national awareness standpoint, you do approach an important point: It may not be a new phenomena, but for the vast majority of Americans that don't follow tech news, this is new and, given the scale of recent attacks, somewhat scary.

It's not enough to just gain access - once you're in you need to compromise other defenses, you need to communicate your demand to the victim, you need to know how much to extort, you need to actually process the payment. Either you do this on a case by case basis or you take advantage of additional exploits that will only be viable for a subset of your potential targets, and this is all a race against time before someone notices your initial exploit. Either way, it's likely impractical for any non-nation state actor to simultaneously attack more than a few thousand targets in one go.

This is combined with a business model resembling patent trolls: you want to extort just a little less than is worth fighting for. If a company gets hit on its own, it's probably not in a position to really do anything about it, but if there is some major hack affecting tons of companies, the odds of an actor with significantly more tech capability like the US government getting involved go way up, and suddenly fighting seems like a good option.

Maybe you’re a state actor and a ransom demand, at least an overt one, is not your objective.

My mind went there as well. Say I'm an affluent oligarch shorting major companies. I'd paying the ransom group to massively attack the company or various companies. Then cash out during the chaos.

Yes, except for the fact that we don't hear about most of these attacks because both the attacker and attacked keep them quiet.

That doesn't jive with your market manipulation hypothesis.

SolarWinds affected 100% of installations that updated their deployments during that 8 month window. Your 1% comes from the ratio of networks that were specifically targeted and received 2nd stage with all the goodies.

The reason why 2nd stage was only given to (relatively) small number of organizations - because the attack wasn’t ransomware, attackers didn’t have economical motives (in fact they were spooks on a government payroll).

EDIT: I can’t spell

Tinfoil hat, but that Solarwinds access was way more valuable than a ransomware payoff. Made sense to keep quiet with it.

Yeah, I'm guessing they're going for steady income over risking a serious retaliation. If the hack is serious enough, there will be consequences.

FWIW though (and I don't have easily available "sources") there was this immediate retaliation where Biden was like "we will completely prosecute these offenders" and within days DarkSide PR department said "Hey sorry we didn't mean to disrupt core services, we just want money" (sic)

So it's a spectrum

That's not even close to what happened.

The administration left it alone for days saying they'll let private business sort it out. (Default investigation notwithstanding.)

When a bunch of news media started reporting the group was Russian and then insinuate it was a state sponsored attack, DarkSide said something along the lines of, "We didn't realize this would start geopolitical conflict. We will be careful to vet clients more carefully in the future."

They also accepted a ransom substantially below their typical going rate. The Darkside people were probably shitting their pants, this is not what they intended at all.

Did they leave it alone for days? The FBI seized the ransom (claiming it was left in a Coinbase account) so clearly someone was doing something.

"Left alone" as in publicly and geopolitically.

The FBI investigated the crime as they always do. It was treated as a standard international monetary theft.

I mean it's reasonably close - but FWIW thanks for the correction, it's been a wild year

Sounds so spooky, do say more! Do you mean Jason Bourne / John Wick shows up at the hackers’ nest?

Well, if the attacker manages to kill a few thousand people, there's precedent for the USA going to war over it. It would depend on the host nation of course, if it was e.g. China, Russia or some state in their sphere of influence, it'd be different than if the hackers were holed out in, say, Afghanistan.

Or Raytheon, yeah, I imagine so.

Because there are plenty of zero-days the NSA can deploy if you step out of your lane.

It’s as much a political game at this point as anything.

If anyone thinks they can hide behind cryptocurrency and hold truly strategic companies hostage they are deluding themselves.

They’ll either end up hacked beyond their wildest imagination or facing literal hellfires.

It’s brinkmanship. When the devs literally die, they think twice.

At some point, some nation-state will get annoyed enough to do something drastic. That's what ended state-sponsored terrorism.

Or even a company. Uber's security chief once became annoyed with an attack from Nigeria. They traced the attack to an Internet cafe and sent some "lawyers" to talk to the attacker.

Someone tried a ransomware attack on the Teamsters Union in 2019.[1] The FBI advised them to pay. The Teamsters didn't pay. There were no further attacks. The Teamsters declined to comment. (For those unfamiliar with American labor history, trying to push around the Teamsters Union usually ends badly for the pushers.)

[1] https://thehill.com/policy/cybersecurity/558066-teamsters-re...

>That's what ended state-sponsored terrorism.

Wait, what? Have you notified the Department of State?


Iran is still on there. I'm pretty sure some people have been "annoyed enough to do something drastic" for quite a while.


You make it sound like the Teamsters Union could do something bad to the attackers, so attackers gave up, but in reality Teamsters just rebuilt from archives, which was perhaps an economical decision:

“Ultimately, the union decided not to pay the ransom based on advice from its insurance company, and instead rebuilt its systems based on archived materials, NBC reported.”

I wish you would have sourced the Uber Nigeria story instead.

It's in the book "Super Pumped: The Battle for Uber".

Given that paying the ransom only outs yourself as a potential repeated target who pays, it was a wise decision

Source: https://searchsecurity.techtarget.com/news/252502519/Repeat-...

Maybe a nation state is already behind it? https://cryptome.org/2021/06/Peck-Barb-1974.pdf

Was Edward Snowdon "https://www.youtube.com/watch?v=1GtVt6quoD8&t=78s" or a psychologically manipulated patsy for the good/bad guys & girls?

https://en.wikipedia.org/wiki/Full-spectrum_dominance isnt just about hacking a few computers, its about getting inside the brain of each and every one of us/you like a https://www.youtube.com/watch?v=lG7DGMgfOb8.

Or is this line of thought just a https://www.youtube.com/watch?v=wmin5WkOuPw&t=48s ?

If i recall correctly solarwinds was more of an espionage operation by russia government actors. Their targets were mainly government agencies in US. The ransomware attack are from private profit-seeking groups, although I remember the head of REvil tweeted once he was neighbours with KGB's number two guy so you could argue the distinction is vague

Attribution is quite hard. When the 3-letter-agency tools leaked a few years ago, one of their leaked tools concerned deliberate false attribution.

The solarwinds attack seemed to be about using a supply-chain attack to gain persistent access for recon and lateral movement. Pivot to Azure via Microsoft via SolarWinds software. Whomever it was tried to stay invisible for as long as possible. Once the game was up, they were not so careful about visible actions.

RansomWare is more smash and grab though it's interesting/sad to see the current trends of Supply Chain attack prevalence and Ransomware attacks converge.

> The SolarWinds attack affected only something like 1% of their customers, when it could easily have been 50% or more!

If it was me (it was not), I’d use it to gain persistance in companies like Kaseya, extending my beachhead as first priority. After that is basically game over, cleaning it would take making new IT systems from scratch. And lets not forget firmware…

> If it was me (it was not)


Sure. No melted craters, no fallout.

You'd need to be able to process all the orders also. Every company needs support to pay the random and unlock.

Also, at some point the military gets involved.

Yeah. If you take down 100 companies, it's crime. If you take down 100,000, it's an attack.

Correct, this won't get better until these groups are physically disbanded.

Didn't it only affect those who were unpatched hence the low percent? Current hack is 0-day.

Solarwinds was distributed by a malicious patch (through legit channels). So all orgs were unpatched and in fact all got at least first stage downloaded (if they patched during that window).

Steady income is definitely the way to play. You don’t want to make a demand so large that there’s cheaper alternatives of dealing with you.

Also you want the company to stay in business so it can continue generating revenue to extract future ransoms, and not have it lose a bunch of its customers from your repeated attacks.

I did write an answer before but now it seems like only Internet facing VSA servers are effected and some other measures may have stopped the attack. It could be all the servers they could find...

I would not be surprised if there is a market for the tools and the knowledge. That the real hackers just sells it and then other people do the attacks and thereby taking the risk. Similar setups existed with botnets.

DarkSide's business model was to professionalize ransomware attacks with a dedicated professional services IT model, finance, and helpdesk support.

Note that i n the case of SolarWinds, there was no demands for ransoms. It was good old state level spying, not a job to get few bitcoins.

> when it could easily have been 50% or more!

Was that down to slow patching cadence at 99% of companies?

In which case those customers have different vulnerabilities to tend to.

At some point you cross the threshold of "this is too much, drone them". Or send an assassin. Yes, even the United States does this occasionally.

I suspect the attackers know this. Or else they aren't in it for the money. One or the other.

Yeah unless they work from an office in e.g. Moscow. The US is powerful for sure but even they would think twice before droning a building in Moscow over some hacks, especially without concrete proof that's where they originated. At least I hope they would because if not then we may be closer to a world war than we thought...

The catchy rhyme being "warheads on foreheads".

I worked for an MSP that used Kaseya VSA. First used the SaaS version. Their "SSO" is not claims-based but an agent that may just run on a DC and copy NTLM hashes to the SaaS instance. Had an admin account compromised. Asked for logs from Kaseya. Attacker traffic came from a Tor exit node. They did zero ingress filtering. Much of their codebase is Classic ASP riddled with comments like "'fixed SQL injection." Beyond the bizarre HTTP traffic, the agent communication protocol is a black box with some VNC. Logging goes to SQL so you have to do custom work to parse or push that to a SIEM. Terrified. Moved to on-prem and stuck a bunch of mitigating controls (blocking known Tor exit nodes, blocking egregious injection attempts, etc.). Wrote custom scripts to ingest logs. I'd like to see a professional penetration test report against their software. It does not look good.

This sounds like something Computer Associates would buy and keep alive for another decade.

These kinds of games, and the all-nighter / weeks long nightmares they cause, make me want to leave this industry. We set up software on a lot of machines and then we answer a million ridiculous user questions until we finally resort to installing remote access so we don't have to stay up all night telling people what to type into a command line. Then the remote access gets hacked en masse. I'm pretty much at the point of thinking people need to learn how to write on paper and whiteboards again. Without a well-trained work force, this shit isn't resilient, and no technical priesthood can keep it running in the face of constant attempts to demolish it. It's too brittle, and the knowledge of the user base is too shallow.

Depth can be provided by reverting to older skill sets. Fallbacks. Businesses should not go down because their computers locked up with ransomware.

I pitched and wrote some software for a company a few years ago to automate a very rigorous daily process that used to take a lot of man-hours. Occasionally, local networks would go down and people would have to revert to the old way of doing things on paper. But as turnover happened at the company, fewer and fewer people knew the "old way". Now they've reached a point where they're locally paralyzed if there's a network outage. They have to call in senior management on their day off to run the shop. I realized I didn't do them a favor. I solved one problem for them and saved them a lot of labor, but I created a whole new problem of reliance on a system that's more convenient, but much less robust than the paper system they used to have. And this doesn't even take into account the potential for security issues.

I think we should try somehow to architect things with offline fallbacks and training for those scenarios. The pace of attack is unsustainable and we're losing the war. If the point is to keep business running, we will lose the war if we lose the skill base and knowledge that we had which was capable of running the economy without a screen in front of them.

[edit] Come to think of it, there's a great startup idea in systematically re-paperizing businesses for failover. Take all that business logic that got written into software, and turn it back into a set of worksheets and training manuals.

> I'm pretty much at the point of thinking people need to learn how to write on paper and whiteboards again.

Health IT here: won't happen.

You need your CT NOW. The patient is about to be opened. There is no time to wait for the printer and it's Sunday night. The radiologist is at home examining the data while the scanner runs.

And man...security is so bad and it's so hard to convince management to invest into proper security. Also everything that breaks or even slightly slows down workflows is just unacceptable.

I'm sweating hard with every wide scale attack out there expecting the next big thing to hit us. The targeted ones I just don't even want to think about.

Well, that's the scariest thing I've read all week. Just reading your level of stress between the lines here gives me the chills. Why is it so hard to convince them to take security seriously? Especially with hospitals, this should be a national security issue. The consequences are right in everyone's face now. In my case, an attack might be expensive, even dire, but no one would die. I know why I have a hard time pushing security reviews, they're costly and intensive and not sexy for management or investors. But things like this need to make it clear to the c-suite how quickly the wheels can come off.

It's hard because "we've been doing it this way all the time and nothing happened" is what I hear most of the times.

Most of the times I still "sneak" in improvements where I can without disturbing operations but the whole thing needs a proper overhaul and it always is, as we say here: "a dance on the razor blade".

What I hear from other colleges and contractors in the sector: it doesn't look better there. I don't want to leave out that there is a certain amount of IT personal which is responsible for it too. Most of them older guys (yes...they really are all guys) who also follow the mantra I mentioned above.

There is hope though...there is a certification requirement coming up here in Germany. It covers most of the basic security measures. We fail to cover a significant part of it. We've just passed one of the deadlines. Two are coming up and than there is a certification process. I've presented management with the measures we'd have to take to fulfil those. They've been ignored. The whole issue is being actively ignored or played down. The day will come when it'll be too late and I wonder what will happen. Wouldn't be surprised if I lose my job about it since somebody will have to be blamed or the certification issue will be "made to work out" somehow. Seen that happening before.

I’ve been thinking it would perhaps be a good idea to shut down the power grid a couple of days a year to get this kind of resilience exercise.

> We set up software on a lot of machines

Windows, right?

In the case I had in mind, the company runs a mix of windows and os x. And some android. Luckily it's mostly mac in the shops now, but personal laptops and tablets that connect to the LANs are also involved, and definitely the most dangerous point of failure.

Yup, if I had to run a company, it will be macbooks and iphones with MDM, like by jamf.com. That will cover device security. Then SSO, separated networks and no Windows whatsoever.

These digital networks and devices have become so complex we can’t reason about them, or in any case can’t easily reason about them given the resources available to most of the organizations running them.

However, from what I’ve seen, most of these attacks are successful because these organizations are simply neglecting best practices (e.g. patch management, whitelisting, security awareness training).

> or in any case can’t easily reason about them given the resources available to most of the organizations running them.

I really feel this. Any new piece of software needs a level of ongoing maintenance that no one seems to realize, not even many software engineers I've worked with.

You can't "just" toss a binary onto a VM and forget about it. But all the work required to secure that and keep it secure is so invisible to management.

And because the work is invisible, it might even hamper career growth. So good luck getting either management or devs to prioritize all the security tasks they should be prioritizing.

Mostly, they're neglecting training their employees to keep the business running when the software is down.

like everything else in america, #1 priority is fèeding ceo salary and shareholder value.

everything in corporate america is derived from the growing wealth inequality and these shake downs are precisely targetting the glut. soon enough, itll still be cheaper to have a bribe fund, just like tax evasion lawyers, lobbiests and the rest of the feeder classes than a holisitic defense.

Such a brave meme. It might carry some water if the same problem didn’t apply to every country with both open and closed source projects.

These particular decisions are driven more by compliance and CYA. It only feeds into executive compensation as executives avoiding getting fired. Even if you're the executive who approved this integration, you'll just say "I chose a known vendor with a respectable client list."

This is a tiresome, meaningless religious mantra nowadays.

Yes there is corruption. No not everybody is corrupt. No it does not only exist in USA nor is USA anywhere near the worst. No you can't blame anything and everything you don't like on corruption and greed.

Perhaps, but of all the leading developed nations on Earth, the US has a particularly corrupt government that sells itself to the highest bidder thanks to Citizens United and armies of lobbyists.

Our healthcare, prison, and student loan systems, for example, prey on US citizens without repercussions at lengths that don’t fly in most developed countries.

I think it’s safe to say that corruption and greed are at the root of most problems in the US, and it’s important to call it like it is.

Oh, you’re one of the kids who thinks that the US had less regulatory capture and influence peddling before Citizens United. That is not how it was at all. Money simply follows the path of least resistance.

Market capitalism is greedy and brutal, but the discussion here is about ransomware, which is one of the things the market should be well equipped to solve. Rather than throwing broad shade at the system in general, consider the opportunity here. Faced with a threat to the increasing automation they rely on for YoY growth, corporations could react by ensuring better job security and higher pay, better workplace conditions and better training, to create more resilience. The market could support those shifts if they see the danger of relying totally on non-human decision making at the local level. The Russians might even be doing us a favor if we're adaptable enough to take advantage of what we're learning from it.

Yes yes I know the scriptures, and yes I'm possessed by the secular-devil (who is that today? Putin? Hitler? Trump?) for not unthinkingly reciting them verbatim.

> of all the leading developed nations on Earth, the US has a particularly corrupt government that sells itself to the highest bidder

I would contend with that. The US government is just very visible. I know HN likes to glorify European nations but we're really really good at wasting taxpayer money, too. It's just less lobbying and more knowing the right people here.

Oddly explosive headline, considering:

> It is not clear what specific companies have been affected - a Kaseya representative contacted by the BBC declined to give details.

So why "colossal"?

> "This is a colossal and devastating supply chain attack," Huntress Labs' senior security researcher John Hammond said in an email to Reuters news agency.

The BBC is going with "colossal" in their headline simply because the guy who discovered the incident said so?

Hacker News hit by "oddly explosive" BBC headline.

The next such attack -- which will be much larger than this one -- will be called super-colossal. The next such attack.. :)

Beautifully executed.

The BBC headline uses 'colossal' in quotation marks. So yes, it's a quote.

After the Equifax breach, everyone learned that until there are actual repercussions for cyber attacks (like fines and people going to jail for negligence), if you can weather the storm, over the course of a year or two, there is effectively zero impact to your bottom line.

You can also see this in the Solarwinds stock price. Year over year, they are down a hair under 4 percent... After being directly responsible for one of the most impactful cyber incidents yet. Hell, if you invested in January, after most of the stuff blew over, you would be up nearly 20% on your investment.

There is even a perverse incentive to not do things and just get cyber insurance to cover you. Since these underwriters generally have no fucking clue what they are doing, you can actually make money on a cyber intrusion if you play your cards right. Only now that insurance companies have paid out the nose with ransomware incidents have they started to wise up. Having worked in the space, its absolutely bonkers what we accept as normal business practices with regards to cybersecurity.

“After the Equifax breach, everyone learned that until there are actual repercussions for cyber attacks (like fines and people going to jail for negligence), if you can weather the storm, over the course of a year or two, there is effectively zero impact to your bottom line.”

It’s even worse than just weathering a storm. Lax security has been incentivized. The Equifax CEO, Richard Smith, stepped down shortly after the public became aware of the breach, with a $90m severance package.


It's almost as if making shareholder returns and CEO pay the only indicator of company success creates terrible consequences.

Juice the returns at all costs for a few quarters and then walk away with riches from total ruins, you say?

This is the way.

Long term shareholder returns are directly correlated to the long term company success.

It's always such an odd criticism to think of "shareholder returns" as a pejorative.

TheOtherHobbes said "only", as in "to the exclusion of all other concerns". Where's the pejorative?

They key is long-term

Success for the company, at the expense of everything else (the environment, public health, individual privacy, etc).

It's worse than PII leaks and CEOs stepping down. Lax security has become scary. The U.S. Nuclear Weapons Agency was breached shortly after SolarWinds. Let's also not forget about OPM.

Except everyone forgot about OPM.

What is OPM ? Office of Personnel Management ?

Yes. In case you're asking what OPM is and not just the acronym intended, OPM is an agency that manages and maintains stewardship of a stupid amount of information about all employees that work for or closely with the federal government.

Background checks and investigations, healthcare related policy information, etc. e-QIP, managed by OPM specifically, collects a lot of highly sensitive information on federal employees working in the national security ecosystem was hit:


Holy hell... no wonder they snuffed it out in the media.

I live in Eastern Europe. A local city with a population of 300-400k was hit with a near total ransomware attack. The hackers asked for 400 bitcoin.

The mayor answered to them on TV "You fools, we still do most things on paper here ! We'll just spend the week-end installing windows and word and F** Y* !!!"

I sometime find wisdom in the approach from olden times :-)

> Holy hell... no wonder they snuffed it out in the media.

The OPM hack wasn’t ‘snuffed out’ by any means - it was fairly well covered for a cyber attack of it’s era. Perhaps it wasn’t covered much in your part of Eastern Europe, but it was definitely not covered up.

The fact that some people have forgotten about it is a completely different issue.

I do watch major networks in US and the coverage on CNN and FOX amounted to 'Russia did it' or 'Russia prolly did it'. There was no meaningful coverage of impact or what the Solarwinds hack amounted to. To be frank, compared to coverage of a hurricane, it got minimal necessary coverage. I agree with parent's assertion that it was snuffed out.

I had recently just moved back from NYC at the time. I was kinda still plugged 24/7 to the US media sphere.

But it's true that I don't remember it at all, even though I worked in a field parallel to CompuSec and usually notice those events.

They should also have the old wisdom of not connecting critical systems to Internet.

Those were not "critical systems". It was all the desktop computer used for basic office work (email, word processors, etc).

Major mission critical systems are managed by the country's Ministry of the Interior, and haven't had a major hack (yet), as far as is publicly known.

And besides, how are those poor souls gonna connect to Facebook during their mandatory 10 o'clock coffee pause ?

It is becoming harder and harder to install software on systems without internet connectivity. More and more things assume they can hit maven or npm or random other places at deploy time; even expensive well regarded third party software. At least Golang deploys are ok. (Source: running prod systems with a mandate of no internet connectivity).

Stupid me never learned the trick of failing upwards.

Isn't Equifax a government organization? How do they have severance packages?

It's a para-state agency; while Americans don't have ID cards because they're afraid of surveillance, a private company having a complete database of everyone and veto power over mortgages is fine because it's a private company.

There's three companies doing it, so they possess the holy blessings of the all-knowing market \s

The existence of credit scores has tangible benefits that we take for granted. Without such databases we would all pay much higher interest rates and many more people would be denied loans. Very wealthy people would have little trouble, but low- and middle-income people would find it far more difficult to buy a house or a car. The reason it is better to be run by a private company than the government is not that surveillance, but the near-certainty (at least after everything we saw happen over the past 5 years) that a government credit scores agency would be politicized. We would have the same problems we have with equifax, and a whole new set of problems as e.g. the political party that rewrote the tax code to punish people who voted against them tried to weaponize credit scores.

As seen from another capitalist country, namely Switzerland, I take the "higher interest" rates as a tired argumentative "canard". It's a false idea perpetrated by lobbyists.

We don't have such databases. The difference here is that the bank's mortgage divisions have much lower profits, because checking somebody out is actually done by humans. It costs the credit provider more. US style mortgage broker do not exist.

Low- and Middle- income people here do not have houses because of high real estate prices due to very restrictive zoning (the country is small), and on average much, much, much more expensive construction than in the US. Here people expect a fully concrete house, near-to-passive level insulation, with 30-40 years free of any big renovation.

In conclusion: we do without an Equifax just fine.

...so low- and middle-income people are not buying their own homes under that system, which is exactly what I said. What is the disagreement here?

You say that interest rates are not higher, but that is a meaningless statement if people do not generally buy their homes on credit. Low- and middle-income Americans typically buy a home using a mortgage, and credit scores are an important part of that system.

My opinion point is that maybe if the US tried to do old style approach to home ownership, old fashioned banking, it wouldn't need that many artifices like rating agencies. Why I think that:

Your position is that the lack of a well informed credit market would make interest rates high, precluding acquisition of houses, hence the need for rating agencies.

My position is that truthful, complete information is enough to keep rates low, a market for that information is not necessary for assets which are not liquid (houses, mortgages). Swiss mortgage rate oscillate between 1-1.5%, depending on your financials.

Absolutely everybody buys houses and buildings on credit in Switzerland, due to huge tax deductibles. Those who don't are a rounding error around 99.9%, mainly due to some rare people's estate planning triggers.

Selling cheaper houses and apartments at lower prices has been repeatedly in the last 20 years (as low as a third of the usual price range). They doesn't sell.

Swiss are conservative, they tend to like long term investments with low degradation risk, regardless of current market price levels. Hence high prices, because they want high, long lasting quality.

Again nothing to do with credit information markets.

Yet another reason why I think Switzerland would be a great country to move to.

Yes and no.

It's not as good as it once was, and purchasing power is slowly but certainly going down. Everything is tightening up. Switzerland is extremely integrated into the western money circuits. If it goes to shit in the US, it'll follow suit at a much slower pace.

However, Eurasia is replete with countries which try to imitate Western European successes by applying the same receppies. If you can swing it, the purchasing power is 3-5 times larger on the same net income, and you don't have pesky invasions of your private sphere at each corner.

Also, as a Swiss, I can tell you that past the superficial welcome, we're a mountain people. We're really not as warm as others peoples. Over time, depending on your character, it may accrues and impact quality of life.

We are also very disciplined in a lot of aspects of life, even outside work. That is a problem for some over time.

But if your character fits, you'll have a blast.

The state has broad enough illegal/illegitimate and legal surveillance tools that a nationwide ID card is unnecessary.

Equifax is a private company.

You'd think that one of the credit bureaus responsible for maintaining the most sensitive data, and making it difficult for people to get affordable housing would be a government institution, but nope.

Would you rather have a government agency assign credit scores? The abuses would be rampant. Right now there is one party openly pushing to restrict voting access to people who are likely to vote for the other party, and a few years ago that same party enacted a new tax code that almost surgically penalized the residents of states that supported the other party; do you really trust such politicians to set up a fair credit rating system? I can see the headlines already: "SCOTUS rules 6-3 in favor of GOP effort to depress credit scores in Democrat-leaning cities," or perhaps, "Northeast states fear wave of foreclosures following GOP overhaul of credit score bureau," or maybe, "Whistleblower: President pressured credit rating agency to attack CNN, NYT reporters."

Equifax and the other ratings agencies have plenty of problems, but none of those problems are solved by having the government run things and many new problems would be introduced.

Then why is the SEC public, it could arbitrarily issue fines and fuck with the share price of any company that didnt donate to your party, maybe it should be private too?

Different role, different scope, different situation. The SEC has limited power to target individuals compared to a credit rating agency. It would be a scandal to politicize the SEC, but it would not be the sort of nightmare that a politicized credit rating agency could become.

It is also worth pointing out that both the credit ratings and audits of publicly traded corporations are conducted by private-sector companies, not government agencies. The SEC's primary role is to ensure that the rules are being followed, which is a straightforward law-enforcement/regulatory role that makes sense for a government agency.

>Would you rather have a government agency assign credit scores? The abuses would be rampant.

Do you think the abuses are any less rampant when power is privatized? The main problem that would be solved by a government institution is a pathway for transparency and citizen recourse against questionable practices. It's admittedly not a lot of transparency or accountability but it can be far more than currently exists.

People talk about government corruption and sure, there's lots of it, but there's just as much if not more private corruption hidden behind privacy protection veils. At the very least, there is some degree of transparency with the government and we can in theory hold them accountable with explicit rights granted to us (more-so than private institutions).

I cannot hold these private institutions that have gamed the system so far they're beyond my grasp accountable for their actions. Ill start a credit rating agency tomorrow and compete with Equifax, Transunion, and Experian so through market forces of competition I can fix these problems! Consumers and market forces will fix these problems! Yea, right, give me a break.

This whole government bad, private good, anti-communism/socialism/whatever argument has grown tiring because we're at a point now where you can chuck private institutions in the same gutter of corruption as different systems of government. We played that fiddle and gave private institutions the benefit and here we are, with rampant corruption in concentrated pockets of business as well, governing our daily lives with little oversight or means of recourse beyond avoiding the system or hoping some competitor can actually change things.

Privatization works well when you can actually hold institutions accountable, when there are competitors that actually compete and give consumers the option to vote with their wallets. When that doesn't exist, it's far worse than a US government agency managing it. It might be cheaper but there's probably a good undesirable reason it's cheaper than a public institution that isn't related to poor management and basic optimization practices to improve efficiency. Those efficiency gains probably exist because the institution is doing something it shouldn't be doing, focusing on profit margins over implications on the consumer.

Yes and: Since contractors aren't subject to FOIA, privatization is a time honored strategy to move activities off book.

Did I say anything about communism? No, that is what you brought up. I mentioned possible abuses that are specific to a government agency, abuses that are the result of politics.

There is no reason to think that a government agency would be any more transparent than Equifax et al. are right now. Consumers have the right to receive a free credit reporter from these companies, and the right to dispute information in that report (also free). Maybe there is a need to adjust the regulations in order to combat particular abuses or problems that are happening right now. That does bring up the question of what specific abuses you would like to see fixed -- you did not actually mention anything in particular that Equifax is doing or how a government agency would avoid such a problem.

The previous president spent 4 years trying to use government agencies to punish political opponents, and just before leaving office he filled those agencies with loyalists in an attempt to sabotage his successor, all without regard for the effect such actions might have on the public. Those are forms of abuse that is specific to government agencies and it would be a disaster if it happened at a credit rating agency. This is not an argument that the government is always worse than the private sector; it is an argument that when it comes to something like credit scores the government should not be in charge.

It is a publicly traded corporation.

This isn't really true. Stock price is not an indicator of a company's "bottom line".

As someone who helps respond to major breaches at big companies, these types of breaches often result in enormous expenditures on company-wide efforts to close security gaps or revamp processes. Either a regulatory agency, or more often the company's board of directors, will make a mandate to the C-suite that something must be done. Some of these expenditure campaigns are low-visibility, some even to the employees of the company, and they are usually not very sexy or noteworthy, so you won't read about them on the front page of CNN but they do happen and they are very costly to the company (in the ballparks of tens to hundreds of millions of dollars).

I do think there should be harsher punishments in the form of fines, etc. But to say that there is "zero impact" just isn't true.

Dude, if you look at Equifaxes and Solarwinds EBITDA/earnings statements following their respective breaches, you will clearly see that there has been no major impact to their bottom line. Sure, expenses rise a bit for a short period of time, but these are not catastrophic by any means.

I mean, I'm looking at Solarwinds last earnings statement and comparing quarters from last year to now, they are up about 3.5% in revenue (3/31/2020 vs 3/31/2021).

>Dude, if you look at Equifaxes and Solarwinds EBITDA/earnings statements following their respective breaches, you will clearly see that there has been no major impact to their bottom line.

I'm looking at Equifax's 2018 statements right now. With Operating Revenue of $3.4 billion and profits of $850 million, they had $400 million of expenses related to the breach. "No major impact" my ass.

If you compare year over year, many of the things they attribute to the breach are actually just IT/overhead costs they were able to shift to a loss. If you look at their EBITDA, everything is essentially static. In the grand scheme of things, it really isn't a huge impact to them.

Lets say you are a CEO: If you underspend on technology/security by ~50-100m/year, for 5 or 10 years... then have a bad breach, which costs you 400m, what do you get?

A: A Ferrari, because you saved the company 500m dollars and got a cyber insurer to pay for your technology/security program.

I'm not even joking you, I have been in meetings with a CEO, CIO and CISO, where they literally joked around that they should have more breaches because they actually made money on the intrusion and that they were able to upgrade a bunch of stuff they were planning on upgrading next year anyways.

>If you compare year over year, many of the things they attribute to the breach are actually just IT/overhead costs they were able to shift to a loss.

No, it's not. Read the 10-K. It includes pages upon pages of the breach-related expenditures, including hundreds of millions of dollars spent on extra stuff like credit monitoring, legal fees, and professional services costs. That's not "just IT/overhead costs".

Just because a company was planning to spend $400 million anyway doesn't mean that having to spend that $400 million on breach-related expenses is no impact. The budget doesn't just come out of thin air, it gets allocated from other places. Spending $400 million on breach-related expenses means not spending that $400 million on something else like product development, research, marketing, or other company initiatives. The impact is enormous.

>In the grand scheme of things, it really isn't a huge impact to them.

You have no clue how businesses work if you seriously think that an additional, unexpected $400 million in expenses (almost 50% of their yearly net profits) "isn't a huge impact to them". That's really all that has to be said here.

> You have no clue how businesses work if you seriously think that an additional, unexpected $400 million in expenses (almost 50% of their yearly net profits) "isn't a huge impact to them". That's really all that has to be said here.

You clearly have no clue how it looks inside the board rooms and executive offices of some of these huge companies. This type of stuff is treated the exact same way as if a 400m building burns down.

define: impact

2) have a strong effect on someone or something.

My point still stands... If a company can weather the storm, there is no long term impact. If you look at equifaxes breach, it hasn't depressed their revenue. They haven't had to massively changed how they operate or had to pivot into new businesses. Over the long term, it has had very little effect on the company long term, which is my entire point.

>You clearly have no clue how it looks inside the board rooms and executive offices of some of these huge companies. This type of stuff is treated the exact same way as if a 400m building burns down.

I sit with CISOs daily discussing this stuff. $400m expenditures is enough to scare the shit out of them. A $400m building burning down would have CEOs fired (see: Equifax CEO being fired after breach). I don't know what fantasy land you live in, but you're either delusional or lying.

>If a company can weather the storm, there is no long term impact.

That's not what impact means.

>If you look at equifaxes breach, it hasn't depressed their revenue.

This means nothing. It's possible that with an additional 50% of their yearly net income freed up, they could have massively increased their revenue by spending that on product development or sales efforts. You cannot draw any conclusions simply from the fact that their revenue hasn't decreased.

>Over the long term, it has had very little effect on the company long term, which is my entire point.

On the other hand, it may have had an enormous impact. In a time period where every other company is seeing massively rising profits and stock prices, Equifax has been relatively stagnant. Your point has no standing.

I agree with most of your points, but I think it's worth noting that "they didn't do as well as they could have" and "their CEO stepped down with a 90M severance" is a tough pill to swallow. Like, yes, Equifax could be doing better had they not been breached. I'm sure CISOs and board members are quite unhappy with a 400M dollar expenditure. But I also think it's very fair to say that that's getting off easy.

YoY it's a bad thing and makes for a bad year. But longer term the effect seems to have been negligible.

That could be because the $400m would likely have gone on dividends and remuneration, not investment.

> A $400m building burning down would have CEOs fired (see: Equifax CEO being fired after breach). I don't know what fantasy land you live in, but you're either delusional or lying.

In what world does getting 90m $ to leave the company constitute "getting fired"? That's early retirement.

> In a time period where every other company is seeing massively rising profits and stock prices, Equifax has been relatively stagnant.

So, it will take them 2 or 3 years longer to reach some arbitrary stock price. Certainly an earth shattering experience.

Equifax’s stock is up 50% from a year ago. I’d say this hack did nothing bad for their stock.

Seems long covid fogs the market analysts brains too.

Roughly 10% of revenue is something, but not that big of a deal, especially since their overall revenue is up.

Don’t you think stronger consequences than that should happen when a company unintentionally discloses tens of millions of people’s personally identifiable information that has been collected without any particularly explicit permission given by those people?

Credit agencies hold a special place in the US economy, and when they messed up this badly, the team threat of some near-going-out-of-business level consequences seem like the only way to truly get other companies to take this seriously. Especially considering that there are other credit agencies in the country - they don’t have a monopoly on this.

Their business model is weaponizing this information against consumers. They work for the businesses that do lending, not for the recipients of the loans.

And you would think that given their one job is to supposedly safeguard this info, the consequences would be more severe or we would re-think this entire business model of consumer credit, but our society is not capable of that kind of consumer advocacy. Likely due to some powerful interest's bottomline.

That doesn't mean anything. in such a year their products should have flown off the shelves.

Remote monitoring\management? in COVID year? just 3.5%

that's horrendous

Are you sure about that?

The customers who had experience with remote work and already knew that SW products would help them in this situation was a fixed number.

The number of companies who had no clue about how to do remote work, and after haphazardly had to switch to it may still have no idea that you need to use products provided by SW.

Also do you really need any of that to do remote work?

Of course not.

I'm sorry but I have pretty good info about SW. I can tell things are rough there.

More than anything, it proved that their model is flawed.

Just the number of gov agencies that are forced to stop working with them is a major blow.

You missed my point.

I agree they are not doing well, but I also do not see why they should’ve, even if the breach didn’t happen.

> they are up about 3.5% in revenue

Revenue != bottom line. Bottom line is profit, ie revenue minus expenses.

Please point out some 10Q/10K filings that go into detail about these enormous expenditures related to security breaches.

The SEC EDGAR database [0] is where you can find public quarterly financial statements and forward guidance from management (which will definitely mention the security breach related expenses), for every US-listed publicly traded company. Good luck!

[0] https://www.sec.gov/edgar/searchedgar/companysearch.html

Literally the first company I pulled up, Capital One, has this in the 2020 10-K:

>During the year ended December 31, 2020, we incurred $66 million of incremental expenses related to the remediation of and response to the Cybersecurity Incident, offset by $39 million of insurance recoveries. To date, we have incurred $138 million of incremental expenses, offset by $73 million of insurance recoveries pursuant to the cyber risk insurance coverage we carry. These expenses mainly consist of customer notifications, credit monitoring, technology costs, and professional and legal support.

Go look at Equifax's 2018 10-K and it has pages upon pages talking about the impact, including:

> During the year ended December 31, 2018, the Company recorded $401.2 million of pre-tax expenses related to the 2017 cybersecurity incident and insurance recoveries of $75.0 million for net expenses of $326.2 million. Costs related to the 2017 cybersecurity incident are defined as incremental costs to transform our information technology infrastructure and data security; legal fees and professional services costs to investigate the 2017 cybersecurity incident and respond to legal, government and regulatory claims; as well as costs to provide the free product and related support to the consumer.

For Equifax, there is also an additional $112 million (net, after insurance recovery) in breach-related expenditures in the 2017 10-K.

I'm not sure which number to use, but Capital One had either 2.4 or 5 Billion in income... .066 billion on cybersecurity remediation isn't an existential threat.

Aren't they just fixing leaks in the ship that should have been adressed years ago? If these are expenses on their infrastructure, thats not really losses, its an investment.

Losses would be their customers abandoning them in droves, or having to pay out massive fines.

If I am being frank, based on anecdotes I have heard, Equifax had their heads so far up their asses that they basically had to rebuild their entire infrastructure because it was an unmitigated disaster.

This was a conscious business decision to not make the necessary changes to address their infrastructure.

Share price doesn't regularly continue to go up while profits continue to contract.

I don't believe you. Give me an example of a company spending 100's of millions as a result of a breach. Companies understand it costs them nothing and if there is a cost it's trivial. When there is no penalty or the fine is a pittance, no company is going to spend 10's to 100's of millions. It makes no business sense first of all and secondly they can blame a foreign actor to mask their own incompetence.

Honestly, I'm shocked by this comment.

As if stock market is a perfect representation of a company performance, it is highly distorted\manipulated market.

SolarWind is fucked, they have a massive drop in new customers, I work with dozens of companies that are now plan to completely abandon their suites(those things take time).

Insurance is a trap. once you read the small letters, they don't fully cover the damage, usually only direct. Some have refused to pay due to some shady conditions that they insert into contracts to deceive customers(like any other insurance sector)

If the stock market has distorted the price of SolarWinds that badly, as per your analysis, that's probably a sign that the stock market is massively overvaluing everything, and that we're headed for a gigantic crash.

Which by coincidence is exactly what Michael Burry, the guy who predicted the 2008 housing crash, has been saying recently.

That's not what I've said. I'm not a financial expert by any means but I think the stock market has proved again and again that it is not reliable and can be manipulated easily.

People short-squeezing stocks, shooting their "value" by 30x in 2 hours making them millionaires.

Hedge funds manipulating stocks to meet their portfolios

IPO's in billions of dollars for new, non-profitable startups just because of hype. when you look at the balance sheet it makes no sense.

The market is volatile and inflated, it is as clear as day. Whether there will be a crash? that's beyond my level.

It will crash, predicting when is something else. I was correct in 2001 and 2008 but I was wrong in the past years as the market has been overheated for quite a while (I thought we would've crashed already) and both in stocks and recently in crypto, I have been hearing 'this is the new normal, all is different this time around'. Which is what people always say just before the carpet gets pulled.

> that's probably a sign that the stock market is massively overvaluing everything

No it's not. The performance of stocks was always only weakly linked to actual company performance.

There are countless examples of companies that are hardly profitable and not even a tenth the size of their competition, but are valued at twice the price of some of their competitors. It's mostly made-up prices created entirely on hype that often make less sense than the soccer trading card market.

I've been hearing the "we're headed for a crash" thing with the same logic since at least mid-2019. Either we're not heading for a crash, or the market has become so irrational that it doesn't even matter any more, and we can build castles in the air forever.


We can definitely build castles in the air for two years.

In 2005 it was clear to some thay the market was heading for a crash, but it can take years.

I believe the market should in theory rise with inflation. Doesn't seem too crazy all things considered.

Isn't inflation above gains essentially a devaluing of the market? If the stock market goes slightly down and inflation ramps up significantly isn't that the same as a crash?

Correct, and that is why you should always analyze inflation-adjusted returns for any long-term investment.

The most expensive and valued stocks are "essentials" they dann just increase there price with inflation.

I dont understand the problem with inflation..

Inflation is not a problem; unexpected changes in the rate of inflation are the problem. There are various reasons, but the most important is that a spike in the inflation rate leads to a spike in interest rates, which is generally harmful to businesses (loans are harder to repay, customers are less able to buy, etc.). There is also a second-order effect: rising interest rates reduce the value of long-dated bonds, which reduces the available investment capital (or worse, it can trigger margin calls and create a "contagion" effect).

Yes that’s exactly my point. People look at absolute value of the stock market but what matters is the relative value to the dollar.

How is this comment shocking to you? I actually was using the stock price and public information on their earnings to make a point. The point being that no, these companies aren't losing customers in droves and if you look at their performance from a 3 or 5 year perspective, most breaches have had very little material impact on the companies.

I disagree with you on SolarWinds being fucked... Sure, lots of folks are going to drop it, but they are closing new deals. The types of people that buy things like SolarWinds aren't buying the products because its a good technology.

Not sure what insurance you have been looking at, but many of the larger businesses will essentially write out what they want covered (for example IR, infrastructure replacement due to hacking, business loss due to downtime, professional service implementation, support, PR assistance, etc.), and then the insurance company will come up with a price based on their calculations of risk.

Sure, if an SMB goes and gets a "cyber policy" they are gonna be lots of technicalities, just like a mass market homeowners policy.

> like fines and people going to jail for negligence

Being bad at your job is not negligence, nor is underestimating the threat.

It’d be nice to see consequences but I really don’t want to have the government locking people up for being well-paid fuck-ups.

Don’t some of these companies have… shareholders?

> I really don’t want to have the government locking people up for being well-paid fuck-ups.

If you go to a doctor and he fucks up: he (or his insurer) has to pay you. If he really fucks up, he ceases to be able to practice medicine.

The same with nurses, lawyers, accountants, architects and other professionals.

Software's much better—then they point to the "we take no liability for any errors" clause in the contract and everyone carries on as if nothing ever happened.

afaik if a doctor fucks up his rights to practice medicine are taken away by a board of (probably) doctors after a examination of the case. Although this sounds all well and good, I've read countless accounts of this not happening as much as it should be happening, almost similar to the police not taking action on their own officials who go bad, corruption runs deep in our systems and imo our psyche

I think a top down approach to enforce anything at scale is never gonna work until people decide to respect their place in the world and do the due diligence from bottom up

Where does "being bad at your job" stop and "negligence" begin?

Some jobs come with certain responsibilities. Of course we need to have some leeway for e.g. doctors making honest mistakes – they're only human after all – but at some point that stops.

People shouldn't go to prison for being fuck-ups.

Those negligently responsible should be fined and go to prison for leaking private data, endangering physical safety, possibly for compromising national security, damages from the toxic sludge they produce etc.

Note that the US does deploy it's national security forces to fix some of those fuck-ups, and at least threatens to use physical forces, so it's not just a private or civil matter.

If you screw it up with a building or a bridge, you might go to jail, and we as a society are fine with that. Why not in this case as well?

It is 100% possible to be really good at InfoSec, do everything right, and still be breached.

I think there’s a (very simplistic) view of IS, where it’s a black and white process of just engineering everything ’correctly’. It’s not like that in the real world…

Nobody is saying you should go to jail or get fined for getting owned by a 0day. I don't think that it is unreasonable to say that someone is negligent in having a CVE from 2014 unpatched, which then allows your customers to get compromised.

In case of Equifax, for example, they're in a quasi-cartel with only two competitors.

It's quasi-monopolistic. It has the same problems : nobody gives a flying furry about actual performance.

The insurance is a joke. I’ve seen requirements from companies that we want to do professional services for that require us to carry $5mil in cyber insurance, but nothing at all mentioned as to requirements on security governance and or policies/procedures.

Nothing will change until government regulates it. Same with auto, airlines and rail. They did not make their products and services safer by choice, they were regulated to do so.

> Only now that insurance companies have paid out the nose with ransomware incidents have they started to wise up.

Exactly right, and eventually they will GET A CLUE, and require serious security audits to get a sane price on incident insurance. Otherwise they will make you pay gobs and gobs of money, and it will just be cheaper to be sane about your security posture.

Otherwise there is zero incentive for the insurance companies to keep paying out the nose on policies they aren't making money from.

This has happened to police stations, as they get mismanaged by idiot police chiefs, the insurance providers say.. uh we aren't going to insure you anymore unless you fix your sh*t. As but one example: https://www.theatlantic.com/politics/archive/2017/06/insuran...

I see this happening to cyper security policies also, they(insurance companies) will wise up or go broke.

UK companies act hundreds of summary criminal offenses covering all aspects of corporate responsibility for any director.

A 1977 case precedent established in the event that a director relies upon the advice of a accountant for making company directions, he or she will be liable to be banned from holding a directorship for life. The appeal failed. This is because the only essential role of a director is to be themselves a competent assessor of the company affairs.

If you can't knobble the board of a UK limited liability company for letting go their own primary competitive asset (the more important consideration for the law designed to govern the behaviour of directors in fulfilling two goals : justify public indemnity to the extent of any shares they own in the company in the event of collapse ; and do their job without prejudice to the shareholders or the crown treasurer to pay negligence.

Summary criminal charges are convicted on bringing proof and a judge not being shown disproof. Criminal intent doesn't come into it.

Hell, if you invested in January, after most of the stuff blew over, you would be up nearly 20% on your investment.

Yeap, I did that with Ubiquity after their incident. Bought at $275 and the stock now is 12% higher. Seems like a good strategy, and I'm looking forward for similar incidents in the future.

It's not bonkers.

We have 20-30 years of data on cyber attacks and Cybersecurity is not that important - https://ubiquity.acm.org/article.cfm?id=3333611

Larger the dumps get the harder they are to exploit or do serious damage. I can hand you all my orgs data and 200 people who work with it everyday and it will still take you years to figure out what anything means.

> I can hand you all my orgs data and 200 people who work with it everyday and it will still take you years to figure out what anything means.

Ye what are you supposed to do with the information. I worked at a place that is paranoid for data leaks of non personal data, like source code.

Even if their direct competitor got the sources they would have almost no use for it since it is an undocumented mess. The source without the dev. departments is useless.

The same applies for business strategy if it leaks. Which competitor is nimble enought to change anything based on that data.

We have operating systems for which remarkably few known exploits have ever been found.

VMS now "open" and recently running in a VM on Xeons.

I have always suspected that the silence that resounded suddenly about security prowess of VMS coincided with the release of extensive POSIX compatibility layers and the vaunted ports of years old open source nix wares as a excuse to play buzzword bingo at that time. But anyone writing a native VMS application I'd firmly embraced by a deep architecture designed to provide accounts for the time when DEC silicon and their own leading fab was creating a explosion in processing power and the number of users capable of being supported by a OS that has still incredibly well integrated system programming tool chain languages including Digital BASIC that can do about anything BLISS can low level. This virtually (sorry) made it a overnight imperative to get the security right and tight. Alpha had hardware security rings almost certainly to give VMS the chance to serve the maximum number of users and steal account wins.

Until it hits something important.. also, data breaches are just one form of cyber attacks.

We do not see a change until shareholders truly get hurt and stock price dives because of neglience. Investors would be suddenly interested in cybersecurity. But I do not what would be the best mechanism to cause this.

I was shocked at how minimal the impact was on Garmin, given that their customers are consumers who are trusting them with very personal data.

I've always said: "There's surprisingly little money in correct software."

hypothesis: security failure by a service provider is evidence of winning at externalizing costs.

strategy: find SaaS corps responsible for catastrophic cyber-attacks and buy them on the the dip?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact