The fact this was paid off, and paid off so rapidly means that targeting major infrastructure for massive payoffs is going to become more and more prominent. The next time though, it'll be $50M. I work with people in the oil fields and I know the numbers they are playing with and the fact that a single well being down can easily be $100,000 lost per hour. So obviously they want these systems back up fast.
$5M for shutting down that major of a pipeline seems like too little, unless, of course, they weren't expecting the company to even pay. Now that these actors know that the oil (and quite likely other utilities) are more than willing to pay big bucks to get back online, they will be targeted far more.
I think these ransoms are net good. I'd rather greedy hackers shake them down for money then having the country get crippled by political terrorists or enemy nation states that can't be negotiated or reasoned with.
There are lots of infrastructure management teams taking security more seriously than they were a month ago. That alone is worth more than $5M
I think you're kidding yourself if you think a company that gets "hacked" by off the shelf cryptoware is going to step up their game enough to have any chance of stopping a targeted state actor.
The fact they caved so quickly tells me they are years away from a reasonable security posture.
It gives other companies with more responsible practices an opportunity to get a competitive advantage from their failure.
Without widespread ransomware scammers, the risk of getting compromised is just theoretical, not tangible. Companies can get away with ignoring security concerns for a long time and might never be impacted by it.
Thus, companies which are paying a premium for better security might never be able to benefit from the mitigations they are implementing, and could be outcompeted by the companies which simply got lucky enough to avoid being attacked.
Eventually we end up with major too-big-to-fail megacorps like Equifax getting hacked by trivial exploits because nobody took advantage of them when they didn't have such a strong market position.
> I think you're kidding yourself if you think a company that gets "hacked" by off the shelf cryptoware is going to step up their game enough
Still, this might lead to their first solid security hire that can bring about change in the form of zero-trust principles, security in depth, etc.
> to have any chance of stopping a targeted state actor.
Given unlimited resources, interest and budget, no participant in the modern digital landscape has a significant chance of stopping motivated threat actors.
> The fact they caved so quickly tells me they are years away from a reasonable security posture.
Yes, obviously, but driving change is about incrementally tending to a desired state. Your fatalism is, quite frankly, unnecessary, not that you're not entitled to your opinion, just that disagreeing with GP or stating they are naive because this won't bring about perfect, all-encompassing change is not useful.
No single participant has a significant chance, but if each target becomes more expensive on average, then state actors can only afford less targets, which makes the society as a whole more resilient.
And if one target is so critical that it could take out a society, perhaps it would be better to either 1. Make it so minimalistic that it can be fully audited and secured or 2. Broken into smaller pieces and decentralised so they can either qualify for #1 or increase the total cost and complexity of compromise.
Also, making society and individuals more prepared and ready to deal with no-more-oil for a while, situations.
E.g. warm blankets at home, and food that doesn't need to be boiled, if cannot heat the house because the oil and electricity system is broken for a while?
Yea, I think I tend to agree with you. It may cause a lot of pain in the short term, but being forced to pay penetration testers seems like it could be a net good in the long term for security in general. I don't think nation state attackers would be so kind as to un-fuck your system after they cripple it, even for a massive fee.
I don’t know. Did any of it matter? It was bad when people started hoarding gas. Just a few unfathomably stupid people - as always in this country. If idiots didn’t hoard gas, nothing would really have gone wrong.
The preppers are the other side of the same coin. The only thing they seem to never run out of is toilet paper. Who the fuck cares? Pentesters have the same energy. They tell you about what software not to use (anything in their automated suite), followed by a bunch of meaningless bullshit. It is a form of anti preparation, it would not have helped Colonial at all.
You talk about crippling, and in the biggest audition for crippling society in the world, time after time it’s the Everyman being an idiot - or a Prepper being too smart for their own good - that is responsible for all the bad.
You know what's way more effective at stopping gas hoarding so it's available for someone who really wants/needs it? Doubling the price per gallon. Anti-price gouging laws caused the shortage, just like with toilet paper and PPE last year.
Addressing legitimate problems of hardship can be dealt with from the other end, by channeling resources to those people. In the mean time, higher prices mean that supply isn't interrupted, and for the vast majority of people that means that you don't fill up your car and your wife's car and your lawnmower and a 55gal drum, because it's not worth it. You just skip a few trips and let your gas tank get below half a tank. And if you were wrong about thinking you could hold out, you can still buy gas because it's not sitting in your neighbor's new gas cans in his garage.
We shouldn't have widespread shortages for the sake of theoretical people whose existences are structured around 5 gallon commutes and razor thin margins tied with no lines of credit to float a week of double gas expenses.
The people who would be hurt by such an increase, absent already-in-place assistance, constitute the vast majority of people. And, unfortunately, nobody is helped by "can be dealt with" -- the policies to prevent their pain, and to prevent disadvantaged folks from being disproportionately impacted by such an increase, need to be in place before the increase.
The reason, which is likely apparent to both of us and everyone reading this, is that such assistance would likely never be put in place, and the only help ordinary folks will have in the event of such an increase is wishful thinking.
> The people who would be hurt by such an increase, absent already-in-place assistance, constitute the vast majority of people.
and so you suggest hurting the vast majority of people even more?
That's right, your suggested approach of keeping artificially-low-prices-that-allow-hoarding leads to zero gas at the pumps, and zero gas at the pumps hurts the vast majority of people very directly and very effectively.
Allowing prices to float higher in a shortage solves the allocation problem for scarce resources by ensuring that people think twice about how they use it, while keeping it available. Ignoring that the resource is scarce not only doesn't solve anything, it makes the problem worse.
>and so you suggest hurting the vast majority of people even more?
1. More people are hurt by price gouging of products with inelastic demand than hurt by limiting the price increase.
2. A completely floating gas price disproportionately affects disadvantaged folks who have no alternative, even when there are no outages.
I mean, I totally agree with OP, there's a straightforward solution: set up a government program that insulates people from the effects of a floating price proportionately to how disadvantaged they are, and then, after that, let the gas price float. That way we get your idea without hurting anyone! Win-win.
Rationing makes simple things complex. For example, rationing assumes that everyone has exactly the same need. Trying to fix that is hopelessly complicated.
I don't see how that's an argument against anti-price gouging laws though. It's not like price gouging solves that problem. Fundamentally a shortage is just never going to give perfect results.
Even simplistic rationing is at least relatively fair and can ensure at least a base level of availability. For the major necessities this can be pretty reasonably accounted for to ensure nobody suffers from some extreme deprivation.
It also lets you at least put some somewhat predictable cap on how fast resources are going out. Allowing price gouging doesn't let you predict much of anything, and I'm not sure it even slows the outflow given the uncertainty of these situations.
For example, before anti-gouging laws, when a hurricane interrupted the gas supply, people from a state over would immediately fill jerry cans with gas and drive into the disaster zone, selling gas out of the back of their pickups.
Anti-gouging laws put a stop to that. Now nobody gets gas until FEMA gets around to it.
I've talked to people who lived through WW2 gas rationing. It was a mess of mis-allocation. People who didn't need their ration turned into criminals selling it on the black market. People who needed it turned into criminals buying it on the black market. It was pure political theater.
Those who buy it are those who can afford it. They also (think) they need it, for some reason. That can include the intention to legally resell it. Buying suddenly comes with even more time pressure. Buying now becomes both a hedge and speculation, with zero legal risks.
Sure, people bringing in supplies on their own sounds good. It can even be good. But how much does it actually bring in? Does it come with other problems, like additional stress on infrastructure? If this really is such a great thing, I don't see how it should be wholly incompatible with anti-price gouging laws. If you want to specifically incentivize private transport of goods from out of the area, you don't have to allow them to be sold for any price, nor do you have to allow it for goods already in the area.
As for the black market, I'm not sure that's a bug. It makes it harder, and adds some risk, to acquiring more. If you actually need it enough, you'll do it. There some be some correlation between actual need and willingness to participate in the black market. With proper use of the discretion available to law enforcement, maybe this even enhances how well the goods end up distributed. That's a big "with" but still.
> As for the black market, I'm not sure that's a bug.
Rationing implies a black market must be illegal. Besides, it enriches random people who don't need gas, at the expense of the people who went to the effort to supply it.
If that isn't a topsy-turvy unjust, inefficient and inequitable way to run an economy, I don't know what is.
Rationing suffers from the delusion that bureaucratic rules can distinguish who needs something and who doesn't, and denial of the existence of normal human motivations.
Sure. I'd guess it still adds enough friction to help result in a better distribution of goods in the end. The time, effort, and (legal) risk you are willing to take to acquire something should correlate decently with how much you really need those goods.
Price of gold and silver is at its peak and people are buying it more than ever convinced it will somehow become extinct. The people who have money to hoard gas are also the people who have the money to hoard gas at double the price. These are not individuals with any knowledge of economics - theyre not doing it for trade, they're doing it out of belief.
> The people who have money to hoard gas are also the people who have the money to hoard gas at double the price.
But then the outcome would not have changed, the people who hoard will be worse off, the people who produce a valuable commodity will be better off and there is incentive for people to contingency plan and have gas reserves for when things get tight.
That is a strict improvement. Plus, you're probably being overly pessimistic - people will stop hoarding once the price gets high enough. The shelves would not be bare.
> These are not individuals with any knowledge of economics - theyre not doing it for trade, they're doing it out of belief.
The people with knowledge of economics don't have much of an advantage though, do they? When has an official body ever been banging the drum before a major crisis issuing panicked warnings? Every crisis it turns out all the people held up as experts had grossly misread the situation.
People with knowledge of economics often get bowled under by people with a knowledge of politics or of statistics when it comes to trading.
"The people who have money to hoard gas", in the current environment, is as worthwhile a qualifier as "the people who have shoes to hoard gas". They're both necessary for going out to get gas, but neither is limiting in any important way. The people hoarding gas had a particular combination of gas containers, lawnmowers, and free time. I want the gas stations hoarding the gas and then trying to figure out what price will make them the most money by pricing high enough to just barely sell out by the end of a shortage.
Last year when all the grocery stores were running out of staples, I was unluckily traveling for work where there are no grocery stores. Upon returning, I conveniently found that my neighborhood co-op still had eggs because, while almost all eggs had been sold out, no one wanted to hoard $7.50/dz eggs, meaning I had the opportunity to get eggs if I really wanted them. I was able to buy eggs and did so, adjusting my habits accordingly to operate on fewer (but not zero) eggs. Everyone else who had visited that store had the same opportunity, and they also had the chance to compare their own desire for eggs against others' desires to recognize that they could do with fewer, as could I. At the end of the day, having reduced amounts of something I want is way less impactful than having none.
The only problem here is if gas stations talk to each other to set a price, but that is collusion, and is illegal independent of gouging laws.
We have to wait and see if future results justify current actions, when the action is preparation for some future event. It is only in hindsight that we can truly point out if someone's risk/reward calculation for the future was flawed.
I'm not buying gold, but the odd thing is that Russian and Chinese central banks are. I would assume those people know a thing or two about economics. I'm curious to see what the future holds.
Because their central banks are concerned about stability of international currencies [1]. A central bank is in a good position to use gold, whereas most individuals aren't.
> Price of gold and silver is at its peak and people are buying it more than ever
no, you've got it exactly backward. People buying it more than ever is driving the price up, and the feedback of higher prices is actually slowing down their purchasing not only by changing their minds, but also because their purchasing budget is spread more thinly.
I disagree that a higher price will dissuade hoarders from hoarding; you've just validated their notion that gas is getting more valuable and more scarce. See: deflation.
I doubt companies care about anything besides profit and that they could care less about gas hoarders. Also, this was just a little taste of havoc that could be done to the economy and society. I'm hoping they lost enough money to knock some sense into them to practice better security and hopefully it makes others think twice about practicing sloppy security. I'm also guessing in the wake of this that ransom attacks will increase in frequency, ransom demands will increase in value, and insurance premiums will increase as well and insurance providers may be forced to do better due diligence about policies that they sell to large corporations to ensure that they don't practice sloppy security. I'm hoping there is some tipping point where it makes financial sense for these large corporations to practice better security. Right now, they gamble that they won't be attacked and so don't invest in security and for the most part they have been rewarded.
>If idiots didn’t hoard gas, nothing would really have gone wrong.
Maybe. But, I tend to lay the blame with the foreign criminals/adversaries who attacked us rather than a panicky handful of my fellow country people.
Not sure why some here are blaming the victims while giving the criminals a pass, and even thanking them as if unsolicited, live pentesting on critical infrastructure with a side order of extortion is a good thing.
Penetration testing is part of a security program. If you don't have a security program, penetration "testing" isn't useful whether it's painful or not.
Haves the careers or investments of anyone significant who brought things to this point been screwed? If not, nothing will change.
I think ransomware gangs will be emboldened and more will go after bigger targets. I also think ransom demands will grow and insurance premiums will continue to grow as well. I'm hoping there is some attainable point where it makes financial sense to practice good security.
As someone with a lot of friends and co-workers who are on the info-sec side, the stories I repeatedly hear of how many times they visit the same company year/year and little if anything is done to harden their networks, and impose stricter security on their users is way more common than it should be.
Most, if not all of these networks should be taken offline and siloed, but you know that won't happen now, the genie is out of the bottle. If they did, it would create a much smaller attack surface for critical infrastructure. As it sits now? Doubtful we would go back to that world.
>> But I also suspect the feds put some pressure on Colonial
This was my thought was as well. I thought they were in on this before it hit the public media. For me, it was like in the movies when the feds are trying to tap the line and the person is trying to keep the bad guy on the line as long as possible so they can trace the call?
My theory is the feds encouraged Colonial to string it out in order for them to get as much information on the hacking team as possible. From what we're seeing now (bitcoin seized, servers seized) it sounds like the Feds have them nailed pretty good and their gamble paid off.
I might agree if I had any faith that the people who paid this ransom would do any more than the bare minimum to close this one specific vulnerability and nothing else.
Well, fines are a fixed cost, the risk can be calculated and offset against a bonus. A ransom has an unknown downside. I'd imagine most ransoms would be priced to likely get paid, but ransomers don't really know the biz inside and out, so they might guess a painful or fatal price. but that's a one time cost. the lost revenue is the killer.
I'd expect there will be some serious talks about how much to pay to prevent things like this. 5 million, once? Meh, why bother taking security seriously? I suspect the lost revenue is tougher to swallow.
I dunno. you gotta pay every month forever, for protection against maybe something bad happening someday? It's an insurance premium, but you don't get made whole.
I guess, I'd expect companies to start paying a little for infrastructure so they can buy good insurance policies. backups would get you a lot.
Yea, it seems like a pretty novel situation where I'm almost happy that these gangs are walking through these companies' unlocked front doors and causing enough havoc to be noticed but not enough to hurt them beyond repair. If it becomes enough of an infectious cesspool with diseases that can't be slept off to the point that these sloppy companies are forced to wear hazmat suits to exist in the environment, maybe that isn't such a bad thing in the long run. Before that tipping point, hopefully we don't just breed a large quantity of super bug diseases/ransomware gangs that laugh at decent security.
We have been in a less than ideal evolutionary equilibrium with respect to security: in the short term, companies that don't fund security can outcompete the prices of companies that do fund security, but they leave themselves vulnerable long-term to attackers.
This is analogous to overspecialization in an ecological niche where there was no predation.
As ransomware becomes more widespread, it becomes more and more detrimental to companies to pursue short-term security savings. That's good for everyone.
I think that would only be true if you thought it possible to obtain perfect security, but we’ve seen that even air-gapped systems are vulnerable to nation states motivated enough, and exploits are always laying in wait. This gets some bugs patched - but it also illustrates US infrastructure weaknesses to others.
Of course political terrorists can be negotiated with, they have an agenda and stated goals. You might not like it but they're not actually mad they just use means you are not comfortable being brought to bear so close to home.
I'd agree. But I would be surprised to see that level of action. At least for the next while. Considering the payment time on an invoice is averaging 270 days now. I would be surprised if they moved on this.
I’m on the fence. I definitely see your point, it’s solid. But I also think this (even only 5 million) incentivizes more of the same. It’s no stretch to see we’re in for an increasing amount of this.
If the terrorists and nation states are content with going after random single targets, causing low disruption, and leaving with some money, then good! That's not the scary scenario.
While I'm not directly trying to claim that this hack was the result of a nation-state actor, there's also no reason to assume such an entity wouldn't test the waters with small scale, targeted interference either.
It adds up quickly. For North Korea the revenue from "criminal enterprises", including hacking and ransomware, are a valuable source of foreign currency.
At the very least, $5M seems a good start to fund an ongoing effort at developing cyber attack capabilities.
I’d rather we pay the $5M in ransom, and then $5T to track the hackers down and eliminate them. Certainly someone died due to the pipeline shutdown. Eliminating the hackers would be fully justified.
>These ransoms are net good...There are a lot of infrastructure teams taking security more seriously.
Nonsense. This is not an academic exercise. Our country is being attacked by "nation states" (do more research) and we need to respond accordingly, treating it as the national security threat it is and making the perpetrators pay a heavy price. If they'd bombed our critical infrastructure, no one would be sitting around saying, "oh this is good for improving our defense. Thank you for dropping bombs on us."
The idea that they are doing us some kind of service and we should just play a game of defensive cat and mouse, hoping for 100% effectiveness (which we know is impossible) is absurd.
With whom? And given that nation states have engaged in this sort of thing for years, and that the US/5-eyes/etc also engage in these activities, do you really want to turn a cyber/cold-war into a hot one?
The solution is defense-in-depth, with liability on the providers of software, which will require them to insure, which will raise prices, which will force them to address security as COGS which will force them to reduce their attack surfaces to reduce their insurance premiums.
>Can they unbomb it for $5 million? Cause that would be cool
Yes, we could "unbomb" it for some amount of money. But, we generally use the term "repair".
But, that's a great point: as long as we can undo the damage for some amount of money (via repair or paying extortion), we should let foreign adversaries dictate the terms on which they'll allow us to operate our infrastructure.
Still, do let us know when you think we should be concerned. 2 more pipelines? 3 more hospitals? 4 more police stations?
$5 trillion ransoms? Maybe? No? Perhaps when infrastructure outages and other attacks cause deaths?
It's kinda the opposite scenario. The hackers knew they were willing to pay more but didn't actually want to cause this much attention and so lowballed so they'd quickly pay.
Krehel (chief executive officer and founder of digital forensics firm LIFARS and a former cyber expert at Loews Corp) said a $5 million ransom for a pipeline was “very low.” “Ransom is usually around $25 million to $35 million for such a company. I think the threat actor realized they stepped on the wrong company and triggered a massive government response,” he said.
Yup. They start doing this kind of damage, and they can start to expect the wrong kind of attention -- a kinetic response, as in cruise missile into the upper left window, circular error probable = 50cm., or similar.
Unless Vlad starts frowning on this behavior, which persists at his pleasure, it may take that (or a seriously escalatory cyber response) to lower the threat to acceptable levels.
The bigger the difference between the cost of the downtime and the ransom, the most likely it is to be paid.
Assuming you were in a TV show, and offered two options: Spin wheel 1 with a 95% chance of winning $5M, or spin wheel 2 with a 50% chance of winning $50M, which one are you going to spin? The EV is higher on the second one, sure, but taking the near-certain 5M may still be a better choice - a bird in the hand is worth two in the bush.
Additionally, this group is said to do its research and adjust ransoms accordingly, so it seems likely that the ransom amount was a carefully thought out choice.
Which option maximizes the long-term rate of growth of your wealth depends on how much money you already have. The Kelly Criterion takes this into account and handles a wide range of situations including this one.
Example: Say you have $10⁵, and you have the option to play game 1, which offers a 95% chance of a $5×10⁶ prize, or game 2, which offers a 50% chance of a $5×10⁷ prize. By the Kelly Criterion, the value of a scenario is the expected value of the logarithm of your wealth under that scenario:
Great example. It is worth emphasising your model will capture the game from the ransom attackers perspective since for them it is repeated many times. It won't give the EV for the attacked as their number of ransom incidents will hopefully be very small rather than tend to infinity. Which likely means the EV will be based on their own personal utility function.
Re. this group doing its research: one of my past employers got hit by a patent troll C&D demand, threatening to sue. It was clearly bogus but also clearly enough of a hassle that the company didn’t want to pick the fight if one could be avoided. Our clients were actually throwing their support behind us fighting it, offering their legal resources. But at the end of the day what the higher ups told us is that this patent troll was “very professional. They did their research on us and know exactly how much we can afford or pay them without going out of business.” So we ended up paying and I guess it all worked out ok from there.
When I was young my father had a new company in a industry that was known for lawsuits. The first came came in and I looked at it and said settle. He said no F’ing way. We won. Even with winning the legal fees, the cost in manhours was more then settling. Then the next one hit. Did not settle and won……and so on. After about 3 years no more lawsuits. His reputation was never settle and the trolls moved on.
Never negotiate with terrorist. As long as they feel there is a chance they will get paid it never stops. Burn it to the ground, but do not give in.
Yeah, unfortunately the games that go into extortion mean that fighting back only makes sense (a) on such a long time scale that no one is considering it while being extorted, or (b) because it's the right thing to do.
The difference in actual value between the two for me at least is much smaller than the difference in numerical value.
Both amounts are enough for me to never work another day in my life, and instead focus on building what I want to build. Past that massive increase in quality of life extra money is relatively meaningless (to me) .
This is the same reason that people who decry spending money on lottery tickets as a stupid thing to do based on the EV alone are thinking far too simplistically.
> The difference in actual value between the two for me at least is much smaller than the difference in numerical value.
That was exactly one of the reasons why I showed that example.
While this may be different for a gang that has to split the money N ways, the "bird in the hand" might still be worth the two (or ten) in the bush, due to this or other factors.
Mostly software - games and related tooling, learning more about type systems and applying it in a homebrew language.
I can start building it now. But after already spending 40 hours a week writing code at my day job and having those problems swimming around in my head all the time it's hard to find the motivation to do more, even when what I do for my day job isn't as creatively fulfilling.
In 2019, ProPublica wrote how paying ransoms benefit insurance companies.
They called this: "The extortion economy: How insurance companies are fueling a rise in ransomware attacks. Even when public agencies and companies hit by ransomware could recover their files on their own, insurers prefer to pay the ransom. Why? The attacks are good for business." [0]
Thank you for posting that link, it was really informative.
> ProPublica has found that they [insurers] often accommodate attackers’ demands, even when alternatives such as saved backup files may be available.
What a perverse set of incentives. Insurance companies are paid to manage risk. Risks go up the more they pay ransoms. As risks go up, the risks are more visible, more companies get policies, insurance gets more money. This is bordering or racketeering.
How many times do people on HN say “back up your files”. Certainly that’s the way to do it if you want less ransomware. It’s not sexy, it’s not “visible”, but it will reduce your risk. But apparently, the goal here is to encourage more ransomware.
For such a long article it's (IMHO) a fairly naive view. Sure insurance companies make some profit, specially in the beginning but, eventually, the price gets higher and higher and the cost to secure becomes less than the cost of insurance.
We had a similar issue with builder's insurance down here in Australia. It's was (and still is) cheaper to get insurance than build a quality building. Eventually that caught up with the builders (and they moved to using a seperate corporate entity for each building and closing it after the building is built, but that's another story of corruption).
Security professionals need to start getting serious about demanding physical write-enable switches to all embedded systems, so malware won't survive a reboot.
Backup media must be append-only unless a physical write-enable switch is pressed.
Physical write-enable switches used to be standard.
I'm not buying the crazy argument that remote update is necessary to remove malware installed using remote update.
This was extremely poor (I'd say weak) leadership by Colonial Pipeline. While continuing to be down is painful, it puts a spotlight on the issue and forces hands of critical infrastructure to improve security and reach out to security companies for audits and consulting. Also, they could have gotten additional support from the US government and political support by continuing to stay down instead of the back alley payout.
Not to mention, rewarding people for bad behavior is never a good idea. I learned this as a child... "If you give a mouse a cookie, then he'll ask you for a glass of milk."
It appeared that they informed the government that they are paying the ransom, Government's view was that it was a private sector matter so it is prerogative of Colonial to act.
What I have heard regarding ransoms like these is that the perpetrators goal is to incentivize the transaction goes smoothly, or it won’t continue to work.
So they have to follow through with unlocking and they have to use an amount of money low enough to make the decision obvious.
Well from the article, the decryption tool was so slow they kept using backups along with it. Sounds like future hackers need to improve their decryption tools, or companies where speed matters (like utilities) won't bother paying.
Given how encryption is generally symmetric in terms of operations and speed, I wonder how long the original encryption took as well? What warnings were overlooked leading up to the attack fully going into place?
Basic game theory dictates that the cost of ransoms will continue to rise until it hits the price point at which the targeted company would have to replace its compromised systems from scratch.
5M, 50M, 500M, 5B, 50B?
I wonder how the government would react if a hacker group held gas/power/clean water/etc. hostage for millions of Americans for a ransom in the tens of billions
The government seems to have no problem with utilities doing this or worse to their own customers (PG&E, Texas power grid, Flint Michigan). But I guess if they could blame a foreign power that's an opportunity for a profitable war.
> The government seems to have no problem with utilities doing this or worse to their own customers (PG&E, Texas power grid, Flint Michigan).
The “utility" acting in Flint was state and state-imposed local government officials, 9 of whom have been criminally indicted for their role, so manifestly the government has something of a problem with it.
> I wonder how the government would react if a hacker group held gas/power/clean water/etc. hostage for millions of Americans for a ransom in the tens of billions
Not if the hacker group is a nation state. Sure, small time hacking is cute and all, but the US isn’t going to just roll over and be all like “oh no, you hit critical infrastructure that had a big impact on peoples life. Carry on”
Even this time too government was aware that Colonial paid the Ransom. The question was asked in the Whitehouse press breifing, it was told that this is essentially a private sector matter, Government will not advice Colonial in this regard. Also they said it was Colonial's prerogative to decide and act.
That's because nothing of consequence happened at this time. Had the actions resulted in deaths of a few hundred or a few thousand Americans the response would most likely be very different.
Taking out a fuel pipeline in non-heating months seems a lot less likely to cause casualties than downing a power grid. Dead people get different responses than theft, even massive theft. Also, knowing how scary oil companies are, I wouldn't be surprised if some people turned up dead over the colonial hack. Even if they get the attribution wrong, a dead hacker group would have a chilling effect on such activities.
As per reports the ransomware had no effect in any of the physical pumping/operational systems required to pump/operate the pipeline, it only affected their billing system and they stopped pumping because later it would have difficult to reconcile the billing for it, not because the ransomware disabled the pumps.
Unfortunately these types of breaches don't net the culprits nearly as expeditiously as we'd all like. Given that they're likely based in a country that could care less or may be adversarial to the US this may even be lauded.
In the grand scheme of things it's very low likelihood we'll see any level of prosecution for this incident in the next year or two, if ever. And even then it will likely only result in attribution in a random report a year from now with no actual consequences for the attackers.
The final thing is the amount of money isn't extraordinary. As others have said it's mostly a rounding error in annual revenue that passes books of a company like this.
There were some news reports yesterday about insurance companies dropping cyber-ransom insurance from their offerings (AXA, I think). Very likely more insurance companies will do the same soon, or at least, refuse to insure the company unless they comply with some cybersecurity standards.
I think you're entirely wrong. I doubt the hackers realized quite what they had hacked, and by the time they did realize, they were probably shaking in their boots.
My guess is they lowballed - $5m is considerably less than other payments I have seen them take before for a much bigger catch.
They could always just keep the "X" unplugged, it is doubtful hackers from Russia go onsite and sabotage things. Maybe this will make companies realize if they can't secure it at least just disconnect it. Everything doesn't have to be online.
The “X” may be unplugged but your day-labor contract industrial cleanup techs are hired through an Uber-for-Cleaners Hygiene-as-a-service Heroku app that’s had every backup silently deleted for a month and has just been replaced with a funny cat photo by someone claiming their name is spelled with a zero in it.
You can’t hire cleaners. Site cleanup is suspended. Your insurance is invalid if workers are onsite while hazmats are left uncleaned. You shut down on the day you were about to ship a full warehouse of product. Manufacturing grinds to a halt because the outbound warehouse is full. You have to cease goods incoming because they were misusing the day-labor app for handling delivery grunt work.
Ph0bos Security Services also hacked your email so now everyone has to use personal phones and WhatsApp but upper management won’t share their numbers with anyone except the foreman so he is now the bottleneck for everyone who needs approval to make out of budget purchases to handle the capacity overload that’s happening everywhere except for hazmat cleaning.
But the pumps were airgapped so at least they didn’t get hacked.
It certainly will fund them to go stronger in the future. (And now invents bad behavior even more) The challenge is this isn’t this Colonial’s problem. It’s the next one.
At some point it will wake up the authorities to go after them more seriously too.
Hardly, they regularly take much more than this. This is an organized criminal enterprise, I just read through a chatlog where they got $12 million and this is just one of many.
$5m was 100% a lowball because of the geopolitical implications of this attack.
It is a scientific fact that there will be more of it. It's called operant conditioning. If you reward behavior, you get more of it.
If you think it's about the ransom though, think bigger. Can you even imagine how many billions of dollars silicon valley is going to make off it? They just paid Microsoft dozens of billions for some AR glasses.
How much for some software to prevent cyber terrorism? How much did TSA get to secure planes? $8 Billion/yr.
$18 Billion for border protection.
How much for cyber border protection? $100 Billion? Where will it go? Google? Microsoft? Palantir? Facebook? Twitter? Amazon?
Trillions.
They haven't even gotten started. $5 Million is pennies. Rounding error.
I don't have much to add here, but I've been going to Def Con and the other Las Vegas security conferences for a few years. Every year there is a section for infrastructure security (factories, refineries, etc). Its always the smallest section and the least populated. But its simultaneously the "most important" in terms of how much damage can be done from a single attack. Every year I went and was always terrified by all the stuff I saw because all the people hosting booths were like "yeah its dead simple to get in and break things." I feel like so many people could see this coming and there are just no consequences for the companies to incentivize them to do better.
In 50 years I hope to find out it was pulled off by the infrastructure teams who have been arguing for more security all along and that they did some good with the money.
Completely agree. If interested check out the documentary Zero Days. Insane, essentially the NSA in tandem with Israel took down Iran's nuclear program by impacting their industrial control units. Many Zero Days were used with nearly an unlimited budget.
Colonial is being widely lambasted for a culture of absolutely lackadaisical security. Call me callous but numerous federal agencies exist to issue security best practices and exploit announcements. numerous vendors also exist. play stupid games, win stupid prizes.
Not paying the ransom would have been tantamount to complete dissolution of the company. it would have tirggered a much wider investigation into the company with shareholders abandoning it as the outage dragged on at the hands of an incompetent leadership.
Unfortunately it seems to have been a Pyrrhic victory as paying the ransom puts their shareholders at risk of serious sanctions and indictment from the US Dept. of the Treasury.
If the US were to be serious about corporate IT security, they'd empower and indemnify DoD, NSA, private industry red teams to pentest against everything with a US point of presence or customers, using commercial available / in the wild methods.
This would have the beneficial side effect of flushing all the incompetent paper-pushers / requirement-box-checkers out of the security industry.
If you're found vulnerable, that's a fine. If something gets accidentally broken in the exercise, that's the price of commitment.
Nothing is going to change until you increase the frequency / likelihood of breaches for these companies. If it's a yearly cost, it gets addressed. If it's a catastrophic possibility, it gets ignored.
>If the US were to be serious about corporate IT security
What happened to the responsibility of corporations for corporate security? Including corporations that are the victims of attacks, and corporations that sell buggy operating systems and applications?
Why does the government have to provide the red teams? The general attitude is all government agencies are wasteful and incompetent, except in this circumstance where the wealthiest corporations in the history of the world apparently can't spend enough to fix their own crap. But the government not only can but should??
This just sounds like externalizing costs to the public while banking record private profits.
How about rather than subsidizing software corporations we talk about liability laws and fines, like any other physical industry that releases dangerous, broken products. Or an insurance system that is funded by a portion of the profits the software industry makes. Then we're actually making the software vendors feel some pain which will incentivize them to release higher quality code.
The problem does not fix itself until the investors start truly losing money, the care, unlike the Equifax case. Until the portfolio value cannot go down 90% there is not going to be a change in corporate actionism.
Agree. The govt need not provide the teams as they must compete for talent like anyone else and don't have much to spare.
The govt only has a relative abundance of talent [largely interspersed with its contractors] in highly regulated activites like making nuclear weapons, where private entities don't participate.
One of the functions of the government is to educate and train its population. I think using that function could resolve the shortage or high cost of talent.
The market has already solved this in the form of ransomware groups. No need to have the government do it and issue a fine, ransomware groups literally are doing what you said.
I guess the government could legalize ransomware hacking to encourage it, but that'll never happen.
> paying the ransom puts their shareholders at risk of serious sanctions and indictment from the US Dept. of the Treasury
This (almost certainly) isn't true. It may put management at risk of sanctions, but shareholders are shielded by the corporate veil.
I say "almost certainly" because in some cases prosecutors can go after shareholders, but this is limited to cases where a specific shareholder is involved in decision making.
I wasn’t aware of this policy. Is it totally apolitical or does the WH need to initiate the sanctions process? Consider the optics of sanctioning the domestic company providing your own country’s critical infrastructure, right after you spend a week discovering just how critical it really is.
And yet Scripps healthcare in California is going on more than a week of all of their IT systems being down for the same reason and it’s disrupting operations enough that they’re diverting a incoming patients to other providers and a lot of their patients have no way of finding out whether their already-schedules procedures will still happen.
Since healthcare is so heavily regulated it’ll be interesting to see what repercussions come of this. The company has been mostly silent on the matter despite it being severe enough that you can’t even get to their website.
You'd damn well hope so. In civilized countries, when you leave the key in the ignition the cops will go after the thieves. The next thing that'll happen is that they'll also go after you because you just made the roads unsafe.
And this isn't a car, this is infrastructure with national security implications. Someone needs to go and do time.
It should be noted that Colonial had several infosec openings at the time of the attack. While having those filled might not have prevented this attack, it also might have or at least put them in a better response position.
There are lots of infosec openings across the country but compensation doesn't seem to be rising in response. It appears that companies are fine with leaving these positions open for long periods of time. As long as the position actually exists, they're not all that concerned with filling it. This might be complacency creep. Everyone staffed up after the cluster of breaches that happened around the time of the Target and Equifax breaches. A lack of other high profile breaches or attacks might be why many companies have become lax in keeping their staffs full.
No. The security problem is not a lack of effort or laxness, it is a fundamental inability to solve the problem. At a $5M payout there are essentially 0 commercial IT systems in the world that can stop such an attack. The absolute best of the best commercial IT systems implemented as envisioned with full support can maybe protect up to the $10M level and I am just extrapolating upwards since I have never had any security professional or executive in a Fortune 500 company with a budget in the tens to hundreds of millions of dollars ever assess their own systems as more than $1M. With an ROI of 5 is it only a matter of time before criminal enterprises can bootstrap themselves up to exploit the entire total addressable market. At best, better, but still inadequate, security means that the thousands of hungry bears eat the slower fish in the barrel first to get the energy to reproduce and make more bears to eat the rest.
This is not a failure to live up to potential or incompetence, though there is a fair amount of both of those. We need solutions that are literally 100x better than the best systems currently available before we get to even adequate for critical infrastructure whose disruption can literally cause hundreds of millions or billions of dollars in damage let alone potential human lives. Anything less than that keeps extortion economically viable for the attackers and paying off extortion economically sound for the victims. That is how far away we are.
What are you going to do with private signing keys? Compromise an iPhone? Unlock an iPhone?
Like how the FBI paid $900,000 to do so and get exactly what they wanted (at least with respect to the phone) in the San Bernardino case which you are referencing? Or how the going price for a iOS zero-click remote code execution with persistence, which basically gives you the ability to arbitrarily compromise any iPhone at any time, on Zerodium is $2M? You can get effectively the same outcome as stealing their signing keys for $2M or less in a way that is far less traceable or detectable. There are so many ways in and to get what you want that the fact that one of them, which is not even clearly the best or easiest way, being untouched is not exactly a cause for celebration or indicative of the quality of that defense. The cash register being untouched because the safe door was wide open is not exactly a very compelling security story. So, no, they do not reach the $10M level. Not even close.
It absolutely does affect backups. If you stand to gain $5M from an attack you can also target the backup systems and still easily end up profitable. Only if you stand to gain less than $100k does the budget actually start to get tight.
As for how you attack the backup system it depends. If it push based you send your payload during the push. If it is pull based you craft your payload in the data that will be backed up. If it is not append-only you can easily nuke the entire available history. If it is append-only, but that is only done in software you just need to take over the software. If it is in hardware you just infiltrate then silently encrypt any new data until it would be painful to revert that far back in time. Given that the mean-time to discovery is on the order of months that is quite painful. If they regularly test their backups you just silently decrypt the data on restore until it is time to strike. There are plenty of ways to beat vulnerable backup systems in that sort of budget.
Like, seriously, with a $5M budget you can literally purchase and burn multiple zero days for every system in the chain and still come out ahead. You can hire 10-50 full time software engineers for a year per attack. Most systems have serious vulnerabilities discovered by lone individuals working for a few months in their free time let alone a team of 50 people. The current backup systems survive because most of these attacks are being done with budgets closer to $10k-$100k to maximize profit and growth rate and that is not really enough money to pay for the second arm of the attack. But with a $5M return they could easily allocate a few million to capitalize on the opportunity if that is what is needed once all the juicier targets have been eaten.
It seems appropriate to regurgitate the one about the bear and the hikers....
Two friends are in the woods, having a picnic. They spot a bear running at them. One friend gets up and starts running away from the bear. The other friend opens his backpack, takes out his running shoes, changes out of his hiking boots, and starts stretching.
“Are you crazy?” the first friend shouts, looking over his shoulder as the bear closes in on his friend. “You can’t outrun a bear!”
“I don’t have to outrun the bear,” said the second friend. “I only have to outrun you.”
In our scenario, the bear is the ransomware attackers, and Colonial Pipeline is one of the runners.
There are hundreds or thousands or tens of thousands more runners that the bear can go after.
You don't need to have perfect security over every aspect of your operation (though you should of course aspire to that). In particular you don't need to give up because in theory someone could infiltrate your offsite backups.
You just need to make things hard enough that the ransomware guys will go after an easier target.
Except that is totally wrong. You are assuming that there is one bear, that you can escape the bear forever, that the bear is not hungry enough to eat everybody, and that the bears are not multiplying ferociously due to nearly unlimited supply of delicious food.
No, reality is more like the story of the dodo. A vast quantity of delicious prey that nobody was eating because nobody knew about them. Then they were discovered and some predators showed up but there were not enough to eat all of them. But then more and more predators showed up to exploit the vast untapped resource until they were all eaten.
We are still in the middle of that process which is borne out by the fact that the frequency of attacks has been increasing on the order of >100% per year and average demands per attack have been doing something similar. That is an utterly ferocious rate of growth that will soon be enough to attack not just the juiciest targets, but every profitable target in a few years.
Being slightly faster or slightly less delicious will not help when there are finally enough bears to eat everybody.
I can’t speak for other industries but in the financial industry (in the US at least) periodic backups are required on physical tapes both off- and on-site.
Barring a Mr. Robot hack of the institution and Iron Mountain to burn the tapes the absolute worst-case scenario in a ransomeware attack on a financial institution is an afternoon of data lost.
You just hack the machines that are loading the data onto the physical tapes or the system that is collecting the data to put onto the tapes. Essentially, at some point the data goes from where it is being used to the tapes and you just takeover one of the systems in that pathway. You then wait for 6 months silently encrypting the data before you make your demands. Now the absolute worst case is that 6 months of data is lost or however long you were hiding. Industry studies indicate that the average time between infiltration and detection of an agent actively exfiltrating data is a few months, so a few months for an agent not even pushing data out over the network, just silently corrupting data going to your off-site backups that you are not looking at is very reasonable.
Backups are not the end of the story unless you are dealing with attackers with only $10k to their name which is essentially what everybody without backups is losing their minds over and being defeated by. That is a literal rounding error of a rounding error of a rounding error for the financial industry. People spend more on lunch than that. A moderately sophisticated attack with a few million behind it is literally 100x the resources of most of these attacks and that is still just a microscopic pittance compared to the financial industry. Think about that, if you want to reach the $1M level you need a system that can defend against an adversary with 100x the resources of a basic ransomware attack. The gap is so large that the capabilities fundamentally change and intuition for how to defeat a $10k attack does not generalize.
And, we have not even considered a system that would even be considered barely adequate for the financial industry. If you want to get to something barely adequate for the financial industry, like say protecting against an attack funded to a level comparable to one day of disrupted operations for JP Morgan, you would need to protect against an attack on the order of $500M, literally 500x more than those "good" systems and 50,000x better than these basic systems. The gaps are ludicrous and the lessons at one scale do not really apply when you go up another 2 or 4 orders of magnitude.
Lol as if majority of those hacks aren’t just some misconfigured s3 permissions or creds that got submitted to Github or an unpatched windows machine. Those are essentially script kiddie hacks 2.0 except they now can get payed thanks to crypto (at least it’s useful for something)
Yeah this is sort of nonsensical. As someone else commented, Apple manages.
It’s well, well, well known that working in ICS security as a security engineer means aggressively lower salaries to secure horribly insecure, outdated tech in a low funding environment.
The issue is less about people unwilling to take those wages, and more about a lack of people whose breath can even fog a security mirror so to speak. I work in security and have been involved with hiring at several “brand name” companies including FAANGs in hot tech markets, and it’s always been a talent pipeline issue more than anything. Given how difficult it is for the biggest players to keep security staffed up, and they still get hacked routinely, I can’t imagine how low quality the applicant pool is at Colonial, and doubt it would have made a difference. Almost every company of moderate size perpetually has openings for security roles.
The other problem is that the industry has an oversupply of by-the-book certified security people who can configure firewalls and run scanners, but who have never dealt with live hackers or hacked anything themselves. But hackers are clever and artistic, and defending against them isn’t like following a recipe for baking a cake.
And as an employer looking to introduce security, there is no way to really evaluate a good security leader vs a charlatan, and then it’s either bad hires all the way down, or talented people on the bottom who lack leadership and are ineffective in the bureaucracy.
Is being a "good" security person really more involved than:
* making sure you have all your ports locked down
* limit connectivity between all instances to only the bare minimum
* any public access is via protocols such as ssh which have zero-to-none vulnerabilities
* any 3rd party software you dont know is secure should never be public
* routinely run employee training on how not to let themselves get hacked via social engineering
I'm sure I'm missing other stuff, but I feel like if you follow these "best practices", you have just made yourself a very hard target and hackers will probably skip over you unless they have some weird reason to target your org specifically. So for 95% of companies out there, this level of security should be sufficient.
I'm legitimately asking - is this sufficient? Or are hackers so creative that even following these basic rules will still not make you a hard target?
This stuff seems fairly easy to do but I agree you need training or an info-sec person making sure your dev teams are doing it all. You can't have any slip ups. Your devs / managers have to take it seriously.
In particular, "routinely run training" might reduce the probability of a breach due to social engineering, but it probably won't.
You also didn't really cover client machine security, which is how compromises often happen. Your awesome security isn't worth much if the admin's machine is compromised.
Your employees need to use computers to do their job. As part of that, they will need to browse the web, which they will do with one of the major browsers. This browser has unknown 0-day vulnerabilities. Whatever security measures you implement must not disrupt business.
They may also need to plug in USB drives. These can come with malware. Whatever security measures you implement must not disrupt business.
They may also need to open documents, possibly with macros. Whatever security measures you implement must not disrupt business.
Your "basic rules" will at best prevent the - still extremely common - social engineering based attacks, but they still won't reliably keep an attacker out of your network. The attacker will compromise a random person, find some company-wide writeable shared network drive (that you didn't even know about) where a team shares their executables, replace one of those, compromise more machines, escalate to domain admin credentials through one of the many ways that exist, then use your own fleet management system to push their backdoor to your entire fleet.
For good security, you need for example:
- an overview of what assets (computers etc.) you actually have
- a decent way to manage these assets
- monitoring so you can hopefully detect when (not if) a compromise happens
- many layers of defense in depth that slow down attackers and limit what they can do once they've compromised one part of your company
- technical barriers to prevent social engineering attacks (binary whitelisting, strong multi-factor authentication)
- protection against insider risks
- physical security
and that's just a few things that popped into my head, the actual list would probably not fit whatever post length limits HN has. And of course all of this needs to be implemented with the limited budget the company is willing to give you, without disrupting the business, etc.
Thanks for the great response! Very informative. I assumed I was way simplifying the problem. It seems like what works for my small remote only startup is not even close to what you need for a large in-person org running who-knows-what software.
I'm sorry, but this just doesn't work in the real-world.
As the "security guy", you're seen as the troll under the bridge. Someone to get past via any means necessary, including lying.
But lets say you get your way.
"making sure you have all your ports locked down"
You can't imagine how much work this actually is on a network with 1,000+ servers running at least 10,000 distinct pieces of software. Most of which don't document their firewall requirements.
Oh, did you know that Active Directory domain controllers -- the single most valuable attack targets -- require essentially all ports open to all computers on the network?
What is your firewall going to do when all modern software communication is over HTTPS and "looks the same"?
How are you going to firewall off just one modern server with 200 Gbps Ethernet? Do you have any idea how much you'd have to spend with CheckPoint or Juniper or Cisco or whomever to do that?
"limit connectivity between all instances to only the bare minimum"
That lasts right up to the point that the shouty guy in finance that talks directly to the CxOs wants PowerBI on his desktop to be able to pull in data directly from all the databases. Did I say desktop? I meant a laptop on unencrypted airport WiFi.
"any public access is via protocols such as ssh which have zero-to-none vulnerabilities"
You don't get to choose the software. Windows doesn't use SSH for anything, and can't be made to.
Also, if you know anything about ransomware attacks, you would know that protocol encryption does nothing to even slow them down. If anything, it makes detecting attacks harder!
"routinely run employee training on how not to let themselves get hacked via social engineering"
Meet Mr Bell's Curve, and its unavoidable left hand side. Some people are just incorrigibly stupid and will routinely fall for phishing attacks, no matter how much training they receive. At any large corporation -- the type worth ransoming -- these people are inevitable. You, Mr Security Person, don't work in HR and don't make hiring and firing decisions.
"I'm sure I'm missing other stuff"
You're missing the fundamentals of the problem, which is that as a security guy:
- You must come up with security solutions that work in the face of morons.
- You must be able to secure software written by morons with no interest in, or ability to write secure code.
- You must do this without impacting the business in any material way, because if you stand in the way of anyone more senior than you -- even once -- you'll never be listened to again.
"Or are hackers so creative that even following these basic rules will still not make you a hard target?"
Currently, for any large org above about 1K staff, security against targetted attacks is basically impossible. Certainly not financially viable. Your competition will not spend the money, make more profit, pay out the ransom, and come out ahead of you.
Thanks for the great response! Very informative. I assumed I was way simplifying the problem. It seems like what works for my small remote only startup is not even close to what you need for a large in-person org running who-knows-what software.
> The issue is less about people unwilling to take those wages, and more about a lack of people whose breath can even fog a security mirror so to speak. I work in security and have been involved with hiring at several “brand name” companies including FAANGs in hot tech markets, and it’s always been a talent pipeline issue more than anything.
Oh come on. It is just an excuse. Look up what FAANG pays for those jobs ( total compensation ). Pay 2x. Get people from FAANG to work for you.
Game theory says nah, just do enough so the other guy gets hit first. I mean I could spend the kids' college fund turning the house into an impenetrable fort with bulletproof glass, booby traps, 2 ton doors and concrete walls; or I can spend a few thousand and get a really grumpy window cat so a burglar moves on to an easier target.
That’s a pretty silly way to look at it. There are a lot of reasons, but the most obvious is that you just moved people around, you didn’t get any new ones. It’s zero sum in the short term, because there are many many years of latency to correct the talent pipeline on something like security.
The problem I see is that there are tradeoffs between security and usability, and again between developing security vs developing features. Security doesn't make money next quarter, while features and ease of use do.
Any software engineer can do security if they spend time learning and working on it. But executives don't seem to care about it.
Weird to see this downvoted, it’s completely accurate and pretty basic economics. Security is a cost center, product development is a revenue multiplier. Investing in the latter as much as you can get away with is the most rational way to allocate resources.
From my experience, the problem is that most infosec positions are powerless to do anything to increase security at the company, and are primarily there for PR or compliance reasons. The positions seem to be mostly filled with people who wanted to make a career change for the money; experienced people usually leave to work at private security companies, or FAANG sized companies.
This. A million times this. I can’t tell you how many netsec roles are staffed by people that are content being a butt in a seat and have zero effect on the overall security of a corporation.
Being hit by ransomware is not an indicator of total IT incompetence.
Having no good options but to pay the ransom absolutely is.
Part of paying the ransom is the promise that the ransomer will not just unlock your system, but will also delete all the data they downloaded (which often includes a pile of PII that the ransomee doesn't want published).
Note that the attackers can also threaten to release data. Backups are no protection against that. They could also corrupt data and not tell you which part was corrupted and when, so even if you have backups you don't know which ones are corrupted unless you have some way of verifying all the data. One example of this would be to plant a backdoor, leave it in place and unused for months, then trigger the ransomeware encryption. The company decides not to pay, they restore from backup, and the attackers use the backdoor to encrypt it all again and demand even more. They could also use access to destroy hardware, say on a timer that triggers after the payment deadline. Backups won't protect against that if you can't get all your systems offline fast enough, or if taking them offline triggers the destruction.
I wouldn’t even pin it on IT. I’d be willing to bet there’s some poor IT person, or perhaps a lot of them, cleaning up this mess who have been begging management to beef up info sec for a long time.
I mean, let's address the elephant in the room: there is no such thing as computer security. As we see with new leaks and hacks and vulnerabilities every single week, the idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors, and we're reaping the rewards of that now. It will take decades of layering on band-aids to approach anything like security, and more likely we will have to rebuild the entire industry from the ground up without that assumption. Both will take a very long time and a lot of money. Hiring some guy with an infosec cert would not have stopped this attack, because there is no way to stop this kind of attack.
Risk cannot be eliminated but it certainly can be reduced. Also response plans for when something happens can be funded and regularly tested. You can't anticipate every possible successful attack but you can reduce the risk of being unprepared to respond to whatever attack happens.
> Idea that a computer that is connected to the Internet can be secure is a joke. The whole industry is built on protocols and tools that assume there will never be any bad actors
There are companies that get hacked a lot and there are companies that don't. It is for sure true to say everyone is vulnerable, but it's also true to say that you can reduce your risk without reducing your revenue.
For many companies, security threats are all theoretical, but they are required to have the positions to meet some compliance requirement. They need to have them, but don’t really want them, which would explain the lack of enthusiasm (as demonstrated by the low salaries) in getting the jobs actually filled.
Also, a lot of infosec positions are just chugging through audits and ticking boxes to say whether you have some control in place or not. Those are more clerical positions that don’t require deep technical knowledge that could command a higher salary.
Are there enough Infosec people to fill every open job for it in the USA? I would imagine that it is like software development, where the unemployed software devs are the kind that can't figure out git.
This is basically accurate but with an added problem. When devs do their job, the product is software. When security does their job, the product is “not getting hacked”, so if you act busy enough, it’s easy to appear as though you’re doing important work, until it’s too late.
Then, paradoxically, you aren’t actually punished, but usually rewarded, when you do get hacked. That’s the one time you’re needed most, and you get to act like the hero for saving the company.
I doubt there are enough infosec people which means in theory that compensation should rise which will then attract more people into the field. Until they're trained and experienced, whoever provides the best place to work (compensation and intangibles that lead to satisfaction) would get the help they need while others would be more vulnerable to attack. But from what I've seen, this isn't happening. There's lots of complaints about there not being enough workers but instead of boosting compensation and/or quality of employment, the positions simply stay open for extended periods of time.
If I’m a leader in a company with a culture and intangibles not yet optimized for the people working in the infosec roles, how would I aproach changing the environment for the better?
Is it viable to cooperate with other companies to share best practices? Wouldn’t they hesitate to share?
Would doing deep interviews with potential employees get me the right information?
Would some hr consultancy provide this info? Arent’t they too old-fashioned for this field yet?
I love the idea of sprinkling bitcoin private keys in text files around your infrastructure, so any hacker that gets access can take the funds, but you'll be alerted to it and can quarantine the box and investigate the intrusion. Maybe include "Email us with a write up of how you got in and a bitcoin address, and we'll send more bitcoin based on how helpful it was"
Rotate the keys periodically and sweep all unstolen bitcoin into a bonus fund split between everyone who had access to the machines which held the private keys. Give devops real skin in the game for keeping boxes secure.
Could also develop a convention for deriving private keys from security secrets - make it so if someone gets your AWS root key, they can test the credentials to see if the company has offered a enough funds that they are willing to announce (and thus burn) their access by transferring those funds away. I wonder if you could 'license' these coins in a way that it would be legal (or at least more-legal) to take them without prior consent: if there was a legal means to monetize 'misplaced' credentials many hackers might choose that over the legally riskier and less-moral traditional alternatives.
Credential rotation would certainly be more fun if it meant I was going to get a bonus!
This might work for your personal system, but in the corporate environment, what's to keep someone with legitimate system access from emptying the wallets periodically and blaming an advanced persistent threat? It would be better to do the same system with standard bank transactions, and just provide a promise to not prosecute people who make contact after pwning your company.
Well, presumably the rest of the team would push for increased monitoring and access control until coworker was no longer confident they could steal the bonus without getting caught, at which point your systems have been hardened and risk from outside attackers is also probably reduced. But, I'm definitely getting into 'hand wavy' territory here.
Since the wallets are canaries the rooted hosts will have to be rebuilt.
Also make the dollar amounts relatively low so an insider is unlikely to risk their position. $1000 is a lot to someone who doesn’t care but is foolish for someone who passed a security check to get access.
I think you substantially underestimate how much these ransomware companies make once they've gained access.
If they've gained access, they'll just do their normal thing and then right as s they're doing it empty out the wallets. $1000 is not going to deter them from a multi million payday
I am doing exactly that on a personal level. I have an unencrypted BTC wallet and one that is encrypted, the unencrypted one contains not that much and will tell me if my device has been compromised.
What I’m more worried about is when the sat is too valuable. We can millisat it with LN, but it’s still not enough. I used to think eth was absurd with 18 but we kinda wish for more than 2.1 quadrillion of the beasties.
There are world currencies used by 100M+ that are worth less than a sat today (IDR). In short while it’s more like 1B+ as BTC grows rarer.
> Once they received the payment, the hackers provided the operator with a decrypting tool to restore its disabled computer network. The tool was so slow that the company continued using its own backups to help restore the system, one of the people familiar with the company’s efforts said.
I thought the protocol for these attacks was to send the decryption keys, not provide a "decrypting tool."
If some kind of software was provided by the attackers, and Colonial installed it, this could be far from over.
Also, if the company has backups, then why not use them instead? If they're incomplete, then that's the real problem.
Probably a reporter/reporting issue. No company that just have been hacked would run a binary received from the hackers in order to restore the systems, they cannot be that stupid. But then again, they did pay the ransom and also seemingly can't restore their systems from backups, so who knows how stupid they really are?
More charitable reading is that the encryption key was sent over, and they started restoring with that but using standard OSS tooling.
What? No, the ransomware people truly do send a decryption tool, or the decryption functionality is built into the ransomware. Do you think they are sending people some AES key and then everyone goes off and builds some python tool to decrypt his data?
This is a fundamental misunderstanding of the ransomware business. The whole reason people pay up is because the hackers don't run and leave you hanging; if you pay they will decrypt your data. Trust and convenience are essential to making this work.
Great, we should get the word out then that some don't.
Perhaps a few cases of high-profile companies falsly claiming „wow, what a load of shit! we got ransommed and after paying up the hackers disappeared! we had to restore from backup, AND the money is gone“.
What are the hackers gonna do, sue those companies? :-)
Also, assume you have the key - what you do with it? You don't know how the files were encrypted, in which way they were stored afterwards, etc. There are many ways one can encrypt and write data, even with the same key - you obviously need the algorithm, but also there are often parameters (e.g. block sizes), storage formats etc. The easiest way to deliver all that is to provide a program.
Otherwise, what a random "press any key" IT person would do with an encryption key? They probably don't even have any tools that can do encryption on any of the systems. Do they have to write those themselves? Use OSS tools - which ones? With which parameters? What if it doesn't work?
Well if the company is already that messed up to not have backups and desperate that they paid criminals...
One would hope they'd just run the decryption program on each computer, not connected to the network. Or maybe hire some experts to extract the decryption key.
> More charitable reading is that the encryption key was sent over, and they started restoring with that but using standard OSS tooling.
That would make a lot more sense but I also bet there's a non-zero chance that in a day some dumb media outlet will conflate those tools as "hacker tools" and the headline will be "Hacker tools used in Colonial pipeline hack available freely on Internet. News at 10."
These inane arguments didn't kill GTA, or virtually anything else. How are they going to kill OSS that hasn't needed mainstream appeal and still doesn't? So, maybe some high school kids end up on the github pages and become 1337 hackers? Quite a stretch..
Pretty hilarious to see all these comments from people who have no idea what they are talking about.
Thinking $5m is a "high" ransom, thinking that there is no way they would send a decryption binary rather than a key.
Why don't you just actually research how these schemes function before commenting? The team that did this has a pretty consistent MO and five million was a massive discount.
> If some kind of software was provided by the attackers, and Colonial installed it, this could be far from over.
To be fair, malicious code has already ran on the affected machines, so if the ransomware authors wanted to do further damage they wouldn't need a malicious decryptor to do that.
So you'd either:
1) not trust the ransomware authors, rebuild everything from scratch (potentially paying the ransom and reverse-engineering the decryptor or running it isolated from the internet) and make sure to not carry over any executable code that could allow potential malware to persist
2) trust the ransomware authors and not rebuild everything, in which case you may as well run their decryptor
I don’t think the hacking group would want to show future targets that paying the ransom won’t get them un-hacked. People would stop paying them. It would be bad for business.
If anything they’re working on speeding up their decrypting tool for the next release :)
Disclaimer: I work as a CISO in a large corporation.
The interesting bit in this article is not necessarily the sum of the ransom, but that Colonial decided to pay quasi-immediately. It seems as if the attackers had full control over their network. Another possibility: Colonial staff could not be sure that if they used their backups, everything would be encrypted immediately again - possibly the backup servers as well. My bet would be on scenario 1.
Having read the release by the attacker, my initial thought is that the immediacy of paying was probably due to the threat of the release of sensitive data, not the ability to restore operations.
I’m sitting here wondering what exactly about the release of their financials and internal procedures prompted them to immediately pay $4-5m in the hopes of preventing it from happening?
If this is the case, then paying the ransom will turn out to be a stupid idea.
If the threat was to release sensitive information, surely the firm would be asking the attackers for details of the sensitive information they claim to have.
If the attackers come back with nothing then it was just a bluff.
However if the attackers come back with real information
then paying the ransom is just stupid, as the attacker still have the sensitive information and can repeat the payment demands ad infinitum.
Just spit balling here but they have had several other pipeline shutdowns in recent years. One was blamed on a third party damaging the pipeline but I believe the others were operational issues. Perhaps there's more information on those issues than the company would like the public to know? Just a wild guess.
I am curious what your thoughts are on other commenters making as if it is possible to prevent these types of attacks by just taking security 'more seriously'. My guess is that you know that no matter how much is spent with a large entity and many employees it's near impossible to prevent this type of attack. People make mistakes people are easily fooled people don't follow what they are told to do and so on.
I can't even begin to imagine the amount of people that could cause an issue in the size company you are a CISO at.
It's certainly possible to achieve serious security but probably not practical for most private entities. I've spent most of my development career making software for the US intelligence community and their systems were definitely not going to get broken into by a ransomware gang. Security measures include multilevel air gapping plus heavily armed physical security, six foot thick concrete walls set back from the street by other concrete barriers, locating facilities on military installations, disabling USB ports on most devices, banning anything radio enabled from being anywhere near your workstations, jamming radio signals anyway, severely punishing, possibly executing, anyone caught working as an intentional insider threat, requiring multiple persons in the custody and approval chains to move any files from one network to another via write-once media like DVDs, having the transfer media itself in a separate locked cabinet in a separate locked room inside the actual classified vault serving as an office. Installing and running everything in a separately sandboxed staging environment even after it gets through all the walls and air gaps and DVDs and running it through some fairly extensive testing and analysis before putting it anywhere near a production system.
Clearly, you can never make it literally impossible, but to my knowledge, nobody has ever managed to get malicious software onto a classified production system. Information leaks are, of course, another story.
Thank you - this is the closest I have read on this thread as to the real security practises we will need in the future - if you can elaborate more that will be helpful.
Are these (i suspect not) published anywhere as "Three letter agency network security standards"?
You cannot completely eliminate risk but you certainly can reduce it and be prepared for what to do when one of those low probability risks ends up happening.
If there’s a business need, you can secure a wooden box on the sidewalk in a way, that it is almost impossible to break in. It will be very costly, but if profit or IP depends on it, one can find a way.
Taking cyber security „seriously“ always depends on who you see as a potential attacker. I don’t think any corporation on the planet has the capacity or willingness to really protect itself against dedicated state actors. This does not include ransomware gangs that are not prosecuted by the Russian Federation or DPRK, but highly specialized forces within the usual intelligence services.
The types of ransomware attacks we see today might not be preventable as well, every company on the planet will get or was already hit. But, the difference between the attacks: the amount of damage. If money is spent on security, that amount will certainly be smaller.
If the attackers had full access, they probably broke into the financial systems, issued the bitcoin transactions and paid themselves directly. I mean, why bother going through the hassle of trying to teach people how to do all of that stuff?
There are other stakeholders involved in a transaction like this, most importantly banks. Payments, especially large ones, are heavily regulated. You cannot hack a finance department and issue a monero transaction of that size without triggering a lot of alarm bells.
I really want to be supportive of pipelines as a better option than trains or trucks, but it's really hard to do when things like this don't result in enormous payouts against these companies.
The US legal system is not capable enough to allow for large pipelines.
Also, this and coal are the sort of stuff that nuclear replaces...
It's also the sort of stuff that renewables and EVs and batteries (EVs = mobile batteries) replace.
Nuclear is fine for baseload, but no good for anything else, costs a fortune, has huge externalized waste processing costs, and is inherently not fail-safe using actual deployed designs.
I agree that this is something that EVs and batteries could replace, but if your baseload is coming from oil, all you've done is transfer the combustion to a more efficient central site and then sent the power through the grid.
Regarding base load, my suspicion is that SMRs inherently accommodate transients better because leakage becomes a bigger factor and starts to compete with poisoning effectively, so maybe you get more bang for your buck out of influencing the moderator. Regardless, even if it's only ever good for base load, that's a lot of ground nuclear could still cover in the US.
Finally, the cost of nuclear comes mostly from the aggressive safety standards. There's space to fix some of that (enormous cost to the fact that radiation workers typically experience less exposure than aircrews) and also space to acknowledge that we're lowballing standards in fossil fuels, with pipeline leaks and ransomware compromise being easy examples. That's before you talk about the pollution released by fossil fuels, including the radioactive contamination released by mining and burning coal.
So, supposedly, Colonial paid the ransom "within hours after the attack". And, supposedly, the attack didn't even hit any ICS, just the payment infrastructure ( https://www.zdnet.com/article/colonial-pipeline-ransomware-a... ). Why are there still gas shortages 6 days later?
Not a rhetorical question at all. To me, the idea that the infrastructure we rely on is controlled by middle managers with no sense of urgency and no grasp of their domain looks like the real fridge horror story here. On the other hand, I have learnt better than to trust everything I read in the press; thus the supposedlies. Either way, "the decryption tool is slow" is not an excuse to not deliver essential supplies.
You do not need actual disruptions in supply to create a shortage. The threat of a disruption or a shortage for such a critical commodity can create a situation that it becomes a self fulfilling prophecy (short term).
That is what can often create bank runs and created the "great toilet paper shortage of 2020".
And the toilet paper shortage was not purely panic-driven. People did shit at work before the pandemic, and that part of demand switched to a different supply chain. The panic-induced bullwhip was probably stronger than the original demand spike, but the whole thing wasn't just memed into existence.
The pipeline was shut down as a preventive measure (we're being told) just in case the attackers had made their way into the control systems. Trucking of gas has been increased to compensate for the closure of the pipeline, and there was emergency legislation passed in Congress to lift regulations that would have prevented these higher levels of trucking. The gas supply has been just fine here in the Northeast; the issues I've heard about have been in the South and Southeast due to panic buying (in areas of the country that shouldn't have been negatively affected by the closure of the TX->NY pipeline).
Preventive measures shouldn't be creating third-world-level shortages in basic supplies; those things are dangerous themselves. I'm not sure where the Northeast ends for you, but DC seems to have had serious shortages as late as 2 days ago: https://twitter.com/GasBuddyGuy/status/1392467605898907652
$5M for shutting down that major of a pipeline seems like too little, unless, of course, they weren't expecting the company to even pay. Now that these actors know that the oil (and quite likely other utilities) are more than willing to pay big bucks to get back online, they will be targeted far more.
There are so many reasons this is very very bad.