I think you can put things as modern baroque software stacks with their effectively vast "attack surfaces" are not going to stand-up to a well-financed, patient, skilled attacker.
I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter).
Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.
> I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter).
Yes, this is the current bleeding edge of cloud security, known as “zero trust.” Amongst other things, it usually involves provisioning mTLS identities in a secured manner to each service, with connections restricted to a whitelisted set of identities.
I found Evan Gilman and Doug Barth’s “Zero Trust Networks: Building Secure Systems in Untrusted Networks” [1] a pretty helpful read in understanding what modern/next-gen cloud security looks like.
Some modern implementations of varying depth and scale include SPIRE [2], Tailscale [3], and BeyondCorp [4].
I really wish https://news.ycombinator.com/user?id=nickpsecurity was still around here to throw his weight in this thread. He'd often geek out in those kinds of discussions, talking how in the '70s people were actually doing security in computing, vs. the current security theatre, and how many things are totally possible to do, just no big company wants to do this now (even though loudly claiming otherwise), because it is expensive, and it's more profitable to focus on profits than security. For some technical references from Nick, see e.g.: https://news.ycombinator.com/item?id=20946537
For example, quoting verbatim from the link above:
> [...] Modern OS's, routers, basic apps, etc aren't as secure as software designed in 1960's-1980's. People are defining secure as mitigates some specific things hackers are doing (they'll do something else) instead of properties the systems must maintain in all executions on all inputs. We have tools and development methods to do this but they're just not applied in general. Some still do, like INTEGRITY-178B and Muen Separation Kernel. [...]
Yeah, but if you're blindly installing a third party's binary blob, it's hard to call that "zero trust".
Edit: It seems like a serious extension of the zero trust concept would involve something like "only allow in source code into the system from sources we trust and then we compile it ourselves". Limit trust to trusted identities and don't allow binaries in any more than people.
Of course, SDLC security and binary verification is also part of zero-trust systems.
But you have to keep in mind - the whole point of zero-trust systems is that even if a malicious third-party blob is present on some nodes, it's blast radius is limited by the whitelist of clients that it can communicate with, which makes exfiltrating information particularly difficult unless you have additional exploits that let you move laterally.
I am suggesting you could limit "binary verification" and just allow "sources you compile yourself", 'cause how much can a binary be verified? (enough to know your blob was hacked at the source and not along the way, yeah but as we see, that's not enough). What describing just involves "we make sure our vendors are sort of trying to be secure" and that's about it 'cause that's all you can be when someone hands you a binary.
And sure, there exist things like Linux drivers that are binary blobs in source code form but at least directs one's eyes to such things.
Edit: Also, even in these environments, some people and some software has to run the network itself, like a sys admin tool. Which makes sys admin tools something everyone should look at. And brings us to the present hack.
I've been asked to look into ZTN just today by a large government customer.
The problem is that network security is a tiny, tiny subset of security in general. You can throw SAML, mTLS, or IPsec at your connections all you like, it would have done exactly nothing to stop the type of attack that hit SolarWinds and their customers.
Let me reiterate this: ZTN would have achieved zero protection.
The problem in general is that there are conflicting interests between what the developers want to do to reduce their work effort and operational application security. Even large vendors are hopelessly bad at producing secure and securable software. The documented and recommended easy path is not the secure one. The products are incredibly hard to secure from the perspective of some poor ops person.
Some random examples:
- Most vendors document their firewall ports, which is great! But not their direction or what roles they apply to.
- Very few vendors provide machine-readable firewall definitions that can be imported into firewall systems automatically.
- Vendors like AWS and Azure that do publish JSON lists of their endpoint addresses and ports use unique, non-standard formats, so very little effort is saved for the end-users.
- Public cloud vendors especially end up "blending" multiple customers into pools of shared IP addresses for all of their PaaS or SaaS services. This is a disaster for security, because it makes it virtually impossible to put a WAF or a WAP in front of certain services.
- mTLS support is a shitshow. Manually uploaded certificates that expire and break everything annually is the norm. CRLs and OCSP are almost never checked. Many products support only a single certificate, and will even crash if talking to products that do support multiple certificates and are in the middle of a rollover. Alerts and visibility of these certificates is poor. Secure certificate storage (i.e.: in a TPM) works better on laptops for securing WiFi than it does for servers processing billions of dollars in transactions.
- The rise of Docker, NuGet, Cargo, and NPM are a security disaster. Incident after incident and warning after warning have been ignored. Why? Because these technologies are soooo very convenient for developers, and they're the ones making the decisions to use these package management platforms. The ops people are then left with the mess. Have you ever tried patching a vulnerable version of .NET Core in a Docker container from a vendor that doesn't even exist any more? Good luck with that.
- Similarly, the plug & play nature of the public cloud with the extensions, plugins, agents, and marketplaces isn't a disaster waiting to happen. It did happen. That's what this SolarWinds thing was all about.
- Pervasive telemetry is largely indistinguishable from data ex-filtration. Not only does it pollute the logs, but vendors are deliberately obfuscating the traffic to work around customers blocking it. Did you know that Microsoft used "eu.vortex-win.data.microsft.com"? Did you spot the deliberate typo? It took me a while to realise why telemetry traffic was still getting through despite firewall blocks on "*.microsoft.com"!
- Back doors in products for "support" are another security disaster waiting to happen. Several SAN array vendors for example include these as standard now.
I could go on and on.
I'm watching some of our larger customers struggle with security, and to be honest it feels hopeless. These orgs have 10K+ endpoints, 2K+ servers, running tens of thousands of distinct pieces of software from at least a few hundred, maybe a few thousand orgs.
For every step forward in security, there's two steps back for someone else's convenience.
> The problem in general is that there are conflicting interests between what the developers want to do to reduce their work effort and operational application security.
The most secure work is the work that doesn’t get done.
Most Enterprises actually use the typical 3-tier bullshit that Cisco put into their Campus network design for everyones global operations. And then you Accenture, McKinsey, Oracle and everyone else advocating for this kind of design. All the different departments poking holes everywhere because they actually can't work. They can't access compute, they can't access storage. It takes 3 months to upload basic data somewhere.
The whole thing is a cluster fuck. I keep getting hired as a person to help retain talent, but enterprises like to put all their cards on Solarwinds. When they can blame Solarwinds, their managers are still safe. There is no compartmentalization of access. You don't need to have all departments globally be in the same LAN to begin with.
We can hire all the "security experts" in the world, but if software and architecture are fatally flawed from the getgo, you can check all the checkboxes, and scan and produce all the pretty reports you want.
The only way out of this mess I see is throwing everything away and starting from scratch, using provably (machine verifiable proofs) correct software.
I'm thinking of properties like "for any api input the reply is independent of $SECRET".
Original Unix wasn't formally verified. seL4 took ~8 years from initial design to release (though apparently the verification was done after 3 years?), and it is a code base smaller than TypeScript. CompCert started in 2005 and it still can't compile all of C, and it doesn't go past the level of Gcc's -O1 optimizations.
Formal verification is an extreme cost amplifier, and the cost is super-linear with the size of the code base. I don't think my estimate is unrealistic, depending on exactly what you would expect to have running before vi (I was assuming a complete OS including a kernel, device drivers, a C stdlib or equivalent, and a userspace similar to the GNU tools).
I'm always amazed that the top-tier security firms use... Linux. And the Militaries of the world use Windows (like come on). You'd think there would be some crazy million-dollar-per-license impenetrable operating system out there. The race to the bottom is palpable.
I recall that many companies have switched from a perimeter model of defense where systems are secured from the outside to a "defense in depth" model where each system is secured on it's own (plus the perimeter).
Perhaps folks should think about tightening the in-depth model and avoiding the consumer model of constant updates from a zillion providers. Or perhaps a single lab could verify updates of the zillion providers rather than leaving them on their own.