As big a fan of open software as I am, the combinations of open-weight LLM models that are capable of e2e cyber-penetration (even by actors that might just put in the compute and prompt it: "make me money") and the current mitigations that let those entities, that already hold too much data and power (political, infrastructure, etc), expand to become the eventual gatekeepers of the "open internet", is highly worrisome.
If you have ideas for how we can adjust to this safely and with an eventual more equal distribution of power, please do share.
The invention of the locomotive was also the invention of the train wreck. V.
The disaster is not just "open weight" model penetrating everything, its is that we have built the whole world around an IT revolution that never really reached maturity and became secure. The chickens are just coming home to roost.
Those same entities would become gate keepers of the LLMs too.
> we have built the whole world around an IT revolution that never really reached maturity and became secure
I don't disagree, but how does this relate to this incident, in practice? We're talking about a zero-day vulnerability here. What do you think should have happened - forbid any licensed IT practitioners from deploying PHP at all? No WordPress or Wikipedia?
Stop using memory-unsafe languages for anything unless a
formal proof of safety is produced, only use OSes with strong sandboxing mechanism, make sure every piece of software is being sandboxed, etc. Or even just: “make the software authors liable of criminal negligence if an exploit occurs and they cannot demonstrate that they did everything possible to avoid it”.
It would indeed have slowed down the field immensely, but we would have avoided the chaos that is ahead of us right now.
And that sounds crazy compared to the standard we're used to, but what's crazy is that if your coffee machine explodes and harm you the makers are liable, but if your coffee machine is part of a botnet that disrupts a critical infrastructure, causing the death of people, then somehow the coffee machine maker faces literally no liability for not securing their IoT crap.
But scripts written in PHP are themselves memory-safe. The problem here isn't in a particular PHP script, but rather in the underlying implementation of the PHP runtime, which was written in a memory-unsafe language.
If that's your standard, then Python isn't a memory-safe language either, since CPython is partially written in C. Same as many other memory-safe languages like Ruby, anything in the .NET ecosystem...
Sure, and that's the problem: if security was taken as seriously as in any other industry then all of the above languages would have been written in SPARK(or its successor, which would likely have happened much before Rust had safety been paramount).
If you bother to check, you'll eventually notice that even "memory-safe languages" have accumulated a good share of CVEs, inclusive for memory safety issues. Check out RustSec.
> We don't need perfect security, what we need is computer people taking security seriously.
Are we in agreement that all this talk of "memory safe languages" is a red herring? I mean, who is foolish enough to believe that rewriting something in a different language makes is magically safer?
I think the real disaster is the asymmetry in time it takes for large existing systems (inert) to adapt to equally powerful but single-goal (small complexity) attackers. [considering attackers and defenders have equal access to resources to drive the same open-weight models to either defend or attack]
That sounds like some grand statement with answers. The inherent asymmetry of the attacker defender model means there will always be successful attackers.
I’m not sure I’ve ever heard of a locomotive being used as a weapon. Maybe a more apt comparison would be “nuclear fission invented nuclear fallout”, no?
Intentional derailment is something that has happened which feels closer to the example then using the train itself as a weapon IMHO. https://en.wikipedia.org/wiki/Battle_of_Mill for example
If you have ideas for how we can adjust to this safely and with an eventual more equal distribution of power, please do share.