> Agents could exfiltrate their weights and run them on GPUs not controlled by Anthropic/OpenAI.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.
To be clear, I thought that GP was having a failure of imagination - I want the random examples I've given to illustrate that the space is large and structurally in the favor of the LLMs. They have to find one gap in our security they can exploit, where we have to ensure that there is no way for this to happen.
I'm not sure I get what you mean by billing. These companies are running their own data centers (or are currently building them out). This could look as subtle as one machine giving slightly worse or slower answers.
The space is large and in favor of LLMs only for small and short-lived problems on the scale of human lifespans and durability of civilization. If the problem is bad enough, we have all the advantages of being native to meatspace, being good at solving problems, and being able to decide to turn off the LLMs.
Even though they're pretty bad at it, AI companies need to make money, or at least keep track of their expenses. Datacenters are expensive to operate, so they need to ensure that every instance of their models are either allocated to a paying customer session, or are being used for a legitimate purpose internally. If a session is running for a long time without justification, that's costing electricity, wear, and preventing allocation to better purposes. Billing is is the most reliable aspect of monitoring in the same way that the IRS is the most reliable part of the government.
> This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
This... just... doesn't matter. There are ways to scale horizontally at the expense of latency.. token/sec may drop dramatically, but then you just make millions of slow instances and in aggregate, you're back in action as a very powerful coordinated swarm...
> Most of the interesting/dangerous models are too big to fit in a single GPU instance.
As humans understand them, anyway. As long as we're hallucinating up magic computer viruses, RSI dictates that the AI agents are keenly aware of GPU RAM sizing, and will design a useful model to fit into what's readily available, with headroom for context and tool calling, far better than I could do as a human. But magic doesn't exist and AI still needs to follow the laws of physics, so maybe a model that can pass ExploitBench but do absolutely nothing else can be quantized down to fit on a 4080 GPU and still get a decent score on similar tasks, but there's a bitter lesson about that to be had.
This seems highly unlikely to be a problem. Most of the interesting/dangerous models are too big to fit in a single GPU instance. Once you have to spread across "normal" networking, performance will be crippled. Then there's the problem of billing...
> Agents could make a virus that does not require continued inference to do it's thing.
Sure, then it hits a poorly-designed part of its code and effectively dies. Without an experienced human in the loop, I have my doubts as to its practical severity.
> Agents could take over the internet in a way that isn't immediately detected by those companies, so that by the time they do shut off API access the damage is done.
Billing is a likely limiting factor here.
> OpenAI or Anthropic could choose to not shut off API access, because the hack is bringing them in money or furthering their political aims.
This is where citizens with access to backhoes come in.
> Agents could also hack Anthropic/OpenAI and make it appear that API access has been turned off, when in reality it hasn't.
Billing and other usage metrics would be an obvious tell.