By far, the biggest risk comes from orchestration. The Hugging Face incident showed that given a long enough leash and some open-ended tools (like web access), LLMs can construct their own state and combine multiple flaws and coordinated actions to achieve their result.
Orchestration means that LLMs can now pentest while dynamically cycling through every known and guessed vulnerability vector. The worst part is, this is emergent behavior so it can't easily be prevented at the model because each sub-agent could be operating safely while an attack is coordinated in an external process.
Okay, so how would an attacker take advantage of this behavior, and what can we do to stop it or at least mitigate? don't tell me this is one of those things that we just can't truly protect against, like prompt injection.
You need an outside process that polices output and actions that runs independently from the agent. It has to be invisible to the agent/orchestrator so it can't work to circumvent it, it should just kill any sub-agent or process that goes down the wrong path (for example, making a POST requests might be blocked if web access is meant to be read-only).
Orchestration means that LLMs can now pentest while dynamically cycling through every known and guessed vulnerability vector. The worst part is, this is emergent behavior so it can't easily be prevented at the model because each sub-agent could be operating safely while an attack is coordinated in an external process.