I can only see LLM's forcing a perpetual stalemate for application exploits. Projects will start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines. Technical exploits being a dead end, attackers focus their agents on large-scale social engineering. Spamming Discord and Facebook is the new war dialing. Multi-year /goal sessions culminating in gaining a position of trust and sabotaging the CICD pipeline's pentest step since getting anything past it would be intractable. Interesting times indeed.
If projects start adding "tell the strongest no-guardrails open weights model to pentest it for 12 hours" to their CICD pipelines, then researchers will prompt a pentest for 14 hours.
Different models test to find different attack paths. Attackers with bigger libraries of attack techniques will be able to train more dangerous models.
There will also be models that make better use of tools, like static analysis and fuzzing, and they will find different defects. Social engineering is going to be a big skill to learn too, as it is a much softer skill.
Right, but the comparison isn't perfect here because wall height is linear in the analogy, but searching for exploits in an existing piece of code reaches a point where either there are none left to find, or finding the next exploit would take years. That's the stalemate point I referred to. (Any high-stakes project may end up having its own "red team" agent/s running 24/7 too.)
