Right, so we can distinguish between "model did an oopsie" and "model was actually hijacked".
The former seems to be a much bigger problem. I've seen lots of reports of "Claude/codex bypassed security / stole API keys to complete a task", where that was the "intended" behavior — at least from the LLM's POV, trained to persist and complete user's task at all costs.*
I've actually seen zero reports of actual hijacking due to prompt injection, though they must be out there too.
---
* See also the recent thing about how every model was willing to do fraud to get a refund:
More often than not these sorts of stories are "I ignored all of Claude's warnings and told it YOLO and I'm totally surprised my production database was deleted!"
The agent crawls some websites and reads some stuff and pushes some buttons. It's my limited understanding that you have
1. Hard limits like "file system permissions" (* never mind it going around for sake of this conversation)
3. On other hand of spectrum, fundamentally soft limits like a prompt "and ask me before you purchase "
2. And then in the middle you have harness enforced limits like "it won't do X with approval"... Those are being sold as hard limits but frequently they have classifiers and heuristics that essentially try to figure out which button on some third party website constitutes a "purchase" and triggers the harness limit.
Bottom line - I don't think such limits are as 100% fool proof as they are advertised / interpreted.
They support payments through Shop and Link. I don't know about the former, but Link requires that you do a separate approval for purchases. The Duet cannot just take a card and do what it wants.
