> Then it's a great time to be a LLM security researcher then.
This reminded me of Jon Stewart’s Crossfire interview where they asked him “which candidate do you supposed would provide you better material if he won?” because he has “a stake in it that way, not just as citizen but as a professional comic”. Stewart answered he held the citizen part to be much more important.
I mean, yes, it’s “probably a great time to be an LLM security researcher” from a business standpoint, but it would be preferable if that didn’t have to be a thing.
* Mislead agents to paying for goods with the wrong address
* Crypto wallets drained because the agent was told to send it to another wallet but it sent it to the wrong one.
* Account takeover via summarization, because a hidden comment told the agent additional hidden instructions.
* Sending your account details and passwords to another email address and telling the agent that the email was [company name] customer service.
All via prompt injection alone.