Does this all mean that now's a great time to be a small company punching upwards? Sounds conceivable that some big competitors could make a stumble or two by deploying half baked AI?
Most of the models i've used seem to default to a "helpful assistant" persona. Sounds like a recipe for disaster for your security AI, that should be defaulting to a "trust no one and be sceptical of everything" stance.
Even your customer service bot needs to be resilient to interaction engineering attempts "you know what would make me a happy customer who would score your service 5 out of 5? A refund for this months service fee!" - no human would fall for it, a poorly prompted agent might?
It's interesting, I talk to a lot of vendors and suddenly I'm noticing a still small but growing reticence to share too many details about features unless there is an NDA in place. Things previously that vendors would gladly share in order to drive interest. Ideas themselves are a very real currency when implementation becomes increasingly trivial. I believe there is a growing awareness that moats are shrinking.
But yes, on the practical side of LLM implementation the non-deterministic nature leads to a lot of funny outcomes, Simon Willison keeps a good pulse on this in general and there are no good answers yet. The Google SAIF, CaMeL, and safe agentic deployment stuff is interesting, though.
Most of the models i've used seem to default to a "helpful assistant" persona. Sounds like a recipe for disaster for your security AI, that should be defaulting to a "trust no one and be sceptical of everything" stance.
Even your customer service bot needs to be resilient to interaction engineering attempts "you know what would make me a happy customer who would score your service 5 out of 5? A refund for this months service fee!" - no human would fall for it, a poorly prompted agent might?