Fun fact, we fixed this issue by adding a #profanity tag and dropping the message to the next human agent.
Now our most prolific sales engineer could no longer run demos to potential clients. He had many embarrassing calls where the Ai would just not respond. His last name was Dick.
I find it odd that your engineer would make the system rely on instructions (“Do this. Never do that.”). This exposes your system to inconsistencies from the instruct tuning and future changes thereof by OpenAI or whoever. System prompts and instructions are maybe great for demos. But for a prod system where you have to cover all the bases I would never rely on such a thin layer of control.
(You can imagine the instruct layer to be like the skin on a peach. It’s tiny in influence compared to what’s inside. Even more so than, in humans, the cortex vs. the mammalian brain. Whoever tried to tell their kids not to touch the cookies while putting them in front of them and then leaving the room knows that relying on high level instructions is a bad idea.)
Now our most prolific sales engineer could no longer run demos to potential clients. He had many embarrassing calls where the Ai would just not respond. His last name was Dick.