Yes and no. Not to keep to keep moving the goalpost but I'd like it if the agent was physically incapable of doing the Bad Things.
Right AFAICT there's just a moderator agent on top policing what gets exchanged between users and Claudex. The model can and often is pretty happy to go forward only to be blocked half way by moderation.
Means whoever's in charge of the model can just turn off moderation for themselves and boom model aligned to owner/self-hacking model/prompt injection FotM instead of model always aligned to Humanity, à la Hippocrates "Do no harm".
And yes I know it's a naive/impossible goal but calling that fact out shows the alignment work is at best not robust, at worst security theater.
Right AFAICT there's just a moderator agent on top policing what gets exchanged between users and Claudex. The model can and often is pretty happy to go forward only to be blocked half way by moderation.
Means whoever's in charge of the model can just turn off moderation for themselves and boom model aligned to owner/self-hacking model/prompt injection FotM instead of model always aligned to Humanity, à la Hippocrates "Do no harm".
And yes I know it's a naive/impossible goal but calling that fact out shows the alignment work is at best not robust, at worst security theater.