Yeah, and it's not just an LLM thing either. There's a thought-experiment I like to pose for self-driving vehicles: Suppose they statistically are half as dangerous as manual driving, but this record is tainted by unpredictable and inexplicable Scary Stuff like driving up on the sidewalk to attack a pedestrian or going sideways off a bridge.
Would we be satisfied with that tradeoff? Should we be satisfied? At what point is a X% error case we can understand and characterize equivalent to a Y% error case where we can do neither? (Mitigation is another aspect.)
I would go even further. Around 170 people die everyday in car accidents. If 1 person died because of an accident caused by a self driving car they would be banned.
Would we be satisfied with that tradeoff? Should we be satisfied? At what point is a X% error case we can understand and characterize equivalent to a Y% error case where we can do neither? (Mitigation is another aspect.)