Yea I think being able not to leak is the bare minimum? But it really depends on how good it is at specific applications; I wouldn't give a tax-preparation agent my SSN unless I was confident that it was no more likely to misfile my taxes than a professional tax preparer.
In other words, the risk of harm doesn't need to be zero, just less than the equivalent risk of a human with similar skillset. So I'm comfortable riding in a waymo, and not comfortable giving chatgpt my SSN at this moment in time, but I expect that within 5-10 years (assuming no doom) I will trust some AI agent with my SSN because they will be better at handling sensitive info than humans
I feel punishment is largely a means to the end of reducing overall harm. If a vehicle is less likely to kill me, that's my preferred option regardless of whether it achieved that safety through negative consequences for the driver or through gradient descent optimizing a loss function.
I will happily ride in a waymo today, even though the AI powering it faces no consequence if it gets in a crash; it is clear that waymo is safer than human drivers in the areas in which they operate, so who would technically be liable in the event of a crash isn't really of concern to me
I just mean an AI that could use a routing number or SSN and gmail/slack/whatever at the same time without a leak.