A human can be held responsible for a mistake. A model can't. For the ethically flexible, the latter is a benefit rather than a hazard: one can use ML to whitewash bias and rely upon public perception of computers as dispassionate and objective.
Humans are expensive and require individual training. Models scale. In the time it would take a human to make a mistake, a broadly-deployed model might make millions of impactful mistakes.
I think analogy with medicine is fruitful. Small molecules can't be held responsible either, but they can be recalled from market and they do. Our pharmacology isn't good enough to design, say, vaccine that works worse for Asians, but we did require clinical trial participants to be a representative sample. Medicine is also mass produced, that's why there is post-market surveillance to catch rare side effects missed in clinical trials.
Even with all this analogy, we don't insist medicine should have known mechanism of action, although we do prefer it. So I think we will regulate models with recalls, testing standards, monitoring etc, but won't insist on understanding.
By the way, don't we already have widely deployed models, such as PhotoDNA, which supposedly removes millions of images a year to filter child pornography? I wonder how it was evaluated to be suitable for deployment.
Humans are expensive and require individual training. Models scale. In the time it would take a human to make a mistake, a broadly-deployed model might make millions of impactful mistakes.