It seems you're then fine with me "lending" your voice to make irritating prank calls to all your relatives and coworkers - since it's, to reiterate, fair use of AI in your opinion
Why do you think that means I'm lending my voice? I still have the ability to talk while you're playing around with a digital representation of my voice. If I lend you my car, I can't use it to drive myself around because you have it.
I mean harassment or libel laws already cover this right? Or revenge porn laws could apply if you did it to your ex. That's not fair use, simply reproducing the sound of a voice using AI is not a crime.