Here’s a parallel construct: It would be trivial for Microsoft to have a key logger in Windows that sent every keystroke to Redmond.
Why does anyone trust that they are not doing it?
Indeed, they could scan your computer for whatever data they like and send it to Redmond. But no one even suspects it, or at least at nowhere near the level people seem to distrust Amazon and Google over sending your voice to them full-time.
This is what I mean by Misplaced Distrust. We already trust Google and Amazon with far more of our personal data without a thought.
But a microphone? Ooh, scary!