You're still relying on a piece of software to make the wake-word assessment and hand off the audio to the cloud. Now you're just adding more hardware parts to fail.
If your argument is that you trust your software more than Amazon's, then you shouldn't need anything more than a single microphone anyways because why would you surveil yourself?
For the same reason nginx usually runs as a separate user: people make mistakes and security vulnerabilities happen. Security in depth is a good thing.