For some reason smart speakers give me pause but not a smartphone. A smartphone is kind of the same thing (microphones listening) when plugged in with the addition of a camera and a host of other sensors. I guess because a phone's main function is to not listen to us that makes the speakers much more worrisome to everyone?
Even something like an Apple Watch is the same thing with the addition of reading your vitals.
If you don't trust home assistant vendors to not randomly send non-queries to the cloud, I'm not sure why you'd trust phone vendors to not have the microphone on when you haven't given it permission.
The latter is more unlikely compared to the former, which is by design. Furthermore, at least on iOS Siri can be completely turned off through a configuration profile.
This is a reasonable approach for someone whose thread model allows them to use a smartphone...
Even something like an Apple Watch is the same thing with the addition of reading your vitals.