As far as I'm aware, the Moto X is very much able to avoid triggering by anyone's voice except the one it's keyed to. I was curious about how well it worked, so I tried repeatedly to launch Voice Search on my girlfriend's phone (even trying to imitate her voice); I wasn't able to do it at all. Trying to do so to an arbitrary person's device would be even harder.
How would this even work if it was stored on the server? Every singkr sound triggers a roundtrip to the server from your phone to check the voice? I can't imagine this working any other way than being stored locally.
I don't really see how this follows, particularly since I don't believe you can sync the voiceprint across phones. If you're saying that it's _possible_ to send voice prints to the server unnecessarily, then sure: but this has nothing to do with always-listening/voice-print storage; it could be the case for any sort of voice command, which means that what we're talking about here isn't any different from any voice command system in that respect.