Your argument is no different than what Apple could do to your iPhone. The fact that it happens on the server changes nothing. Apple could push a button and have your iPhone upload whatever they want to their servers. In other words, based on your argument, you shouldn't trust anything, including locally run AI. You're probably right, but it isn't practical.
Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well:
> Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good.
> In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
I think he has a nice pragmatic view on things. I’m EU enterprise we basically view things like picking cloud providers as a question of who we want to spy on us. Typically it comes down to AWS or Azure if you’re pocking a “everything included” service. That being said, I’m not really sure I’m on board with this part:
> As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.
Isn’t that sort of where the pragmatism ends? All the parts aren’t going to be secure… Unless I misunderstood his intention, I think the conclusion should be more along the lines of approaching the cloud without trust.
Edit: The final couple tweets from the Matthew Green tweet thread posted in another comment sum it up well:
> Wrapping up on a more positive note: it’s worth keeping in mind that sometimes the perfect is the enemy of the really good.
> In practice the alternative to on-device is: ship private data to OpenAI or someplace sketchier, where who knows what might happen to it. And of course, keep in mind that super-spies aren’t your biggest adversary. For many people your biggest adversary is the company who sold you your device/software. This PCC system represents a real commitment by Apple not to “peek” at your data. That’s a big deal. In any case, this is the world we’re moving to. Your phone might seem to be in your pocket, but a part of it lives 2,000 miles away in a data center. As security folks we probably need to get used to that fact, and do the best we can to make sure all parts are secure.