This is how the police gets warrantless searches "for drugs" - having identified the subject, call for a dog and have it "alert" on the subject (or the vehicle belonging to the subject), after which there's a probable cause for a search and everything that is found can be used against the subject. The fact that there was no drugs is irrelevant.
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account