The false positive rate has to be a lot lower for this to work.
If it is 1/1000, it is only 1/1,000,000,000 if they have only 3 of images from a customer. They typically have thousands, though. A 1:1000 false positive rate would mean several ones in many iCloud photo databases.
On the plus side, in case of multiple hits, they would have a human look at the images.
The whole thing was intended as a way to make that human check economically viable. Instead of having people look at every picture uploaded to iCloud, they would filter out almost all of them, and only let humans look at the few remaining (where, I guess, ‘few’ still could be a lot, given their number of users)
They wouldn't have a human look at it. They don't have the images, only the hashes–that's the point.
And it's somewhat irrelevant how the probability of collisions is specifically calulated (1/1000 already assumed 1:n comparisons), as long as we agree it's easy to calculate for a given user. The algorithm does know about the sizes of the respective image libraries, for example, and could adjust the threshold with precision.
“The device creates a cryptographic safety voucher that encodes the match result. It also encrypts the image’s NeuralHash and a visual derivative. This voucher is uploaded to iCloud Photos along with the image.
[…]
Once more than a threshold number of matches has occurred, Apple has enough shares that the server can combine the shares it has retrieved, and reconstruct the decryption key for the ciphertexts it has collected, thereby revealing the NeuralHash and visual derivative for the known CSAM matches.”
If it is 1/1000, it is only 1/1,000,000,000 if they have only 3 of images from a customer. They typically have thousands, though. A 1:1000 false positive rate would mean several ones in many iCloud photo databases.
On the plus side, in case of multiple hits, they would have a human look at the images.
The whole thing was intended as a way to make that human check economically viable. Instead of having people look at every picture uploaded to iCloud, they would filter out almost all of them, and only let humans look at the few remaining (where, I guess, ‘few’ still could be a lot, given their number of users)