
Show HN: Perceptual hashing tools for detecting child sexual abuse material - faustomorales
https://github.com/thorn-oss/perception
======
faustomorales
Hi HN! I'm a data scientist at Thorn, a non-profit dedicated to defending
children from sexual abuse. We're excited to open source some elements of our
perceptual hashing tooling as a Python package. We've tried to make it very
flexible both for ourselves and hopefully also for others. Our aim with is to
provide tools that (1) help more people eliminate child sexual abuse material
from the internet and (2) assist with common tasks where perceptual hashing
can be helpful (e.g., media deduplication). We hope you'll take a look, get
some use out of the package (check out the example use cases), and even
contribute feedback and/or code to make it better.

For more information on the issue, I urge you to check out our CEO's TED talk
here: [https://www.thorn.org/blog/time-is-now-eliminate-
csam/](https://www.thorn.org/blog/time-is-now-eliminate-csam/)

Documentation for the package here:
[https://perception.thorn.engineering/](https://perception.thorn.engineering/)

~~~
akersten
If I may, I'm curious about your thoughts on few things, in the context of use
case #1 (abuse), not #2 (reduplication):

* What are the challenges surrounding verification that your system functions properly, given that the test material is illicit?

* Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates?

* Are you aware of any large organizations leveraging your solution?

* Do you feel that the availability of these tools obligates service providers to use them, either morally or legally?

~~~
faustomorales
Thanks for asking these important questions!

* What are the challenges surrounding verification that your system functions properly, given that the test material is illicit?

You're right that storing child sexual abuse material (CSAM) is illegal,
unless you are the National Center for Missing and Exploited Children (NCMEC)
or law enforcement. What is legal is to maintain a hash of known CSAM. NCMEC,
Law Enforcement, and large tech companies maintain their own data sets of
known CSAM hashes and, where appropriate, share them. The Technology Coalition
[1] has more information on this. All that said, we can and do simulate the
system to verify that it works properly using bench testing with non-illegal
content [2].

* Can you speak to the reliability of the system in a sensitivity/specificity kind of way? In other words, what are the false positive and false negative rates?

The false positive rate in practice is very low. We set our thresholds based
on bench tests with an expected false positive rate of less than 1/1000 (the
thresholds vary based on which hash function was used). Different hash
functions are more resilient to some transformations than others (e.g.,
cropping, watermarks, etc.).

For the false negative rate, it depends entirely on the kind of modification
made to the image. For many common operations, it is close to zero.

* Are you aware of any large organizations leveraging your solution?

Thorn builds technology to defend children from sexual abuse, one of the
products we build for this purpose is Safer [3]. Perception provides an easy
way to get started using the Safer matching service. Safer provides a more
robust and complete solution including handling a queue of content and
reporting tools. Some organizations using Safer include Imgur, Flickr, and
Slack.

But this technology (perceptual hashing) is used by many companies who don't
use our tools. Our goal is just to make it easier for more people to get
started.

* Do you feel that the availability of these tools obligates service providers to use them, either morally or legally?

Not being a lawyer or a public policy expert, what I can say is that the law,
as I understand it, requires companies to report CSAM once they are aware of
it. Working in this field I’ve learned two things pertinent to this question:
(1) Most people don’t know how pervasive of an issue this is, and (2) There
aren’t a lot of easy ways to start protecting your platform from this abuse.
No one wants the cool new products and platforms they make to be used to abuse
children. Privacy is important too, which is why solutions that preserve
privacy and avoid leaking private information to third parties are critical,
and perceptual hashing allows us to do both.

[1]
[https://www.technologycoalition.org/](https://www.technologycoalition.org/)

[2]
[https://perception.thorn.engineering/en/latest/examples/benc...](https://perception.thorn.engineering/en/latest/examples/benchmarking.html)

[3] [https://getsafer.io](https://getsafer.io)

EDIT: Line breaks

~~~
nieve
A false positive rate of 1/1000 is hard to assess without actual prevalence
stats, but with a decent-sized userbase it seems likely you're still going to
get a significant number of false positives. Is it intended that users of your
system would have employees manually vet all positives (with legal and mental
health concerns) or just submit them without review? I'm coming from having
built tools to support a large manual sweep in the 2000s and watching the toll
it took on my coworkers.

~~~
faustomorales
Great question — organizations decide how to handle reviews internally. So the
answer to your question on “review all” versus “automatically submit” is a,
perhaps unsatisfying, but honest: it depends. We provide a guide [1] to help
organizations formulate their own policies. And we're currently working on a
content moderation tool that focuses on helping organizations operationally
handle problematic content and considers the wellness and resiliency of
reviewers.

[1] [https://www.thorn.org/sound-practices-guide-stopping-
child-a...](https://www.thorn.org/sound-practices-guide-stopping-child-abuse/)

~~~
nieve
Oh good, I'm sure most organizations can use something like that guide as well
as the tools. There's a lot of legitimate worry about both the wellness side &
the legal exposure issues, but it seems like beyond the common wisdom to be
very careful (somehow) I think in a lot of minds there's a lack of clarity as
to what exactly that means. Is there a particular reason access to the guide
requires handing over contact information?

------
tempaccount8354
If you dare to look into this topic, please also take a moment to look to
another child protection non-profit[1] that (quite unusually) considers human
and civil rights and sex positivity as one of its core values.

You might feel uneasy about "sex positivity" being associated with preventing
child sexual abuse. They also have a lot of other messages that might
initially repel you. But I also think that "when you associate shame and guilt
with sex, you are facilitating sexual abuse"[2].

Here is a relevant Twitter thread about CSEM content filtering[3] and the
secrecy around it. Secrecy that got them ejected out of a public National
Center for Missing & Exploited Children meeting merely for tweeting about what
was being said.

This Show HN open source project by Thorn seems to be an enormous improvement
on that front. Would it be resistant to adversarial hashing (false positives)?

In my opinion, Thorn focuses way too much on technological solutions, and has
an outright hollow message beyond that. Looking at 10+ of their website's
pages, they don't dare to try to confront or explain the actual child sexual
abuse itself, but only its most visible ill effects.

[1] [https://prostasia.org/about/](https://prostasia.org/about/)

[2] [https://prostasia.org/blog/the-weapon-of-
shame/](https://prostasia.org/blog/the-weapon-of-shame/)

[3]
[https://twitter.com/ProstasiaInc/status/1178783074328424448](https://twitter.com/ProstasiaInc/status/1178783074328424448)

~~~
claudiawerner
I think it's good you posted this; the goals listed on the first link seem
very reasonable to get behind, and it's something both I and researchers in
child sexual abuse material (e.g. Amy Adler's paper, which is one of the most
cited in the field) have noticed. The rhetoric of "think of the children" can
be and has been used many times to silence sexual activity between adults,
including fictional representations (such as manga and comics, some of which
are illegal to possess in the UK), kinks (online ageplay roleplay is often
targeted under obscenity law, despite being between two adults), and other
cases.

In my view, the protection of children based on hard evidence showing causal
links is paramount, not suspicion and anecdota (it was, in fact, anecdota
given before Parliament which was used in the creation of the bill in England
and Wales to illegalize _lolicon_ manga) based on the "social harms" that seem
obvious at first sight but are refuted by cultural anthropology (e.g Patrick
Galbraith and Mark McLelland have come to very interesting conclusions
regarding how adults consume fantasy material). There is no group to fight for
the few thousand people caught under this law so far (according to the English
VAWG report 2017), nor do I imagine their being. The organization you linked
seems, at least, to have this sort of thing on the agenda.

------
amitport
I don't get it. Sound nice but what exactly does Thorn do? what is this
spotlight services they're providing? Seriously I've just spend time on their
site but I don't get it beyond "using data". How they use it?

Lets say I have friends in a local DA office and I want to sell them this idea
of using data they have. They just know that giving access to their data is a
big legal headache. what exactly to I tell them to get them on-board with
this? ( _exactly how the data / what data is handled?_)

------
GistNoesis
How did you build, label and store your dataset ? What's the legal framework
for manipulating such data ?

------
natch
What are some of the ways you can improve on the n-squared nature of these
distance calculations at scale?

In your use case you have a limited set of hashes you are comparing against,
but I'm thinking of the more difficult use case of comparing every image
against every other image (the context is ML training and weeding out overly
similar examples so that overfitting doesn't happen, not related to CSAM).
This quickly becomes untenable if you get, say, millions or billions of
images.

"Don't do that" is one answer but are there any other ways you have come
across to mitigate that?

~~~
faustomorales
Hi there! Our example on deduplication [1] takes the case of deduplicating of
the Caltech256 [2] dataset for the purpose I think you're describing, which I
interpreted as avoiding duplicates in a dataset so that you don't have images
that end up in both your training and test sets. Even though the dataset
contains >30K images, you can still do this in memory (and find a handful of
duplicates!) because each category is relatively small and you probably don't
need to deduplicate images of trains with images of dogs.

On the same page, we mention two tools (specifically, FAISS [3] and Annoy [4])
to help with doing approximate search for scales where computing the distance
matrix is impractical.

[1]
[https://perception.thorn.engineering/en/latest/examples/dedu...](https://perception.thorn.engineering/en/latest/examples/deduplication.html#real-
world-example)

[2]
[https://authors.library.caltech.edu/7694/](https://authors.library.caltech.edu/7694/)

[3]
[https://github.com/facebookresearch/faiss](https://github.com/facebookresearch/faiss)

[4] [https://github.com/spotify/annoy](https://github.com/spotify/annoy)

~~~
natch
One more thing: I tried the different hashes that would run on my system (some
had a dependency on opencv-contrib that was not met, and which python -m pip
couldn't find) and found they aren't what I'm looking for. They may be good at
finding different versions of the same image, but what I want is more like
environment similarity. So that two pictures taken on the same street would be
closer than a picture of the street versus a picture of a forest. They don't
seem to be tuned for this task at all. So, my search continues. Let me know if
you have suggestions on this as well.

~~~
faustomorales
Perceptual hashes in this family will probably not be the right fit for that
use case. There is work out there to build hashes that are intended to find
find semantically similar content, typically using CNNs. Imagededup [1] seems
to have one of these though I have never used it myself.

[1]
[https://idealo.github.io/imagededup/](https://idealo.github.io/imagededup/)

------
GistNoesis
How do you defend against semi transparent overlaying ?

Take an image you want to censor, overlay with a very transparent offensive
image. Also publish the original image (or a second image with a different
transparency value), and an in-browser extension can reconstruct the offensive
image.

Your hash database will be flooded with wrong values.

~~~
lonelappde
That seems like symmetric key encryption with extra steps.

~~~
GistNoesis
This is a general attack against perceptual hashing. It tries to achieve
multiple objectives. Using stenography you could add any data in the image,
like you would with encryption as you suggest. But here it is deeper.

By superposing a very transparent bad image with the good one, the perceptual
hash won't be affected but the image would be still labelled as offensive by
the poor guy labeling the data because the transparent offensive image is
still visible. This mean that the good data will be marked as potentially
offensive.

This allows anyone to target any individual user, or website. You take some of
their published content, overlay offensive image and republish. The image will
be reported as offensive, yet have a similar hash (i.e. a collision) to the
good image which automated systems will pick-up as the original image being
offensive and blacklisting its user or tanking down the website on search
engine results.

Because this is sensitive data, by law it should be deleted as soon as it is
detected which means the proofs get deleted automatically.

------
m0zg
When I read about something like this, I can't help but wonder, how do you
deal with error analysis and such? I personally would not be able to view such
disturbing and revolting material. And if you don't do error analysis, how do
you know this stuff actually works?

------
totoasticot
Reminds me of [https://github.com/CIRCL/douglas-
quaid](https://github.com/CIRCL/douglas-quaid) which also use SIFT-similar
techniques in addition to perceptual hashing

------
olliej
Seriously this is a topic that I can’t imagine having to think about :-/

I really hope that there’s support in place for people who have to work on
this, even if they aren’t exposed the actual abuse.

------
httpsterio
Without reading the link, can this be used to help find child porn? If yes,
are there any steps you can or are going to take in order to prevent this?

~~~
Nasrudith
It only filters images already there - child porn isn't already among the
chaff only thing it can find by definition would be false positives. Which
could be an interesting vetting exercise in itself - feed in tons of cat
pictures until it dings on one as inappropriate.

Technically you can always try cycling through image combinations match the
hash/looks like child pornography and nothing could technically prevent it
except that it would take a very, very, long time.

Even given adversarial networks I would be very surprised to get anything more
than vague figures and I don't even expect the result to look human. In which
case it is really generating instead of finding.

------
callmejorge
This is great work. Not often we see posts about using tech directly for
humanitarian efforts!

------
pbalau
From what I've read, this is supposed to catch the bad thing after it
happened, not to prevent it. Then you say that 1 in 1000 is an acceptable
error rate. 1 in 7 billions is not an acceptable error rate!

~~~
catalogia
> _" Then you say that 1 in 1000 is an acceptable error rate. 1 in 7 billions
> is not an acceptable error rate!"_

When systems like this return a positive match, that doesn't result in a
summary execution. Rather, it prompts a deeper investigation.

