
Non-Consensual Intimate Image Pilot - JumpCrisscross
https://newsroom.fb.com/news/h/non-consensual-intimate-image-pilot-the-facts/
======
Xeoncross
The better solution is to have the image hashed and NOT sent to facebook. If
the hash matches an existing image [A]. If the hash isn't found [B].

[A]: human can review the image they already have

[B]: Facebook waits until someone uploads an image that matches and then
reviews the image (as normal) but with a marker alerting the problem.

The benefit is that people NOT affected won't have to upload lots of images of
themselves to facebook personal for "review" _just to be sure_.

The problem is that should facebook update it's hashing or find better ways to
match images, having the original image would allow them to transition. This
point is moot though since facebook claims not to save the image anyway.

~~~
trjordan
There was an excellent Twitter thread on why they didn't do this:
[https://twitter.com/alexstamos/status/928740488395608065](https://twitter.com/alexstamos/status/928740488395608065)

The salient bit:

> A quick note to everybody who says 'calculate the hash locally':

> A) Photo fingerprinting algorithms are usually not included in clients to
> prevent the development of circumvention techniques.

> B) Humans need to review to prevent adversarial reporting.

Further down the thread, they discuss further that PhotoDNA and similar
algorithms are, in particular, not resistant to being tricked. Until they are,
it's another layer to prevent adversaries from fully decomposing how they're
hashing it, which is a good thing.

It's a trade-off, for sure, but this is a hard problem with no cut and dry
answer. Facebook already has the images in question, because you have to be
sending them to somebody in order to mark them as NCII. Asking yet another
Facebook service to look at them without storing them seems reasonable to me.
You may disagree.

~~~
topmonk
First, Facebook isn't the only place to share revenge porn, so they aren't
really solving the problem, only reducing it slightly.

Second, security by obscurity is not a good thing. Not running the algorithm
locally, not to mention producing the source to their algorithm is not going
to help in the long run.

Third, regardless of whether the source of the algorithm is known or not, it
would be a simple task to upload a test image of oneself, and then attempt to
circumvent the algorithm using dummy accounts.

Fourth, if someone abuses the service by uploading the hashes of a benign
images, it can be reported at that time. No need to pre-emptively guard
against this.

------
LukaAl
As other has pointed out, this is extremely creepy. The easiest solution would
have been this one:

\- The user uses a JS solution to hash the images on the client, without the
image being uploaded

\- She compiles a form with additional information (e.g: capture her account,
reasons for uploading, suspect person sharing the picture)

\- The picture is saved in the DB as un-verified revenge porn.

\- The first time someone uploads a picture that matches the hash, the pic is
quarantined and the specially trained individual manually check them

\- A scoring system could be used to check the reliability of the submission.
If multiple photos marked revenge porn get rejected, the control becomes ex-
post. For even more violations, the user get banned from using the tool and
should directly contact Facebook. Submitting the same hash that has been
rejected, will count as a "red mark"

Now, I understand this system is very complex, what Facebook has done is an
MVP and as a product manager, this is what I prefer. But considering the issue
(revenge porn, not something I necessarily want to test the impact on
retention :-) ). Also, yes, it requires resources, but Facebook has a problem
with trust lately, better to do the best...

[edited for formatting]

~~~
evgen
Now figure out a way to do step #1 (user does a hash client-side) without
making it trivial for someone else to create a filter that adds enough noise
to invalidate step #4 (uploaded pics that match a hash are quarantined.)

~~~
wjh_
PhotoDNA would be an option, I imagine. Or at least something similar!

[https://en.wikipedia.org/wiki/PhotoDNA](https://en.wikipedia.org/wiki/PhotoDNA)

~~~
moyix
Honestly, I doubt that most of these sorts of algorithms would survive
concerted attacks – that's why they tend to be closely guarded.

Alex Stamos (Facebook's CISO) implies this is why they can't do it client-
side:

[https://twitter.com/alexstamos/status/928646228472078336](https://twitter.com/alexstamos/status/928646228472078336)

~~~
dannyw
The attacker isn’t a national state adversary. The attacker is a vengeful ex.
The chances of such attacker even been aware of what hashing is, is close to
zero.

Whoever designed this system is probably heavy on security, but low on
product.

~~~
kelnos
No, the attacker is a bored hacker who wants to create problems for FB, just
because they feel like it.

------
alexggordon
> To establish which image is of concern, people will be asked to send the
> image to themselves on Messenger.

> Once we receive this notification, a specially trained representative from
> our Community Operations team reviews and hashes the image, which creates a
> human-unreadable, numerical fingerprint of it.

This clearly implicates that Facebook has, almost without reserve, the ability
to read messages from user to user (even though it might limited to messages
to oneself), and exposes that ability to its employees.

While I stopped using Facebook a long time ago, Zuckerberg's quote, "they
'trust me'; dumb fucks" seems relevant here. If I were Facebook, I would do
this in a different, much more privacy conscious way.

1\. Obviously, Facebook is already hashing all images on its platforms and
storing the hashes. Given that, let the user, using a frontend only platform,
upload an image and generate the hash. Make sure this image is a real photo
(not a meme or something heavily photoshopped) in addition to making sure:

1a) The image contains a human[0]. Fairly easily doable for most situations.

1b) If not, let the user know that "we can't automatically detect a human in
this photo, do you want to still submit this photo for removal, this could get
a strike against your account if (not in some list of reasons for removal)".
Maybe the photo has a picture of a credit card or something.

2\. Submit the hash to the backend. Given that hash is not used on the
platform past a significant amount of views, automatically ban the photo, and
allow other users to petition for reinstatement of the photo, knowing that a
non-valid petition for reinstatement (the user doesn't have rights to the
photo) COULD result in a strike also.

3\. Have humans review the petitions, and in cases where the users explicitly
allow, the images.

Doing this would limit another human seeing the intimate photo in well over
90% of the cases I'd bet. In addition to that, even if the photo is shared
with another human, it would let the end user decide if they'd like a Facebook
employee to view the image too. IE, basic user privacy. Come on Facebook. Dumb
fucks.

[0] very doable, [https://trackingjs.com/](https://trackingjs.com/)

~~~
danbruc
_This clearly implicates that Facebook has, almost without reserve, the
ability to read messages from user to user (even though it might limited to
messages to oneself), and exposes that ability to its employees._

You can view and search your message history on Facebook, so there is not
really any question that they have and can read all messages. I never
explicitly checked it but at least I never noticed any gaps, i.e. missing
messages because I sent them using the Messenger app.

I believe to remember that Messenger is supposed to use Axolotl Ratchet for
end-to-end encryption but that is hard to reconcile with the availability of
your message history on facebook.com. So maybe it's - quote - end-to-end -
unquote - between the phone and the server?

I never really thought about that. Or is it just not available or disabled in
my Windows Phone version of the app?

~~~
xanderstrike
You can download a complete copy of the data Facebook has on you, including
all messages you've sent since your account was created [1]. I did this some
years ago to graph changes in sentiment with friends.

1\.
[https://www.facebook.com/help/131112897028467](https://www.facebook.com/help/131112897028467)

~~~
mavhc
Well, all data Facebook has on you that you uploaded, if other people upload
your address, that's apparently their data.

------
deusofnull
The discussion surrounding this made me think of a feature I'd love to have
along these lines. I'd like to be able to blacklist people / certain people
from uploading pictures that facebook detects my face in. Fb as a platform
must already have this functionality, what with the auto face identification
and privacy / review post settings.

I can imagine images of people that aren't "non-consensual intimate images"
that they'd reasonably still like to be able to block being posted.

One use case I can imagine is doxxing prevention. Say there is a national news
story involving some random person in some small town that exposes the persons
face and name and rough location. Some group of enraged internet denizens
start sharing the person's face and name all over the place as a means to
spread their pitchfork & torches mob against them.

Seems like a reasonable feature to me. Recently a professor in my state made
some remarks about white privilege and the doxxing of him was so violent and
pervasive that he and his family had to leave the state for months, and he
still gets threats and needs protection on campus.

~~~
15155
> I can imagine images of people that aren't "non-consensual intimate images"
> that they'd reasonably still like to be able to block being posted.

If I take a photo of a public landscape and you happen to be there, do you
have the right to prevent me from publishing it unaltered?

Keep in mind: you're not the primary subject (and no reasonable person would
believe that you were) nor owner of the photo. Your likeness is not reasonably
intended to alter the value of the photo.

~~~
yjftsjthsd-h
Excellent! Then it should be short work for you to photoshop my face out so it
won't be blocked. Everybody wins.

~~~
15155
Why is Facebook obligated to fulfill your desires for privacy by censoring
content you have no legal claim to?

~~~
yjftsjthsd-h
I think the claim is that I _should_ have a legal claim to my likeness.

Note that I'm not personally convinced of this, I just felt that it was really
weak to say that it was fine because the faces aren't an important part of the
picture.

~~~
15155
> I just felt that it was really weak to say that it was fine because the
> faces aren't an important part of the picture.

This was very intentional, because it is how courts look at the _monetization_
of this kind of content today.

Your likeness cannot be commercialized without your consent (which is why
things like photo/video releases exist) - that being said, the line
determining your importance to the photograph's value is up to what a
reasonable person (a judge) would believe.

This comparison, while not perfect, was used to preempt an argument from the
"$ without permission" angle.

\--

As a private company, Facebook is free to add this feature for its users.

As a citizen: I am free to publish my photo containing your likeness elsewhere
until legislatively or judicially prohibited. And then I am free to challenge
that censorship (as many have on the commercialization issue) on obvious First
Amendment grounds.

Aside: I won't get into any European arguments about the "right to be
forgotten" \- we (USA) don't recognize this as a right.

~~~
Y_Y
But Facebook operates in Europe with photos of EU citizens. So regardless of
where the photographer is from, the photographee should be able to assert that
right.

------
marrone12
Why do they need a human representative to hash the image? I don't understand
why this can't all be done automatically. Creeps me out that a real human
would see intimate pictures before they block them.

~~~
BinaryIdiot
I'm _assuming_ the only reason a human representative is doing this is so the
system doesn't get abused. Like say I upload a photo claiming it is of myself
but it's actually the advertisement a competitor of mine is using. If they
didn't verify it was something that SHOULD be removed they'd automatically
remove my competitor's posting of their advertisement every time they posted
it.

Granted I really wish this was all done on the client so Facebook didn't gain
access to the images themselves but I'm not sure of a good way around it to
verify the image is something they should remove and not an abuse of the
system.

~~~
tantalor
_they 'd automatically remove my competitor's posting of their advertisement
every time they posted it_

Since that will be very rare case why not make _that_ the human-required step?

~~~
BinaryIdiot
Why would that be a "very rare" case? Facebook and Twitter have (or have had)
hundreds of thousands of bots and constant abuses, why would this feature be
rarely abused?

------
intopieces
It’s interesting that the general consensus is strangers seeing your nudes is
creepy (referring to the manual processing aspect.) I understand it on a
fundamentally emotional level: I want absolute control of my private life /
photos / etc. I am embarrassed at the idea of being seen naked without my
active participation. On a logical level, though, what difference does it make
if every person you will never meet has seen your dick?

It’s a philosophical question, to be sure.

Also, what happens with these manual reviews that turn up underage nudes? Teen
sexting is a thing. For that matter, isn’t Snapchat a massive repository of
child pornography?

~~~
cowpig
What's really bizarre to me is that people here don't think using facebook in
general is creepy...

~~~
intopieces
I think it's the same deal: your data is being seen / used by algorithms, your
information is just a bunch of 1s and 0s being copied and processed and re-
copied and re-processed. It's all very impersonal, so it's easy to abstract it
away.

------
wellboy
Ok, I had to laugh. FB is literally the worst company to make this service
with the worst privacy reputation ever.

Then the service makes the worst mistake that they actually upload nudes to
their platform, PLUS they have a human look at it, which adds another layer of
security leaks.

Did they do this intentionally or did they have such a huge lack of common
sense when they decided how to design this and nobody in the project had the
idea to address this.

~~~
dude01
I think they want to see how much can they get away with. Let's see if we can
actually get people to send us all their private porn.

------
dmitrygr
So they basically openly admit that a human will review nudes? How is this in
any way a sane idea? If you were already scarred (for life, likely) by said
revenge porn existing and leaking out somewhere non-facebook, likely the last
thing you want is to be forced, YOURSELF, to send it to someone (facebook)

~~~
skybrian
From the article: "people can already report if their intimate images have
been shared on our platform without their consent".

If it already happened, they don't need to upload it again. Presumably this
new process is for prevention when it didn't happen yet but they have good
reason to suspect that it will.

~~~
dmitrygr
"if it already happened _on facebook_ " != "if it already happened"

There is, in fact, a whole world outside of www.thefacebook.com

------
BadassFractal
Tricky. Sending them all of your potentially leaked embarrassing photos so
that they can store them and prevent them from leaking out. If they get hacked
now someone has a lifetime supply of blackmail material. Not clear if the cure
is better than the ailment.

~~~
moyix
From the fine article:

"We store the photo hash—not the photo—to prevent someone from uploading the
photo in the future. If someone tries to upload the image to our platform,
like all photos on Facebook, it is run through a database of these hashes and
if it matches we do not allow it to be posted or shared."

~~~
tonyarkles
Is there a way for me to audit that the image is never stored? (No) It seems
like that's a huge honeypot; someone compromises the "upload your naughty
photos here" endpoint and has an endless supply of "non-consensual intimate
images".

Edit: Thinking about it a bit, I'd be _way_ more comfortable with the idea of
the image hashes being computed on-device and only the hashes being sent to
the server. This opens a different door of abuse (by essentially permitting
individuals to ban someone from posting an image by uploading the hash of the
image), but it's definitely preferable to encouraging people to send all their
selfies to the privacy commissioner.

~~~
evgen
While this (client-side hashing) would be better in terms of privacy
protection for this specific case it is not going to happen because Facebook
is not allowed to perform the specific photo hashing client-side as this would
expose the hashing mechanism to analysis.

~~~
dmitrygr

      > this would expose the hashing mechanism to analysis
    

Security through obscurity. Works every time (tm)

~~~
kelnos
I mean, it worked pretty well for Google's ranking algorithm, so much so that
it spawned an entire industry (SEO, which still doesn't let you get away with
much in the way of black-hat type stuff).

Even just the rate-limiting aspect of keeping the algo on the server (in this
case behind a human, but that's even not necessary) is useful: I can't sit
offline with the algo and keep beating it with slightly different images until
the hash changes enough to be considered unique.

------
orastor
To everyone asking why they need a human to review to image before hashing it:
it's currently the only way to prevent abuse of this system -- Without this,
everyone would upload random images that would get taken down, effectively
mass-trolling

~~~
Pulcinella
At most, they would only need a human to review after the hash matches, not
before.

Facebooks plan is awful.

------
Apreche
Ok, so now all the harassers out there are just going to edit a few pixels on
the images before uploading, so the hash isn't caught. All the same tricks use
don YouTube to avoid the copyright bot, will work here as well.

~~~
BinaryIdiot
Yeah I'm curious how their hashing mechanism works. Like, is it a straight up
sha hash of the image or something a little more sophisticated that wouldn't
be fooled by minor edits? Though in the latter case can that even technically
be considered hashing? I guess it all depends on how they're doing it which
brings us back to my first question...

~~~
evgen
It is called PhotoDNA and was developed by Microsoft. This is the same tech
that is used to find child porn images even when people go to various lengths
to prevent simple hashing from being able to determine photo similarity. It
works surprisingly well at the task from what I have seen/heard.

~~~
b4lancesh33t
Photo dna is just one of a class of algorithms called perceptual image hashes
(there are also perceptual audio and video hash algorithms). They by no means
invented the technique, just one particular branded implementation of it.

------
lukejduncan
Great idea, absolutely horrible implementation even as an MVP.

Never in a million years would I trust that kind of data residing on Facebooks
servers even fleetingly. Years back, Facebook allowed you to have private
profile photos. My then-girlfriend-now-wife has a picture of the two of us as
hers which for cultural and religious reasons she had private since we hadn’t
decided to tell her parents yet. Facebook decided without telling anyone one
day that Facebook photos were now public. She hasn’t spoken to her father
since that day. Facebook can and will do whatever they want with the data they
have on you. Maybe permissions change on messenger (or there’s a bug) and data
from this flow leaks. Maybe a year from now this flow still exists but
messager makes all picture uploads a soft delete because they want to keep
things as training data. Maybe the upload gets replicated to their CDNs in a
way that is discoverable and recoverable. Maybe documentation for this manual
process outlives the software, someone finds it, and uploads a picture only to
find out they did it in the wrong place that now has no guarantees of even an
effort for privacy.

That’s a lot of maybes for something so sensitive. Facebook doesn’t deserve
your trust with this kind of data. They should come up with something where
the hashing happens locally and the user never has to put faith in them to
begin with.

------
scoot
> Once we receive this notification, a specially trained representative from
> our Community Operations team reviews and hashes the image

At first I wondered "Why?!" Why does a "specially trained" person have to look
at the very same intimate images that their users wish to prevent being
shared.

Then it struck me that without review, _any_ image could be blocked, simply by
submitting it through this process. This would also explain why the offline
hashing approaches being suggested aren't practical - they're just too prone
to abuse by flagging images that are not of a sexual nature, but that you
would like to see blocked for whatever reason.

An ML approach to masking the most sensitive portions of the submitted images
would both protect the user, and reduce the burden on those tasked with
reviewing the submitted images.

If this could be handled on the client, and sent with a verifiable hash of the
original image, that would solve the majority of concerns raised in the
discussions here.

------
aerovistae
It's a cool idea, but isn't this pretty easily defeated by an adversarial
attack, or just altering the image in any simple way?

E.g. as the perpetrator, open the image of your ex in MS Paint, add a stripe
across the bottom in such a manner as to not obscure the primary content at
all, ta-dah, upload, image will not have a matching hash.

~~~
averagewall
Already addressed in this other comment:
[https://news.ycombinator.com/threads?id=turc1656](https://news.ycombinator.com/threads?id=turc1656)

It seems like HN comment pages might have a critical mass beyond which it's
impractical to see if somebody's already said what you want to say so people
repeat the same idea - ballooning the comments page even further.

------
iam-TJ
What happens when the FB employee[1] finds a way to collect the images and
personal data they are reviewing and at some later time distributes the images
using some other outlet?

[1] "... a specially trained representative from our Community Operations team
reviews and hashes the image, ..."

------
beager
In this thread there are a lot of valid concerns about client-side
fingerprinting and how this would allow sophisticated algorithms to be broken.
Putting something like PhotoDNA on clients is probably not good.

However, I think a simpler approach would handle >95% of cases here, and
that’s to just calculate the image sha and use that. It does nothing for
resizes/obfuscations, but if Facebook users are dumb enough to volunteer their
nudes to human review, they’re dumb enough to not modify images before
reposting them as revenge porn.

Everything else with respect to modified images and elaborate fingerprinting
and perceptual hashing can happen at the next tier, which I trust Facebook
already has an intricate (and publicly tolerated) system to handle.

------
lasfter
Facebook: "Send Nudes"

------
f_allwein
Even if this worked as intended, what’s the point? The person wanting to
spread such images could just use another platform to do so, or modify the
image slightly... Would this really stop much abuse?

~~~
gdulli
The same hubris that lets Facebook believe they can do this without it
backfiring on them is the same hubris that lets them believe that theirs is
the only platform that matters.

------
nvahalik
Is there anyone else out here scratching their head as to why anyone would
think it's a good idea to actually take (or otherwise) send (let's be honest,
here) pornographic images of themselves to anyone else?

Back a long time ago, if you had a Polaroid you might have some reasonable
expectation that picture was the only one... but now? Once it is digital you
cannot guarantee a single thing that happens to it unless you do it offline.

Why would anyone want to take that risk?

~~~
sneak
Your threat model is not my threat model.

~~~
nvahalik
Well, there is this. Part of the reason why I took the cameras out of my step-
daughter's phone. No ability to consider past the here-and-now.

------
oh_sigh
> Once we receive this notification, a specially trained representative from
> our Community Operations team reviews and hashes the image, which creates a
> human-unreadable, numerical fingerprint of it.

I'm wondering how long it will be before we start seeing cell phone pics of
screens with peoples intimate images on them.

------
chatmasta
How long will it be before "nudes" are so normalized that people post them to
their own profile? 10 years? 20 years? 100 years?

If you doubt this will ever happen, look up some images of "1920s beachwear"
and ask yourself where the trend is going.

------
borplk
Lots of people caught up here about nitty gritty details of hash type client
side vs server side and so on. The elephant in the room in my view is the fact
that the person can just upload the images on another image/file sharing
platform.

------
Freak_NL
Is it legal for Facebook to solicit pornographic images from minors, or are
they (nominally) limiting this feature to adults?

------
macawfish
Something just doesn't add up here.

------
beaconstudios
isn't nudity generally banned on facebook anyway? I didn't realise this
would/could be an issue. Unless they're talking about private communications
in eg. messenger.

~~~
sp332
Revenge porn isn't all about nudity. And this does cover Messenger (and
Instagram). The point isn't to get an image taken down after the fact, but to
proactively block it from being posted or sent in the first place.

~~~
beaconstudios
I thought facebook would run posted images through their AI nudity filter
before allowing them to be posted? It's good that this covers messenger as
I've read stories in the past of people being blackmailed over it, but surely
there must be a better way than asking people to upload their private images
to facebook. Plus, what's the case with younger facebook members? Do they
upload the images and are therefore sharing child pornography with facebook,
or not upload and don't get to be protected from revenge porn postings, when
they're one of the most at-risk demographics?

~~~
sp332
I didn't know Facebook had an AI nudity filter, I thought it was all based on
reported images. They softened their stance on some things, especially after
censoring a Pulizter-prize winning photo from Vietnam, and eventually changed
their policy on breastfeeding too
[https://www.facebook.com/help/340974655932193/](https://www.facebook.com/help/340974655932193/)

I'm sure they don't have a legal exception for underage photos, but those laws
do vary from state to state.

------
ryanmarsh
OMG the mad rush to get there first for libraries of revenge porn.

------
loa_in_
Twins with different opinions about this are going to have a hard time...

------
pault
This is insane. If they can fingerprint your images, why can't they provide a
tool for processing the image and sending the fingerprint only? Can you
imagine how big of a target a tailor made database of blackmail material
linked to facebook accounts would be?

~~~
DiThi
If you can send only the fingerprint, what stops you from uploading photos you
are not in?

A better idea may be an uploading tool where you can obscure some parts for
verification.

~~~
karthikshan
That's an interesting idea, but might break whatever hashing method is used?

~~~
DiThi
The idea is to hash the original image, but to send the obscured one for human
verification. However this requires a different hashing method (such a Merkle
tree) to know the unobscured parts match.

------
nether
Fuck Facebook

------
Breefield
Am I the only person reading this who thinks it won't work at all...? All it
takes is the image being sent via screenshot, or otherwise manipulated to
change the hash

------
turc1656
This is dumb. A hash? That's only going to deter the most incompetent internet
users when it comes to a file type that is subject to alteration without
losing it's information value (pictures, video, music, etc.).

All someone has to do is change a single bit/pixel in the image and the hash
will be different. No one will notice that and it defeats the hash. Hell, you
don't even have to do that. You can just rotate the image, save it, then
rotate it back, and re-save it. The odds are that program that you used will
most certainly not write the image data in the same exact way.

Hashes would be better for files that cannot be altered without breaking the
contents. Although, these could be subject to compression, which would create
a new hash.

Either way, I think they need to use something similar to Google's image
search that actually examines the photo contents for similarity.

~~~
sveiss
In this case, 'hash' doesn't necessarily mean 'a cryptographic hash of the raw
file contents'.

There are perceptual hash techniques which still match after changes to the
file, and are tuned in a similar manner to lossy compression algorithms to
prioritize comparing information relevant to human perceptions.

[https://www.phash.org/](https://www.phash.org/) is one open-source example.

