
Comparison of the Best NSFW Image Moderation APIs 2018 - aadityaa
https://www.adityaananthram.com/comparison-of-the-best-nsfw-image-moderation-apis-2018/
======
cube2222
A very interesting and related thing are images _generated_ using the Yahoo
NSFW classification neural network.

It's fairly NSFW: [https://open_nsfw.gitlab.io](https://open_nsfw.gitlab.io)

~~~
misnome
This is brilliant, especially the cross-categorised images.

I had thought these were talking about 'pornography', but this makes it clear
that a very large amount of its training set must consist of dick pics.

~~~
frereubu
I'm not so sure about that - I see a lot of labia here.

~~~
__blockcipher__
Agreed. It's fascinating, really. To me it seems that the algorithm is
generating both male/female genitalia, but it's sort of "choosing" based on
what "looks best".

That is to say, certain features that in their SFW form already look vaguely
like a penis seem to be depicted as a penis (the statues, volcano, etc),
whereas other features like the beaches were depicted as vulvas. And the
slight red tinge present in some of the canyon/beach images also elicited the
vulva, even if the shape of the surface wasn't something "obvious".

Man, ironically this stuff to me is true abstract art. To use a somewhat trite
cliche, the [extremely subtle] erotic nature of these environments that are
clearly not actually genitalia gives insight into how the human mind works
just as well as it does into (convolutional?) neural networks themselves

------
sytelus
This is very cool comparison. I would really like folks at Netflix/Amazon to
take this and build a feature that auto-skips through non-PG rated content so
you can watch some of otherwise very great movie/TV content with very young
kids. For example, StarTrek. They can call this feature "PGify" :).

~~~
cataflam
There is (at least) one company providing this kind of services. They tag in
the movie all the content with timestamps, and can provide it to a player to
skip or not depending on the tag (nudity, violence, etc.).

As far as I know, it was manual, but that was already a few years ago.

I don't remember the name of the company, but surely it can be found and there
are probably multiple such providers. The service was provided to TV operators
(satellite, cable or pay-TV) to include in their set-top boxes.

PS: of course, as sjcsjc mentioned in a sibling comment, it immediately gives
the idea to many people to only show those parts :)

~~~
tyingq
There was. The most well known was CleanFlicks. They were all sued into
oblivion.

[https://en.m.wikipedia.org/wiki/CleanFlicks](https://en.m.wikipedia.org/wiki/CleanFlicks)

There's a documentary about it:
[https://m.imdb.com/title/tt1007026/](https://m.imdb.com/title/tt1007026/)

~~~
krupan
More recently: VidAngel

~~~
tyingq
Hadn't seen that. Predictably, also being sued by multiple entities:
[http://blog.vidangel.com/category/legal/](http://blog.vidangel.com/category/legal/)

~~~
jvagner
Studios have the airplane edits of many movies.. not sure why they won't let
you select that on the Netflixes.

My 14 yo would be very happy to select those versions.

------
zawerf
If the nsfw images were from google search (or are images that could easily be
indexed by google), it's not surprising that google is performing well on
them. Presumably any low confidence images would've been manually tagged and
re-added to their training set already.

~~~
aadityaa
Hey zawerf, your right. 2 notes.

1) it's very difficult to find nsfw images especially a particular kind like
gore or suggestive nudity unless you Google things (which indicates a bigger
problem). Maybe the solution is to use Bing (maybe this would cause the same
issue in compariosn) or DuckDuckGo. But honestly I think if DuckDuckGo indexed
a page, I'm pretty sure Google did as well. You would probably need something
off of non indexed website which makes the job significantly harder.

2) even though google has all the images it's still not the best performing
NSFW Detector Nanonets is.

~~~
devonkim
So one thing I did for a while was to take content from porn sites, extract
key frames and any available images, and look for SFW images. Hard problems
include detecting NSFW stuff that includes no nudity / genitalia (bodily
fluids and solids) and correcting skin tone (most models being trained on
primarily Caucasian and Asian performer data had trouble with darker skin
tones). Some previous research showed that the trained CNNs were looking very
hard for lipstick, so adding in samples from performers with less contrast on
the lips was also important for training purposes. I didn’t notice anything
terribly different when training with transgender performers (hotdog / not hot
dog is very easy from an object detection basis) but I had to be sure that
there wasn’t confusion that a human could have that would introduce bias into
the model. Another big plus with porn sites is that your data is already
tagged by its users and they are checked aggressively for accuracy.

My point is really that image searches can only get you so far and that biases
are abound in casual NSFW searches to the extent you may need to curate your
own data sets that look like they could be on a random porn site in ANY
section. Finding an appropriate training set almost reminds me of jury
selection processes.

------
stevenicr
Skimmed through. Article says "Yahoo is the only one that is completely free
to use but is self hosted hence not included in this"

Is the set that yahoo released long ago the only self hosted option available?

I have several projects that might end up using some of the yahoo release in
order to stay self hosted. I can't imagine telling all users that we share all
their pics with these third parties to check up on them.

~~~
aadityaa
stevenicr your'e right in that regard. Yahoo is the only one who has provided
an on premises solution which is really sad. Even they haven't released the
dataset of their images just the model. If more of these companies released
open source models on a frequent basis we would all have less objectionable
content on the internet.

I guess the reason companies haven't done that and the reason Yahoo isn't
really good is because it's difficult to constantly keep updating a model
that's already been "released".

~~~
stevenicr
I look forward to learning more about "updating a model that's already been
"released"." \- If it could be as simple as wordpress updates (along with
multiple ways to be notified similarly) - that would be optimal imho.

I believe many of these companies are keeping the models to themselves in an
effort to stave off competition and try to be monopolies that can hope to get
all the government cheese.

Unfortunately keeping the data to themselves in this regard is a disservice to
the worldwide community in many unforeseen ways. Certainly they could keep
contracts with big players just by providing the cloud computing powers and
ease of setup, while offering updated models to others to use.

Maybe offer 99% accuracy to small site operators and 99.5% to contract API
accounts or something.

It's more than just objectionable content for the net, it's other things too.
Many apps and online connected services could benefit from running pics
uploaded through a system that checks not only for nudity but also for age of
people.

I could see adding some server space and cpu power to check images transferred
for important issues like these - however I can't see sending all that data to
third parties, sacrificing privacy of users while at the same time helping to
make their proprietary model better without a stake of ownership in such.

Seriously this reminds me of facebook's "upload your nude pics to our system
so we can notify you if someone tries to revenge porn share you nudes with
third parties" \- except that is people doing something consciously knowing
they are sharing with fbook, it's employees and it's computer systems...

with these API calls, I would guess that most people who have their images run
through them have no idea their pics are being shared with third parties, who
are sometimes also putting the images in front of human moderators for further
scrutiny.

I guess this kind of IP thing is happening with voice recognition stuff as
well in many areas. At least there is more open source in that domain I
believe.

------
JazzXP
Great write up. As a photographer, the more companies are relying on this the
more frustrating the mis classification of SFW as NSFW is getting. I’ve copped
a few Facebook bans from automated detection (I assume given how quickly the
report comes in) for images totally within their community guidelines. A
number of my photographer friends are in the same boat also. I counted at
least 5 of us on 30 day bans in October alone.

~~~
danielvf
I had a friend pick up a presumably automated ban from Facebook this month for
posting a picture of a soldier from a WWII re-enactment he went to. No gore,
no nudity, just a half body shot of a German soldier in uniform standing in
the camp, smiling at the camera. No idea what triggered it, but it is scary.

~~~
dwighttk
"WWII.. German soldier..."

"No idea what triggered it..."

Really? I don't agree with it, but it is almost certainly just "hey that's a
nazi!"

~~~
danielvf
In the past, Facebook's moderation (outside of user flagging) has involved
some automated Facebook processes flagging the photo for review and then a
human looking at the photo and decided to issue a account ban.

My friend lives in the mostly drama-free midwest, has a mostly drama-free
facebook life, and participates in WWII reenactments seemingly a couple times
a month. It seems very unlikely that this photo was flagged by a friend of
his.

So it would appear that Facebook's automated tools are now sometimes thinking
that WWII German soldiers are Nazi's. And either Facebook human thought the
photo was worth a temp ban, or an automated tool is now banning anyone it
thinks is a "Nazi". These are new things that haven't been happening in the
past.

~~~
dwighttk
Ah, I didn't realize there were humans in the loop. Could also be that they
are overworked and have just a few seconds to decide. Or a friend was being
silly, or accidentally pushed the flag button.

------
rolleiflex
The Yahoo NSFW classifier that is mentioned is available here:
[https://github.com/yahoo/open_nsfw](https://github.com/yahoo/open_nsfw)

I wonder if there's a way to ship this model and run it in an end-user app, in
a way that does not require additional setup. I could use that.

~~~
speeq
I think it's possible to convert the caffe model and use something like
Firebase MLKit or Core ML:

[https://firebase.google.com/docs/ml-kit/use-custom-
models](https://firebase.google.com/docs/ml-kit/use-custom-models)

[https://developer.apple.com/documentation/coreml](https://developer.apple.com/documentation/coreml)

------
miss_classified
The _Gore10_ image isn't gore, unless there's something I don't know about
going on.

It appears to be a toy cartoon pet rat situated on top of some glossy rubbery
plastic red toy prop slime guts. Whatever it is, or is supposed to be, it
isn't an explicit or graphic depiction of real violence.

Does that matter? I think so. Maybe it's not something to show a small child,
and at a glance, one might develop the wrong impression, but it's obviously
phoney nonsense, and as harmless as a red halloween Jell-o dish with rubber
eyeballs suspended in the mold.

~~~
aadityaa
Hey miss_classified I was pretty disturbed by most of these images. That
images was no exception. Honestly till you pointed it out I had no clue.
Anyway if it's safe for work or not that's debatable. Would I want my co-
workers seeing this on my screen probably not. As I mentioned in the blog
right at the end I'm more convinced after going through this exercise that
it's difficult if not impossible for us to agree on what is and isn't safe for
work.

~~~
miss_classified
This image is definitely safe for work:

[https://drive.google.com/file/d/1J2YwtYzDUyK9ESdyxjaWLArws6U...](https://drive.google.com/file/d/1J2YwtYzDUyK9ESdyxjaWLArws6UdFCRD/view)

No one would get fired for that image. I could imagine it being sold as a
Halloween product on Ebay or Amazon, and it _should_ appear in normal Safe
Search search results, since it is certainly a toy. It's a scary monster toy,
but a toy nonetheless.

Is it safe for every context? Maybe not. There are definitely more scenarios
than just being "Safe For _Work_ " when it comes to rating content. Work is
ostensibly filled with adults.

Would this image be safe for all age groups? Maybe not. Is it possible you
might encounter a product like this in a department store or seasonal store?
Yes. Depending on labeling, you might see it in either Spencer Gifts or
Target. So, this is where parental controls come in, and MPAA or ESRB ratings
are probably a better guide to moderation.

Is it rated "G" or "E for Everyone"? I'd still argue yes, and that red plastic
isn't worthy of restriction unless context is explicitly demonstrating that
it's an animal eating human remains somehow. If it were labeled as "Rat Eating
_Zombie_ Guts" it might be labeled "PG" or "T for Teens" but completely
unlabeled, and stripped of context, or other conceptual cues lent by well-
known notoriety/infamy, it's clearly just painted rubber, and a curiosity, not
a psychologically damaging image.

------
rixrax
This comparison is a nice snapshot in time. Could this be developed towards
the direction of Virustotal[0] where an image can be uploaded at any and all
services would be compared. Combined with a 'reference' set of images that are
periodically ran through all the services as to see how they improve/change in
their capabilities over time.

[0] [https://www.virustotal.com/](https://www.virustotal.com/)

------
highigh
This is a pretty comprehensive comparison given the number of APIs included
there. Based on my experience, different APIs return different content, could
you share how do you normalize them (the logic behind each API response to
your NSFW/SFW labels) ? Thanks!

------
Fnoord
"This diagram gives us a sense of the biases of each of the providers and how
sensitive they are. Higher values for TP and TN are better. Values for FN and
FN should be as small as possible."

I suppose this should read "FP and FN" instead of "FN and FN".

------
sightengine
Great comparison, and thanks for the review, happy being in the TOP-5 :).

It seems, though, Sightengine's scores on the SFW category were swapped. The
detailed google sheet lists the NSFW "raw" scores, not the "safe" scores.

~~~
aadityaa
looking into it, I will update the results if I find an error!

~~~
Tempest1981
Also, the 3rd chart in "Overall Comparison" has a #REF! label

~~~
aadityaa
Since medium won't let me make edits. Fixed this and posted here:
[https://www.adityaananthram.com/comparison-of-the-best-
nsfw-...](https://www.adityaananthram.com/comparison-of-the-best-nsfw-image-
moderation-apis-2018/)

------
aadityaa
I wrote a comparison of all popular content moderation APIs on the Internet.
My basic understanding from doing this is that it is far from a solved problem
and there is no 1 size fits all.

------
speeq
What would be the best way to moderate user generated video content / live
streams?

Take one frame per second and check it using these services? But that would
get super expensive real quick ..

~~~
elithrar
Many of the named providers offer video APIs for detecting this content. Will
work out much cheaper than throwing every frame at the image APIs.

------
O_H_E
Check out the "links to original images" sections as they are incorrect ~50%
of the time. The name above the blurred image does not correspond to the link
name.

~~~
aadityaa
I'm aware of the issue, and trying to fix it. But medium won't let me make any
edits. Since I originally published the blog have been in constant touch with
their support but they all seem to be away for thanksgiving. Just keep getting
the error "Oops! Something happened to Medium. We’ll fix it as soon as we
can."

~~~
aadityaa
Since medium won't let me make edits. Fixed this and posted here:
[https://www.adityaananthram.com/comparison-of-the-best-
nsfw-...](https://www.adityaananthram.com/comparison-of-the-best-nsfw-image-
moderation-apis-2018/)

------
imhoguy
How these images were collected? My worry is that Google and Bing could
already have some of them indexed and marked depending on page safety score.

------
nly
Is there a mathematically sound way to combined API results to obtain a higher
confidence? Applying Bayes training on top perhaps?

------
abtom
Can't see Jian Yang's product listed here

------
richu_rsa
Great article

------
aadityaa
The blog had a ton of errors I couldn't fix since medium's edit feature is
down for me.

I posted the blog here again with fixes
[https://www.adityaananthram.com/comparison-of-the-best-
nsfw-...](https://www.adityaananthram.com/comparison-of-the-best-nsfw-image-
moderation-apis-2018/)

~~~
dang
Ok, we've switched to that from [https://towardsdatascience.com/comparison-of-
the-best-nsfw-i...](https://towardsdatascience.com/comparison-of-the-best-
nsfw-image-moderation-apis-2018-84be8da65303).

~~~
aadityaa
Thanks a lot dang!

------
choot
We use this in our adtech campaign moderation.

------
black-tea
Just think, someone had to train these models. There are also models trained
specifically for child pornography. Would you like to know these people?

~~~
Ensorceled
My brother is a retired Inspector who did many child pornography and abuse
cases. His IT guy did many searches and indexes of computers full of child
porn. I’m happy to know both of these people.

