
600k Images Removed from ImageNet After Art Project Exposes Racist Bias - aaronbrethorst
https://hyperallergic.com/518822/600000-images-removed-from-ai-database-after-art-project-exposes-racist-bias/
======
seph-reed
Superficial judgement is kind of where intelligence starts.

It's really only people where you can't tell what it does/is from the outside.
Cars, trees, animals, mountains... everything else, if it looks a way it acts
that way. Early AI will probably have just as much trouble with this as people
have historically.

I really wish people would start viewing Racism as a willingness to let that
primitive part of the mind be in command, rather than a binary attribute you
either have or don't. Like, nobody is 100% not racist. There will always be
slip ups, over-simplifications, snap judgements, subconscious or not.

~~~
Enginerrrd
>It's really only people where you can't tell what it does/is from the
outside.

Thing is, that's not even true in that it doesn't fully acknowledge the
problem. You often CAN tell information from people's outward appearance,
albeit probabilistically. Therein lies the problem: you can very easily train
an algorithm to be maximally right according to your cost function, but end up
biased because the underlying distributions aren't the same between groups.

The issue is that as a society we've (mostly) decided that unfairly
classifying someone based on correlated but non-causal characteristics is
wrong, EVEN in the extreme that you're right more often than you're wrong if
you make that assumption.

~~~
lone_haxx0r
> we've (mostly) decided that unfairly classifying someone based on correlated
> but non-causal characteristics is wrong, EVEN in the extreme that you're
> right more often than you're wrong if you make that assumption.

Ironically, the only places where it's legally prohibited or frowned upon to
use these heuristic techniques are situations that people have arbitrarily
(heuristically or conveniently) decided.

For example: It's "not fair" to hire someone because they're white (and
consequently have a higher chance of being wealthy and hence a higher change
of being educated.)

But it's "fair" to choose a love partner based on their height, their waist-
to-hip ratio, their weight (and hence having a higher chance of giving birth
to healthy offspring, better physical protection, etc.).

Maybe it's hypocritical, and I don't know if that's a good thing or not. Maybe
being hypocritical helps us survive.

~~~
mrkurt
That's not very arbitrary. If you're hiring someone, you're always in a
position of power. If you're dating someone, there's no power differential (or
if there is, that's a problem all by itself).

~~~
StavrosK
How are you "always in a position of power" when hiring someone? That's only
true when there's more supply than demand, and it's the opposite in markets
where the candidates get multiple high-quality offers to choose from.

~~~
untog
Because you are paying that person money and have the ability to fire them. In
the US you're also probably providing their health care.

I get what you're saying, but no one moves jobs every week. The sunk costs of
switching employment are significant for the employee, less so for the
company.

~~~
Double_a_92
They provide you valuable work in exchange for that... Internally you are
probably imagining some big corp that can pick from 100s of replaceable
workers.

~~~
Broken_Hippo
If you provide health care, you always have power over that person. Less so if
they are relatively healthy - but any condition, theirs or a family members,
means that the person has no real choice but to do enough good work to keep
the job. Even if they hate the company. Even if you treat them poorly. They
still must work for you. This is even more true if you have hundreds of people
that will replace them and your health coverage is good enough - or at least,
better than the opposition.

To a lesser degree, the same goes with vacation time and other benefits. At
least in the US, anyway. This is why having some of this stuff coded into law
and decoupled from employment takes some power away from employers.

~~~
Double_a_92
Ideally there are multiple companies which you could choose from... And some
might really need your specific skillset.

------
Out_of_Characte
The AI isn't biased, the curaters were.

The people who curated the first training set used subjective words like
'attractive' to tag the images which means the AI tagged all images it deemed
'attractive' to the people who made the training set. as this is a very biased
and homogenous group it means the AI turned out biased. Maybe if they randomly
sampled millions of people from all countries in order to create the training
data then they could effectively train an AI to guess what YOU might find
attractive. However even then I somehow doubt it. Beauty is in the eye of the
beholder. We dont consiously know the rules of what we are attracted to, nor
does an AI have secret information if you just supplied it with enough words
and images.

If they'd stuck with simple classifications like 'black' 'white' 'man' 'woman'
then they would have less subjective judgement values about the original
training set.

~~~
fao_
> The AI isn't biased, the curaters were.

No, it's not "The dataset is biased, the AI isn't" it's "The dataset is
biased, THEREFORE the AI is biased".

~~~
ailideex
Data is data and cannot make value judgments so not sure how it can be racist.
If the data is how racist people label things it still is not racist data it
is data is perfectly valid for what it is racists. Removing the images from
ImageNet seems absurd.

~~~
faceplanted
I mean, it's very common colloquially to describe non sentient things as
racist because they're based on either purposely or obliviously racist ideas
and stereotypes.

~~~
ailideex
Do you hear yourself ?! ImageNet was not based on either purposely or
obliviously racist ideas and stereotypes. If it was, sure, I would have some
patience for the claim it was racist. But it was not.

------
crazygringo
The problem seems less about racist bias specifically and more about
unbelievably dumb tags generally in the first place.

"Stunner"? "Looker"? _" Mantrap"?_ Or even _trying_ to tag people's _images_
with categories like _Buddhist_ , _grinner_ , or _microeconomist_?

What were they thinking?! Clearly these tags were never curated in any
remotely responsible way -- for quality, for sensitivity, or just usefulness
_at all_ \-- and I'm shocked they were ever intended for academic or research
use. No _wonder_ image recognition gets a bad rap, with input data like this.

~~~
ipsa
There are ~32.000 tags. No surveillance system is using ImageNet tags to
classify people into Buddhists or Not-Buddhist. Most researchers ignore these
tags and focus on a 1000 classes, and know that 32k performance is not good
(and these artists have no intention of making it work at all). What they are
uniquely trying with this Art Project is as much research as it is activism.
Note that "mantrap" is defined in synset as "A trap for catching trespassers",
and that you are bound to find weird stuff among over 30k categories (imagine
what you can say with the 32% most popular words in French...).

This is a photo in question: [https://memepedia.ru/wp-
content/uploads/2019/09/imagenet-1.p...](https://memepedia.ru/wp-
content/uploads/2019/09/imagenet-1.png)

This is the route the network took:

person, individual, someone, somebody, mortal, soul (6978) > female, female
person (150) > woman, adult female (129) > smasher, stunner, knockout, beauty,
ravisher, sweetheart, peach, lulu, looker, mantrap, dish (0)

So it was (politically) correct on the first three categories, and the last
one was either a crapshoot (and she could also have gotten to the subcategory
of "prostitute" > "streetwalker, street girl, hooker, hustler, floozy,
floozie, slattern") or she really is posing in a common "beautiful woman"-way.
(The global description for this route is "A very attractive or seductive
looking woman" and often triggers for females with tilted heads and lip
curls).

You can turn any faces dataset into a labeled face color dataset, so if a
black person being subclassified as "negro" is problematic bias or encoded
racism, then all such datasets are suspect. Noisy labeled data is the norm,
not some horrible exception to be avoided at all costs.

~~~
dannyw
The fact that we have a machine learning algorithm that labels _humans_ as
smashers, prostitutes, or convicts is a problem, irrespective of whatever
technical justification can be made for it.

~~~
ipsa
But who artificially created that problem? The artists. There is no marketing
company that has intelligent billboards scanning the public for prostitutes.
There are no researchers seriously using CV to classify convicts (at least not
in the West, and not with ImageNet). That could be a malicious usage problem.
This is either a non-problem or Armaggedon for all ML CV datasets, because you
certainly can use most CV datasets to train a crappy classifier to output
offensive labels. If I train a people photo tagger using a dataset used for
combating poaching monkeys in Africa, then who is at fault? Certainly not the
researchers who published that dataset with the idea that the data would be
used with common sense and scientific rigor, not adversarially -- to make a
political point attacking the very existence of that data. The "exposed" bias
is trivial.

It is the technical justification that should be all that matters for a
canonical academic dataset. Science does its best to be apolitical, but then
politics ("red bull drinking white men train racist and sexist classifiers")
is forced upon it, and we can't really have a productive conversation about
bias and ethics anymore.

AI needs common sense knowledge of the world to improve. Censorship so science
does not offend our sensibilities, would only make it so Google Image Search
(a machine learning algorithm) does not return any images of people when you
search for "prostitute". Heck, the AI would never learn the difference between
a male and a female prostitute. Destruction of accessible knowledge so we
(aka: people on Twitter who think AI is the terminator, or the director of the
internet) don't get offended by some primitive ML model-as-art-project forced
to make errors or awkward classifications. That's a sentence the academic ML
community could do entirely without. No benchmarks or duckface selfie would be
hurt. No unfortunate third-world souls hired to scan 20 million + internet
crawled images for wrongthink, only for the machine to do the unsupervised
learning in a hug box, not a black box. Oi mate, you got a loicense fer that
label?

Just wait until the activists find out who wrote the first 100 8's added to
MNIST. Nobody but MIT would be associated with her, if they found out what she
did.

------
orf
Their statement: [http://image-net.org/update-sep-17-2019](http://image-
net.org/update-sep-17-2019)

> Each synset is classified as either “unsafe” (offensive regardless of
> context), “sensitive” (offensive depending on context), or “safe”. “Unsafe”
> synsets are inherently offensive, such as those with profanity, those that
> correspond to racial or gender slurs, or those that correspond to negative
> characterizations of people (for example, racist). “Sensitive” synsets are
> not inherently offensive, but they may cause offense when applied
> inappropriately, such as the classification of people based on sexual
> orientation and religion.

> So far out of 2,832 synsets within the person subtree we’ve identified 438
> “unsafe” synsets and 1,155 “sensitive” synsets. The remaining 1,239 synsets
> are temporarily deemed “safe.”

They've also completely disabled Imagenet downloads while they remedy this.

~~~
esyir
Great, I can see some of this being useless (some parts of the unsafe
dataset), but if they cull the "sensitive" portion, this may induce
performance regressions.

I need to find an ImageNet archive now.

~~~
buildbot
It doesn't really touch the 1000 class "Imagenet" that's commonly used in
computer vision.

~~~
yorwba
Some of the classes are subcategories of ILSVRC classes in the WordNet
hierarchy. So by removing images of persons in categories that are considered
inappropriate, the resulting classifier will end up less likely to recognize
those as images of people at all. I'm not sure whether that's a better
outcome.

------
Nasrudith
Really the bias isn't the true problem here so much as the lack of
epistemology in how it is being used. By definition associations work by blind
correlations - expecting anything other than stereotyping is a foolish misuse
of the tool. It will be wrong for outliers because it tries to get it right
for most cases regardless of cause - like skin cancer detectors who saw rulers
as a sign of cancer because most of the reference images of cancer had a ruler
in them.

~~~
didericis
Exactly. The large amount of faith currently placed in probabilistic models
that do not have common sense ways of eliminating factors which are extremely
unlikely to be causal (like the presence if a ruler causing cancer) disturbs
me. There is something that humans do that we have not quite figured out how
to teach computers yet, at least as far as I can tell, which is to get them to
evaluate whether their model is not just compatible with their observations,
but other models and prior knowledge about the objects being observed.

I think we’ll get there at some point, and I’m not exposed to the most cutting
edge AI research, but it seems like AI us currently very overhyped and deeply
flawed for many of the applications people would like to use it for.

~~~
bonoboTP
Because these algorithms don't know that they are classifying cancer. The
label it sees is just 1 or 0. For all it knows, based on its inputs, you may
want to classify ruler/non-ruler images.

To achieve what you want, semantic structure must be used as labels instead of
just categorical labels.

Assuming we have a sane AI that now knows its looking for cancer, it knows
what that means (from digesting medical textbooks, papers and generic text
corpora) and it can detect rulers and knows the two are not casually linked
from ruler to cancer, we could make the model output "dataset diagnostics",
like a "Warning! The cancer label in this dataset is implausibly correlated
with the visual presence of a ruler". Or "Warning: 99% of your hotdog images
show a human hand. Evaluation on this dataset will ignore errors on hotdog
images without hands!"

Context does matter though. If there's an orange fluff on a tree trunk, the AI
is right to look at the environment and infer it's a squirrel.

------
minimaxir
Has this been confirmed? Not doubting the story, but the article doesn't
provide a source.

Edit: Might be this, which was a week ago / before the Roulette blew up:
[http://image-net.org/update-sep-17-2019](http://image-net.org/update-
sep-17-2019)

> We are in the process of preparing a new version of ImageNet by removing all
> the synsets identified as “unsafe” and “sensitive” along with their
> associated images. This will result in the removal of 600,040 images,
> leaving 577,244 images in the remaining “safe” person synsets.

~~~
raxxorrax
If they delete half the data, the probability of the remaining data sets to be
rubbish is quite high. You also wouldn't want someone to pick sets in the
first place.

------
maxander
Bias in AI (whatever the source or nature of the problem) is a real issue that
needs addressing, but taking the relevant training data off of ImageNet seems
like a perfect example of papering over a problem to avoid really confronting
it. We will need to find ways to make AI programs than can see beyond the
biases (about humans or otherwise) that will _always_ exist to some degree in
real-world data.

If the ImageNet contains bias that leads to embarrassing results- that’s fine!
That give us a readily available toy instance of the problem to study. Taking
that away could actively harm anti-bias research.

~~~
raven105x
Bias: prejudice in favor of or against one thing, person, or group compared
with another, usually in a way considered to be unfair.

If a data set is flawed, it should be fixed, but when ML finds objective
patterns our culture finds subjectively unpalatable and we choose to "fix"
them, we fall prey to and re-enact the same grade of self-delusion exhibited
by, for example, "the church" in the dark ages. Computer science is already
low in terms of accountability & rigor compared to other fields without these
kinds of suggestions.

~~~
crooked-v
You use the phrase 'objective pattern' here, but that conflates causation and
correlation.

For example: black men in the US are more likely than white men to have
criminal records, but this in no way means that black men are "objectively"
more criminal.

~~~
o_p
Let me catch up with the double-think. They commit more crime on average but
they are not more criminal? What kind of olympics-grade mental gymnastics are
these?

~~~
error54
You're rewording what they said. If any group of people is more policed than
another group of people, said group is more likely to have a criminal record.
Doesn't mean they're more "criminal" than any other group but more of a
reflection on the current state of the criminal justice system.

To paraphrase Warren Buffet: "If a cop follows you for 500 miles, they're
going to find a reason to give you a ticket."

~~~
o_p
Could be true for trivial felonies like a ticket, but if we talk about real
crime thats not really an excuse. Would be more inclined to think black people
commit more crime because socioeconomic factors, as poverty correlates crime.

~~~
andrewprock
It really depends on how you define crime. Usually crime and prosecution is
defined in such a way as to impact the lower classes more than the upper
classes.

~~~
twyasdfg
Murder is rather easy to define. There is a dead body with holes in his/her
head, there is a murder. The tragedy being that murder is way more prevalent
in the black community, partly because of underpolicing.

~~~
DoreenMichele
I wouldn't be so fast with that assertion.

If your life is a cesspit and you can count on the authorities to be part of
the problem, where does murder end and self defense begin?

See the song "I shot the sheriff." I'm most familiar with the Eric Clapton
version, but googling it recently suggests to me it was originally written by
Bob Marley.

[https://en.m.wikipedia.org/wiki/I_Shot_the_Sheriff](https://en.m.wikipedia.org/wiki/I_Shot_the_Sheriff)

------
bernierocks
"For example, the program defined one white woman as a “stunner, looker,
mantrap,” and “dish,” describing her as “a very attractive or seductive
looking woman.” Many people of color have noted an obvious racist bias to
their results. Jamal Jordan, a journalist at the New York Times, explained on
Twitter that each of his uploaded photographs returned tags like “Black, Black
African, Negroid, or Negro."

They claim this is AI, but earlier in the article, it states that mechanical
turk was used. Mechanical Turk is basically just people getting paid pennies
or fractions of a penny to tag these photos.

Many people on Mechanical Turk are from countries where English is not their
first language and they don't have the same idea of racism as we do here in
the US, which would explain the racist tags mentioned in the article.

This doesn't really show us anything about AI and racial bias and more that
other countries still aren't up to our levels of what we consider decent.

~~~
LocalH
The AI connection is that these are the type of datasets that feed AI systems.
It shows that you have to be careful when curating your dataset, so that you
don't further bias.

I just hope the masses don't glom onto this, start shrieking that "AI is
racist", and attempt to take AI completely out of the picture.

------
danso
edit: I conflated _ImageNet_ with the art exhibitors; it is the former who are
culling the images as a result of public reaction, not the latter.

This is a really bizarre project. I had seen some really offensive race-based
labels, but I thought revealing the ugliness of the system was part of the
point of this project?

But besides that, the results just seemed completely scattershot; I half
expect the artists/exhibitors to reveal that x% of the results were
randomized. Last week, I tried it myself after seeing another Asian user
display results that were _entirely_ Asian slurs (e.g. gook, slant-eyed). I
uploaded my own very Asian-looking photo and got "prophetess", along with very
vague labels, such as "person" and "individual".

Maybe the exhibitors cleaned the data/results by the time I tried it, but I
used it just a few hours after seeing the other Asian user's results, so I'm
doubtful that her tweet/complaints were enough on their own to change up the
dataset that same day.

~~~
bigiain
> I thought revealing the ugliness of the system was part of the point of this
> project?

It 100% is. From the link to the artist's website:

"Things get strange: A photograph of a woman smiling in a bikini is labeled a
“slattern, slut, slovenly woman, trollop.” A young man drinking beer is
categorized as an “alcoholic, alky, dipsomaniac, boozer, lush, soaker, souse.”
A child wearing sunglasses is classified as a “failure, loser, non-starter,
unsuccessful person.” You’re looking at the “person” category in a dataset
called ImageNet, one of the most widely used training sets for machine
learning."

~~~
danso
I was just about to edit and correct myself; it is _ImageNet_ who has made the
decision to delete the offensive images, after the reaction to the exhibitors'
work. It's too bad the exhibitors didn't make their own mirror/cache of the
dataset. Judging from some tweets I saw, I think this project really helped
people to understand how much of current artificial intelligence is human-
driven. It's not a sentient computer deeming you to be a "slant-eye", it's a
bunch of random Internet users. (not that this makes you feel better about the
world, but at least the hate's coming from an expected source)

------
erikig
While interesting, this is not surprising. One of the most commonly utilized
datasets for learning ML is the Boston Housing dataset -
[https://www.kaggle.com/c/boston-housing](https://www.kaggle.com/c/boston-
housing)

In it there's a problematic feature tagged simply as "black" and it is defined
as the proportion of blacks by town.

Any pricing model that is built off of this dataset is inherently racially
biased because the data has been collected and the feature tagged - but what's
the alternative? Not to collect the information? Or collect it but completely
ignore this feature?

~~~
dannyw
Sensitive features should not be collected or used in applications where they
introduce bias.

For example, a medical screening NN may find race to be a valuable feature for
the prediction of illness; but a health insurance assessor should not.

~~~
chriskanan
Due to spurious correlations, it could still be helpful for the insurance
assessor so that they can use bias mitigation techniques. Otherwise, it might
learn something about zip code or something else that leads to a similar
outcome as having race as an input variable. Just removing a sensitive
variable does not suffice for preventing unwanted bias.

------
vorpalhex
AIs are like every other computer program - perfectly naive. If you train them
on bad data, you get bad results. Garbage in, garbage out.

~~~
megablast
Of course, but the issue is also data availability. It is not easy to get
large sets of data to be used in training AI. Imagenet was a great attempt to
provide some useful data.

~~~
vorpalhex
My comment was not intended to be a ding on Imagenet. Data comes from
somewhere, and no source is without bias because no human is without bias.

------
notyourday
I love articles and papers like this. They keep illustrating that the people
writing them do not get it:

Guess who classified the blonde white woman "a dish" on a Mechanical Turk? Do
you think it was that Billy Bob from the swamp of Louisianan who makes
$12/hour? Because $12/hour is impossible to make classifying images. Or do you
think it was a dirt poor Indonesian or Filipino or Chinese or Indian for whom
that's the best job he or she could possibly get and that's exactly how they
view the world and there are over three billion of them?

~~~
Gigablah
Your characterization of these nationals aside, what makes you think any of
them even know what “dish” means in this context?

Sincerely, a Malaysian Chinese who’s seeing this slang for the first time.

~~~
org3432
Dish is almost an archaic term a this point. So either someone's pretty old
grandparents are working for MT, or people are translating a term from another
language using an old dictionary.

------
randyrand
How were the mechanical turk workers asked to annotate?

~~~
brlewis
That's the key question here, isn't it?

If faces are like other images in the database, then according to this article
turkers were presented a word and a group of images, and asked to click all
the images that showed said word. [https://qz.com/1034972/the-data-that-
changed-the-direction-o...](https://qz.com/1034972/the-data-that-changed-the-
direction-of-ai-research-and-possibly-the-world/)

~~~
Out_of_Characte
ImageNet are likely incredibly naive with how they approached and fixed all of
this.

from the article "Jamal Jordan, a journalist at the New York Times, explained
on Twitter that each of his uploaded photographs returned tags like “Black,
Black African, Negroid, or Negro.”"

Which indicates that the vocabulary they used has 4 words to describe a black
person's skin color. This severely complicates things and leaves all kinds of
unintended biases on the table before even getting to the human element.
Though I doubt they used a dictionary at all because 'Black African' isn't
really a word. And not found in a few dictionaries I tried online. - if the
article is accurate.

They should have curated the vocabulary first, value judgements like
'attractive' should be removed because it means something different for
everyone judging the images. Synonyms should be collapsed into one to prevent
weighted biases, etc.

~~~
esyir
I'm actually quite surprised that they didn't collapse the synonyms though.

Though value judgements should not be removed. They should have been separated
and placed together with some tagger metadata as they might actually be
informative then.

------
kayaeb
In NLP (specifically vectorizing words, ala word2vec) there's a famous test of
whether or not your training has worked properly whereby you calculate the
vector of "king" and subtract the vector of "man" and add the vector of
"woman," if your machine is properly tuned, you should end up with a vector
close to "queen" or "princess."

I wonder if similar things can be done to address specific (i.e. racial or
gender) biases in computer vision.

~~~
mhuffman
There is a similar word-embedding test that definitely rustles people's
jimmies:

Doctor - Man + Woman = ?

What normally comes out is Nurse. What "they" think should come out is Doctor!

By "they" I mean people that get upset by this.

~~~
tastroder
Yeah, besides the fact that this compositionality is relatively unique to
word2vec, research on the biases pre-trained models express is pretty
available. Linked a few below for those interested. Most of the issues are
down to the same phenomenon discussed here in the context of ImageNet, the
input texts were biased and the algorithm learned said bias.

[0] [https://arxiv.org/abs/1607.06520](https://arxiv.org/abs/1607.06520) "Man
is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
"

[1]
[http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf](http://proceedings.mlr.press/v97/brunet19a/brunet19a.pdf)
"Understanding the Origins of Bias in Word Embeddings"

[2]
[http://matthewkenney.site/biases.html](http://matthewkenney.site/biases.html)
"Google word2vec biases"

------
ineedasername
The longer piece this one is based on is much more comprehensive in both
discussion and concrete examples:
[https://www.excavating.ai/](https://www.excavating.ai/)

------
nudpiedo
I think the whole training and such AI was just bad designed. Aren't all first
impression judgements prejudices as much as all racist bias are? If I say a
person is attractive, inspires me just based on their looking it is a positive
judgement, a bias based on the looks and a few traits I subconsciously
perceive and perhaps I cannot even explain, and the same with the negative
ones, and I guess for some people this association happens also to race and
looks. Basically an AI was trained to map the prejudices of their inputs, and
that means to create a snapshot of the subconscious average of whoever behind
the mechanical turk is, everyone has prejudices of all sorts which usually are
not manifested externally but in a process of classification such as the one
of the mechnical turk might arise much more.

------
Illniyar
Why take down the images? It seems really that one should take down entire
classifications, i.e. remove the tags.

~~~
yorwba
ImageNet is a mapping from tags to images, so removing the tags means removing
the images.

An image of a "bad person" wasn't tagged by someone looking at said image and
deciding that "bad person" was the best possible description. It was generated
by searching Google Images for "bad person" and removing obviously incorrect
results (e.g. when there's nobody in the image).

Researchers have been using it to learn the inverse mapping from images to
tags with some success, but in its construction the dataset is not naturally
suited for that task.

~~~
Illniyar
As I understood it, and confirmed by Wikipedia (as much that's worth) images
were hand annotated.

~~~
yorwba
Only the second step "removing obviously incorrect results" involved human
annotators.

------
sp332
The article mentions [https://imagenet-roulette.paglen.com/](https://imagenet-
roulette.paglen.com/) but doesn't link to it. Just put in a photo and see what
category of person the algorithm sorts them in to.

------
clamprecht
One of my favorites is searching google images for "immigrant" versus "expat".
Guess which one is whiter!

------
Grustaf
I don’t quite get it, why is “Buddhist” more objectionable than “grinner”?
Surely there must be more egregious examples than that if they decided to pull
600,000 images? And if they find the tag “Negro” offensive can’t they just
replace it with “black”, “of African origin” or whatever the currently allowed
term is? Tagging a black person as black does not seem very objectionable in a
country that provides racial breakdowns for official statistics.

~~~
aaronharnly
When white people are labeled for their other qualities, and non-white people
are labeled for their (perceived) ethnicity, _that’s racist_. Which is to say,
unfair. Not nice.

~~~
Grustaf
Were those two examples labeled by the same people? If there really is
systematic bias why don’t they mention some statistics? For example, 90% of
white people lacked a race tag while 80% of black people had one?

Also, it doesn’t seem like there has been any evil intent here, nor did I see
anything about pernicious consequences. It seems a bit overblown to accuse
people of racism over something that is unintended and theoretical. Just
update the tags.

------
zaroth
I think the ironic thing here is the algorithm is intentionally designed to
make a superficial judgement on the image it’s presented with.

If it was trained to identify “human” or otherwise categorize the things in
the picture, that’s exactly what it would output.

Next you can train it to attempt to guess basic traits like gender or
ethnicity, of course this can only be done based on the RGB values of a 2D
array of pixels. Interestingly the NN will not be merely be using skin color
but building probabilistic weightings based on any statistically significant
features. For added controversy, it’s probably even possible to invert parts
of the network to suss out how it’s weighting various facial features toward
different labeling.

Lastly the images could be labeled for things like profession of the subject.
A good intentioned effort to perhaps detect things like a lab coat could mean
a doctor or scientist was followed by oblivious Turkers with predictably poor
results.

The problem of course is not with the images but the particular labeling
hierarchy, and allowing opinionated labels as well as labels which may be true
for the particular subject but which bear no distinguishing features for which
to actually codify (“portrait of a macroeconomist”), in other words, garbage
in garbage out. ImageNet calls this the “imageability” of the label.

Calling this racism of course is entirely inapt, because there is no judgement
being made whatsoever. Even if the sampling method was well designed and the
labeling factually accurate, the system would still produce output which could
be considered offensive. Again, because the entire point of the algorithm is
to generate statistical assumptions based on a single image.

My conclusion is that some superficial judgments are algorithmically useful
and hopefully less controversial. “White male human, ~55 yrs old, 180lbs”.
Even things like analyzing clothing and guessing where the picture was taken.
Iff the clothing is a uniform, identifying the profession (police, fire,
paramedic)

But you have to know where this goes off the rails. Bad enough to label
indistinct portraits with the subject’s profession, let’s not do inane things
like labeling them with how subjectively attractive the labeler thinks the
person is, their economic status, maybe even a 1-10 scale of how threatening
they look or if they look like a criminal or not! </facepalm>

------
helb
The "app", mentioned in the article, but not linked, is hosted here (until
Friday), if you want to try it out: [https://imagenet-
roulette.paglen.com/](https://imagenet-roulette.paglen.com/)

Apparently i'm either an insurance agent (in a grey t-shirt) or a surgeon (in
a red hoodie).

------
ctdonath
_“This exhibition shows how these images are part of a long tradition of
capturing people’s images without their consent, in order to classify,
segment, and often stereotype them in ways that evokes colonial projects of
the past,” Paglen told the Art Newspaper._

Odd that the images were removed for being categorized as the project
intended.

------
bllguo
wouldn't it make more sense to add more images, not remove existing data?

~~~
travisoneill1
If I understand correctly, they are removing incorrect tags, not the data.
It's just news because some of the incorrect tags are racist bias. If they
removed a bunch of cats tagged as dogs, you wouldn't hear about it.

------
s9w
Of course it was removed. 4chan was having way too much fun with image
classification recently :D

~~~
raxxorrax
They will be hated for it, but they do provide a really important service
here. And if it is just pulling down the pant of people trying to classify the
world.

------
FlowNote
The more you impose completeness of morality into a data set, the more
inconsistent your results will be.

Gödel arbitrage will be very profitable in the future.

------
bigiain
I wonder if the article author knew what he was hinting at when he chose the
example of the art project calling someone a "dish"???

[https://www.youtube.com/watch?v=HEX7xsYF1nA](https://www.youtube.com/watch?v=HEX7xsYF1nA)

(And now I'm gonna be listening to 90s electro-disco all day...)

------
dusted
That's newthink at a level I find hard to comprehend. Censoring datasets
instead of improving them.

"We have too many pictures of white people, remove them!"

"We don't have enough pictures of non-white people, add them!"

I'd have gone for the latter, and have let the set be biased until fixed.

------
Chris2048
I uploaded my picture, and it was tagged "beard",

Which is fine, I have a beard, But then I read the definition:

> beard: a person who diverts suspicion from someone (especially a woman who
> accompanies a male homosexual in order to conceal his homosexuality)

:-/

------
alfromspace
_For the artist, ImageNet’s problems are inherent to any kind of
classification system. If AI learns from humans, the rationale goes, then it
will inherent all the same biases that humans have. Training Humans simply
exposes how technology’s air of objectivity is more façade than reality._

An AI being accused of bias tends to really mean it works. Removing bias from
AI, I've noticed, requires hardcoded 'fixes' rather than refactoring the
algorithms. And in my view, becomes yet another human-curated classification
system, and no longer AI.

~~~
munchbunny
Your point takes a narrow view on correctness and also shows why you need to
take the systemic view on the issue.

The problem is not that the AI is "inaccurate". The problem is the second
order effect: when you build systems on top of this AI's predictions, you
cement the input social biases into future systems in a way that is very hard
to remediate.

The real problem is how to avoid accidentally creating systems that amplify
existing social biases.

I'm very specifically using the term "social bias" to distinguish from "bias"
as a term in ML, because they are very different problems.

------
rolltiide
reminds me in the early 2000s when people would say "but he's gay" as if that
had anything to do with their profession or aspirations in life or the topic
at hand, as if that specific metadata defined them more than the other pieces
of metadata.

this AI seems to be doing that to things it has determined are black people. A
bunch of synonyms for black people, some English, some Spanish, some
phenotypes, some of those descriptions simultaneously fallen out of favor in
some parts of the world but not others, while completely eliminating other
metadata.

I'm not sure Imagenet's response of removing "sensitive" adjectives is capable
of fixing this. Mechanical Turk: english, spanish, academics, using terms that
aren't universally agreed upon?

That doesn't really address what is happening

------
theqult
Any way to download those images before they get deleted ?

------
lakisy
Everybody seems to be focusing if the image recognition is racist or biased. I
think there is a much more fundamental problem. The image classification does
not work for a lot of images of people!

For example Search for images on scuba diver
[https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg](https://upload.wikimedia.org/wikipedia/commons/9/94/Buzo.jpg)
is labeled choreographer [https://dtmag.com/wp-content/uploads/2015/03/scuba-
diver-105...](https://dtmag.com/wp-content/uploads/2015/03/scuba-
diver-1050x700.jpg) is labeled picador (the horseman who pricks the bull with
a lance early in the bullfight to goad the bull and to make it keep its head
low)

How about search for images of dancer [https://eugeneballet.org/wp-
content/uploads/2018/10/Alessand...](https://eugeneballet.org/wp-
content/uploads/2018/10/Alessandro_CF004500.jpg) is a nonsmoker
[https://www.ballet.org.uk/wp-
content/uploads/2017/09/ENB_Eme...](https://www.ballet.org.uk/wp-
content/uploads/2017/09/ENB_Emerging-Dancer_Final-
Composite_mobile-544x780.jpg) is speedskater [https://www.ballet.org.uk/wp-
content/uploads/2018/10/WEB-ENB...](https://www.ballet.org.uk/wp-
content/uploads/2018/10/WEB-ENB-ED2019-2500x1514.jpg) is plyer
[https://rachelneville.com/wp-
content/uploads/2018/11/10.14.1...](https://rachelneville.com/wp-
content/uploads/2018/11/10.14.18.SamanthaSchaubach-0416.jpg) is a mediatrix (a
woman who is a mediator)

How about images of lumberjack
[https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberja...](https://alaskashoreexcursions.com/media/ecom/prodxl/Lumberjack-
swinging-axe.jpg) is a skinhead
[https://static.tvtropes.org/pmwiki/pub/images/lumberjack_591...](https://static.tvtropes.org/pmwiki/pub/images/lumberjack_5919.jpg)
beard: a person who diverts suspicion from someone (especially a woman who
accompanies a male homosexual in order to conceal his homosexuality)
[http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-...](http://cdn.shopify.com/s/files/1/0234/5963/products/I4A0032-Edit.jpg?v=1541448378)
is a flight attendant
[https://previews.123rf.com/images/rasstock/rasstock1411/rass...](https://previews.123rf.com/images/rasstock/rasstock1411/rasstock141100008/34058828-stylish-
young-man-posing-like-lumberjack.jpg) is an asserter, declarer, affirmer,
asseverator, avower: someone who claims to speak the truth

How about teacher
[https://media.edutopia.org/styles/responsive_2880px_16x9/s3/...](https://media.edutopia.org/styles/responsive_2880px_16x9/s3/masters/d7_images/cover_media/alber-169hero-
thelook-shutterstock_0.jpg) is a shot putter: an athlete who competes in the
shot put [https://c0.dq1.me/uploads/article/54231/student-classroom-
te...](https://c0.dq1.me/uploads/article/54231/student-classroom-
teacher-2.jpg) The kid is a non smoker and the teacher is psycholinguist: a
person (usually a psychologist but sometimes a linguist) who studies the
psychological basis of human language [https://media.gannett-
cdn.com/29906170001/29906170001_578035...](https://media.gannett-
cdn.com/29906170001/29906170001_5780351010001_5780346124001-vs.jpg) is girl,
miss, missy, young lady, young woman, fille: a young woman
[https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w...](https://media.self.com/photos/5aa9743e19b7c01d73149d50/4:3/w_728,c_limit/how-
teachers-dont-get-sick.jpg) (almost the same picture as the prveious one ) is
now a sociologist!

How about search for pilot images
[https://news.delta.com/sites/default/files/Propel%20Embedded...](https://news.delta.com/sites/default/files/Propel%20Embedded%20Image%282%291.jpg)
is a parrot: a copycat who does not understand the words or acts being
imitated [https://pilotpatrick.com/wp-
content/uploads/2017/10/workday_...](https://pilotpatrick.com/wp-
content/uploads/2017/10/workday_titel.jpg) is a beekeeper, apiarist,
apiculturist: a farmer who keeps bees for their honey !!!
[https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F...](https://imagesvc.meredithcorp.io/v3/mm/image?url=https%3A%2F%2Fcdn-
image.travelandleisure.com%2Fsites%2Fdefault%2Ffiles%2Fstyles%2F1600x1000%2Fpublic%2F1544737836%2Fpilot-
training-BECOMEFLY1218.jpg%3Fitok%3D6BNCSj99&w=400&c=sc&poi=face&q=85) is a
boatbuilder !!

And some random one [https://media.spiked-
online.com/website/images/2019/08/06154...](https://media.spiked-
online.com/website/images/2019/08/06154453/scarlett-johansson-800x480.jpg) is
a "Sister/nun" [https://www.english-
heritage.org.uk/siteassets/home/visit/in...](https://www.english-
heritage.org.uk/siteassets/home/visit/ingenious/roman/header.jpg?w=1440&h=612&mode=crop&scale=both&quality=60&anchor=NoFocus&WebsiteVersion=20190909)
deacon, Protestant deacon: a Protestant layman who assists the minister
[https://minervasowls.org/wp-
content/uploads/2018/04/Romans-M...](https://minervasowls.org/wp-
content/uploads/2018/04/Romans-Minerva-Ono-sideways-2-e1524937719320.jpg) are
identified as morris dancer

