
Semantics derived automatically from language corpora contain human-like biases - akarve
http://scim.ag/2p0WGK9
======
Houshalter
The paper and title implies it's absorbing these stereotypes from humans. I
think there is another explanation. Remember these models are trained on a
dataset of news or Wikipedia articles. And it's 'goal' is to find vectors that
predict what contexts words are more likely to appear in.

So if 34% of doctors are female, then you would expect 34% of doctors in news
or Wikipedia articles to be female. Even if the articles are completely
unbiased and the writers have no stereotypes whatsoever. And so the word
vector would naturally label "doctor" something like "66% likely to occur in a
male context".

And in fact this paper confirms that. Figure 1 shows that the word vectors are
highly predictive of the actual gender distribution of various occupations.
Probably much more accurate than most people would be. So it's not mindlessly
absorbing human stereotypes. It's learning reality's stereotypes.

This result is completely expected and desirable. What makes word vectors so
powerful is how they can learn complicated correlations between words and
their contexts. The famous example is how it learns that "Queen" is the female
equivalent of "King". Which is a gender stereotype as well. If it wasn't able
to learn that doctors were a bit more likely to be male, that would be more
surprising.

~~~
andreasvc
I think you are making a distinction without a difference. If the word vectors
pick up biases from wikipedia text, than for all practical purposes, they are
(indirectly) absorbing stereotypes from humans. This is an expected result,
but not necessarily desirable in the end.

~~~
scribu
The GP is saying that the bias isn't an attribute of the Wikipedia text, but
of reality.

If the reality is that only 34% of doctors are female, why is it not desirable
for the machine to learn that?

~~~
jrkatz
It depends on what you want the machine to do. If you are making a gambling
machine that looks at pairs of names and makes bets as to which name belongs
to a doctor, you want it to learn that.

If the machine looks at names and decides who to award a "become a doctor"
scholarship to, based on who it thinks is most likely to succeed, you don't
want it to learn that.

~~~
scribu
I agree that if your goal is to build a machine that decides who gets to
become a doctor, you need to do more than just let it loose on a bunch of
text.

But I don't think preventing it from learning the current state of the world
is a good strategy. Adding a separate "morality system" seems like a more
robust solution.

~~~
rspeer
What do you think of Bolukbasi's approach that's mentioned in the article? In
short, you let a system learn the "current state of the world" (as reflected
by your corpus), then put it through an algebraic transformation that
subtracts known biases.

Do you consider that algebraic transformation enough of a "morality system"?

I hope you're not saying we shouldn't work on this problem until we have AGI
that has an actual representation of "morality", because that would be a
setback of decades at least.

~~~
scribu
> put it through an algebraic transformation that subtracts known biases

> Do you consider that algebraic transformation enough of a "morality system"?

I would consider it a sort of morality, yes. But keep in mind that the list of
"known biases" would itself be biased toward a particular goal, be it
political correctness or something else.

~~~
rspeer
Yes, every step of machine learning has potential bias, we know that, that's
what this whole discussion is about. Nobody would responsibly claim that they
have solved bias. But they should be able to do something about it without
their progress being denied by facile moral relativism.

If we can't agree that one can improve a system that automatically thinks
"terrorist" when it sees the word "Arab" by making it not do that, we don't
have much to talk about.

------
randomwalker
Coauthor here. Some of the press articles about our work didn't have a lot of
nuance (unsurprisingly), but in the paper we're careful about what we say,
what we don't say, and what the implications are. Happy to engage in informed
discussion :)

~~~
yummyfajitas
Do you have any evidence that this effect results in machines making
systematically wrong inferences?

Near as I can tell, your paper shows that these "biases" result in
significantly more accurate predictions. For example, Fig 1 shows that a
machine trained on human language can accurately predict the % female of many
professions. Fig 2 shows the machine can accurately predict the gender of
humans.

Normally I'd expect a "bias" to result in wrong predictions - but in this case
(due to an unusual redefinition of "bias") the exact opposite seems to occur.

(Drawing on your analogy with stereotypes, it's probably also worth linking to
a pointer on stereotype accuracy: [http://emilkirkegaard.dk/en/wp-
content/uploads/Jussim-et-al-...](http://emilkirkegaard.dk/en/wp-
content/uploads/Jussim-et-al-unbearable-accuracy-of-stereotypes.pdf)
[http://spsp.org/blog/stereotype-accuracy-
response](http://spsp.org/blog/stereotype-accuracy-response) )

~~~
andreasvc
I think your questions would be answered by reading the article. Particularly:

"In AI and machine learning, bias refers generally to prior information, a
necessary prerequisite for intelligent action (4). Yet bias can be problematic
where such information is derived from aspects of human culture known to lead
to harmful behavior. Here, we will call such biases “stereotyped” and actions
taken on their basis “prejudiced.”"

This definition is not unusual. This is about inferences that are wrong in the
sense of prejudiced, not necessarily inaccurate.

~~~
yummyfajitas
The usual definition of bias in ML papers is E[theta_estimator - theta]. That
is explicitly a systematically wrong prediction.

In any case, the paper suggests that this "bias" or "prejudice" is better
described as "truths I don't like". I'm asking if the author knows of any
cases where they are actually not truthful. The paper does not suggest any,
but maybe there are some?

~~~
andreasvc
Again, per the article "bias refers generally to prior information, a
necessary prerequisite for intelligent action (4)." This includes a citation
to a well-known ML text. This seems broader than the statistical definition
you cite.

Think for example of an inductive bias. If I see a couple of white swans, I
may conclude that all swans are white, and we all know this is wrong.
Similarly, I may conclude the sun rises everyday, and for all practical
purposes this is correct. This kind of bias is neither wrong nor right, but,
in the words of the article "a necessary prerequisite for intelligent action",
because no induction/generalization would be possible without it.

There are undoubtedly examples where the prejudiced kind of biases lead to
both truthful and untruthful predictions, but that seems beside the point,
which is to design a system with the biases you want, and without the ones you
don't.

------
ramblenode
This was a neat study, but the authors really do throw around a number of
speculative and unsupported claims in the discussion about linguistic
relativity/determinism.

> Our findings are also sure to contribute to the debate concerning the Sapir-
> Whorf hypothesis (17), because our work suggests that behavior can be driven
> by cultural history embedded in a term’s historic use. Such histories can
> evidently vary between languages.

No, not really. Linguistic relativity makes a claim about the direction of
causation--from language to thought. This study does nothing to test the
direction of causation, which is usually considered possible only with
controlled experiments. This chicken and egg debate has been going on for the
better part of the last century--it is not an easy problem.

> Our results also suggest a null hypothesis for explaining origins of
> prejudicial behavior in humans, namely, the implicit transmission of
> ingroup/outgroup identity information through language. That is, before
> providing an explicit or institutional explanation for why individuals make
> prejudiced decisions, one must show that it was not a simple outcome of
> unthinking reproduction of statistical regularities absorbed with language.

Once again, this was not an experiment capable of producing causal evidence.
The authors have shown that human biases can be replicated by statistical
learning of language corpora--admirable work, but nothing new here for Sapir-
Whorf.

------
mnarayan01
> That is, before providing an explicit or institutional explanation for why
> individuals make prejudiced decisions, one must show that it was not a
> simple outcome of unthinking reproduction of statistical regularities
> absorbed with language.

This is an extraordinarily bold claim. I'd be quite interested in how peoples'
responses to the article changed if this was the lead.

~~~
Osmium
> This is an extraordinarily bold claim.

Very bold. The full quote is worth reproducing:

> Our results also suggest a null hypothesis for explaining origins of
> prejudicial behavior in humans, namely, the implicit transmission of
> ingroup/outgroup identity information through language. That is, before
> providing an explicit or institutional explanation for why individuals make
> prejudiced decisions, one must show that it was not a simple outcome of
> unthinking reproduction of statistical regularities absorbed with language.

I'm reminded of Parable of the Polygons[0] which illustrates Shelling's model
of segregation, showing how quite small initial biases can be amplified and
result in very large segregation.

It would be seem very sad if tribalism in all its forms is simply an emergent
behaviour, a result of a random fluctuation (e.g. one or two racist
individuals) causing a chain reaction throughout society, where biases become
gradually amplified even if most individuals are, themselves, generally well-
meaning. How do we escape from that?

[0] [http://ncase.me/polygons/](http://ncase.me/polygons/)

~~~
TeMPOraL
> _It would be seem very sad if tribalism in all its forms is simply an
> emergent behaviour, a result of a random fluctuation (e.g. one or two racist
> individuals) causing a chain reaction throughout society, where biases
> become gradually amplified even if most individuals are, themselves,
> generally well-meaning. How do we escape from that?_

Sadly, that seems to be the case.

I wouldn't even turn racism into a "special case" here. Humans are capable of
dividing themselves into ingroups and outgroups over everything, no matter how
trivial. I suspect the segregation process will occur with all in/outgroup
divisions. Separations along the race and gender lines are particularly
prevalent because those are the most obvious, noticeable differentiators
between people.

------
Avshalom
[https://www.youtube.com/watch?v=cfRDUsvu5fE](https://www.youtube.com/watch?v=cfRDUsvu5fE)

------
throwaway91111
Be careful with the word truth. You don't know what it means.

~~~
dang
We detached this subthread from
[https://news.ycombinator.com/item?id=14116195](https://news.ycombinator.com/item?id=14116195)
and marked it off-topic.

