
Language contains human biases, and so will machines trained on language corpora - r721
https://freedom-to-tinker.com/blog/randomwalker/language-necessarily-contains-human-biases-and-so-will-machines-trained-on-language-corpora/
======
randomwalker
Coauthor here. The blog post is written in relatively non-technical language
for a general audience, but our paper has tons of technical details that HN
readers might enjoy. Give it a read!

[http://randomwalker.info/publications/language-
bias.pdf](http://randomwalker.info/publications/language-bias.pdf)

~~~
cs702
Doing this was a great idea. Great paper: easy-to-follow and to-the-point.

The results are not too surprising, as the models for learning word embeddings
like GloVe, word2vec, etc. learn to map to vectors existing relationships
between words in training corpora. If a corpus is biased, the embeddings
learned from it will necessarily be biased too.

However, the implications of this finding are wide-ranging. For starters, any
machine learning system that relies on word embeddings learned from biased
corpora to make predictions (or to make decisions!) will necessarily be biased
in favor of certain groups of people and against others.

Moreover, it's not obvious to me how one would go about obtaining "unbiased"
corpora without somehow relying on subjective societal values that are
different everywhere and continually evolving. You have raised an important,
non-trivial problem.

~~~
yummyfajitas
_For starters, any machine learning system that relies on word embeddings
learned from biased corpora to make predictions (or to make decisions!) will
necessarily be biased in favor of certain groups of people and against
others._

This is not true.

Here's an oversimplified example. Suppose your machine learning system wants
to predict something, e.g. loan repayment probabilities. One input might be a
written evaluation by a loan officer.

When trained on a corpora of group X, the predicted probability might be:

    
    
        pred = a*written_evaluation + other_factors
    

(Using linear regression to make example simple.)

However, now lets suppose the written evaluation is biased to the tune of 25%
against group Y. I.e., group Y has written scores that are 25% less than group
X.

Then a new predictor which includes pairwise terms, trained on a corpora of
group X and Y, will work out to be:

    
    
        pred = a*written_evaluation + 0.33*written_evaluation*isY + other_factors
    

This predictor would be unbiased. In general, if you have a biased input and
the biasing factor is also present in your input, your model should correct
the bias. (Obvious caveats: your model needs to be sufficiently expressive,
etc.)

Interestingly, everyone's favorite bogeyman, namely redundant encoding (
[http://deliprao.com/archives/129](http://deliprao.com/archives/129) ) will
actually help fix this problem *even if you don't include the biasing factors
in the model.

~~~
cs702
_...now lets suppose the written evaluation is biased to the tune of 25%
against group Y..._

How do you find out that the written evaluation is biased "to the tune of 25%
against group Y?"

THAT is the problem. It's not obvious to me how you would go about determining
written evaluations are biased (and to what extent!) against group Y without
somehow relying on subjective societal values that are different everywhere
and continually evolving.

~~~
yummyfajitas
Finding out is the easy part. I don't mean to trivialize it, because doing
stats right is actually a very technical matter, but this is just ordinary
statistics.

You build a sufficiently expressive statistical model and include the
potentially biasing factors as features in the model. Then the model will
correct the bias _all by itself_ because correcting for bias maximizes
accuracy.

In the example above, you find the bias by doing linear regression and
including (written_evaluation x isY) as a term. Least squares will handle the
rest. If you using something fancier than least squares (e.g. deep neural
networks, SVMs with interesting kernels), you probably don't even need to
explicitly include potentially debiasing terms - the model will do it for you.

I give toy examples (designed to illustrate the point and also be easy to
understand) here:
[https://www.chrisstucchio.com/blog/2016/alien_intelligences_...](https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html)

This paper does the same thing - it discovers that standard predictors of
college performance (grades, GPA) are biased in favor of blacks and men,
against Asians and women, and the model itself fixes these biases:
[http://ftp.iza.org/dp8733.pdf](http://ftp.iza.org/dp8733.pdf)

Statistics turns fixing racism into a math problem.

If the topic were anything less emotionally charged, you wouldn't even think
twice about it. If I suggested including `isMobile`, `isDesktop` and
`isTablet` as features in an ad-targeting algorithm to deal with the fact that
users on mobile and desktop browse differently, you'd yawn.

~~~
cs702
_...include the potentially biasing factors as features..._

 _Who_ decides what the "potentially biasing factors" are? How is that decided
_without somehow relying on subjective societal values_?

Factors that no one thought were biased in the past are considered biased
today; factors that no one thinks are biased today may be considered biased in
the future; and factors that you and I consider biased today may not be
considered biased by people in other parts of the world. I don't know how one
would go about finding those "potentially biasing factors" without relying on
subjective societal values that are different everywhere and always evolving.

~~~
yummyfajitas
A potentially biasing factor is a factor that you think would be predictive if
you included it in the model. If it's _actually_ predictive, you win, your
model becomes more accurate and you make more money.

Go read the wikipedia article on the topic:
[https://en.wikipedia.org/wiki/Omitted-
variable_bias](https://en.wikipedia.org/wiki/Omitted-variable_bias)

It's true that as we learn more things we discover new predictive factors.
That doesn't make them subjective. A lung cancer model that excludes smoking
is not subjective, it's just wrong. And the way to fix the model is to add
smoking as a feature and re-run your regression.

Again, would you make the same argument you just made if I said I had an
accurate ad-targeting model?

~~~
cs702
OK, I see where the disconnect is. I think the best way to address it is with
an example.

Many people today would object _a priori_ to businesses using race as a factor
to predict loan default risk, _regardless of whether doing that makes the
predictions more accurate or not._ In many cases, using race as a factor WILL
get you in trouble with the law (e.g., redlining is illegal in the US).

Please tell me, how would you predict what factors society will find
objectionable in the future (like race today)?

~~~
yummyfajitas
My claim is very specific. If you tell an algorithm to predict loan default
probabilities, and you give it inputs (race, other_factor), the algorithm will
usually correct for the bias in other_factor.

I claimed a paperclip maximizer will maximize paperclips, I didn't claim a
paperclip maximizer will actually determine that the descendants of it's
creators really wanted it to really maximize sticky tape.

Now, if you want an algorithm not to use race as a factor, that's also a math
problem. Just don't use race as an input and you've solved it. But if you
refuse to use race and race is important, then you can't get an optimal
outcome. The world simply won't allow you to have everything you want.

A fundamental flaw in modern left wing thought is that it rejects analytical
philosophy. Analytical philosophy requires us to think about our tradeoffs
carefully - e.g., how many unqualified employees is racial diversity worth?
How many bad loans should we make in order to have racial equity?

These are uncomfortable questions - google how angry the phrase "lowering the
bar" makes left wing types. If you have an answer to these questions you can
simply encode it into the objective function of your ML system and get what
you want.

Modern left wing thought refuses to answer these questions and simply takes a
religious belief that multiple different objective functions are
simultaneously maximizable. But then machine learning systems come along,
maximize one objective, and the others aren't maximized. In much the same way,
faith healing doesn't work.

The solution here is to actually answer the uncomfortable questions and come
up with a coherent ideology, not to double down on faith and declare reality
to be "biased".

~~~
cs702
My claim was specific too: if a corpus is biased -- as defined by evolving
societal values -- then the word embeddings learned from that corpus will
necessarily be biased too -- according to those same societal values,
regardless of whether you think those values are rational and coherent.

------
protomyth
Did you research focus on English or does the nearly the same biases hold for
all modern languages? I would love to see a competently trained person try
this with the Asian and Native American languages.

I dearly hope we will let the bias "humanity is good, don't kill us" stay in.

~~~
Terribledactyl
My German professor mentioned a few times in class about how automated
translators had picked up sexist biases because of older training materials.

~~~
wolfgke
Which is to me rather a sign of how society changes meaning of the language to
indoctrinate a kind of social propaganda. Greetings from George Orwell's 1984.

~~~
throwanem
In fairness, there's a distinction to be made here between directed and
undirected change.

~~~
wolfgke
When SJW warrior groups make campaigns that from now on some list of words has
to be considered as racist, sexist, ...ist etc. I think one can talk of
directed change.

~~~
akhilcacharya
..what if those words are?

~~~
wolfgke
At least people before did not consider them to be. This shows that proving
that say are is not so easy.

~~~
hamax
Or people before didn't care if they are?

------
yummyfajitas
There was a great comment here discussing the Pathetic fallacy, and how we
humans apply it to machine learning systems. (It's now deleted and I don't
know why.) Specifically, we treat machine learning systems as anthropomorphic,
and assume they will reproduce our biases.

[https://en.wikipedia.org/wiki/Pathetic_fallacy](https://en.wikipedia.org/wiki/Pathetic_fallacy)

In reality, ML systems will generally correct human _biases_. And by _bias_ ,
I really do mean bias in the statistical sense - systematically getting things
wrong in a particular direction.

Now this article does a great job of explaining how ML systems might
understand the _meaning_ of words, and that meaning may contain bias. However,
such a system is merely an input into a separate system which actually makes
decisions based on those inputs. Extracting meaning from text makes no
decisions of it's own. If that later system wants to make accurate decisions,
then the best way to do that is to correct for the aforementioned bias,
assuming that bias is really _bias_ as opposed to just a correct but
undesirable belief about the world [1].

I wrote a blog post a while back that goes into this idea with a bit more
math, and which demonstrates some real world "learning" algorithms (mostly
linear regression) actually correcting biases:
[https://www.chrisstucchio.com/blog/2016/alien_intelligences_...](https://www.chrisstucchio.com/blog/2016/alien_intelligences_and_discriminatory_algorithms.html)

~~~
wyager
One problem you will run into here is that political use of the word "bias"
isn't any sort of statistical claim; it's often just used to mean that
something violates vague social expectations about what information is
"acceptable" to use when making decisions. ML algorithms don't care; they will
make the best possible decision based on the information they have, even if
their though process (so to speak) is "biased" in the political sense that it
may use gender, race, nationality, etc. to help make optimal decisions.

~~~
yummyfajitas
Yes, unfortunately the term is overloaded. "Bias" can also mean "making
correct decisions that I wish were incorrect". That's why I explicitly defined
"bias".

In the past era, e.g. 1980-2010, it was possible to use vague emotive language
to support all kinds of disparate things. As a concrete example that I touch
on in my post, and since racism is the loaded undercurrent of this example, we
like to pretend that eliminating racial or sexual _bias_ (in the sense of
making wrong decisions) will get us proportional representation.

Algorithms are bringing us to an age where analytic philosophy is becoming
really important. You can tell an algorithm to give you proportional
representation, or you can tell it to be racially/sexually unbiased. But the
algorithm will reflect reality and reality may not agree with your
assumptions; you can't assume that asking for one will give you the other. So
now we get into trolley problems: how much meritocracy/equal opportunity will
you sacrifice to get proportional representation?

Unlike before, this is now a choice you need to explicitly and openly state
and acknowledge.

------
skybrian
Here's an example of how to remove bias from a dataset:

[https://socialmediacollective.org/2016/08/06/amplifying-
and-...](https://socialmediacollective.org/2016/08/06/amplifying-and-
neutralizing-gender-bias-using-machine-learning-algorithms/)

~~~
randomwalker
OP here. We address this argument in detail in our paper, and we're deeply
skeptical of it. See the sections titled "Challenges in addressing bias" and
"Awareness is better than blindness".

Here's the short version:

 _We view the approach of "debiasing" word embeddings (Bolukbasi et al., 2016)
with skepticism. If we view AI as perception followed by action, debiasing
alters the AI’s perception (and model) of the world, rather than how it acts
on that perception. This gives the AI an incomplete understanding of the
world. We see debiasing as "fairness through blindness". It has its place, but
also important limits: prejudice can creep back in through proxies (although
we should note that Bolukbasi et al. (2016) do consider "indirect bias" in
their paper). Efforts to fight prejudice at the level of the initial
representation will necessarily hurt meaning and accuracy, and will themselves
be hard to adapt as societal understanding of fairness evolves_

Direct link to our paper: [http://randomwalker.info/publications/language-
bias.pdf](http://randomwalker.info/publications/language-bias.pdf)

~~~
igravious
Interesting paper, thanks!

See how you sway the argument in your favour using words with negative
connotations like "fairness through _blindness_ " and " _hurt_ meaning and
accuracy". Nobody would want to deliberately blind or hurt something, would
they? How about _rebalance_ or _recalibrate_ or _re-correct_.

A concrete analogy:

1) I have a meter measuring stick but I discover that it was made wrong, it is
actually 2mm shorter than advertised. Every time I make a measurement with it
I have to add 2mm to the measurement. Would it not be better to use a more
accurate stick and not have to continually compensate?

~~~
im4w1l
Stereotypes have great predictive power. The reason we sometimes avoid them is
they can lead to outcomes that are seen as undesirable.

------
jonbarker
This is related to the alphago series. Human language describing expert
intuition created norms about the game that became "standard expert practice"
but not truly efficient play.

------
jcoffland
This whole concept makes the silly assumption that today's machine learning
results in machine understanding. Without understanding human biases, as they
are embedded in language, are irrelevant.

~~~
jhanschoo
That's a bold, ill-defined, and unsubstantiated claim. Leaving machine
understanding undefined, I can think of an example in which these biases are
relevant. For example, if you're offering a search engine solution
incorporating clustering, you'd be concerned that your search retrievals
aren't associating racial minorities with pejorative prejudices.

~~~
wolfgke
> For example, if you're offering a search engine solution incorporating
> clustering, you'd be concerned that your search retrievals aren't
> associating racial minorities with pejorative prejudices.

If in the data racial minorities are associated with pejorative prejudices,
this is plain distorting of the truth. If you are concerned about this
associations, don't shoot the messenger, but the people writing the original
texts.

~~~
jhanschoo
> this is plain distorting of the truth.

In the context of business and many other interests, it's expedient to present
a more polite face. You're providing a service, not a mirror. In the context
of a search engine service, say, a specialized, domain-specific search for a
certain profession, such biases often distract and detract from the quality of
the service you provide to your customers.

------
mc32
There are many instances where bias is not a desired result. We rather not
preassociate negative denoting words with a generally good set of people. On
the other hand, there ate situations where we want to have bias. A bias
against unjust discrimination, a bias against psychopaths, a bias against
repeat offenders, etc. So to me, there are places where we want to retain bias
and places we want to rid bias and that will be up to society to decide, as
they decide everyday laws and conduct, etc.

------
troels
"Instead, we suggest that mitigating prejudice should be a separate component
of an AI system."

Much like I imagine we humans deals with the concept.

------
nix0n
Similarly, algorithms trained to mimic human decisions will mimic human
biases. See
[https://news.ycombinator.com/item?id=11753805](https://news.ycombinator.com/item?id=11753805)
(probably others too)

~~~
yummyfajitas
That article is fundamentally dishonest. The author's own statistical analysis
(see their R-script and my comments on the article) cannot reject the null
hypothesis that the algorithm is unbiased.

I wrote a more detailed critique here:
[https://www.chrisstucchio.com/blog/2016/propublica_is_lying....](https://www.chrisstucchio.com/blog/2016/propublica_is_lying.html)

------
shanacarp
Can you measure linguistic shifts with your model?

