
How a Machine Learns Prejudice - matthberg
https://www.scientificamerican.com/article/how-a-machine-learns-prejudice/
======
KODeKarnage
> In June 2015, for example, Google’s photo categorization system identified
> two African Americans as “gorillas.”

> Law enforcement officials have already been criticized, for example, for
> using computer algorithms that allegedly tag black defendants as more likely
> to commit a future crime, even though the program was not designed to
> explicitly consider race.

Consider these two examples from the article. One is when the machine failed
in a particular undesirable fashion. The other is when the machine "worked^"
but in an undesirable fashion. (^That is, overall it might have worked well,
but the bias in errors wasn't sufficiently explored.)

Modelling "undesirable fashion", which is a dynamic, subjective social
construct, is far more difficult than either of the tasks originally set in
the examples.

In the first example, how many images of people were confused with non-humans?
I can't believe this was the only failed result. The only reason this
particular example was problematic was that these specific people were
specifically identified as gorillas.

No problem if they were identified as a car, or a chess piece, or a satellite.
Gorillas though, that's a special failure. We all know why. And we know why
the error occurred. If the machine sees predominantly white people and
gorillas, then "people" are white and "almost people" but with darker pixels
is "gorilla". It is human prejudice in interpreting the results that made the
error an issue.

------
andrewclunn
There seems to be a bit of explaining away all bias as the result of human
creators here. If an algorithm trained on real world data displayed some
traditional bias, would we assume such a cause and dismiss it? One could just
as easily make this argument to dismiss all correlation based evidence. Yes
the code / methodology matters, but we'd best get ready for deep learning
algorithms to confront us with some uncomfortable truths.

~~~
empath75
Let's assume you have red and green people who are otherwise identical, and
that there was systemic discrimination against green people in the past that
led them to have lower incomes and lower education levels and live in areas
with lower property values and higher crime rates. A naive computer model
trained on objective data might find correlations between green people and
high default rates on loans, for example, which might be accurate, but which
aren't inherent to green people but were forced on them by external
circumstances. A reasonable bank might then conclude that they should
discriminate against green people without any personal bias against green
people at all.

The question you have to ask yourself is: is that fair? And what should the
law say about it?

~~~
atrendyguy
Forcing non-optimal decisions based on an arbitrary notion of "fairness" will
only weaken our institutions and breed resentment: "group A gets loan approval
with a lower credit score than members of my group, what gives!?"

It also increases the fragility of the group you're trying to help by
acclimating them to lower standards.

The success of Asian and Jewish people suggests that things can correct
themselves over time. The law should stay out of it.

[http://www.pewresearch.org/daily-number/asian-americans-
lead...](http://www.pewresearch.org/daily-number/asian-americans-lead-all-
others-in-household-income/)

[http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-
Chine...](http://www.zakkeith.com/articles%2Cblogs%2Cforums/anti-Chinese-
persecution-in-the-USA-history-timeline.htm)

------
h4nkoslo
There is literally no support in the article for the contention that
"artificial intelligence picks up bias from human creators" as opposed to
making correct inferences from reality. All of the examples they provide,
modulo blurry tank pix, are of the latter.

------
randyrand
Oh no! the computer is making uncomfortable findings that we don't like. The
computer must be wrong!

~~~
jjoonathan
What makes me sad is that there are good reasons to throw out information
which don't involve denying mathematical truth, yet people, for whatever
reason, seem to prefer denial.

What should happen: "I understand that this person has black skin and
therefore according to elementary statistics is more likely to be a criminal,
but acting on the information would create an unfair double standard that
punishes people for crimes they didn't commit, it would help perpetuate a
cycle, and it would violate individual human dignity, so I won't."

What actually happens: exactly what parent said. "Oh no! the computer is
making uncomfortable findings that we don't like. The computer must be wrong!"

 _facepalm_.

------
dominotw
whats with shitty scientificamerican stories on HN today.

~~~
grzm
If you don't think the submission is appropriate for HN, flag it if
appropriate and move on.

------
kahrkunne
"racist computers" is the funniest trend since "sexist babies"

It should be obvious that if an ML algorithm is trained on sufficient real-
world data, it won't be racist - it'll just not be politically correct

~~~
arjie
Why is that obvious? Take the case of criminal predictions. If you're using
convictions as ground truth, then you're just mirroring existing societal
biases. For instance, in a society with pervasive racism (like, say 1960s
America), the more real-world data you give it, the more your model is likely
to converge on racism.

Computers aren't racist but to characterise the article as referring to
something like that is to fail to understand it.

~~~
randyrand
Not as bleak as you imply. Just don't give the model race or strong indicators
of race as an input.

~~~
KODeKarnage
That's the problem. If there is ANY bias shown in your results, your algo
(actually you) will be accused of prejudice. That is what happened with the
Re-Offending Risk Assessment in the linked article. It didn't have race or
anything like it. It had suburb.

~~~
jonathankoren
Please. Zip code is a well known proxy for race given the history of racist
housing practices. Don't believe it? Ask yourself: What does a person from
Harlem look like?

~~~
KODeKarnage
Zip code is not a proxy for race and it is idiotic the say that it is. The zip
code of Karl Malone is the same as Chuck Norris. Zip code is a proxy for
COMMUNITY.

~~~
jonathankoren
You're trying to shift the words because it's uncomfortable, but the
underlying facts remain the same. Community is often dominated, and in fact is
defined by, by race. You can cherry pick a millionaire here and there but
those are exceptions but it's disingenuous and ahistorical to assert that this
isn't true. Redlining existed to maintain racial divisions. Cities, counties,
and even states explicitly maintained racial divisions, and even though these
laws do not exist today, their effects are maintained.

Do you think it's a coincidence that the only community with a significant
number of black people on the peninsula is East Palo Alto? It's not. (
[https://techcrunch.com/2015/01/10/east-of-palo-altos-
eden/](https://techcrunch.com/2015/01/10/east-of-palo-altos-eden/) ) Do you
think it's a coincidence that white people live north 8 Mile Road in Detroit?
Shouldn't Marshall Mathers had grown up south of it? It's not. I suggest you
read up on your history
[http://www.theatlantic.com/business/archive/2014/05/the-
raci...](http://www.theatlantic.com/business/archive/2014/05/the-racist-
housing-policy-that-made-your-neighborhood/371439/)

------
h4nkoslo
The interesting thing is that literally all of the inferences we are "worried"
about computers making were social consensuses not too long ago, with ample
first-order evidence to support them even in the absence of fancy machine
learning algorithms.

