
A.I. Systems Echo Biases They’re Fed, Putting Scientists on Guard - wglb
https://www.oodaloop.com/technology/2019/11/12/a-i-systems-echo-biases-theyre-fed-putting-scientists-on-guard/
======
shantly
Am I wrong in thinking that "echo the biases they're fed" is a pretty
complete, concise description for what "AI"/ML systems are _supposed_ to do?

~~~
wpasc
Kind of. AI/ML systems are supposed to learn things from the data they are fed
to accomplish some sort of more generalizable goal or task. Echoing the biases
they're fed is probably more akin to overfitting to their training data.

~~~
rcthompson
No, it's not the same as overfitting. It's nothing to do with the algorithm at
all. The problem is with the training data, which contains patterns induced by
the way the data was collected rather than the reality of what you want to
classify.

For example, imagine that you wanted to train an algorithm to distinguish
photos of dogs from photos humans. So you collect a bunch of photos of both
dogs and humans and use them to train a classifier. You do all the proper
cross-validation, bootstrapping, etc. to ensure that you are not overfitting,
and you get really good results. Then, looking at the mis-classifications, you
notice something: all the photos that are taken looking at an angle down
toward the ground are classified as dog photos, and all the photos taken
looking straight ahead are classified as human photos. It turns out that in
your training set, most of the dog photos are taken at a downward angle while
must of the human photos are taken facing straight ahead, because humans are
taller than dogs, and your machine learning algorithm identified this feature
as the most reliable way to distinguish the two groups of photos in your
training set.

In this hypothetical example, no overfitting occurred. The difference in photo
angles is a real difference in the training sets that you provided to the
algorithm, and the algorithm did its job and correctly identified this
difference between the two groups of photos as a reliable predictor. The
problem is that your training set has a variable (photo angle) that is highly
correlated with what you want to classify (species). This is considered an
unwanted bias (and not a reliable indicator) because the correlation is caused
by the means of data collection (most photos are taken from human head height)
and has nothing to do with the subject of the photos.

~~~
CardenB
I think you’re arguing semantics a bit. What you’re saying checks out, but one
could say that overfitting was occurring but the test dataset distribution was
not wide enough to catch it.

~~~
Izkata
Overfitting is about being too precise because of the sample inputs, such as
if "downward angle" \+ "brown blob" (one specific dog breed) + "leash" \+
"lots of green" (grass) was required to identify a dog. GP's example wasn't
that, it was just identifying the wrong thing.

~~~
XuMiao
Instead of overfitting , it's more related to exploitation vs exploration. We
see more men related to programming might be just that women are not given
opportunities to explore the programming as a career.

When AI makes a decision, right now, people only uses the probability output.
Hiring A has .6 probability while hiring B has .4. then we will hire A instead
of B. However, if we consider the confidence intervals, the decision might not
be that clear. Say +/\- .5 to hire A but .2 to hire B. If exploration is
considered too, very likely that we will give B a chance.

AI is in the realm of probabilistic decision making, while normal people don't
follow. The bias is not from the training side. It's the decision making
process incorporating AI should change.

------
kempbellt
Try and wrap your mind around the idea of an "unbiased" AI. It is not
possible.

As a trainer of an AI, simply looking at a picture and saying that it's a cat,
and then telling the AI that it's a cat, is biased. You are automatically
assuming you are correctly identifying it as a cat, and teaching the mind of
an AI to follow your presumption.

The only way to make an AI _less_ biased (but still biased), is to diversify
your labeled training data. And by diversify, you need to diversify the
presumptions on the labels of the data themselves. What is the confidence
among _several_ people that this picture is of a cat? Rather than one trainer.

This is a simplistic example. If you scale it to more complicated tasks, such
as understanding natural language, body language, intent of users, or anything
related to ethics, you'll likely end up with a crazy AI. And yes, I mean
crazy. As in, simultaneously holding very seemingly opposing understandings of
reality and spitting out data that reflects those oppositions.

If you've ever had more than one authoritative figure in your life give you
contradicting advice to another - one parent holding differing beliefs to
another, or one teacher to another in school, you may notice yourself starting
to qualify your teacher's competence as a factor in the fidelity of their
teachings.

A scary, but simultaneously cool thought: An AI that questions my capacity to
teach it, while I'm teaching it.

Bias will always remain an element in the equation however.

One AI will likely have interacted with different trainers or data, and create
contradicting understandings of reality to that of another AI.

~~~
crooked-v
"Duck or rabbit" is another simple example that would require bias in visual
interpretation to reliably decide.

[https://www.independent.co.uk/news/science/duck-or-rabbit-
th...](https://www.independent.co.uk/news/science/duck-or-rabbit-the-100-year-
old-optical-illusion-that-tells-you-how-creative-you-are-a6873106.html)

~~~
kempbellt
Hah. Thank you for this reminder. This is a great example to illustrate the
point.

I would say that a less biased (wiser) AI, would learn that this duck/rabbit
image is, itself, questionable, and can be perceived both ways (high
confidence values for both identifiers).

Me: Is this a duck or a rabbit, AI?

Padawan AI: It's a rabbit (51% confidence as rabbit. 49% as a duck).

Master Yoda AI: Rethink your question, you must. _Yes_ , the most correct
answer is.

------
humanistbot
"Garbage In, Garbage Out" has long been a foundational saying in computer
science. I'm genuinely curious why so many in this generation of AI
researchers and developers seem to think they're immune to this.

~~~
LeifCarrotson
> _On two occasions I have been asked, — "Pray, Mr. Babbage, if you put into
> the machine wrong figures, will the right answers come out?" In one case a
> member of the Upper, and in the other a member of the Lower, House put this
> question. I am not able rightly to apprehend the kind of confusion of ideas
> that could provoke such a question._

> _Passages from the Life of a Philosopher (1864), ch. 5 "Difference Engine
> No. 1"_

A very long time indeed!

I think the key source of confusion (both in 1864 and today) is that non-
technical people hope that the machine has some _wisdom_ correlated with its
arithmetic prowess. Charles Babbage's mechanical calculator did not have
wisdom. Even when you give a bleeding-edge algorithm Big Data and lots of
processing power, it doesn't yet have wisdom. But hope springs eternal.

------
dhairya
The oodaloop article is a summary of the NYtimes
([https://www.nytimes.com/2019/11/11/technology/artificial-
int...](https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-
bias.html)) article which is more detailed. Also pleasantly surprised in the
NYTimes accurate description of what BERT was trained to do (masked word
prediction and next sentence prediction) and the implications of that.

I have worked closely with BERT and other language models for our startup.
There is a disconnect between the capabilities and the state of AI research
today and the public's expectations and imagination. There's also fundamental
confusion between scale and intelligence. That is the public largely believes
the efficacy of many "AI" models on large scale problems is equivalent to
intelligence. That assumption is problematic in both overestimating the
capabilities of these models and misdirecting the focus of critical inquiry.

Hopefully, there is more education around the limitations and capabilities of
these technologies. We should be more cautious in apply them to usecases where
there is high potential of negative consequences.

------
manfredo
Is the problem that AI is biased? Or is the problem that AI _lacks_ the biases
that we expect people to have?

For instance, people have detected that AI models associate technical words
with men more often than women and point to this as evidence of bias. I would
argue that this is the opposite situation. There's social stigma attached to
acknowledging differences between groups, so people have developed biased
against acknowledging those differences. The AI on the other hand sees that
70-80% of technologists are men, and associates accordingly. So the problem
isn't that the AI is biased it's that it lacks the biases we expect.

Now, there are good arguments why this bias may be a good thing. Unbiased does
not automatically mean good.

~~~
gbrown
I think you've very severely missed the point. Differences like those you
describe generally reflect biases in our socal structure, which are then
(correctly, but problematically) reflected in statistical models and
algorithms of all stripes. This becomes more problematic when we overinterpret
the resulting findings, or uncritically incorporate the resulting predictions
into real systems.

A good example is the criminal justice system in the US. Minority communities
are much more intensively policed, prosecuted, and jailed. A model will,
without care, use race or poxies thereof to predict "criminality" without
understanding or accounting for the bias inherent in the label. If that is
used in policy, it runs the risk of amplifying existing social problems and
injustices.

We alway have to ask: "bias for what?", or the conversation will be hopelessly
confused.

~~~
nostromo
It's unpopular to talk about, but there is quantifiably more crime in some
minority communities in the US.

There are some crimes, like drug use, that are indeed over-reported in some
communities due to over-policing.

But there are other crimes, like homicide, that are nearly universally
reported, regardless of where they are committed. And those crimes are much
more frequent in minority-majority parts of the country.

Turning a blind eye to this problem is a disservice to those communities,
because they are the same communities that are most commonly the victims of
crimes.

~~~
dmwallin
The problem is that these systems are fundamentally unable to distinguish
causation from correlation. Which is admittedly a hard prob even for humans,
but at least we have some capacity to tease these out.

In this case, the increased crime in these communities are not caused by their
minority status, but rather from a multitude of other factors with historical
and societal origins.

~~~
manfredo
> The problem is that these systems are fundamentally unable to distinguish
> causation from correlation. Which is admittedly a hard prob even for humans,
> but at least we have some capacity to tease these out.

I'm not sure why causation vs. correlation is relevant here. When, say,
someone is up for parole the judge will review their past criminal history,
conduct in prison, whether or not they will have a support structure when they
are released, etc. None of these factors are causal. A judge cannot point to a
past crime, or to conduct in prison and say "this factor will _cause_ you to
reoffend". No, the decision is made based on factors that are _correlated_
with higher rates of recidivism.

~~~
iknowalot
The USA has spent 4+ centuries kicking certain communities down the Maslow
Heirarchy, this was done on purpose through laws introducing redlining, Jim
Crow, no access to the GI Bill, slavery etc...I'd like a world where we could
judge people fairly on the same standard, but we purposely created economic
underclasses and that comes with a certain amount of desperation that leads to
crime.

White Americans = 70% of pop. , $100 trillion in wealth.

African Americans = 14% of the pop, $2.9 trillion in wealth

African Americans in particular owned almost 10-12% of the land in this
country (true wealth) and were promised more from the government in
reparations (40 acres) not long ago, but discriminatory policies stripped that
land away from them over 100 years.

Being in the USA for 150-200 years results in atleast 500k-1million $ in
wealth purely due to land and home value appreciation.

The average black person has about ~500$ in wealth. This isn't a fluke, this
was designed... This is a country that criminalizes being poor more and more
in many ways, so we can fall into the trap of revictimizing the underclass we
created, this time using algos, if we are not careful.

~~~
DuskStar
And you're _also_ confusing correlation with causation here. Yes,
"governmental policies have discriminated against African Americans and
prevented the accumulation of wealth" _is_ a valid hypothesis, but the data
you've presented doesn't show it.

Showing causation with whole-group statistics is very, very hard.

~~~
iknowalot
It's not a hypothesis. It's hard to track money flows but not as hard as the
anti-reparations crowd make it seem.

The majority of the $500k-1m baseline in white middle-upper class wealth is in
homes and land handed down as inheritances, this is not debatable. That
property allows a certain amount of leverage to invest in education,
businesses etc...

The ben & jerry's founder talked about this in vivid detail on the campaign
trail with Bernie Sanders. If he was black where he grew up, no GI bill = no
cheap housing = no appreciation of property/land over his childhood = no
financial leverage to build his company.

I can take risks and fail without going bankrupt thanks to familial wealth,
that is a tremendous luxury not afforded to the group I'm mentioning.

~~~
DuskStar
Unfortunately for that argument, the vast majority of white wealth is not in
the hands of the middle-upper class, and I highly doubt that the wealth of the
1% (or .1%) is mostly in homes and land inheritance. And if you're counting
"middle-upper" as 80-95%, I think the top 5% has ~70% of the total wealth in
the US... (for reference, 95th percentile is ~2.5m, or comfortably above your
500k-1m baseline. 1m is 88th percentile, 500k is 80th)

(And in any case, my point was just that it wasn't shown as _causation_ in the
stats you cited)

~~~
iknowalot
>...the vast majority of white wealth is not in the hands of the middle-upper
class

You are correct on that point. More than 50% of the 100$ trillion is held by
the top 10%.

On the other point, my point is the first $500k to 1million of wealth was due
to inheritance... not the total wealth of the 1%.

Follow up: At one point, any white male could move westward and get free land
(whether indegeous owned or not). 40-100 acres after a century+ of property
value appreciation is quite a bit of wealth. The $500k-1m number I use as a
baseline for white wealth is very conservative.

------
acd
I am concerned about AI algorithms and especially recommendations algorithms
in regards to biases. If someone is in a temporary psychological condition.
The ad recommendation algorithms will keep reinforcing that bias behavior in
order to maximize ad revenue. As long as the viewer is hooked on the site and
keep coming back for more content. The algorithms have little or no sense what
bias they are reinforcing. I wish there was more public debate and awareness
from programmers about that. Ie we have moral obligations on what algorithms
recommend.

~~~
x220
Engineers at Instagram, Facebook dot com, Youtube, and Twitter DO NOT want to
think about how they are affecting the mental health of their users. All they
want to do is spout some platitudes and wash their hands of all
responsibility.

------
ropiwqefjnpoa
I'm curious if some of these are actual discriminatory bias' or just harmless
cultural norms. Something like associating men with football can hardly be
considered bias.

~~~
loopz
It goes both ways: Even though you remove the sex/gender variables, the system
could correlate football to identify men, and sockermoms!

------
jimmaswell
Wayback link:
[https://web.archive.org/web/20191113162629/https://www.oodal...](https://web.archive.org/web/20191113162629/https://www.oodaloop.com/technology/2019/11/12/a-i-
systems-echo-biases-theyre-fed-putting-scientists-on-guard/)

------
kazinator
Scientists should be sophisticated computer users who understand the GIGO
(garbage-in, gabrage-out) principle; that's what should put them on guard with
regard to any manner of massaging their data.

------
dmtroyer
the new york times article that this links to:

[https://www.nytimes.com/2019/11/11/technology/artificial-
int...](https://www.nytimes.com/2019/11/11/technology/artificial-intelligence-
bias.html)

------
40acres
Good.

I hope that AI can be used as a mirror for the engineers working with it. AI
allows amplification of the subtle assumptions we make in design and hopefully
that amplification leads to better understand and appropriate measures to
reduce bias.

~~~
strken
Less a mirror of the engineers working on it and more a mirror of the data
it's fed, which is a really important difference. Nobody thinks "today I'll
create a model that hates women", they just aim their shiny new model at e.g.
every New York Times article since 1851, then only check for the results
they're interested in.

------
Digit-Al
Is there an opening here for some enterprising persons to try to develop
systems to try to determine the biases present in training data?

------
bobloblaw45
This reminds of of Tay from a few years ago.

------
swebs
>and is more likely to associate men with computer programming

So would any reasonable human. I think the real problem these advocacy groups
have with AI models is that they're _not_ biased. They reflect the real world
based on data and evidence, rather than conforming to progressive dogma. I
fear that instead of using AI to overcome any wrong assumptions we may have,
we're just going to get AI diversity officers to "correct" models that draw
any uncomfortable conclusions.

~~~
gbrown
> "So would any reasonable human."

That's the entire point of the article, our biases, as reflected in the world,
are learned by models. Consider the following:

* More computer programmers are men than women (descriptive statement, no problem)

* A predictive model correctly identifies that more computer programmers are men than women (a prediction based on observed data, no problem)

* A recruiting agency uses a predictive model to recruit computer programmers. Due to the way the model was trained, it excludes qualified women (not OK, a clear misapplication of an algorithm)

Feel free to try this thought experiment in other contexts where existing
biases can be amplified through algorithms.

~~~
Excel_Wizard
Let's say that there is a population bell curve of the variable "propensity to
enjoy being a programmer". One curve for men, and one curve for women. In some
dystopian future, everyone takes a career aptitude test, and if an individual
falls within the top 10% of the overall population, they get turned into a
programmer by the government. This is assuming a perfectly-trained AI judges
the variable, and that it controls against the underlying variables of
societal bias in order to ensure that long-term human resources are properly
allocated, and that long-term biases, informed by people's interactions with
eachother, trend toward actual biological differences.

It may well be the case that the gender split is 80% / 20% male to female (as
is roughly the case today). It may not. However, the left-leaning zeitgeist
opinion would seem to be that this outcome is impossible, and current observed
differences are only due to systematic oppression. The right-leaning zeitgeist
opinion would seem to be that this outcome would make sense.

I tend to think that the left-leaning opinion on this is so wrapped up in
double-think that it can't even understand itself- it tends to argue too much
in favor of "biological differences would mean permanent, uncorrectable
injustice, therefore it is impossible that biological differences exist."

~~~
sgslo
> in favor of "biological differences would mean permanent, uncorrectable
> injustice, therefore it is impossible that biological differences exist."

That's because the issue of men vs women in programming really is an all-or-
nothing topic. Either you believe that a female is capable of being an
equivalently skilled programmer to a male, or you don't.

Referencing biological differences always cascades to a question about the
innate ability of a female to program. The best example I can point to is the
infamous internal Google manifesto on male vs female programmers. If you read
that text, it appears reasonable enough: the author thinks that there are
biological differences, and these differences might lead to differences in
programming strengths. But it is a wolf in sheep's clothing; as soon as you
believe that there are differences, it follows that one set of differences
must be advantageous to the other.

I can _understand_ that there are biological differences between females and
males, but I absolutely and vehemently choose to believe that there is no
inherent difference in ability - females are 100% as capable as males when it
comes to programming. Full stop.

Yes, this is double think. But I'd rather be a hypocrite than hold a secret
belief that my biological sex makes me a better engineer.

~~~
WhiskeyJack55
What if the question isn't one of capability, but a question of self selection
and preference?

Does those things even play a role? If yes, how does that play a role? How
large of a role? If it does play a role, why? What's important to women in
career choice vs what's important to men? Why is that the case? Is it nature
or nurture or both (and to what degree of each influence those choices)? Is it
upbringing? Is it pressure from society? Is it barrier's to entry? And to what
degree does all that play a role?

I see the potential for a much more nuanced conversation with this topic.

Saying women aren't capable of being a programmer or being successful in STEM
is in my mind a garbage assertion.

~~~
commandlinefan
> Saying women aren't capable of being a programmer or being successful in
> STEM is in my mind a garbage assertion.

Good, then, that nobody has said, or even implied, that.

~~~
rurp
That's literally the argument that sgslo was countering. I agree that it does
not seem like the most charitable position to argue against though.

~~~
commandlinefan
That isn't even remotely what the person that sgslo was arguing with said.

------
Doingmything123
Ironically, I think this shows how human-like we have been able to make AI
systems.

~~~
AstralStorm
Not entirely. Humans are able to use well described and backed logic in
decision making. Ever seen AI write out its decision logic, in a form that's
portable to other AI?

People past few years old can output at least partial rationale for behavior
or decisions. Systems like BERT are at best comparable to a pre-linguistic 2
year old.

~~~
Doingmything123
I admit that it's unfortunate that AI can't write out their decision logic but
I would argue that is because there hasn't been enough resources put into
explainable AI. Considering the increasing use of these algorithms, I don't
know if that is even a high priority.

I tend to think that people are not as logical as they like to think they are,
myself included. Not to say there isn't good reasoning, just that much of our
decision making is emotional and habitual over some pure sense of logic.

Systems like BERT seem perfectly rational to me. Are they not just following a
set of rules on a given input to modify a state?(In the most simplistic sense
of computation). I think the confusion is more over what the goal of these
programs are and how do we encode that. This reminds me of the ai system that
would pause the game of tetris so that it could never lose. Not we it's
programmers intended but still accomplished it's "goal".

~~~
bluGill
While people are not as logical as we like to be, we all are logical. I can
teach someone the rules of math - my method of teaching might be (probably is)
bad, but if the student tries he will learn those rules. Latter on when given
a test the student can show his work and it will be much the same as every
other student trying to explain his reasoning: operations like "complete the
squares" have been well described and reasoned out.

Likewise chess masters can explain their thought process while looking at the
next move and other chess masters will agree the lines of thought are good
(they will probably ask why not some other equally good line...). We know this
explanation is good because students can watch the experts explain their
thought process and replicate it in games to a small extent and to better.

------
pixelbath
TFA is not responding and Google's cache is showing a 404.

So, at the risk of a post that's purely snark: "In other news, scientists
discover water is wet."

