
The Implicit Association Test falls short of basic scientific standards - sohkamyung
https://qz.com/1144504/the-world-is-relying-on-a-flawed-psychological-test-to-fight-racism/
======
ewjordan
Being hesitant to uncritically embrace iffy research that supports our
politics can be really hard. In myself I even notice a tendency when looking
at research on controversial subjects to put more weight on articles that I
"like" going in, and I will sometimes go as far as to interpret articles that
disagree with my views as partially supporting them, if read in a certain way.

The thing is, it's 100% correct to evaluate new claims in light of your
priors. That's absolutely necessary if you want to make good decisions - my
expectation that special relativity will not be shown to be inconsistent is so
strong that no matter how interesting or new an argument is against it, I will
shred it to bits way more viciously than I'd pick apart a simple replication
of special relativity's claims.

But.

Some priors don't come from reliable places, and we have to be way more
careful about applying them. In general, the more politically or socially
relevant an issue is (gender, race, money, morals) the less any of us should
trust both a) our own priors, and b) the priors of researchers in the area,
who have many ways to unwittingly and without bad intention put fingers on the
scale. In those situations we really have to throw our hands up and wait for
meta-analyses to tell us what's up.

In this case, it's super worth noting that even if the IAT is a garbage test,
that doesn't mean that implicit bias doesn't exist, or that racism is not a
problem. It just means the test doesn't measure it, and we shouldn't be
funding further research that leans on it as a tool. Which is why I really
don't understand why criticisms of the test draw such intense opposition.

~~~
throwanem
I'd guess it has to do with the loss of a measure thought to be able to
reliably quantify unexamined racism in a novel way.

It's one thing to say to someone, "even if you don't realize it and don't
think it's true, you're at least a little racist," and have to leave it at
that. Without some means of substantiating the claim, you don't have a good
counter for the immediate "no, I'm not!" that such a statement is very likely
to provoke. You're likely to end up with pointless recrimination on both
sides, because what else, really, have you got? It's semiotically equivalent
to telling someone they've "sinned in their heart", in the absence of any
tangible evidence beyond your fervent belief this must be so - and exactly as
useful as that, besides.

The IAT offered a means of substantiating the claim. Armed therewith, you
could say, "even if you don't realize it, you're at least a little bit racist,
_and I can prove it_." That's a different and rather stronger proposition, and
it's something which I think I don't too far miss the mark in saying that it's
something a lot of people have been looking for for a long time. I can see why
someone who believes he's finally found that thing might find it difficult to
acknowledge that there's less substance to it than initially imagined. And for
similar reasons, I can understand why an otherwise objective researcher might
have the same trouble.

------
wutbrodo
I'd like to say that "everyone already knows this", but this particular topic
is one that seems to be a poster child for people believing what fits their
politics instead of what fits reality, unfortunately. So it's always good to
see it getting more exposure.

I honestly don't get what's in it for the people pushing bad science related
to social-justice issues (excluding those who are actually part of the
industry of seminars around same). This is a problem that many people want to
solve, and the bad science just makes it _harder_ to solve these problems, and
lessens support for the movement to do so.

~~~
closed
> I honestly don't get what's in it for the people pushing bad science related
> to social-justice issues

as a first step, I would try to imagine the situations where you might produce
bad science, in the name confirming something about an issue important to you.

(I think most researchers have felt the pull of striking results, and that
little twinge of pain from realizing they might have been produced by a bug).

~~~
wutbrodo
The fact that I completely don't understand it is because it's something I
very much wouldn't, and don't, do. I tend to confuse people I'm arguing with
about any issue that ends up being politically polarizing, because from their
perspective, my worldview is a muddled mix of things. What's actually
happening is that I don't conflate facts with values, and I try to only
believe facts that are true, instead of taking my beliefs wholesale from the
political package that makes me feel good.

By the way, I'm not talking about researchers here, I'm talking about people
_pushing_ bad science. Bad science happens, and it's not even the fault of
scientists lots of the time. Pushing it into policy before it's good science
is the phenomenon I'm complaining about.

------
jaggederest
I would be pretty amazed if the test didn't indicate bias between a complete
A-A test (i.e. associate white people and white people with red words and
green words), even leaving aside consistent scores as they discuss in the
article.

Which means it's not reliably going to indicate anything in particular by
itself. This is a challenge for a lot of these psychology tests and is part of
the root crisis of reproducibility.

------
joelkevinjones
If all variants of this test act as the author describes, wouldn't the fact
that the same keys are bound to different things in different phases of the
test be a problem?

------
oh_sigh
Surely everyone who quotes this study will amend their statements at this
point.

------
bob_theslob646
>There are various psychological tests purporting to measure implicit bias;
the IAT is by far the most widely used. When social psychologists Banaji (now
at Harvard University) and Anthony Greenwald of the University of Washington
first made the test public almost 20 years ago, the accompanying press release
described it as revealing “the roots of” unconscious prejudice in 90-95% of
people. It has been promoted as such in the years since then, most vigorously
by “Project Implicit,” a nonprofit based at Harvard University and founded by
the creators of the test, along with University of Virginia social
psychologist Brian Nosek. Project Implicit’s stated aim is to “educate the
public about hidden biases”; some 17 million implicit bias tests had been
taken online by October 2015, courtesy of the nonprofit.

“Implicit bias” became a buzzword largely thanks to claims that the IAT could
measure unconscious prejudice. The IAT itself doesn’t purport to increase
diversity or put an end to discriminatory managers. But it has certainly been
deployed that way, partly due to its creators’ outreach.

HR departments quickly picked up the theory, and implicit-bias workshops are
now relied on by companies hoping to create more egalitarian workplaces.
Google, Facebook, and other Silicon Valley giants proudly crow about their
implicit-bias trainings. The results are underwhelming, at best. Facebook has
made just incremental improvements in diversity; Google insists it’s trying
but can’t show real results; and Pinterest found that unconscious bias
training simply didn’t make a difference.

Current studies have found the race IAT to have a test-retest reliability
score of 0.44, while the IAT overall is around 0.5 (pdf); even the high end of
that range is considered “unacceptable” in psychology. It means users get
wildly different scores whenever they retake the test.

“For other aspects of psychology if you have a test that’s not replicated at
0.7, 0.8, you just don’t use it,” says Machery.

The second major concern is the IAT’s “validity,” a measure of how effective a
test is at gauging what it aims to test. Validity is firmly established by
showing that test results can predict related behaviors, and the creators of
the IAT have long insisted their test can predict discriminatory behavior.
This point is absolutely crucial: after all, if a test claiming to expose
unconscious prejudice does not correlate with evidence of prejudice, there’s
little reason to take it seriously.

Essentially it all boils down to this statement: >No psychologist or
neuroscientist can convincingly point to a clear divide between conscious and
unconscious thought. And so psychology’s attempt to solve discrimination by
delineating between an amorphous collection of conscious and unconscious
biases is both simplistic and misguided.

Pretty amazing.

~~~
dang
Could you please not paste large blobs of articles into HN threads?
Discussions here are supposed to be conversations, not regurgitations, and
people can read the articles for themselves.

~~~
noobiemcfoob
I'd agree, but that was a gigantic article. Picking subsections to talk about
is going to be largely required.

~~~
dang
Each user who reads the article is welcome to talk about their subsection of
interest.

