
Classic and contemporary psychology experiments replicated - mathgenius
https://cos.io/about/news/28-classic-and-contemporary-psychology-findings-replicated-more-60-laboratories-each-across-three-dozen-nations-and-territories/
======
andrelaszlo
Short summary:

\- The 28 findings was repeated in more than 60 laboratories all over the
world resulting in a median sample size of 7,157

\- Overall, 14 of the 28 findings (50%) replicated successfully

\- The replication team obtained the original materials to ensure faithful
replications of the original findings

\- All 28 studies underwent formal peer review at the journal prior to
conducting the studies

\- All of the studies were preregistered at OSF

\- Researchers can predict whether findings will replicate or not in surveys
and prediction markets with pretty good accuracy. (prediction market)

Full paper (100 pages):
[https://psyarxiv.com/9654g/](https://psyarxiv.com/9654g/)

------
iandanforth
"It was surprising that even with these efforts we were only able to obtain
support for the original findings for half of the studies. These results do
not definitively mean that the original findings were wrong, but they do
suggest that they are not as robust as might have been assumed." \-
Michelangelo Vianello, one of the project leads.

Yes, yes they do. It's ok to say the original findings were wrong. It's ok to
re-assert the null hypothesis. The unwillingness to do so undermines the
process and is disconcerting to say the least.

~~~
tlb
Failure to replicate doesn't prove the original findings were wrong. Other
things that can cause replication failure are:

\- Poor explanation of experimental procedure, so the replication doesn't
match the original experiment.

\- Awareness of the effect may invalidate it. For example, it'd be hard to
replicate the Stanford Prison Experiment because every psychology undergrad
has heard the story.

\- Change in social conditions or prevailing attitudes. Consumer behavior (the
subject of some of these studies) has changed significantly over the decades.
Homophobia (another study) varies dramatically over time and place.

~~~
iandanforth
We should continue this argument over coffee, hope you're doing well!

~~~
skrebbel
Please record the audio, i'm curious who convinces who.

(or just continue here on HN of course)

------
_pastel
Fun game: read through the summary of each experiment (psyarxiv.com/9654g) and
try to guess whether it replicated or not before they tell you. I got 20/28.

Some of them were obvious (#3 - no, using a bad font does not make people
answer questions more carefully; #22 - yes, people attribute more
responsibility to adults than to babies). But there were some big surprises
for me:

#6 - Participants read a hypothetical water conservation dilemma, then rate
how much they trust the people in the scenario to conserve water. Sometimes
the people are called "consumers" and sometimes "individuals". Does it make a
difference? Apparently yes. When they're called "consumers", you expect them
to conserve significantly less water than when they're "individuals".

#24 - if you ask "How satisfied are you with your relationship?" and then "How
satisfied are you with your life-as-a-whole?", are the results more highly
correlated than if you ask those questions in the opposite order? I thought
for sure they would be - relationship happiness is one of several contributing
factors to your overall happiness, so if it's fresh on your mind, you'll weigh
it more heavily when you answer the life-as-a-whole question. Apparently not.

~~~
c3534l
I'm about halfway through doing just that and I think a pattern has emerged:
does there exists a plausible, concrete (you could put it into words)
mechanism for cause and effect? That is, can a person tell you _why_ the
finding exists. For instance, a semantic association between capitalism and
greed does not provide a plausible mechanism for why mentioning capitalism
should make one greedy. The gay one failed that test, but otherwise that seems
to be what's going on. A lot of these papers aren't testing a scientific
theory. They're testing vague, half-baked ideas and associations.

------
andrelaszlo
List of studies, and the outcome:

1 Direction & SES (Huang et al., 2014) p<.001

2 Structure and goal-pursuit (Kay et al., 2014) p=0.97

3 Incidental disfluency (Alter et al., 2007) p=0.67

4 Moral Foundations (Graham et al., 2009) p<.001

5 Affect and Risk (Rottenstreich & Hsee, 2001) p=0.77

6 Priming consumerism (Bauer et al., 2012) p=0.15

7 Correspondence Bias (Miyamoto & Kitayama, 2002) p<.001

8 Disgust & Homophobia (Inbar et al., 2009) p=0.56

9 Incidental Anchors (Critcher & Gilovich, 2008) p=0.25

10 Social Value Orientation (Van Lange et al., 1997) p<.001

11 Trolley Dilemma 1 (Hauser et al., 2007) p<.001

12 SMS & Well-Being (Anderson et al., 2012) p=0.58

13 False Consensus 1 (Ross et al., 1977) p=0.23

14 False Consensus 2 (Ross et al., 1977) p<.001

15 Position & Power (Giessner & Schubert, 2007) p=0.31

16 Framing (Tversky & Kahneman, 1981) p=0.43

17 Trolley Dilemma 2 (Hauser et al., 2007) p=0.42

18 Tempting Fate (Risen & Gilovich, 2008) p=0.01

19 Actions are Choices (Savani et al. 2010) p<.001

20 Intuitive Reasoning (Norenzayan et al. 2002) p<.001

21 Less is Better (Hsee, 1998) p<.001

22 Moral Typecasting (Gray & Wegner, 2009) p<.001

23 Moral Cleansing (Zhong & Liljenquist, 2006) p=0.08

24 Assimilation and Contrast (Schwarz et al., 1991) p=0.39

25 Choosing or Rejecting (Shafir, 1993) p=0.10

26 Priming warmth (Zaval et al., 2014) p=0.01

27 Intentional Side Effects (Knobe, 2003) p<.001

28 Direction and Similarity (Tversky & Gati, 1978) p=0.99

~~~
sys_64738
None of the classics.

~~~
conjectures
I'm not much of a psychologist, but I can see at least one paper by Khaneman,
who won the economics Nobel for that sort of stuff. So not sure if you're
being sarcastic or not?

~~~
sys_64738
20 of the 28 are from this decade which hardly makes them classics. Nothing
from the 50s, for example. This list isn't The List.

~~~
Wohlf
To be fair, the title says classic and contemporary. Probably would have been
better to just say "High Profile" or something instead.

------
tvanantwerp
Looking through the paper, even the half that replicated generally seem to
have done so with greatly diminished magnitude.

------
throw2016
Poor choice of headline for the original article given 'replicated' indicates
success. This was an intensive attempt to replicate but 14 of the 28 did not
replicate.

That is a 50 percent failure rate inspite of using over 186 researchers in 60
labs worldwide. The widely reported 2012 study could not replicate nearly 100
of 100 studies. And in this case if this kind of comprehensive effort can't
replicate half the studies it raises serious questions about the original
studies and adds more fuel to the reproducibility crisis.

Given studies are routinely used to justify and implement policies [1], and
taken seriously as points of authority it's not just academic.

There needs to be much larger discussion on the use of studies to direct
policy that impact real lives, and the level of oversight, additional
replication and other credibility building measures that need to be
considered.

[1]
[https://www.sciencedirect.com/science/article/pii/S001632871...](https://www.sciencedirect.com/science/article/pii/S0016328717300472)

------
tylerjwilk00
The thing with psychology experiments is that the fundamental thing being
tested is not immutable.

This makes it different than many other science experiments where concrete
parts of reality are tested.

Psychology really only tests the minds of humans in a global society at a
point in time. If future minds change enough the results may no longer be
valid.

~~~
jhedwards
Your statement in itself is a hypothesis to be proved. There are psychology
experiments designed specifically to ascertain what parts of the mind are
hard-wired and which are depended on nurture/culture.

I am not a Psychologist but I can give one example: Piaget hypothesized that
the mind developed by first accumulating low-level sensory/motor data, and
then progressively building higher levels of abstraction on that foundation.
If that is true, then that process itself is not related to society or culture
and should be observable so long as the biology underpinning cognitive
development doesn't radically change.

~~~
tylerjwilk00
The mind is mutable.

Psychology is different than other "hard" sciences.

I only say "results _may_ no longer be valid" and I'd say this study proves
that.

If you go deep enough into the mind to the point where social environment no
longer influences the behavior you may be getting closer to biology than
psychology but point taken.

------
hnmullany
Here are some of the details of some of the unreplicated studies (not
scientifically specific - but you can always read the paper)

Replication Failures

\- Disgust sensitivity predicts homophobia (no it doesn’t)

\- Assimilation and contrast - the order of asking specific vs. general
questions about the same domain matters (no it doesn’t)

\- Priming Consumerism - being told you’re acting as a consumer reduces your
trust in others (no it doesn’t)

\- Moral Cleansing - people exposed to unethical things have an urge to
cleanse themselves physically (no they don't)

\- Priming Warmth - on warm days people have higher belief in global warming
(no they don’t)

\- Structure & Goal Pursuit - being exposed to structure improves goal pursuit
(no it doesn't)

\- Incidental Disfluency - making tasks incidentally difficult (hard to read
fonts) - improves analytic performance (no it doesn’t)

------
dorchadas
I'd love for someone to go through and do this with education research. It's
an absolute joke, and I'd argue it's _worse_ than psychology research. And it
doesn't help teachers are making students refuse to use old literature (or
maybe I'm just pissed I can't quote older studies in my 'literature review'
for my 'research paper'...)

------
bluetwo
I'm just going to point out that the ridiculous idea that using a hard to read
font improves retention of information was disproven.

~~~
pbhjpbhj
Why do you class that as "ridiculous"? A mechanism by which that might be true
leaps readily to mind.

~~~
bluetwo
Because the science behind building better instruction already exists and does
not rely on parlor tricks to make poor instruction more memorable.

------
sometime
Is there a list of the experiments and the effect sizes and how they differed
from the original studies?

~~~
bluetwo
Yes, in the paper, you can to follow the links.

------
jonnycomputer
I read psych papers. Regularly. I don't know any of these aside from
Tversky's.

Psychology is a big field. I just find it hard to accept that these papers are
representiative of much of anything.

------
ppeetteerr
TLDR; only 50% succeeded replication.

You don't have to feel so bad about not reading that Malcolm Gladwell book,
now.

