
Daniel Kahneman calls for ‘daisy chain’ of psychology replications - tokenadult
http://blogs.discovermagazine.com/notrocketscience/2012/10/04/daniel-kahneman-daisy-chain-replications-priming-psychology/
======
dfc
This is a meta-piece about another article. The actual article:

[http://www.nature.com/news/nobel-laureate-challenges-
psychol...](http://www.nature.com/news/nobel-laureate-challenges-
psychologists-to-clean-up-their-act-1.11535)

The actual article is worth a read, I was a little intimidated after reading
"here’s some overmatter that didn’t make it into the piece because of length"
but the actual article is only 712 words. The over-matter is 500 words.

More background about the problem:

"Replication studies: Bad copy" [http://www.nature.com/news/replication-
studies-bad-copy-1.10...](http://www.nature.com/news/replication-studies-bad-
copy-1.10634)

~~~
bambax
"Replication studies: Bad copy" is very interesting. Sample quote: _“I've seen
students spending their entire PhD period trying to replicate a phenomenon,
failing, and quitting academia because they had nothing to show for their
time.”_

We say that a study "succeeds" when it illustrates a novel behavior and
"fails" when the expected behavior doesn't happen.

This is a very slippery use of words.

A finding that can't be replicated isn't a finding. It's a rumor, an urban
myth, but not a scientific step forward.

A study that "fails to replicate" a previous finding isn't a "failure" -- it
succeeded at debunking a false belief! And it should be granted wide access to
publishing.

(Same thing happens in medical studies BTW, where it's arguably much more
dangerous).

~~~
chii
> A study that "fails to replicate" a previous finding isn't a "failure".

Yep. the problem is that to the "laymen", it seems a bunch of people wasted
time trying to re-do what someone else has "already" done, and failed. Whether
the failure was due to incompetance, chance, or an actual failure to replicate
(meaning it is a significant), to the mind of the laymen, they are one and the
same. this then end up being difficult for people who might fund such studies
to fund these sorts of studies (despite it being quite useful), and leads to
people seeking "newer" things to research.

its a bit of a chilling effect really.

~~~
StavrosK
Who are these laymen, and why are they reviewing papers?

~~~
chii
they are not reviewing papers, nor are they the fund contributors. They are
the general population, whose sentiments affect the way society sees things
overall. For example, a lot of people are now rightfully scared of nuclear
tech (due to recent disasters). And yet, most people dont understand that
neclear energy is quite safe (compared to coal, say). If a political leader
decides to invest in nuclear energy, they instantly become unpopular, and thus
lies the problem of funding via such channels.

~~~
StavrosK
Sure, but they aren't the ones deciding what gets published or not. The study
that tried to replicate another study but failed has already been done, and
it's not up to the laymen to decide whether or not to publish the results. Nor
can we predict which studies will fail to duplicate results and not fund them,
otherwise we wouldn't have to make them.

Results of failure are still results, and they represent valuable knowledge.
Too bad the people who got them consider them a failure.

~~~
bambax
> _Results of failure are still results_

Yes! I would add that failing to replicate existing results is the only
interesting outcome of a replicating study.

If we try to replicate a study and arrive at the same conclusions, we have
learned little (the original result is a little stronger but that's about it).

If however, a series of new studies fail to replicate original results, we
have learned a lot -- we have learned that those original results were false.

I think the whole problem is, ironically, psychological in nature. Results of
studies become articles of faith; people identify with their beliefs and hate
to admit they have been fooled...

~~~
StavrosK
> Results of studies become articles of faith; people identify with their
> beliefs and hate to admit they have been fooled...

Exactly, and that's precisely the opposite of what science is. We should be
excited, not intimidated, by falsifications and getting results we didn't
expect.

------
FrojoS
Richard P. Feynman on pseudo science:
<http://www.youtube.com/watch?v=IaO69CF5mbY>

BTW, I come from the field of robotics and there, hardly any published result
is replicated either.

~~~
choxi
thank god for whoever had the foresight to sit Richard Feynman down and just
have him talk about everything

------
overbroad
IMHO, Bargh should redo some of his old expts himself. If he's so sure they
are repeatable, then why not? His stuff is foundational to the entire field.

~~~
icebraining
As an ignorant non-scientist, I assumed it'd be better for someone else to
replicate the studies, to avoid the same biases; wouldn't it?

~~~
overbroad
Yeah, that's usually how it'd done. But from what I've read (not just from the
Nature article, I did actually read his papers years ago), this guy basically
founded the entire field of what became "priming". It's all on shaky ground
now, whether that's justified or not. I'm not sure it would hurt for him to
question his own conclusions, in addition to having others repeat some
studies. Time to swallow the pride.

Studies are supposed to be repeatable. But in practice a lot of research never
gets repeated. I'm sure he must feel a little insulted to have his life's work
called into question, even if the cause is other investigators who were
fraudulent or incompetent, or even if the questions are arising only
indirectly because of unrelated research that happens to use priming.

IANAS, but psychology is obviously not the hardest of the sciences. I think in
other areas like e.g. chemistry and biology, it's more difficult to pass off
methods and results that are not repeatable _if_ they are going to be
foundational and used for lots of future research by others. If some other lab
can't get the right results using your methods, I would think they will make
some noise. They might first think they are themselves at fault, but they
would probably make contact with others outside their lab and make it known,
in some way, that these methods were not working for them. Whereas I can't see
that as being as common in psychological research. It's too easy if some
method is not working to just "make it work". ;)

~~~
com
I can corroborate the (gp?) note above regarding numbers of PhD students
dropping out because foundational research results that do not appear to be
correct in molecular biology. It's just anecdata, but biology and chemistry,
especially the expensive and difficult kinds (>1-2 years to attempt to repeat)
is probably not reproduced as often as you'd like, and as far as I am aware,
it's hard to get a PhD if you have identified serious shortcomings in
important, but foundational, research. We may be wasting a lot of talent this
way.

------
lutusp
> Daniel Kahneman calls for ‘daisy chain’ of psychology replications

A welcome and overdue call to action. But I predict the challenge will be
ignored, because the practitioners of this psychological specialty know the
effort will fail, and that failure will destroy what remains of their
credibility.

------
aba_sababa
We act like psychology was ever a legitimate science in the first place...

~~~
aggie
Psychology is a broad field with many subfields. Some areas are harder
sciences than others (psychobiology/physiology, sensation and perception,
neuroscience, some cognitive science, etc.), and some are admittedly very
hand-wavy (emotion, personality, just about anything Freud said, etc.), but to
dismiss all of psychological science as illegitimate is to ignore a tremendous
amount of progress in understanding and human well-being that has come in the
past century.

~~~
lutusp
You're missing an important structural difference between psychology and truly
scientific fields like physics. The difference is that scientific fields are
united by theories that define the entire the field, and that inform all work
in the field.

If a particle physicist discovers something basic in his own specialty, it
affects all of physics -- for example, the mass of neutrinos or the presence
or absence of a Higgs boson. Cosmologists must pay attention to particle
physics, even though particle physics and cosmology would seem to be
spectacularly dissimilar.

Such a scientific theory is a two-way street. If cosmology comes up with
something like dark matter or dark energy, based on solid observation but not
explained, this affects particle physics, and work begins immediately to
explain what has only been described. The reason is that the central
scientific theory, that defines physics, also unites it.

Psychology doesn't have such a theory -- there is no overarching theory that
unites the 53 subfields recognized by the APA, which consequently operate
independently of each other and of any serious concern about theoretical
implications.

So when you say that there is real science taking place in some of
psychology's subfields, you're absolutely right. But if someone wants to imply
that this means psychology is a science in the same way that physics or
biology are sciences, they're absolutely wrong.

> to dismiss all of psychological science as illegitimate is to ignore a
> tremendous amount of progress in understanding and human well-being that has
> come in the past century.

When Michelson and Morley falsified the ether theory in 1887, every living
physicist knew exactly what that result meant -- it pulled the rug from under
electromagnetic theory and left it unexplained. No one tried to say, "but
those other physicists are still doing good work." That's not how science
works.

I will know psychology _as a field_ has reached the threshold of science when
psychologists courageously propose a central, unifying, falsifiable theory,
amenable to practical test that, if it fails, invalidates the entire field.
And I won't hold my breath for that development.

Physics has such a theory. Biology has such a theory. Geology has one. Any of
those theories can be tested -- indeed, is regularly tested --and the failure
to falsify those theories is the only reason for the standing of those fields
as sciences.

Science is not scientists working in isolation, in a theoretical vacuum, on
studies that cannot be replicated, as with psychology. Science is instead the
building of a defining theoretical structure, constructed on a scaffolding of
evidence.

> and human well-being that has come in the past century.

That claim is completely falsified by the recovered memory therapy debacle of
the 1990s, which singlehandedly erased any public goodwill toward the field of
psychology and sowed an atmosphere of distrust that will require many years to
dissipate. And recovered memory therapy is just one of many examples in recent
times that reveal the danger of not shaping a guiding theory that
simultaneously informs research and controls the behavior of clinicians.

~~~
cschwarm
It's easy to dismiss a field when you compare a complicated problem to an easy
problem, and then object to a lack of answers for the complicated problem,
because there are so much more and better answers for the easy problem.

Let's face it: Physics is easy! It's objects are often cheap to study, and
usually doesn't react to the behavior of the observer. Additonally, the
observed behavior is often regular and easy to model with maths. There are
millions of research grants, and job opportunites because its findings are
often economically useful. Also, people in general don't think they 'know' its
findings in advance, so it's much easier to overcome your own biases, and
opinions. In most cases, there is often no politics involved about the
results.

In contrast, the social sciences are really hard. Lots of causes make research
harder: Objects often react to the researcher, or the setup. They want to get
payed for their time. Sometimes, they react to the experimental setup due a
political stance. There are ethical constraints for experiments. The shere
variability of human behaviour makes it hard to use mathemathical models to
describe theories. Its findings are not as useful economically, because they
cannot be impletemented on a large scale. Potential users of the research
often think, they 'know' how people react, anyway (folk psychology). For some
findings, many people are also motivated to question the results because it
conflicts with their political opinion.

Overall, the barriers are much higher for the social sciences to make
progress. I think these fields deserver the label 'hard', for finding the
truth is really hard here.

~~~
lutusp
> It's easy to dismiss a field when you compare a complicated problem to an
> easy problem, and then object to a lack of answers for the complicated
> problem, because there are so much more and better answers for the easy
> problem.

Psychology is not dismissed as a science because it's complicated, it's
dismissed because it's not science. The fact that it's complicated is
irrelevant to its standing among sciences.

And offering the explanation that it's _comparatively_ complicated fails any
test of common sense -- remember quantum theory? It's more complex than any
psychological theory, and yet we acquire perfectly reliable results to ten
decimal places. In fact, quantum theory is the single most successful
scientific theory in existence, yet no one fully understands it. "Anyone who
is not disturbed by quantum physics has not understood it." -- Neils Bohr.

> Let's face it: Physics is easy!

Only to those who don't understand the subject.

> In contrast, the social sciences are really hard.

Psychology is not a science because psychological research is so difficult?
Okay, but if I were a psychologist, I would ask you not to be on my side.

> ... for finding the truth is really hard here.

Science is not about finding truth, and scientific theories never become true.
Some of them resist falsification for extended periods, but all of them are
perpetually falsifiable in principle by new evidence.

------
tokenadult
There is now an effort to set up a new open-access journal of psychology that
will encourage best research practices like sharing data with other
researchers as a built-in part of submitting articles, and taking other steps
to communicate (and thus allow checking of) the study method and hypothesis.

[http://chronicle.com/blogs/percolator/building-the-
perfect-j...](http://chronicle.com/blogs/percolator/building-the-perfect-
journal/30794)

A local friend (a mathematician who works as a staff researcher on human
behavioral genetics research) and I will be in the weekly journal club meeting
later today at the University of Minnesota journal club. This week's topic in
the journal club is discussion of Uri Simonsohn, the "data detective." Several
of my recent submissions to Hacker News have been of readings related to our
discussion today. My local friend sent out this background discussion by email
yesterday:

"On Wednesday, September 26, Econ Nobelist and psychologist Daniel Kahneman
sent an open email message to a group of social priming researchers and one of
them forwarded the message to Ed Yong, the science journalist who interviewed
Uri Simonsohn for one of this week's readings. This is that email message:

[http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinf...](http://www.nature.com/polopoly_fs/7.6716.1349271308!/suppinfoFile/Kahneman%20Letter.pdf)

"Ed Yong contacted Kahneman and interviewed him, then he wrote this short
article for Nature News which appeared Wednesday, October 3

Nobel laureate challenges psychologists to clean up their act: Social-priming
research needs “daisy chain” of replication

[http://www.nature.com/news/nobel-laureate-challenges-
psychol...](http://www.nature.com/news/nobel-laureate-challenges-
psychologists-to-clean-up-their-act-1.11535)

"and this entry in his blog:

Nobel laureate Daniel Kahneman calls for ‘daisy chain’ of psychology
replications

[http://blogs.discovermagazine.com/notrocketscience/2012/10/0...](http://blogs.discovermagazine.com/notrocketscience/2012/10/04/daniel-
kahneman-daisy-chain-replications-priming-psychology/)

"The story was also picked up by Chronicle of Higher Education on Thursday
night, October 4:

Daniel Kahneman Sees ‘Train-Wreck Looming’ for Social Psychology

[http://chronicle.com/blogs/percolator/daniel-kahneman-
sees-t...](http://chronicle.com/blogs/percolator/daniel-kahneman-sees-train-
wreck-looming-for-social-psychology/31338)

"In related news, Uri Simonsohn posted his paper on statistical detection of
data fabrication on the Social Science Research Network site on July 22, 2012
(draft dated 7/21). He has since posted a revised draft, dated July 29, which
can be downloaded from this page:

<http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2114571>

