
How Priming Research Went off the Rails (2017) - scott_s
https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wreck-how-priming-research-went-of-the-rails/
======
camjohnson26
What’s frustrating about this situation is how obviously wrong the experiments
were as described in the book, and yet how unshakably the findings were
defended by Kahneman and others. I read the book before the crisis and stopped
reading after the priming chapter, it was clearly fantasy, especially the
experiment about reading words related to old age and then walking more slowly
down a hallway. Now that the crisis is public knowledge it’s hard to believe
these findings were ever accepted by the mainstream, but they were.

You just have to think for yourself and weigh the evidence, be humble and open
to other perspectives. Mindlessly following perceived experts won’t save you.

~~~
0xddd
It seems to me that people put too much faith in statistics despite not having
a fundamental understanding of the methods. I'm sure in plenty of cases these
frivolous results get published due to academic fraud and the pressure of
publish or perish. But in other cases, it seems like researchers really think
"p<.05" translates to "your alternative hypothesis is irrefutably true" and
will believe their flimsy conclusions because, as far as they're concerned,
numbers don't lie. (If I remember correctly, when Brian Wansink was exposed
for p-hacking, it was partly because he wrote a blog post openly describing
his sketchy research methods, where he seemed totally unaware that he was
churning out illegitimate results.)

~~~
anitil
I've been trying to find this blog post, any chance you can remember more of
it to help my googling?

~~~
0xddd
I managed to dig it up:
[https://web.archive.org/web/20170312041524/http:/www.brianwa...](https://web.archive.org/web/20170312041524/http:/www.brianwansink.com/phd-
advice/the-grad-student-who-never-said-no). It really is incredible to read
Wansink's clueless non-rebuttals to the comments that point out his academic
fraud.

~~~
lowdose
Staggering replies from him on the negative comments on his methods and
thinking, what a character.

What a gem to read, absurt funny title "The Grad Student Who Never Said "No".

And the fact that at least 54 times people took a snapshot of the webpage.

We should print this on toilet paper and sell it to universities.

------
c3534l
It's really weird when you hear people saying things like:

> disbelief is not an option. The results are not made up, nor are they
> statistical flukes. You have no choice but to accept that the major
> conclusions of these studies are true.

It's as if they know they're wrong, but are trying to force themselves to
believe it any way.

~~~
0xddd
I think that is quite harsh on Kahneman, especially considering how reasonable
he sounds in the open letter that is linked. But I am also completely
perplexed as to how he was so confident about these priming studies given the
research on cognitive biases that he's so well-known for (as mentioned in the
article).

~~~
aabhay
A great deal of his fame is premised on this priming effect. It’s been taken
almost as gospel for decades, to the point that a typical applied psychology
paper will actually _include_ a priming step even if unnecessary just to
spruce up the methodology. E.g. “subjects were primed by having them read
articles on spicy food” for a study on water fountains, etc.

~~~
0xddd
Oh, wow. Is any of his research still taken seriously? For whatever reason, I
was under the impression his work on researcher bias like "the law of small
numbers" was quite influential and still cited here and there by trustworthy
figures.

------
Vinnl
The selection effect is also easily explained. 'Top' journals have as explicit
policy not to publish the most robust research, but the research they think
will have most impact in the field. (Hence 'impact factor'.)

That's why with the project I volunteer for (Plaudit.pub) researchers
recommending research have to explicitly indicate whether they think a work is
robust and/or exciting, where neither block publication nor the
recommendation.

------
thu2111
Something I've been reflecting on over the past couple of months is how the
university system loudly claims to promote critical thinking and peer review,
whilst it seems to me that it actually does everything possible to discourage
it.

If you look at the unfolding replication disaster in epidemiology, re:
Ferguson and his totally non-replicable Report 9, it's really turning over the
rocks on an academic culture in which:

• Academics flat-out refuse to question each other across disciplines,
practically as a matter of honour.

• Nobody else being able to understand your research is something boasted
about on Twitter and in the press rather than being a source of embarrassment.

• Scientists routinely refuse to reveal their data and code, making a mockery
of both peer review and replication as a process.

• Anyone who _does_ engage in critical thinking about their work gets shut
down, censored, attacked as an "armchair epidemiologist" etc - often it seems
by people with academic backgrounds.

After years of non-stop hiring, tech firms are now flooded with naive post-
docs steeped in academic culture, who appear to have totally bought into the
notion that critical thinking about the output of "experts" is not only
morally wrong but outright dangerous and needs to be suppressed. This has led
to the absurd sight of Google/Twitter trying to delete anyone who disagrees
with the WHO at a time when the WHO routinely disagrees with itself as of a
few weeks ago, or where the WHO disgrees with the CDC, or where experienced
doctors disagree with both, and it ends up being the _doctors_ that get shut
down because they are mere doctors and not "experts"!

It feels like this whole conflation of academia with "expertise" is leading
society to disaster; why should any of us believe universities when they say
they teach critical thinking? Someone trained in critical thinking wouldn't
just accept such a claim at face value, they'd demand a weight of evidence.
But no such evidence is ever presented.

~~~
pjc50
This situation has been exacerbated by the "barbarians at the gates": stepping
outside the zone of people who are trying to do inquiry honestly, however
badly, leads rapidly into the swirling internet drain of quacks and cranks who
feel under no such obligation to try. Hence all the "miracle mineral solution"
nonsense.

The landmark of bad epidemiology is of course Andrew Wakefield, who almost
single-handedly destroyed the effectiveness of measles vaccination.

Experts are in trouble. Some of the "non-experts" are right. But as often as
not it's the confidently extremely wrong who get promoted by the media and
social networks.

~~~
thu2111
Perhaps. I used to view things like that too.

Over time it seems like the gap between the cranks and experts keeps closing.
How cranky are these cranks, really?

You mention vaccination. Anti-vaxxer has pretty much become a byword for
dangerous quack science denier. It conjures up stereotypes of soccer moms who
are certain _their_ special snowflake child can't take the "risk". But
vaccines are perfectly safe; doesn't everyone know that?

A few weeks ago I got a reply to a comment of mine here on Hacker News that
mentioned Pandemrix, a treatment I'd never heard of:

[https://news.ycombinator.com/item?id=22922864](https://news.ycombinator.com/item?id=22922864)

DanBC, who I often end up debating with on these forums as we are somewhat
different in political outlook, remarked _" Pandemrix causes narcolepsy in 1
in 60,000 people who take it. (so, for the entire US that's about 5000 with a
life long debilitating illness that requires constant care.)"_

At the time I replied that I'd not heard of this and without doing more
research couldn't really comment. I took his claim at face value and didn't
think much more of it, filing it away to look at later. Well, turns out this
claim is both completely true and much more importantly, to my surprise
Pandemrix is a vaccine developed for the swine flu:

 _" In August 2010, The Swedish Medical Products Agency (MPA) and The Finnish
National Institute for Health and Welfare (THL) launched investigations
regarding the development of narcolepsy as a possible side effect to Pandemrix
flu vaccination in children,[5] and found a 6.6-fold increased risk among
children and youths, resulting in 3.6 additional cases of narcolepsy per
100,000 vaccinated subjects"_

[https://en.wikipedia.org/wiki/Pandemrix](https://en.wikipedia.org/wiki/Pandemrix)

So, erm, yeah. About those anti-vaxxers. Who think vaccines might sometimes be
dangerous for their children. And who get laughed at and shat on constantly.
And shut down. And censored. And told they don't understand the science. About
them.

Damn.

~~~
0xddd
This is a coarse mis-characterization of both sides of that "debate" if we can
even call it that. Are anti-vaxxers mostly worried about proven risks, like
narcolepsy in this case, or is a much larger part of their movement concerned
with making paranoid claims like that the MMR vaccine causes autism? Are the
experts guilty of denying the narcolepsy-Pandermix link, or are they spending
more time trying to refute bogus statements because they'd like to see herd
immunity maintained for certain diseases?

~~~
thu2111
I don't know what they think. How would I? Do you think the anti-vaxxer
viewpoint is covered by the media, do we ever discuss it here on HN, is it
something that's a part of mainstream debate? No, it isn't.

These people never really get taken seriously regardless of what claims
they're making. The moment a position starts being labelled
quack/crank/dangerous misinformation, you're already at the point of "coarse"
characterisations, or mis-characterisations.

So now look what's happened. Apparently there are two kinds of anti-vaxxer,
the ones worried about "proven risks" and then the _real_ cranks making
"paranoid claims". But the sort of anti-vaxxers who would have been refusing
to give their child Pandemrix _before_ it was understood to be linked with
higher narcolepsy rates would have come across as paranoid, wouldn't they?
They'd have had to say something like:

"I don't know. It's a new vaccine. It might be dangerous. I'm not sure swine
flu is dangerous, I'd rather not vaccinate my child."

and they'd have got an answer from us HN-reading types of the form:

"Swine flu is very serious, the WHO has declared a global pandemic and clinics
are being overwhelmed with infected people. Vaccines are well understood and
very safe. Experts say the MMR scare was just debunked crank science. If you
don't vaccinate your child you're an ignoramus who is putting both your child
and other people at risk, you shouldn't even really have a choice. I will
report your tweets to Twitter for misinformation."

And we'd have been wrong and they'd have been right.

[https://en.wikipedia.org/wiki/2009_swine_flu_pandemic#Histor...](https://en.wikipedia.org/wiki/2009_swine_flu_pandemic#History)

"The Mexican government closed most of Mexico City's public and private
facilities in an attempt to contain the spread of the virus; however, it
continued to spread globally, and clinics in some areas were overwhelmed by
infected people ... In late April, the World Health Organization (WHO)
declared its first ever "public health emergency of international concern," or
PHEIC,[38] and in June, the WHO and the U.S. CDC stopped counting cases and
declared the outbreak a pandemic"

~~~
rleigh
I disagree that the "anti-vaxxers" would have been "right".

They fail to trade off the risks appropriately.

No one can claim vaccines are "safe". They induce a host immune response so
that the immune system is primed to fight a subsequent infection. The immune
system can cause damage to the host, which is why we have a whole catalogue of
autoimmune disorders, many of which can be modulated by infectious diseases
and parasites. There is a risk a vaccine could induce a host immune response
which causes damage. But it's a small risk.

You have to trade off the risk of vaccination with the risk of being infected
with the disease, and the risk to society at large through spread of the
disease when vaccination rates are too low.

In pretty much all cases, the risk of the disease is many orders of magnitude
higher than the risk of the vaccine. If there's a 0.000001 chance of the
vaccine causing problems vs a 0.0001 of dying or suffering long-term damage
from the disease then the choice is obvious: get vaccinated.

Unless the "anti-vaxxers" are basing their arguments upon quantitative data,
then I'll continue to view them as crackpots with little knowledge or
understanding.

(I'm an immunologist, by the way.)

~~~
sifar
>> If there's a 0.000001 chance of the vaccine causing problems vs a 0.0001

The problem is how do I trust these numbers. That the vaccine risks aren't
underestimated and the disease damage over-estimated. Something that can be
confirmed only when enough time/data has elapsed.

The current pandemic is an example. Almost everyone who publishes data has
some motive to push a viewpoint (knowingly or unknowingly). It will take some
time before one can trust the data and its inferences. But then meanwhile, one
is being asked to trust the experts who are not open about the data and
methodologies.

------
jimmySixDOF
Big shout out to the "Reproducibility Project: Psychology" [1] by Brian Nosek
and the Center for Open Science in 2015 that started this trend to shine light
on "establishment" "conclusions" in the social sciences. People are an awfully
tricky subject of study.

[1]
[https://en.m.wikipedia.org/wiki/Reproducibility_Project](https://en.m.wikipedia.org/wiki/Reproducibility_Project)

