
Risk of false positives in fMRI of post-mortem Atlantic salmon (2010) [pdf] - zitterbewegung
https://www.dropbox.com/s/o7gb13j26v9f4dp/Bennett-Salmon-2010.pdf?dl=0
======
wakkaflokka
I was doing fMRI work around the time this paper was published. It astonished
me that people would simply set an uncorrected voxel-level threshold and call
it a day. No FWE-correction, no cluster-threshold - just an 0.001 uncorrected
threshold. It was sad that this paper needed to be published to get
researchers to start paying attention to that.

I'll be honest - when the paper was published I was thinking "no shit - why do
we need a paper to tell us what we all learned in stats 101 about multiple
comparisons??" And then realized the quantity of fMRI papers that used
uncorrected thresholds.

Very similar feeling when the "Voodoo Correlations" paper came out. Except I
was admittedly guilty of having presented correlation coefficients from
clusters that had already been identified using thresholding. So that paper
really did make me take a closer look at some of my figures/conclusions.

~~~
SubiculumCode
Let us be a little bit fair to the researchers who adopted the p=.001 or
p=.0001 uncorrected approaches. Their approach wasn't completely unreasoned,
and was even justifiable at one time given available methods.

There were mainly two approaches to multiple comparison corrections:
Bonferroni and setting an uncorrected threshold. People here might say, well
yeah, use Bonferroni.

However, Bonferroni is really only appropriate when comparisons are
independent. Voxels (3D pixels) which are adjacent are highly dependent, and
indeed the brain is generally correlated. This dependency makes Bonferrnoi
correction (very) inappropriately conservative. Given the average dependence
of voxels, some researchers estimated that the average number of true
comparisons might be on the order of hundreds to a few thousand. In practice,
researchers corrected with Bonferroni, either found a really strong effect, or
reset using uncorrected threshold. Some reported results using both. People
who read the results interpreted results that way too. Bonferroni = reliable,
uncorrected = provisional

The contribution of the salmon study and other research papers is that they
truly demonstrated that the typical uncorrected thresholds in use were
insufficient to control false positives.

~~~
wakkaflokka
You're right. I definitely don't mean to sound like I was an enlightened
graduate student. Nothing ever passed FWE using Bonferroni, so we almost
always resorted to using uncorrected p-values with cluster thresholding, with
the cluster and voxel thresholds set from using alphasim (which gets the
probability of having a cluster of that size significant from a random
dataset, given the smoothness of your actual images).

If I recall correctly, all the major neuroimaging packages (AFNI, SPM, FSL)
had options for cluster-size thresholding at the time. Along with tools like
alpha sim to estimate cluster-level FDR (but I think that ultimately had
issues with it's algorithm, discovered only a few years later...).

I just remember thinking that if the salmon paper had a reasonable cluster-
threshold, none of the spurious voxels would have been considered in the final
analysis.

Granted, several years later, a paper came out suggesting that method would
inflate false positives
([http://www.pnas.org/content/113/28/7900.full](http://www.pnas.org/content/113/28/7900.full)).

I imagine the neuroimaging field, particularly the stats part, has changed
rapidly since I left.

~~~
SubiculumCode
Sorry, I was writing to HN more than responding to you particularly. It is
sometimes easy for non-scientists to underestimate scientists and think of
them as fools, when in fact the problems are frequently hard.

I believe that you are correct that about the time of the salmon poster there
were other methods available for multiple comparison correction. The work in
the early- to mid-2000's was much more "wild-west" however.

Indeed cluster correction may have its own issues, re your link. I think that
a good approach these days is to eschew whole-brain approaches for theory-
drive, a prior i ROIs, then supplement those analyses with a whole brain
exploratory analysis.

------
prefrontal
Hey all. I’m the author of the poster and subsequent paper. Happy to answer
any questions that you might have.

Glad to see this on Hacker News so many years after the poster first went up!

~~~
zitterbewegung
Do you have any other papers that would give me karma I mean that are
interesfing?

Also, which FMRI papers have real results that you think are significant?

~~~
neuromantik8086
Josh Carp also wrote a paper on methdological misbehavior in neuroimaging a
while back:

[https://www.ncbi.nlm.nih.gov/pubmed/22796459](https://www.ncbi.nlm.nih.gov/pubmed/22796459)

------
pmoriarty
_" Task: The task administered to the salmon involved completing an open-ended
mentalizing task. The salmon was shown a series of photographs depicting human
individuals in social situations with a specified emotional valence. The
salmon was asked to determine what emotion the individual in the photo must
have been experiencing."_

~~~
jxramos
I'm thinking
[https://en.wikipedia.org/wiki/Weekend_at_Bernie's](https://en.wikipedia.org/wiki/Weekend_at_Bernie's)
Salmon style!

~~~
aaron695
Slight spoiler, but if you liked this movie watch

Derren Brown - Pushed to the Edge. (It's very educational and fun)

~~~
Luc
'This is a true story' \- first scene of Fargo.

(it's not a true story)

~~~
js2
"A body is found in the frozen North Dakota woods. The cops say the dead
Japanese woman was looking for the $1m she saw buried in the film Fargo. But
the story didn't end there."

[https://www.theguardian.com/culture/2003/jun/06/artsfeatures...](https://www.theguardian.com/culture/2003/jun/06/artsfeatures1)

------
techdragon
This is and remains one of my all time favourite scientific research papers.

Understandable, obvious, and not just relevant, actually providing a
meaningful contribution to their field by virtue of quantifying existing
techniques as not sufficient to fully eliminate spurious data from the 'noise
floor' inherent in fMRI machine data.

------
loeg
I assume this was posted in response to the recent fMRI study from the front
page:
[https://news.ycombinator.com/item?id=15597068](https://news.ycombinator.com/item?id=15597068)

~~~
zitterbewegung
Yes you are correct :) it reminded me of that paper.

------
omginternets
Ahh such a classic :)

tl;dr: correcting for multiple comparisons is important.

------
rav
The linked Dropbox page appears to be password-protected. Here's the original
poster: [http://prefrontal.org/files/posters/Bennett-
Salmon-2009.pdf](http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf)

~~~
prefrontal
I think it was just asking you to register for Dropbox. If you close that
popup the PDF is already open behind it.

Here's a link from my server that should just work:
[http://prefrontal.org/files/papers/Bennett-
Salmon-2010.pdf](http://prefrontal.org/files/papers/Bennett-Salmon-2010.pdf)

~~~
rav
Huh! When I click "Cancel", the PDF disappears and then there's just a
centered box saying "This file is password-protected. Bennett-Salmon-2010.pdf
· 0.94 MB". However, the Download link works just fine and lets me download
and read the PDF. Dropbox is weird.

