
Software faults raise questions about the validity of brain studies - ingve
http://arstechnica.com/science/2016/07/algorithms-used-to-study-brain-activity-may-be-exaggerating-results/
======
Toenex
I spent a few years in the late 90's and early 00's working on fMRI and I've
been stunned by many of the claims that appear to arise from it's use. What's
being measured relates to blood flow changes that correlate with an input
block paradigm, typically from a visual stimulus. This isn't brain activity in
the electrical sense but rather which areas of the brain appear to regulate
their blood flow in relation to the stimulus. The assumption being that these
areas require more 'fuel' to support their increased neural activity in
response to the stimulus. Or as I like to think of it, attempting to estimate
your home electricity usage by monitoring your water bill. Doing so with MRI,
a technology I've spent most of my adult life analysing the data from and
still can best describe what it does as 'magical', adding another not
insignificant level of complexity.

I'm know there is good science going on in this field, by people that
understand the limitations of the techniques and the technologies. However, I
worked with psychiatrists - a clinical discipline starved of quantitative
measurements until fMRI - that would happily ignore statistically significant
activation in the air around the subjects head whilst laying claim as to the
importance of those in the frontal cortex. Seeing the visual cortex 'light up'
in response to flashing chequerboards is one thing, isolating those areas of
the brain responsible for 'forgiveness' is something quite different.

Of course the software has bugs, I personally know people who wrote the
package in question and they are extremely smart and also very human. I doubt
I've ever published a paper using software that wasn't bug ridden. That's why
open source is such an important part of the process, laying bare every last
detail of what was done and not just what felt worth mentioning in the paper.
The medical imaging research community is particularly good at this with most
of the industry software making source code available. The problem with fMRI
is not the software.

~~~
amelius
I guess that if we're measuring bloodflow instead of neuron activity, then
perhaps our knowledge of _where_ in the brain something is happening might be
off, but the results are not per se invalid. Things might only get tricky if
more than one vessel feeds the same neuron.

~~~
gaze
Each blood vessel feeds hundreds of thousands of neurons.

~~~
amelius
That was not the point. Say you have a vessel A, and a vessel B, both feeding
hundreds of thousands of neurons. Assume A and B both feed neuron X (one of
those many neurons). Then A and/or B might fire up in a scan when X is active.
The "and/or" is the tricky part.

------
jcrites
I've recently made the argument that science publications should include code
and data. See comment history e.g.
[https://news.ycombinator.com/item?id=11606278](https://news.ycombinator.com/item?id=11606278)

Glad to see that concrete good is coming from such efforts:

> The researchers took advantage of a recent trend toward making data open for
> anyone to use or analyze. They were able to download hundreds of fMRI scans
> used in other studies to perform their analysis.

The more data and code that's made openly available, the stronger science will
be. I hope that we can work toward a future where most if not all code and
data are expected socially and by funding policies to be included in the
publication process.

~~~
ThePhysicist
Agree 100 %. The data of publicly funded studies should be available online so
that people can run their own analyses on it. There could be a non-disclosure
period of course to give the scientists that have done the original work an
advantage when publishing their results.

I'm confident that we will see cloud-based systems for scientific data
analysis in the near future though, especially for data which is uniform and
for which good standards exists. For some areas like genomics we already see
this, and I'm sure others will follow suit.

I also predict that being able to try out analysis algorithms on peta- or
exabyte datasets will be a huge game changer for many areas of science, and
will invalidate many current findings that are based on smaller datasets,
while hopefully producing many new ones as well.

------
cafebeen
Despite the title, software was not at fault here. Rather, the paper found a
higher than expected false positive rate due to a choice in statistical
modeling. The conclusion was that nonparametric tests can avoid the issue.

So, none of the results are truly invalid in the way of a hypothetical
software bug, since we know what modeling assumptions were used in the
previous studies. Personally, I don't see a major problem, since most fMRI
papers are exploratory, and we should be reproducing the major findings
anyway. We should certainly start using nonparametric tests from here on out
though!

------
efoto
A 15 year old bug in a basic package used by fMRI software was, according to
the researchers, causing numerous false positives in previously published
studies. Which means some of the accepted fMRI based brain studies results are
dubious.

~~~
zbjornson
No, that was a side note about a bug that caused a ~10% increase in false
positives. The real primary issue is that the clustering methods assume
Gaussian autocorrelation distributions, which don't exist in reality. They
estimated a 70% false-positive rate because of this

[http://m.pnas.org/content/early/2016/06/27/1602413113](http://m.pnas.org/content/early/2016/06/27/1602413113)

------
nabla9
Related classic: Neural correlates of interspecies perspective taking in the
post-mortem atlantic salmon: an argument for proper multiple comparisons
correction
[http://pages.vassar.edu/abigailbaird/files/2014/06/bennett_s...](http://pages.vassar.edu/abigailbaird/files/2014/06/bennett_salmon.pdf)

[http://blogs.scientificamerican.com/scicurious-
brain/ignobel...](http://blogs.scientificamerican.com/scicurious-
brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/)

[http://scan.oxfordjournals.org/content/4/4/417.full](http://scan.oxfordjournals.org/content/4/4/417.full)

------
dzdt
There is a real problem with incentives in science strongly encouraging bad
science. It is a publish or perish world, so every researcher needs to be
finding some (apparantly) statistically significant result on a regular basis.
Quantity is rewarded more than quality. Being super careful with the stats
makes it much harder to get publishable results but doesn't significantly
increase the rewards (i.e. academic credit). So the culture tends to sloppy
stats making garbage or weak results look strong. And there is no academic
credit for being open with data, and more of a risk that someone will
discredit your results if you are open, so that is rare as well.

------
cbebdhhd
There really ought to be some sort of PE-equivalent certification for mission
critical software, where they're required to sign off on the code before it
can be legally distributed. It seems like the majority of today's catastrophic
failures are due to faulty software and currently there's no accountability
when it happens. If engineers were routinely crashing aircraft and breaking
MRI's they would be going to prison because of the accountability procedures
that are in place.

------
cs2818
I am not sure if this would really be much of a shock to researchers who have
spent time working with fMRI data.

I'm a computer science grad student, but spend at least half of my time taking
courses and reading literature from the fields of psychology and neuroscience.
Brain imaging studies have become very popular within many subfields of
psychology, yet often the published analyses I encounter make inferences which
are not necessarily well supported by the data.

It is encouraging that more researchers are making their raw data available.
Currently many academics treat fMRI studies as some kind of infallible truth,
when in reality it would be wise to give more consideration to the many
sources of error that contribute to conclusions reached from imaging data.
Hopefully the availability of raw data as a supplement to publications will
help us gain more complete understandings.

------
raverbashing
So how many neuroscience papers can now be used as a scratch pad?

------
sjg007
I am surprised it took this long to discover this..

