
How statistically significant correlations are not always significant - fogus
http://neuroskeptic.blogspot.com/2009/02/voodoo-correlations-in-fmri-whose.html
======
jordanb
I recently read the book "Common Errors in Statistics." It was really
illuminating. They spent a particularly long time discussing errors in various
medical studies, although I think that's because the authors' backgrounds are
in that area rather than medicine having particularly bad statistics.

To attempt to summarize the book, it seems that statistical errors tend to
boil down to one of three problems:

* Improperly building the experiment, failing to ensure independence or account for confounding variables.

* Misunderstanding significance. Reading too much meaning into p-values, and performing post-hoc analysis. Remember that a 95 percent significance level means that in 5 percent of the trials, you'll get a positive result by random chance.

* Improperly using tests, by failing to account for their limitations. For instance, using a test that assumes normality when you don't have a strong case that the data is normal. Or using a test that requires a certain sample size on a dataset that is too small.

When you think about it, most scientists receive a few semesters of statistics
strung out through their education. They learn a little about probability,
then how to do a few significance tests and something here and there about
regression. They then start doing work where in pretty much every field
nowadays, publishing a paper involves statistical analysis at some point. And
most of them grope their way through it based on hazy memories of introductory
stats classes taken years before.

------
otakucode
I don't like the title of this post, just because they're really pointing out
that the correlations are actually not statistically significant, they're in
error. But, beyond that, it's a very good article with some very important
information contained within. We are approaching a point where neuroscience is
getting good enough that we will finally be able to abandon the non-objective
"social" sciences for actual science, and have a much better chance of
approaching the truth.

I fear that the main point of the paper discussed might end up being largely
ignored, however. This would certainly not be the first time where it has been
shown that a certain level of 'accepted' evidence is simply not objectively
true, but a large portion of the community continues to rely on it simply
because finding the truth is difficult. The real problem comes when this
sloppy research is used to drive public policy. Law enforcement is chomping at
the bit to use fMRIs as lie detectors, for example. They have consistently
ignored the scientific invalidity of polygraph examinations, of voice
modulation tests, and I expect no better from them with regards to fMRI use.

------
tel
Very clean writeup on Vul's paper.

It's also a fantastic real world example of "lying with statistics" that is
subtle and easily overlooked. The simple presentation of a subtle (and
possibly misguided) descriptive statistic can easily mislead a casual or
biased reader.

