
Rein in the Four Horsemen of Irreproducibility - bcaulfield
https://www.nature.com/articles/d41586-019-01307-2
======
bunderbunder
> Second, the growing use of meta-analyses, which combine results across
> studies, has started to make clear that the tendency not to publish negative
> results gives misleading impressions.

Mark Crislip, host of PusCast, has made a few good wisecracks on this subject.
Perhaps the most poetic is (paraphrasing here): "When you mix cow pie and
apple pie, you do not make the cow pie better. You make the apple pie worse."

~~~
theobon
Meta-analysis is a sound principle but undermined by selection bias. This is
why every meta-analysis includes selection criteria for which studies are
incorporated into the analysis. Unfortunately publication bias is systemic and
biases the results.

The quote referenced is poetic and pithy but I fail to understand how it
applies to meta-analysis and selection bias. What is the cow pie in this
scenario?

~~~
bunderbunder
The cow pie is biased work.

The root problem with using meta-analysis is that it just wasn't designed to
work with real-world science. It generally assumes that it's being applied to
an unbiased sample of unbiased results. It's now pretty well understood that
the published literature is really a biased sample of (oftentimes) biased
results. No amount of selection criteria can fix that; the best you can hope
for is that they will yield a biased sample of unbiased results.

I'm no expert on health science, I'm just taking potshots from the peanut
gallery, but I'd guess it's pretty much always better to ditch the shiny
mathematical bauble and its false promise of providing an easy, simple,
objective answer to an inherently subtle and complex problem, roll up your
sleeves, and get to work on a proper systematic review.

------
astazangasta
I dont think this will solve the problem. The problem results from the deep
barriers to proper study construction, especially in biomedical science, where
enrolling patients in a controlled study is often extremely challenging. This
means samples are acquired as available and not prospectively.

Does this mean we cant do "science" on such a dataset? Maybe we cant do the
"usual" science, of case vs. control => p.value. But a lot of domains of
science operate on uncontrolled data sets and make do with post-hoc
interpretation (geology, I'm looking at you).

We need different mathematical tools and epistemologies to confront these
datatsets; we cant just hope that everyone will suddenly start doing it
"right".

~~~
SiempreViernes
> But a lot of domains of science operate on uncontrolled data sets and make
> do with post-hoc interpretation (geology, I'm looking at you)

Bah, the real star of that sort of science is _Astronomy_!

