
Unknown Knowns - barry-cotter
https://thezvi.wordpress.com/2018/08/28/unknown-knowns/#comments
======
barry-cotter
It seems that scientists had a very good idea of which psychology studies
would and would not replicate, of a sample of high impact, highly cited
studies. Both a simple survey method and a prediction market identified the
studies that didn’t replicate though the prediction market was better at the
rank order and effect sizes. This suggests that which studies are bullshit is
widely distributed knowledge. It also very strongly suggests that something as
simple and cheap as surveys are powerful tools for making common knowledge
which results and experiments are probably crap. As individuals at the moment
the scientific community knows which experiments are crap. Surveys or
prediction markets would make it common knowledge, would ensure that everyone
knows everyone knows which results are crap.

~~~
Eridrus
So I took a version of that survey that told you how you were doing as you
went along myself to see how it would go. And the main component to predicting
correctly was really some form of occam's razor/causal reasoning, where the
studies that seemed to make more intuitive sense, or you could imagine a
causal mechanism, were the ones more likely to replicate. But this actually
doesn't say anything about the quality of the actual study, it just shows that
we should probably ask for more statistical rigor/power when the results are
unexpected, because we expect the base rate of surprising findings to be lower
in general.

