Hacker News new | comments | show | ask | jobs | submit login

Perhaps part of this is statistical errors and selection bias. Most studies are done with a 95% confidence interval, but only studies that reject the null hypothesis are published. If we assume only 20% of all experiments reject the null hypothesis, then we should expect ~25% of those to be fasle positives. Journals might also be biased to include more sensationalistic results, which are more likely to be statistical errors.

My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.




> My guess is that if you included all test results, that reject the null hypothesis or not, and are published or not, then the reproducibility rate would be closer to 90%-95%.

No; you need to bring in power, not just assume an alpha of 0.05 and a base-rate of real results of 20%. (Consider the recent neuroscience paper estimating the experiments average a power of like 0.3...)




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: