Hacker News new | comments | show | ask | jobs | submit login

Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar.

What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible?

If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not.

What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?




>Is it reproducible if the same scientist repeats the >experiment in the same lab and gets the same results? >Because that's the current bar of reproducibility, and that >89% that is not "reproducible" certainly passed that bar.

I don't follow you here. The above does not seem to be the current meaning of "reproducible":

http://en.wikipedia.org/wiki/Reproducibility

The same person doing the same experiment is repeatable, not reproducible. And I don't believe even the repeatable bar has been met, as very few projects have funding to do the same experiment twice.

The fact that a given investigator can "repeat" his experiment have very low weight among professional scientists, because we are all human. Irving Langmuir's famous talk about Pathological Science, and especially the sad story of N-rays, is a warning to every scientist.

http://en.wikipedia.org/wiki/Pathological_science

http://www.cs.princeton.edu/~ken/Langmuir/langB.htm#Nrays


So there is a theory that if something doesn't reproduce it's because the other guy was just incompetent, and that may be the case, but just like everyone wants to believe they're above average, everyone will want to go to the theory that the other guys just aren't any good, when I suspect that that will be much less of a factor. At any rate, when you start trying these drugs on the wide diversity of the patient population, if they're not super robust, they don't be of much use anyways.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: