This is a really insightful article that contains a number of points that qualify the result and present lessons for any one interested in data science.
A few points that stand out to me:
!) The 11% number comes not from a randomized sample of cancer papers but from attempts made by AmGen to reproduce results that might be useful in drug development. This means that they made good effort to reproduce these results but there is also a strong selection bias that needs to be acknowledged.
2) They point out that using survival time as the measurement complicates things. I've done a lot of machine learning and statistics on cancer and medical data and, in my experience, this can not be overemphasized. and There are loads of confounding factors that contribute to survival time. I expect that big break throughs will come as we develop rigorous ways for measuring the behavior of a tumor (does it metastasize? how does it feed itself?) and use those as the targets of our regression. (Currently these thins are measured by human visual inspection if at all.)
3) They point out that the studies that were repeatable were the ones that were careful about using blinded controls, and eliminating investigator bias. This is basic stuff but easy to overlook. In terms applicable to a startup, a data scientist needs to me motivated to vet existing and proposed practices and identify flawed ones as much or more then they are motivated to maximize gain.