Hacker News new | comments | ask | show | jobs | submit login

Maybe conferences should have a "reproducibility" track for that purpose? Also, I don't know about other fields, but I'm pretty sure that in CS, if you just took a paper and tried to reproduce the results, you'll get rejected on the ground that you offer no original contribution; no original contribution => no publication => no funding.

I don't think reproducibility should necessarily mean literally repeating the same study over again.

Whether or not a paper is fully correct is less important that the further work it stimulates or informs-- either via more papers (impact factor) or via APPLICATIONS of the research.

In the example you're giving, to get accepted, the author doesn't need to do much more than extend the research in some small way, compare it to other research, or explore some aspect more fully. If someone is going to take the time to repeat something, they might as well go a little further.

If someone takes the time to apply research results they're going to, in a way, test the validity of the results. Maybe the author's experiences in the pharma world is very different, but I doubt that.

"I don't think reproducibility should necessarily mean literally repeating the same study over again."

Yes, it should. We have too many demonstrated instances of studies built on other studies that turned out to be flawed, but the second study, rather than showing the first one was flawed, was rationalized and massaged until it conformed to the first study because the first study already had the imprimatur of peer-reviewed correctness on it.

Only someone replicating the original study directly (give or take sample size or duration or other simple such changes) will have the guts and moral authority to stand up and say "I can't replicate this. The original study may be wrong."

But there should also be replication of the concepts of the study under a different paradigm, otherwise you might just be finding out effects of the particular procedure. An original study is rarely "wrong" (fraud is a serious problem, but not a typical one), it's that it's merely one data point, from one procedure, and a lot of the time the effect that can be replicated is "real," it's just a side effect of the process. Basically, there's way more replication that should be going on in science at many levels.

Yes. But replication is a starting point. It isn't just about fraud, it's also about error and failure to communicate the experiment. (It's also beyond time for paper journals to just contain the relevant summaries and conclusions for given studies and for "what fits in a journal" to cease being "what fits in publication". Authors ought to be able to publish as much stuff as they want with a paper, including full source code sets or data sets or whatever, with the paper limitations affecting only the top-level summary.)

So I work for a scientific journal (insert comments about how evil I am here) and we have much bigger problems with authors not providing full data than with us refusing it. It is almost always possible to upload a full data set and code to a paper; what we're working on now is standardizing formats as much as possible so it's easy to immediately see what equipment and reagents were used, where the experimental procedures came from, what data was gathered and when, and how exactly it was analyzed without grabbing tons of files in different formats. "As much information as authors want" is high for an excellent but small set of authors who are probably right that the journal should do more to facilitate hosting their data--for most authors, that's far less information than we actually need.

Yes, I do not mean to just criticize journals, authors also need to start realizing they need to produce that as well. 10-20ish pages of paper for the work of a dozen people for three years (and that for the good-to-great papers) is just not an acceptable payoff for the amount of money involved.

For CS, reproducing should be easy given the code and input data right? Other sciences' input isn't so easily shared

Ideally you should re-implement the algorithm based on the description in the paper to verify that the description of the algorithm is correct. You should also test with your own data to make sure that the algorithm works on all reasonable data and not only on some provided cherry picked data. If you can't get the expected results with your own implementation and your own data then the results aren't reproduced.

Yes. So being able to rerun with the same code and same inputs to get the same outputs is a lower bar. Many papers don't meet even that bar.

(Mostly because they don't publish code nor data; and academic code is often a horrible mess, and the code was mucked around with between different stages of running.)

Great points, thanks!

Rerunning code does not count as reproduction as it is not an independent test of what is being claimed. You do have the option of verifying the code against the goals of the study, though that is a review rather than a reproduction (which is still valuable.)

most cs papers i read (actually all) do -not- include code (or input data). which makes this an issue even in the case of cs.

Not necessarily. There are fields like complexity and computability wich are pureley mathematical in nature and have little to do with actual code, but much with the abstract theoretical fundamentals of code in general. And on the other hand you have fields like machine learning, where you can have an algorithm, wich you can implement (or have as code), but you don't nessessarily know the values of certain parameters, specific for your problem space.

I mean, if a paper gives an algorithm, proves its correctness, and you are convinced by the proofs, then you're done. I don't see how implementing the algorithm gives you more insight. I'm doing my master degree in computational geometry and most people in my lab don't even implement their algorithms. They just know they are correct from their proofs.

Because evaluating algorithms is not always that straight forward. For some algorithms runtime is hugely important, and I don't mean the asymptotic complexity but a hard benchmark of how much it can do in what time span. Stuff like a SLAM algorithm being O(n^2) is nice and all, but to compare it to other SLAM algorithms, I need hard numbers on what it can do in how many milliseconds.

Often, I find that authors don't publish their code. If they do publish their code, they rarely publish their code for their benchmarks.

Yeah sure but at that point, it's more optimization than algorithm design. Of course, with any algorithm, you always have the hidden constant that you must account for. Also, what I was saying does not apply to the entier CS field. It only applies when you try to design an algorithm for a problem that does not yet have an efficient algorithm. I don't have much experience but I don't think it is really hard to see the cost of the hidden constant in most algorithms.

“Beware of bugs in the above code; I have only proved it correct, not tried it.”

— Donald Knuth


Good luck getting the code. Abandon all hope if you want to make it run on a machine different from the laptop on which the original grad student implemented it.

Real issue is that the way science is funded is broken.

Broken assumes it was working well at some point. Nothing new here.

It worked pretty well back when scientific inquiry was funded by selling inventions or other work that the scientist did "as a day job." That was a long time ago, though.

This has worked and still works for technology, but is not how basic science research was done. There's never been "enough" money for sciences, nor they ever been allocated in a manner agreeable to all parties. For a long time, research was either a gentleman's hobby or a side project under auspices of unsuspecting donor. Lagrange has developed variational calculus while doing his artillery school tenure. Einstein's annus mirabilis happened at his stint in Swiss patent office.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact