
The Need for Reproducibility in Academic Research - bmahmood
http://blog.scienceexchange.com/2012/04/the-need-for-reproducibility-in-academic-research/
======
joe_the_user
This topic presents a serious dilemma.

One aspect of science that doesn't get much attention in this debate is the
role of the scientist as an ethical and idealistic actor; to be a scientist is
(or was) have a higher calling, to help humanity get closer to the truth. And
this is crucial to science itself because scientists need to be able to
_trust_ other scientists. And neither Everyone-watches-everyone-style trust
nor you-will-be-punished-harshly-if-caught trust works. You need I-do-it-
because-I-believe-in-it trust to make science work.

Now, the more that graduate students are made disposable, the more that
professors live in a ruthless, sink-or-swim environment and so-forth, the less
a scientist is likely to remain an idealist interested first and foremost in
discovering the truth and the less that crucial element of trust will remain.

The latest fad is "outsourcing science". If we want to make science less
broken, it seems like we should be going in the opposite direction.

~~~
yummyfajitas
Why do you believe that "everyone watches everyone" together with "you will be
punished if caught" won't work?

Your policy of "I do it because I believe in it" seems quite unstable - under
such a policy, it seems bad actors can come in and take advantage of things
with no mechanism to prevent it.

I'm not necessarily against the "make life nicer for scientists" policy you
seem to be advocating, I'm just not understanding your reasoning.

~~~
mechanical_fish
It's a question of efficiency.

Technically, science _can_ work if your colleagues are untrustworthy. This is
one of its big, famous features. Over the centuries, scientists have published
a great many howlers, ranging from honest mistakes to rushed procedures,
deliberate disinformation, and straight-up fraudulent data. These things get
caught, their perpetrators get punished to some extent, and science makes
progress. Eventually.

The problem is that "eventually" can be a really long time: Years or even
decades. (The Piltdown Man hoax wasn't exposed for forty years.) In the
meantime, bad science will confuse the analysis, corrupt the textbooks, and
injure the careers of a few unlucky grad students. It will waste a great deal
of time and money, perhaps that of the most prominent people in the field.

For example, when cold fusion hit in 1989 dozens of scientists dropped
everything for at least six months to try and replicate it. Millions of
dollars were spent. Obviously, while those folks were tinkering with cold
fusion they weren't tinkering with anything more interesting or useful.

We've made a lot of progress since Galileo, the frontier has moved a long way,
so it takes more than a couple of pendulums to replicate most modern
scientific papers. It could take half a decade, the entire productive career
of one or two grad students, an entire research grant, a lab full of
equipment, and the lives of two hundred mice just to replicate one paper. So
the mutual trust is essential for speed: You have to be able to gamble your
time on the results of other people's experiments with some hope of a positive
return [1], or the speed of science slows down to the speed of one person's
work. (Even _that_ could work - you can discover things even as a sole
practitioner - but it would be incredibly slow. Particularly because a
scientist working without good criticism _will_ make mistakes, lots of them.)

\---

[1] The return will never be 100%. One of the things that disturbed me as a
physicist switching to biology was that even the best biology papers are
inevitably _riddled_ with likely sources of error: The subject is just too
complex to control everything perfectly. There are, for example, systematic
sources of error that underlie entire fields, like the fact that most results
are tested either on one highly inbred species of lab animal or on lines of
human cells that have been selected to thrive well in dishes, and which are
therefore, at some level, unlike any cells seen in any living human. So,
science is inevitably a kind of gambling: Will you see consistent and useful
results from this particular corner of experiment space? If the thing kills
cancer in the dish, will it work in mice? if it works in mice, will it work in
humans? if it works for 10% of humans, will it work for 40%? You gamble and
you hope. You hope that you aren't wasting your grant, your career, or your
entire field. The good news is that we do tend to win, in the long run, but
anything that improves the odds on the bet is helpful.

~~~
joe_the_user
Thanks for the detailed explanation.

Maybe I've been too close to science for too long but the whole line of
argument seems so obvious to me that my reaction is "I don't know where to
begin" when someone implies science could be done through coercion.

~~~
mechanical_fish
Yeah, that is the way it works, isn't it? The idea of just straight-up lying
to your colleagues is unthinkable, directly analogous to releasing an open-
source library with a deliberate flaw in it, or loaning your fraternity
brother a bicycle with broken brakes and neglecting to tell him about it.

Of course, just because it makes no sense and is terrifyingly sociopathic
doesn't mean it doesn't happen. Among other things: Mental illness happens.
It's scary when it does.

------
alttag
I'm not in a medical field, but the problem likely exists for our discipline
as well.

The issue, I suspect, stems from the nature of publishing: top-tier journals
only publish "interesting" research, which means reproducing research is less
welcomed and if performed needs to be accompanied by a serious value-add.

There is no incentive to reproduce. It makes it more difficult to publish. It
doesn't lead to tenure. Why bother?

~~~
egis
From my experience as an undergraduate CS student this problem certainly does
exist.

I was once implementing an algorithm I've read about in a paper and could not
reproduce the published results. After lots of frustration I contacted the
author just to find out that the associated dataset he published on his
website was modified and is no longer suitable for testing of the original
research. He also said he has lost the original dataset, which meant I could
neither verify the correctness of my implementation, nor his published
results...

~~~
amcintyre
For some fields, it seems like the model should shift to "you _will_ publish
your source code and data alongside the writeup, else you won't be publishing
here."

Not doing so seems equivalent to publishing a paper about applications and
characteristics of a new mathematical proof without actually including the
proof.

~~~
garenp
Source code should absolutely be required. If you can't reproduce results, you
aren't doing science. I do not understand why publishers would even accept
papers while _knowing_ that there is code out there that would verify the
papers claims. The phrase _willfully ignorant_ comes to mind.

Not too long ago I emailed someone at UIUC about a tool they wrote which was
mentioned in a research paper I ran into online. I wanted to see if their
method really was much better than previous ones, and if the trade-offs they
made were worth it.

Did I get it? No. Instead, they sent me a link to some new company founded
based on the tool. I apparently had to be a "researcher" to receive the
magical tarball.

It also seemed to me to be a conflict of interest for a Professor to be
working for a University and company at the same time - all while selectively
choosing who can and cannot get access to their results.

------
jboggan
Coming from a computationally intensive discipline in academia it is
astounding how difficult it can be for researchers to reproduce their own
results. The tendency is to write enough code to generate an impressive
diagram for a journal illustration or presentation slide and move on. It's not
uncommon to not know what date or version of a constantly shifting public data
set the original result was generated from, or even where the scripts are
located 6 months down the road. I tied myself in knots trying to iron out data
bugs and irregularities that forced me to dump a year of research and recreate
the entire upstream data pipeline in my lab.

In another example a very promising cancer drug prediction algorithm (with
fascinating in vitro results tested by an affiliated lab) was abandoned
because of a key researcher's untimely death and the complete lack of version
control anywhere in the lab. The paper had already been published (thankfully)
but we literally had no idea where the code and the intermediary data were. We
had a ~5,000 node GPFS cluster with rolling backups but it didn't help at all
because all the development was done locally; the situation was the same
across the lab. The decision of the PI in the wake of this compound tragedy
was to have lab members pair up and "cross train" each other for an hour and
verbally tell them where they kept their important data.

Referring to the corrupted data issue I personally experienced, I
unfortunately discovered it the night before a multi-departmental research
presentation. There were numerous reversed edges in a large digraph due to
improper integration of two data sets before my involvement (I was also at
fault for trusting internal data). I told the PI about it in the morning since
the problem was so deep and said I couldn't present anything because every
single result of the past year was invalidated by the bug I had found. His
response: present anyway. I refused. That did not go over well.

I'd like to see every computational paper (especially in biology where these
methods end up influencing human clinical medicine) include all source code in
a public repository but it isn't going to happen. Labs would lose their edge
if they had to tell competitors what model weights they had iterated to in
creating their newest prediction algorithms and university technology transfer
departments would have greater difficulties patenting these methods and
selling them to drug companies. The current model will not change but a new
one might supplant it.

I wasn't on the cancer drug prediction project but I probably know enough
about it to reconstruct it. It actually seems like a great candidate for an
open source project.

------
ExpiredLink
The "the development of new therapies to treat disease" shouldn't be called
"Academic Research". It's part of the pharmaceutical industry.

~~~
bmahmood
Increasingly, pre-clinical studies to identify potential drug targets are
conducted in academic settings.

------
gtani
Good analysis of providing source code, datasets and potential burdens on
reviewers and authors.

[http://nlpers.blogspot.com/2011/03/some-thoughts-on-
suppleme...](http://nlpers.blogspot.com/2011/03/some-thoughts-on-
supplementary.html)

