
What happens when you try to publish a failure to replicate in 2015/2016 - brianchu
http://xeniaschmalz.blogspot.com/2016/06/what-happens-when-you-try-to-publish.html
======
tehwalrus
The paper was a meta-analysis of nine unpublished studies. While the issue of
non-publication of negative results is extremely important, this should not be
the test case for it.

I agree with other comments that the lack of nul-results in journals is a
massive showstopper for epistemology and confidence in scientific work. In my
experience during my PhD, academics were unwilling to submit nuls, they wanted
to find the actual answer and publish that instead - which leads to delays of
years, leaving well known and obvious errors in existing papers unchallenged
in public, and potentially leading to that scientist _never even submitting
the nul result._

~~~
stdbrouw
> The paper was a meta-analysis of nine unpublished studies. While the issue
> of non-publication of negative results is extremely important, this should
> not be the test case for it.

I think they probably made a mistake in calling it a meta-analysis, because
indeed a meta-analysis relies on the idea that the studies themselves have
already been peer reviewed and as such don't need to be questioned
individually. Here, however, we're dealing with eight attempts at replication
by the author that have not previously been published, and as such those
studies _should_ be discussed in detail so we can assess the merits of the
individual replication attempts. Any meta-analysis of those experiments is
then really just the cherry on top.

~~~
cwingrav
> the studies themselves have already been peer reviewed and as such don't
> need to be questioned individually

Just to add a bit to your comment: Peer review is a community's way of saying
a paper stands up to their best understanding of methods at the time. The
author could have accidentally left something out, misrepresented their
approach or fabricated things. Peer review would not catch that, but
replication or failure to replicate would. They are two different things.

For example, I remember one experiment could never be replicated by anyone
except the author and his students. So, it got a bit heated. Ultimately, they
found it was an experimental setup, not reported in the paper because they
didn't think it was important, that allowed the experiment to work. So, it
passed peer review, failed replication, but ultimately, because of failed
replications and the academic process, further information came to light and
knowledge was created.

The academic process can be slow and frustrating. But, when applied properly,
it is our best approach to expanding knowledge. That isn't to say it can't be
improved, especially regarding failure to replicate and null-hypothesis
papers. Academics know this and are working on methods around this, methods
called out in these discussions.

Some of the most innovative people i've found, seeking ways to improve this
process have been NSF and NIH officials. So, it's promising, but slow.

~~~
mattkrause
I think you're putting a little too much weight on peer review here. The way
it is presented to non-academics often makes it sound more like a trial or
inquest, where a large group of people carefully weigh the evidence.

In practice, peer review means that 1-3 people each spent 1-3 hours thinking
about the paper and didn't find anything horribly wrong with it. It is more
like a code review--it's good if it catches something, but not terribly
surprising if some bugs slip through.

You're definitely right about the the importance of continuing to discuss and
replicate work after it has been published.

~~~
ska
The level of rigor in peer review is quite variable across disciplines and
even specialties. This is a cultural thing.

The code review parallel is pretty good (and also captures that, code review
is very variable also)

~~~
mattkrause
Sure. Peer reviews in pure math are often exceedingly careful, or so I am told
(not in the field).

Still, I think it's a little much to say that it's "a community's way of
saying a paper stands up." At best, it's the opinion of handful of people from
that community.

------
faux_intellect
At this point, I think Google Scholar should step in and just put a
replications section beside every scientific publication. People should be
able to quickly and easily know how many times a study has been attempted to
replicate and, of those attempted, how many times it has actually successfully
replicated.

It's unfortunate that replications aren't taken more seriously these days, but
it also doesn't help that, when there are actual replications, you have to
scour the internet for them rather than having them readily available to you.

~~~
kriro
I think that's a pretty great idea actually. If anyone here has the reach to
get this on their table please do reach out. It would be even greater if
they'd use some replication metric in their ranking algorithm for the papers.

The danger however is that this would just lead to only me-too studies being
accepted and failed replications still being rejected.

I'd also love if they'd add (possibly Google hosted) repositories where the
data/scripts etc. that belong to the paper can be uploaded and archived for
all eternity.

~~~
mwambua
I saw something like this a while ago... Codalab
([http://codalab.org/](http://codalab.org/)). It's run by Percy Liang
([https://cs.stanford.edu/~pliang/](https://cs.stanford.edu/~pliang/)). It
lets you create executable worksheets to go with your papers; and people can
validate both the code and data used in your experiments.

~~~
chronic81
[http://gitxiv.com](http://gitxiv.com) is far superior than Codalab. Source: I
go to Stanford.

~~~
kriro
Looks pretty cool indeed, bookmarked. However I believe it is focused on
Computer Science. Nothing wrong with that but I think they key is convincing
scientists that are not exposed to ideas like version control, open source
(and I'd argue data sharing due to the rise of data mining) by default (or are
less computer savvy in general).

------
jknoepfler
The narrative would be more persuasive if it incorporated a story of how the
paper evolved meaningfully in response to peer criticism. The question
lingering in my mind after reading this is whether and how the paper was
substantially revised (in light of reviewer feedback) between rejections. I'm
sure it was (it has to have been, right?), but we don't get that feeling from
the blog post. The author(s?) should have received a large amount of very good
feedback between rejections from well-meaning peers in their scientific
community. I don't recall reading about incorporating any of that feedback
into subsequent revisions of the paper. The term "meta-analysis" probably
should have been dropped after the first (pointed) rejection, for example, and
the paper should probably have been broken down into two or three smaller
papers rather than submitted as a 'meta-analysis' of unpublished work.

This is not to say that peer feedback wasn't taken seriously. I don't know
that at all. But if the goal is to persuade a skeptical audience that academic
publishing is broken, the author should articulate how they followed best
practices in response to rejection letters from peer-reviewed journals. The
alternative is to sound arrogant and self-defeating, which I'm sure was not
the intent!

------
schlipity
Forgive me for not being an academic, so maybe this question is moot.

Why isn't there a place that links to a given paper so that discussion about
the paper can be centralized? It could also contain links to papers that link
to that paper, among them would/could be the failure to replicate information,
adding to the discussion. And I don't really mean a topical "this is what's
new" site, I mean a historical "This is the paper, and this is what people
have said about it." sort of site.

This seems like a fairly elementary idea. The only seeming difficult bits I
see are:

a) Getting (legal?) access to these papers.

b) Dealing with a large number of papers (millions?).

c) Authenticating users to keep the discussion level high.

d) Moderating the discussion in a way that doesn't piss off academia
(impossible?).

e) Keeping the number of these sites (competition, if you will) low so that
the discussion is not fractured between them.

It would seem like one of the "Information wants to be free" sites that host
the papers that everyone shares with each other would be a great place to
start something like this.

~~~
jacobolus
In general the way academics have a “discussion about the paper” is by writing
another paper in response and citing the original.

So what you’re looking for is the citation graph, and you can find it e.g. on
Google scholar. Search for the paper you are interested in, and then click the
link which lists all the papers which cite that one.

You’ll probably need to be on a university campus or ask your anonymous
internet friends for help to get around paywalls.

~~~
semi-extrinsic
Yes, we write comment papers. But the process is very very slow, so many don't
bother.

I submitted a comment paper to a journal in March last year. The paper I was
commenting on was from November 2014. My comment got accepted, after two
rounds of peer review, in May this year. It will appear in the journal in July
some time.

~~~
sevensor
And that's fast by academic publishing standards. A flamewar that would take a
day in an internet forum can smoulder for decades in academia, while the
principals gather supporters and snipe at each other in conference sessions.

------
bandrami
There's a psychology journal[1] dedicated to only publishing null-hypothesis
results.

[1] [http://www.jasnh.com/about.html](http://www.jasnh.com/about.html)

~~~
argonaut
A journal dedicated to negative results is IMO not the solution here.
Academics must publish in reputable, prestigious journals in order to advance
their career (a publication in a non-prestigious journal basically counts for
nothing). A journal dedicated to negative results is never going to be a
prestigious, impactful, or widely-disseminated venue.

~~~
mcbits
But that's a self-fulfilling prophecy. If more people were publishing their
null results in such a journal, one of those journals would emerge as the most
prestigious null-result journals. It seems like an acceptable solution,
especially if the alternative is going to be not publishing at all and
increasing the positive result bias.

~~~
argonaut
Being the most prestigious null-result journal doesn't rise to being a
prestigious journal, though.

Regardless, this is the reality today. I don't think trying to boost a null-
result journal to the level of Nature or Science is a better way forwards than
pressuring Nature or Science (and similar caliber journals) to publish more
null result works.

~~~
mcbits
As far as career advancement goes, I don't work in academia, but I would be
more confident walking into an interview with a list of difficult and
impactful experiments finding no effects than having even one highly cited
paper in a prestigious journal that was later debunked by a couple of grad
students in a null-result journal. The latter prospect should also frighten
journal publishers into taking null results more seriously (especially if they
published the earlier work), but only if the threat is real.

~~~
Al-Khwarizmi
Most assessment processes in academia are purely administrative processes
where they count the number of cookies that you have earned. They don't take
the actual quality of the work into account.

In my country, it goes as far as having "objective" point scales like:
publication in ISI JCR-indexed journals, 1 point for Q4, 2 points for Q3, 3
points for Q2, 5 points for Q1. Publication in non-indexed journal, 0.1
points.

No one ever looks into whether the paper is good or crap, or whether the
author has managed to submit almost the same paper to five journals. In fact,
I know cases of honest people that _tried_ to look into that kind of thing,
but they didn't let them because it was against the published "objective"
scale.

I don't think the system is equally rotten in every country/institution, my
country is probably among the most extreme places in this respect, but anyway
this kind of assessment is the most common AFAICT, as the impact factor cult
is often denounced by international scientific societies.

~~~
argonaut
This does not reflect hiring practices at reputable US research universities.

------
arcticfox
So broken. I'm not involved in academia, so the most I can contribute are
upvotes here and there, and giving respect to those who push against the
current.

~~~
Waterluvian
There needs to be a journal called "Failures" that focuses exclusively on
failed experiments and unreproducible results. I'm far more interested in
learning why something may have gone wrong.

Maybe it's just me but I feel it would surprise many people with its
popularity.

~~~
mattkrause
I'm not sure this would work as a journal.

Who would read this? I skim Nature, Science, and some journals in my
particular subfield every week/month because I am sure I will find something
relevant to my research or generally interesting. With the exception of some
hilarious or spectacular ones, I don't think _Failures_ would be so engaging,
so the only way I would find an article in _Failures_ is if I specifically set
out to search for it.

I might do that (though searching by technique is hard), but even if I found
something, read it, and decided it was helpful, what would I do with it? For
better or worse, citations are the currency of academia and academic
publishing. Where would I cite a paper that specifically _warned me away_ from
a possible project? "Dear Nature, I was going to study XZY via ABC, but [1]
indicates that's a dead end."

So, with no regular readers and no citations, _Failures_ becomes more like a
database than a journal. There currently aren't any good incentives to submit
things to databases, so Failures becomes a....failure.

~~~
startling
"There has been some promising research with alternative technique X, but that
has shown to be unreliable. [1] Because of this and [other factors], we have
decided on technique Y."

------
skosuri
1\. There are many places this could have been published without an importance
review, eg PLoS ONE.

2\. I think anyone interested in the replication problem needs to read this
piece [1] by Peter Walter. As he put it: "It is much harder to replicate than
to declare failure.".

[1] [http://www.ascb.org/on-reproducibility-and-
clocks/](http://www.ascb.org/on-reproducibility-and-clocks/)

~~~
mattkrause
Alternately, it would have worked really well as a "Registered Report."

 _Cortex_ now offers a format where the paper is submitted in two stages. One
first submits the introduction/background, methods, and proposed analysis,
__before __collecting data. If the reviews are favorable, the paper is
accepted, regardless of how the results turn out. There is a second round of
reviews for the final manuscript to assess the quality of the data and ensure
that the pre-determined analysis plan was followed. More
here:[http://cdn.elsevier.com/promis_misc/PROMIS%20pub_idt_CORTEX%...](http://cdn.elsevier.com/promis_misc/PROMIS%20pub_idt_CORTEX%20Guidelines_RR_29_04_2013.pdf)

I've had a hard time selling collaborators on it ("TWO ROUNDS OF REVIEW?!
Isn't one slow and painful enough?!"), but I really like this idea. It rewards
interesting hypotheses and good experimental design, but avoids the ask-a-
stupid-question-get-a-stupid-answer issue with null results.

------
danbmil99
Seems to me this issue is getting to the point where it could become an
existential threat to the credibility of science in general. Note how climate-
change deniers have recently used these sorts of arguments to challenge the
consensus - is it really so far-fetched to argue that perhaps climate
scientists are as biased as researchers in areas such as medicine and
linguistics?

The pay-wall, blind peer review process seems broken beyond repair. There
needs to be a better, robust method to publish every relevant study that is
not utter crank, and get some sort of crowd-sourced consensus from researchers
with credible reputations.

~~~
robotresearcher
Every few days I read that science is broken or losing credibility. And yet
somehow the torrent of new technology, medicine and understanding of the world
just keeps coming, and often makes my life better. So maybe not completely
broken.

CRISPR, Higgs' boson, gravity waves, deep networks, self-driving cars. Not
broken.

~~~
praptak
It's not about technology. Bullshit science affects policies. Also you don't
see what did _not_ get discovered/invented because of people having wasted
time trying to build on bad theories.

Also, nobody's saying it's _completely_ broken.

------
mcguire
This seems to be the original work in question:
[https://www.researchgate.net/publication/8098564_Reading_Acq...](https://www.researchgate.net/publication/8098564_Reading_Acquisition_Developmental_Dyslexia_and_Skilled_Reading_Across_Languages_A_Psycholinguistic_Grain_Size_Theory)

------
LanceH
There should be a failure to replicate journal. The standards committee would
should be all about rigor so that just getting published there would be a
demonstration of technique and ability if not headlines.

------
harry8
Any journal that refuses to publish a failure to replicate research they
originally published without proper reasoning should be closed down. That
journal should have such a reputational black mark next to it that nobody
would want to publish there and anyone who already had should be at the door
with pitchforks and torchers for tarnishing the researchers' reputations.

If it's was important enough to publish research saying "here's something"
then it's important enough to publish properly done research showing
"actually, probably it's nothing." By definition. Or it's not science, it's f
__king marketing and the journal should be treated with the same scientific
reverence we reserve for pepsi-cola advertisements from the 1990s.

~~~
Someone
I somewhat disagree. They still should have the right/plight to vet the
failure to replicate on quality. Without that, everyone and his dog would have
publications in Nature and Science tomorrow.

~~~
TJSomething
I believe objections on the grounds of low quality would count as "proper
reasoning."

------
guard-of-terra
Some of those reviews are good materials for
[http://shitmyreviewerssay.tumblr.com/](http://shitmyreviewerssay.tumblr.com/)

------
cpncrunch
PLoS one specifically says they will publish "Studies reporting negative
results".

~~~
apathy
Or save yourself the APC and put it on arXiv. Even Eisen (who was one of the
founders) suggests this.

~~~
cpncrunch
Its not peer reviewed so wont be found by researchers searching in pubmed or
whatever. Thats the main point of publishing...so reviews will find the
negative data.

------
apathy
Put it on arXiv or f1000 for fucks sake. Who actually believes psychology
papers anyways? The vast majority are fishing expeditions as best i can tell.

When the field starts enforcing minimal standards (as expected for, say,
clinical trials, or even genetics studies nowadays) maybe someone will give a
shit. Until then people like this guy who actually seek the truth will be
ostracized.

~~~
jmpeax
arXiv is not a peer reviewed journal.

~~~
lorenzhs
That is correct, and the author did make their research available online (on
researchgate and OSF). The arXiv simply isn't the right preprint server for
this, as its range of subjects doesn't cover the author's: "Open access to
1,160,864 e-prints in Physics, Mathematics, Computer Science, Quantitative
Biology, Quantitative Finance and Statistics".

For all the hate that peer review gets on HN, it plays a significant role in
science and it's important to have a quality screening for papers. Preprint
servers and peer-reviewed conferences and journals work best in tandem. The
way it works here is that when we submit something to a conference, we also
submit a technical report to arXiv, which we update to incorporate the
reviewers' feedback. The "complete" and preferred version is usually the one
on arXiv (no length restriction, so you can actually explain and prove stuff).
Conferences are much more than just publication venues, though: lots of
collaborations start there, and interesting discussions can be had.

~~~
apathy
The entire replication chain is a statistics exercise. But whatever -- your
point is sound, preprints should complement peer review. It's only Elsevuer
and a few others that try to prevent this. I guess we can hope that fat old
professors die off & younger academics decline to edit for journals that
impede progress. It works great in physics.

------
arviewer
There should be a Nulled Science Magazine!

~~~
TorKlingberg
There are dedicated journals for negative results actually.

* [https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in...](https://en.wikipedia.org/wiki/Journal_of_Negative_Results_in_Biomedicine)

* [http://www.pnrjournal.com/](http://www.pnrjournal.com/)

* [http://www.jinr.org/](http://www.jinr.org/)

* [http://www.psychfiledrawer.org/journal_of_negative_results.p...](http://www.psychfiledrawer.org/journal_of_negative_results.php)

