
The Arsenic DNA paper exposes flaws in peer review - ilamont
http://www.michaeleisen.org/blog/?p=1439#sthash.y6DGan5d.dpuf
======
rubidium
``we need to get past the antiquated idea that the singular act of publication
– or publication in a particular journal – should signal for all eternity that
a paper is valid, let alone important''

Most scientists I know (physics) don't have this idea. They know publication
is just a step. Most grad students learn at some point that just because it's
published doesn't mean it's correct.

Publication is a checkpoint, not a finish line, for any theory or experimental
result.

~~~
jofer
Very true, but he does go on to state:

> If we had, instead, a system where the review process was transparent and
> persisted for the useful life of a work

That is a very interesting idea. (Reviews are currently confidential and
usually anonymous -- only the author and editor sees the reviews, and only the
editor knows who wrote them.)

Not only would this reduce blatantly petty reviews, it would also give some
recognition to the work that goes into reviewing a paper. It's currently an
important but very thankless job. It's considered polite to acknowledge
reviewers at the end of the paper (even if they remain anonymous), but it's
often not done.

~~~
arjunnarayan
I'm going to respond to this in the context of academic computer science
(which works off conferences), in particular, the systems subfield. YMMV if
you extrapolate this to non-CS science and journals, so take this with a grain
of salt.

The bottom line is I, as a powerless grad student, am _very_ comfortable with
the level of reviewing competence I have witnessed (and been a part of).

The proposal to deanonymize reviews has a huge flaw: Namely, there's no way I
(as a grad student with no political power) will write a critical review of
somebody famous if I know I won't remain anonymous.

The risk of blowback is too damn high. (And given the small pool, writing
style would probably be sufficient to deanonymize, if my entire trail of
reviews were public). There's no way I have the balls to write a critical
review of a paper by, say, Robert Morris or Larry Peterson, for some
prestigious venue such as SOSP, when I'm asked to write a review. Unless I
know for sure that there's absolutely no risk whatsoever of being identified.

Look. Reviewing is thankless and hard. Anonymity exists so that we can be
critical and not risk career suicide. Last year at OSDI I received 9 reviews,
and they were all exemplarily thoughtful, even the obligatory "strong reject"
one of them. The problem of "petty reviews" _has_ a solution: The program
committee is self policing, and the reviews are not anonymous within the PC.
(For example, at SOSP this year, there are 28 PC members.) You know that most
of your reviews come from this gang of 28. You might have one or two
externally solicited experts in addition, but that too is not anonymous _to
the PC members_. If there's a culture of meanness, the PC members and PC
chairs can fix this by internal policing (and they do).

~~~
jkimmel
Your point regarding anonymity is highly valid, and I appreciate that you
bring it up. There is actually a concept I'm rather fond of that seeks to
address this, usually referred to as peer-reviewed peer review [1].

Essentially, anonymous experts review publications in much the manner they do
now, with no risk of their identities being revealed. A _separate_ set of
review-reviewers then rates the publication-reviewers themselves, scoring them
on a standardized quantitative scale (1-10) based on the validity of their
claims and the depth of their assessment. In this manner, experts collect an
"aggregate reviewer rating" of sorts analogous to an h-index [2], call it an
R-index.

This R-index can then serve to select accurate reviewers for future
publications. A high R-index could also be included as a positive bullet point
on a CV, providing some reward for reviewers to put a high degree of effort
into their otherwise unrewarded reviewing work.

A few potential flaws are evident: what is one to do if early reviews are
scored poorly, and you are no longer able to provide new reviews to increase
your R-index? A grace period before scoring could potentially remedy this, ie)
an R-index is not assigned until a reviewer has written >10 reviews.

[1] -
[http://www.frontiersin.org/Computational_Neuroscience/10.338...](http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full)
[2] -
[http://en.wikipedia.org/wiki/H-index‎](http://en.wikipedia.org/wiki/H-index‎)

~~~
arjunnarayan
I like the R-index idea. It's basically Slashdot's meta-moderation system? I
always thought that had some promise. However, it does add even more _work_ to
a system that is extremely stressed (writing reviews is so much work! Imagine
having to read papers, read reviews and then review the review on top of
that!)

But basically existing program committees (in computer science) do serve that
meta-moderation purpose (without the empirical R-index metric). The entire PC
does get to see the deanonymized reviews of the other program committee
members. If someone's just being an unhelpful jerk, they're probably not going
to get invited back in the future.

------
suprgeek
This post takes the lead-in with the Felisa Wolfe-Simon "Arsenic DNA" paper.
In recent times I remember this was one of the most Hyped-up papers something
akin to the Cold fusion paper by Fleischmann and Pons.

The contrast in the public response to the two incidents is fascinating. While
Cold-fusion became a career-ender for anybody associated with it and
Fleischmann & Pons both suffered pretty much banishment, repeated experiments
have shown that there is something funny going on with that particular mix of
compounds (Heavy water, Palladium, Nickel etc) - excess heat.

With arsenic life however, follow-up experiments have proven that it was
contamination that was a cause of the result. So there is NO ARSENIC based
life. Yet the lead authors of pretty much a gigantic false claim -orchestrated
shamelessly by NASA- seem to be doing pretty well, dodging the basic questions
with that most famous catch-all - more work is needed (hence more money should
be sent my way) to prove the (bogus) claims made.

~~~
ars
> excess heat

There is no excess heat, it's measurement error caused by miscalculating how
much hydrogen recombines in the cell.

If you build a calorimeter that recombines all hydrogen you find no excess
(but it's really hard to do accurately).

~~~
leephillips
And, even more tellingly, there are no fusion neutrons emitted.

~~~
maaku
That's not very telling when what they were claiming was aneutronic fusion.

~~~
leephillips
The Pons / Fleischmann claims about their "cold fusion" experiment had nothing
to do with aneutronic fusion. In fact, the definitive debunking of their
claims consisted in the failure of other labs, and of P & F themselves, to
detect fusion byproducts, including the expected neutrons:
[http://partners.nytimes.com/library/national/science/050399s...](http://partners.nytimes.com/library/national/science/050399sci-
cold-fusion.html)

------
dekhn
His title is a red herring. he admits later he didn't actually do this. His
point is actually a rebuttal to John Bohannon's submission of a fake article
to public/open journals, demonstrating the Science Magazine also makes the
same poor errors in review as the open journals.

~~~
klmr
His title is the _hook_ , not the catch. So it can’t be a red herring.

~~~
dekhn
I actually knew that when I wrote that, somebody was gonna call me on it.

------
tgb
Warning! This title is a total lie! I know it's a lie done intentionally and
calculatingly, but most of the time most people don't read past the title and
will assume that it's true. I really think this needs to be edited! Even if it
'spoils' the article. Please do this, whoever can!

At the time of writing the posted title here is the same as the article title:
"I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at
subscription based journals"

------
eli
If you want to talk about honesty and integrity in publishing, you shouldn't
give your blog post a blatantly misleading linkbait title.

~~~
donohoe
True - but to make a very valid point about the state of peer review in
science journals it is perfect.

~~~
eli
You needed a demonstration to see that people can easily mislead each other on
the internet with bogus headlines?

~~~
donohoe
No, I was disagreeing with you that the post was about "honesty and integrity
in publishing".

------
badman_ting
The Last Psychiatrist talks in his/her blog sometimes about what a joke peer
review is. Here is one entry:
[http://thelastpsychiatrist.com/2011/01/wakefield_and_the_aut...](http://thelastpsychiatrist.com/2011/01/wakefield_and_the_autism_fraud.html)

~~~
judk
LP is so sarcastic and snarly that it is difficult to understand what they are
trying to say. Not an effective tone in an article about the problems of
misinformation.

------
justncase80
There should be an "Open Source" science journal. Where anyone can submit a
paper and anyone can peer review or freely read said articles. User accounts
could have a vetting process to verify their real world credentials and the
articles they review would be weighted accordingly. Funding by donations and
for-pay apis to do advanced searches perhaps? How awesome would it be to be
able to freely read a repository of this kind of information? And to also be
able to accurately say what is or isn't the current "scientific concensus".

I feel like this shouldn't be too difficult to make.

~~~
robotresearcher
[http://www.plosone.org/](http://www.plosone.org/)

PLOS ONE is this. It is well respected already. The initial invited reviews
are mainly a junk filter to establish baseline methodological quality and NOT
to evaluate significance.

From their "About us" page:

"Unlike many journals which attempt to use the peer review process to
determine whether or not an article reaches the level of 'importance' required
by a given journal, PLOS ONE uses peer review to determine whether a paper is
technically sound and worthy of inclusion in the published scientific record.
Once the work is published in PLOS ONE, the broader community is then able to
discuss and evaluate the significance of the article (through the number of
citations it attracts; the downloads it achieves; the media and blog coverage
it receives; and the post-publication Notes, Comments and Ratings that it
receives on PLOS ONE etc)."

~~~
jamesjporter
I'll just mention for those unaware that the author of the post is one of the
founders of PLOS, which is an umbrella organization under which a number of
journals (including PLOS ONE) are published.

------
IgorPartola
Here is a real case of this happening:
[http://en.wikipedia.org/wiki/Sokal_affair](http://en.wikipedia.org/wiki/Sokal_affair)

~~~
housel
The point of the Sokal affair seems to lean more towards proving that
postmodern cultural studies as a field is lacking in substance than to
demonstrating the ineffectiveness of peer review in general.

~~~
primelens
What bugs me about the Sokal affair is the implication that they didn't catch
a bad review process but proved an entire discipline to be a sham (a
discipline which they understood relatively little of to begin with).

So, when you get junk accepted in a humanities journal, its because the field
is about bullshitting anyway - when you get junk into a science journal it's a
hitch with the review process?

~~~
lmm
Was the Sokal paper published with inadequate review? Or was it passed by
experts in the field?

~~~
magicalist
All it takes is reading the wikipedia article...

But yes it was and no it didn't. That's always bugged me about the Sokal
affair and people holding it up as an example of anything. While the editors
did publish the paper, which is a basic problem in and of itself, they _claim_
they wanted big revisions that Sokal refused to make, but then they published
it anyway. Since the whole point of that journal at the time was to get wide-
ranging viewpoints, I think there is some merit in the editor's claim of a
simple betrayal of trust: they assumed Sokal was being straightforward, and
the worst that would come of publishing his piece would be a bunch of letters
telling him to stick to his day job. In any case, I don't know if we'll ever a
definitive story of what really happened, how carefully they vetted the paper,
etc.

Regardless, the journal he submitted the paper to was not peer reviewed, so
the whole thing demonstrates nothing about the peer review process.

------
utopkara
The problem is not the opaque peer review system.

Instead of making the reviews transparent, some journals solicit notes from
other researchers to discuss the published articles. It is an ethical
responsibility to publish notes which criticize previously published articles,
regardless of how painful it is for the publisher to publish them.

e.g.
[http://www.sciencemag.org/content/335/6069/665.summary](http://www.sciencemag.org/content/335/6069/665.summary)

Reading an article that falsifies claims of another article is pure scientific
joy. And such articles are not uncommon at all!

The problem is, as pointed out in the beginning of the paragraph "we need to
get past the antiquated idea that the singular act of publication – or
publication in a particular journal – should signal for all eternity that a
paper is valid, let alone important".

------
mwc
In legal practice, decisions are tested on appeal or by subsequent tribunals.

While this process is currently conducted by paid-for editorial teams, most
practitioners rely on citation tools which for any judgment provide links
through to subsequent decisions which have, for example, "approved",
"considered", "distinguished" (as in disapplied due to different facts, but
this can have the practical effect of confining an earlier decision to its
particular facts) or "overturned" that decision.

Although Google Scholar seems to provide some authority measure through the
"cited by" tool, I've often wondered why academia (or perhaps I've just missed
it) hasn't developed a comparable "precedent" system for research.

------
zokier
> Any scientist can quickly point to dozens of papers – including, and perhaps
> especially, in high impact journals – that are deeply, deeply flawed – the
> arsenic DNA story is one of many recent examples

Dozens of flawed papers out of how many total papers? Numbers are meaningless
without a context.

------
joshlegs
peer-review scientific analysis has been an issue for some time if you ask me.
As the guy points out, all you have to do is find a few people already
receptive to your idea. I'm not really sure why "peer-review" is so lofty a
concept anyway.

~~~
jpmattia
> _I 'm not really sure why "peer-review" is so lofty a concept anyway._

I agree. Will someone tell pg to turn off the upvotes?

------
jvdh
There are 250+ journals that have cancer growth as part of their field? That
shows you the problem right there.

There is absolutely no way that a researcher in that field can keep up with
publications in all of those journals.

------
unreal37
Hey Michael Eisen, we get it. You don't like the peer review journal process
and having to pay for access to scientific research. If you don't like it,
make a better system.

Also -1 for having the worst form of linkbaity title, which is one that is
utterly false and misleading, and lying for the first two paragraphs of that
blog post.

~~~
tedsanders
To me, the title was obviously satire from the moment I read it.

Also, he did make a better system. He co-founded the Public Library of Science
(PLOS).

~~~
judk
Does PLOS protect against false claims somehow?

------
lutusp
Quote: "So I created a manuscript that claimed something extraordinary - that
I’d discovered a bacteria that uses arsenic in its DNA instead of phosphorous
[sic]."

This tongue-in-cheek piece would be more plausible if the author could spell
"phosphorus".

~~~
piyush_soni
Really, get away from your computer screen for a while. Get a passport, visit
places. Know about the rest of the world, what they wear, what they eat, how
they talk, _how they write_.

~~~
judk
Where do they write "phosphorous"? They don't indiscriminately sprinkle 'u'
after every 'o'. That would be "phosphourous".

~~~
DanBC
phousphourous?

([https://en.wikipedia.org/wiki/Red_Phosphorous#Spelling_and_e...](https://en.wikipedia.org/wiki/Red_Phosphorous#Spelling_and_etymology))

> According to the Oxford English Dictionary, the correct spelling of the
> element is phosphorus. The word phosphorous is the adjectival form of the
> P3+ valence: so, just as sulfur forms sulfurous and sulfuric compounds,
> phosphorus forms phosphorous compounds (e.g., phosphorous acid) and P5+
> valence phosphoric compounds (e.g., phosphoric acids and phosphates).

