
Scientific Peer Review Is Broken – Fighting to Fix It with Anonymity - gkuan
http://www.wired.com/2014/12/pubpeer-fights-for-anonymity/
======
lisper
Anonymity can't fix scientific peer review, it can only replace type I errors
with type II errors. Instead of suppressing criticism that ought not to be
suppressed, anonymity can (and often does) fail to suppress criticism that
really ought to be suppressed because it is in fact false and defamatory. And
indeed, what this article is really about is a lawsuit that alleges that this
kind of error has in fact taken place.

Ironically, the very title of this article is a model of non-scientific
thinking that ought to be subject to criticism, but attempts to inoculate
itself against criticism by asserting that the subjects of the piece are
protagonists "fighting to fix" a broken system. They're not. They're fighting
to replace one broken system with a different broken system.

There is no question that scientific peer review is broken and needs to be
fixed. But anonymity is not the answer. And holding anonymity up as something
that should itself be beyond criticism is _certainly_ not the answer.

~~~
seccess
I agree completely. In my experience as a PhD student (comp sci), all
conferences make reviewers anonymous. As a result, there is very little
accountability regarding the reviews you receive for your work. More than
once, I have had papers rejected simply because a single reviewer barely read
the paper and dismissed it. These kinds of reviews are very frustrating to
receive, not only because they failed to understand the basic premises of your
paper, but because these reviews contain no useful information on how to make
your paper better for future submissions.

Of course, the opposite can happen where mediocre works slips through, and the
reviewers that allowed that should be held accountable too. Its painful to me
that so much of the acceptance process for research papers (in my field at
least) is based on luck.

Moral of the story: reviewer anonymity is good but it comes at the expensive
of accountability.

~~~
javert
I'm also a computer science PhD student. I don't disagree with anything you
said, but I am even more cynical.

> Of course, the opposite can happen where mediocre works slips through

Slips through? Mediocre work is the norm, or at least close to being the norm.
(And, as you say, good work is often rejected.)

> Its painful to me that so much of the acceptance process for research papers
> (in my field at least) is based on luck.

I don't think it's based on luck, it's based on politics. Academic science is
a racket. There is some luck too, though.

~~~
seccess
> Slips through? Mediocre work is the norm, or at least close to being the
> norm.

In computer science, this is likely because of the ridiculous number of
conferences there are and how eager each one is to fill out their roster. I
want to hear of the conference that published no papers this year because none
were good enough for publication.

> I don't think it's based on luck, it's based on politics. Academic science
> is a racket.

I've heard (more than once) of people who submit a paper, have it rejected,
and then get scooped by a professor who was on the PC of the conference that
rejected them. There is a very real possibility of work being accepted or
rejected purely because of what the reviewer thinks will be most beneficial to
their career.

So, I think your cynicism is well-placed.

~~~
javert
> In computer science, this is likely because of the ridiculous number of
> conferences there are and how eager each one is to fill out their roster. I
> want to hear of the conference that published no papers this year because
> none were good enough for publication.

Yep. But there is a reason it is this way, which is that to be successul on
the academic feeding trough/career path, you have to serve on PC committees
and you have to publish a lot of papers.

> There is a very real possibility of work being accepted or rejected purely
> because of what the reviewer thinks will be most beneficial to their career.

I think this is right, but it's less often so work can be scooped, and more
often about building up your "camp." As a research professor, you need your
area of expertise to be popular and you need to get a lot of citations for
your work. You want to be a leader in an area that _actually matters,_ where
your definition of "actually matters" is actually purely herd mentality: what
do other people consider to be important? So you want to reject work that
bolsters your camp, cites you, etc., and reject work that poses a threat to
the ascendancy of your area of expertise.

~~~
blueMist
Agree with a lot of what is being said here. Have seen vicious anonymous
reviews in CS - as we all have - both due to someone misunderstanding the work
but also due to people from "a different camp" simply disliking a given
approach. This must be common in other fields as well. Someone said on this
thread that anonymity is not a guarantee of quality - completely agree. The
move towards "open reviewing" for conferences in CS is very encouraging,
however we will see how far it gets.

~~~
javert
I'm going to say something controversial.

Minor reform of the referee system is treating the symptoms, not the disease.

The disease is that there is no honest, mutual, voluntary exchange between a
party that values a given research project and the party that produces it.

Rather, we have federal bureaucrats handing out money taken from taxpayers
willy-nilly via an old-boy network as described above. (The difference between
taxpayers and slaves is that taxpayers only must sacrifice a fraction of their
productivity to this absurd system, not all of it.)

In CS, all the research we do either goes to benefit shareholders of companies
that ultimately profit from it, or (the majority) is just ignored because it's
part of the paper mill competition. Those companies should be the ones paying
for the research, not the taxpayers. Those companies are free riding, and "we
the people" should put an end to it.

~~~
wmf
I'm not sure what the one has to do with the other. My research is corporate-
funded and my papers still get reviews from people who didn't read the paper
carefully. Unless you're proposing to eliminate the concept of publishing
research at all...

~~~
javert
What one has to do with the other is that the entire system is broken, here's
my explanation for why the system is fundamentally messed up, and my
explanation for how it should be. Simply trying to change refeering by itself
is treating the symptom, not the disease.

> Unless you're proposing to eliminate the concept of publishing research at
> all...

I think putting my research on my website would be just as good as publishing
it in a conference proceeding. I'm not against sharing research and having it
be public, but I think conference proceeidngs and journals in CS are of very
little value. If people want my research, they can find it on my website. And
I think this generalizes... if I want your research, I can (presumably) find
it on your website.

Ultimately, it may be useful to have some kind of aggregation of what research
is coming out in various subfields. Internet fora and the like would serve the
purpose fine. But sure, having a group of referees and some way to screen
research and highlight what is good could certainly be useful. All that stuff
should develop organically as it is needed, though.

The model we have now, with peer review, journals made out of dead trees, etc.
is a hold-over from a pre-Internet time when a few people in every sphere of
life controlled the information that was disseminated, because it wasn't
possible for things to be open, since we lacked the technology. That is no
longer an issue. Of course, academics will hold onto their little racket as
long as they can (probably indefinitely).

------
dougmccune
The actual complaint filed shows pretty nasty behavior. Here's the full
complaint: [https://retractionwatch.files.wordpress.com/2014/10/filed-
co...](https://retractionwatch.files.wordpress.com/2014/10/filed-
complaint.pdf)

Someone very clearly was trying to get Dr. Sarkar fired. The PubPeer comments
are only one small aspect. But the person then also (I assume anonymously)
emailed his new employer to allege fraud, and then went so far as to print out
these allegations and stamped them with official looking nonsense about being
from the an NIH investigation (which didn't exist), and distributed them
throughout Sarkar's department mail boxes.

The complaint makes a decent case for why they think pretty much all the
negativity directed at this researcher is likely from one angry person. They
obviously can't prove that multiple anonymous comments are from the same
person, but reading the content it certainly seems likely. What is assumed to
be the same person then took things way past the line of what most of us would
consider ethical.

There may very well be a place for anonymous calling out of potential research
misconduct. But making fraud allegations anonymously online, then printing out
those comments, trying to fake them to look like an official government
inquiry, and physically delivering them to the researcher's boss at his place
of work isn't the way to do that. This case _might_ be one of those cases
where the anonymity should be protected at all costs out of principle, but
it's a really shitty case to wave your "we're the good guys" flag for.

~~~
tokenadult
I am also not sure (as was a commenter in a reply) what was objectionable
about my first reply to you, but now that I see that I can no longer edit that
reply anyway, I should link here to what PubPeer says in reply to the
plaintiff's motion to force disclosure of the name of the PubPeer
commenter(s).

[http://retractionwatch.com/wp-
content/uploads/2014/12/2014.1...](http://retractionwatch.com/wp-
content/uploads/2014/12/2014.12.10-PubPeer-Motion-to-Quash-Brief.pdf)

As an aside, I'll mention that the PubPeer lawyer is doing a much better job
in filing pleadings, perhaps because PubPeer is a more above-board client,
than Sarkar's lawyer is doing. Yeesh, what a mess the Sarkar complaint is.
(Yes, I am a laywer, and I have read pleadings before and have a gut sense of
which pleadings are convincing.) Note that the American Civil Liberties Union
signed off as lawyers for PubPeer.

~~~
dougmccune
The response from PubPeer's lawyer is certainly interesting, particularly the
analysis of the images in question. But I wasn't trying to argue one way or
the other about the actual merit of the accusations really. I was more
pointing out that it seems fairly clear that someone, acting entirely
anonymously, was set on getting this man fired. He did indeed lose a job (in
fact two!), seemingly because of the efforts of this anonymous source, without
any kind of formal inquiry or detailed analysis of misconduct. And the alleged
actions of the anonymous attacker were entirely unprofessional and just plain
shitty. Does the guy deserve to have his career destroyed? I have no idea, but
trying to do so in such an underhanded mean-spirited way is just nasty. But
this is academia after all...

------
Fede_V
I think the increase in fraudulent papers getting published is a symptom, not
the disease. The disease is that there are too many scientists doing research
for a given research budget.

Due to the ferocious competition for grant money, people are either sloppy and
cut corners, or do whatever they need to do and outright cheat to publish in
top tier journals.

As a society, we need to make a decision about how much we want to fund
scientific research - then, once we've made that decision, insure that we put
in place a sustainable system in place - we cannot put in a put of money to
finance 100 grants, but then build a pipeline that funnels an ever increasing
amount of people into a pool that remains constant.

~~~
snowballsteve
One could argue that papers are the disease. It was a good medium, but with
today's communication technology, something better needs to arise. The need to
"publish to exist" in academia has gotten to the point that it promotes sub-
standard work and hinders advancement of knowledge.

------
jedbrown
I take the opposite approach: I sign my reviews and have ever since I was a
grad student. It compels me to do a better job reviewing and often leads to
further discussion and sometimes collaboration with the authors. Some people
don't like what I have to say, but by and large, they respect it. For many
topics that I review for, the authors will have a pretty good idea that I
wrote the review unless I am intentionally vague. I would rather write clearly
and directly and stand behind it. It is a professional risk that I don't think
anyone should be compelled to take, but I think signing reviews is generally
good for science.

~~~
derf_
Yes, I think anonymity in reviews is largely a myth, even in "double blind"
reviews. For many topics, there are only a few research groups actively
pursuing them, and you can often tell from what other work they cite, what
datasets they use, etc., which lab it is. So identifying at least some of the
authors is relatively easy. You might not get the exact grad student right,
but you probably know their advisor.

Identifying reviewers is a little harder, but it's usually not difficult to
make educated guesses unless they are, as you put it intentionally vague.
Otherwise the other comments on this article about bad reviews from 'people
from "a different camp" simply disliking a given approach' and the like would
be impossible to justify.

------
zmanian
It seems plausible to imagine a system where PeerPub could retain their
"published author to register

This model relies on t a blind cryptographic
signature.([https://en.wikipedia.org/wiki/Blind_signature](https://en.wikipedia.org/wiki/Blind_signature))

PeerPub generates a public/private key pair.

1\. Alice wishes to register on PeerPub. She generates a Nonce N.

2\. She blinds the nonce with a factor B.

3\. She submits the blinded nonce and her identity to PeerPub. PeerPub checks
her credentials and executes a blind signature of the number and returns it to
Alice.

4.Alice now separately registers an account perhaps using a privacy protecting
system like Tor. She uses the original N and the unblinded Signature.

5\. PeerPub verifies N + signature and registers the account. PeerPub will
have no way of linking N to the original credentials. PeerPub can record N and
make sure it can only be used once to register an account.

~~~
sweis
Not specific to PeerPub, but I helped work on a prototype of a federated login
system based on OpenID that used blind signatures in a similar fashion:

[http://private-idp.appspot.com/](http://private-idp.appspot.com/)

[https://code.google.com/p/pseudoid/](https://code.google.com/p/pseudoid/)

[http://blind-signer.appspot.com/](http://blind-signer.appspot.com/)

I haven't touched it for years and it looks like the login flow is broken now
with the demo sites.

Also, this uses unsafe server-side JS crypto for demo purposes only.

~~~
zmanian
I emailed the PeerPub people about something like this. Will see if they bite.

------
p4bl0
Peer review might be broken, but it's not anonymity that will fix it. For the
simple reason that we already have double-blind reviews in many fields and the
peer reviewing process is not any less broken there.

There are people who are thinking of multiple solutions, one that I think is
interesting is the proposition of Open Scholar, dubbed "Independant Peer-
Review". I submitted it to HN so we can discuss it without flooding this
thread, if it interests HNers:
[https://news.ycombinator.com/item?id=8734271](https://news.ycombinator.com/item?id=8734271).

In there scheme, the papers are open access from the beginning, the peer
reviewing process is open, and reviews are citable. I believe it would
encourage better and deeper reviews.

It would also be very nice for young scientists or students who want to apply
for a PhD grant to be able to show that they are able to write a comprehensive
review of a paper in their field, and that their review was good enough to
significantly improve the paper or to be selected by the authors and/or
publishers to be released alongside the paper, for instance.

------
esbio
As a person who did research until a few years ago, I must say that the
problem is the exact opposite. When you send an article to a journal, the
paper gets reviewed by a number of peers, which send their comments back to
the Editor on the appropriateness of the claimed work.

The problem with this mechanism is that reviewers have no liability, because
their comment is anonymous to the author and won't be available to the
readers, as it won't be published as part of the article. The result is that
reviewers are not made accountable now or in the future for inaccuracies in
their review, blatant attacks, or tactical requests for additional irrelevant
investigation just out of spite or to stall you so that they can scoop your
paper.

Occasionally, the Editor can step in and disregard a particularly obnoxious
reviewer, but it depends on the editor, the journal, and the
political/scientific strength of the reviewer.

~~~
seanmcdirmid
> because their comment is anonymous to the author and won't be available to
> the readers

You can make comments on your papers available; there is nothing against that.
I really think, however, peer review comments should actually be published
with rejected and accepted papers (at the author's consent, not reviewers) so
that conferences can be more transparent.

~~~
esbio
You can, but 1) there's no official channel 2) nobody will do it if they are
not forced.

------
kazinator
> A prominent cancer scientist, unhappy with the attention his research papers
> have received on PubPeer, is suing some of our anonymous commenters for
> defamation

On the other hand, should anonymous commenters have the balance of power: in
other words, say whatever they want with impunity, even if it actually is
defamatory?

(Not saying that is the case in this situation, but in general).

The problem is that defamation is a legal concept, which can only be tried
legally. So for instance, whereas a site can rigorously enforce rules which
say that all comments are directed at the research material, and not at
persons, and have a factual basis in that material, those measures cannot take
away the right of someone, who feels they have been defamed, to take the
matter to court (where they will almost certainly lose, which is neither here
nor there).

You can't just create a site and declare it above the law, so to speak.

The only way to protect the identities of the anonymous is for the site to
take responsibility for the statements it publishes: to assert that the
anonymous statements are subject to rigorous standards of review, and when
published, they in fact reflect the views of PubPeer, and PubPeer alone, and
not of any anonymous persons (who do not publish any statements, but act only
as sources of information).

Then if someone feels they have been the target of defamation, the defendant
shall be PubPeer.

~~~
jessriedel
This situation is reminiscent of websites hosting copyrighted material posted
by anonymous users. Does the relevant law (DMCA, I think) say anything about
defamation? A reasonable compromise might be that websites are held harmless
so long as they respond to official defamation complaints in some prescribed
manner. (Of course, this might not work as well because defamation is
significantly less clear-cut than copyright violations, which already has
plenty of cases of over-reach due to disputes over fair use, etc.)

~~~
joshuacc
I think you may mean "prescribe" rather than "proscribe." :-)

~~~
jessriedel
Yes, that is an especially confusing error. Thanks.

------
ignostic
What we need here is a more nuanced approach. Anonymity can solve some
problems in research, but it will make other worse.

When people are anonymous, they ARE more likely to be truthful in their
criticism. They have less incentive to hold back, and it's just human nature
to tone down critical feedback when you're critiquing the work of someone with
is either influential or an acquaintance. No one likes to make enemies.

On the other hand, anonymity can pretty clearly bring out some of the worst in
us. Some people feel little obligation to be fair or honest when their
reputation isn't on the line, and so you see people trying to knock down
rivals, people they don't like, or random strangers just for the "thrill of
the troll."

Imagine if every time you applied for a job your potential employer had access
to anonymous feedback on your past work. Some of it might be fair and honest
(whether positive or negative), but some of it might be lies from an anonymous
coworker with a grudge. Maybe someone is trying to take you down a rung
because you got the promotion over them. You could be penalized for any petty
reason, and it would stick with you.

Anonymous feedback communicated publicly is much the same. It holds the
reviewer and the object of review on unequal footing. Anonymous feedback would
be great for an author or even an editor, but it's just not fair to allow the
pettiest of people to attack the works of others while wearing a mask. I'd
like a system that helps researchers and others invested in the work to
solicit anonymous feedback to make the work better. Public-facing commentary,
on the other hand, should be tied to an identity.

------
ejz
One of the main issues is that this is just a really tedious process. No one
wants to go through algebra; that's why it's siphoned off to grad students. Do
you really think a tenured professor is going to spend time checking the grunt
work on /someone else's/ paper when they won't do it for their own? I'm
intrigued by the possibility of using natural language processing and logical
system tools like Wolfram Alpha. Wolfram recently posted on his blog about
building machines that could store data about complex mathematical objects,
and already you can build machines that confirm first order logical
statements.

Farming out low-level tasks to automated systems would be interesting. Imagine
if the format of papers changed entirely, ie, you had to submit your proofs in
certain formats, or at least certain parts in specific ways. I'm sure that
many professors would be elated to see the number of papers they have to
review go down drastically; although, I'm sure many will be disappointed to
get a return letter that says, "I'm sorry, but the low level flaws were so
serious that they were automatically rejected and are not fit for review."

~~~
ylem
I'm a permanent staff member and I do actually check "algebra" and integrals
when reviewing papers. Especially "technique" papers, because after the
initial paper, people tend to make black boxes, so if the initial errors
aren't caught then they can propagate for quite some time...I view it as
community service.

------
colechristensen
Let's be sure to point out some terrible journalism, reading the following
makes me completely disinterested in finishing the article and have
considerably less respect for Wired.

> Have you ever questioned the claims that scientists make? For example, last
> year’s discovery of the so-called “God particle,”

Using the nickname for the Higgs in the context of questioning science claim
is nothing but bait for the foolish and misinformed, and in this way anyone
scientifically literate should seriously question any claims or opinions in
this publication.

------
ylem
This is an interesting problem. On the one hand, there are merits to a referee
being anonymous to the submitter. Part of this may be to avoid reprisal for
younger referees, but even for more established referees, you may be freer to
comment if you are anonymous to the author. BUT, you are not completely
anonymous. Hopefully (though there have been some recent scandals related to
this) an editor of the journal knows your work and has chosen to use you as a
referee based on that. This can help to keep down some of the noise that
another poster (lisper) mentioned. Also, the fact that the editor knows who
you are may provide some constraint on how you may phrase a review as compared
to if you were completely anonymous.

I don't think this is something one should issue a lawsuit over--but I also
don't think that their proposal of completely anonymous review is at all
useful.

------
weissadam
My understanding is that most peer review systems in place at various journals
and funding agencies today are already anonymous (except for when people are
identified by their well known viewpoints.) If you ask me, the real problem
can't be solved with communications technology, the real problem must be
solved at the source: The funding agencies need to take the importance of
reproduction of results seriously and require their grantees to do a certain
amount of rote reproduction work in order to qualify for grants for novel
research. Will it slow the pace of things down? Certainly. Will it increase
the quality of the science? Certainly.

------
pc2g4d
Much of the discussion here seems to have become "anonymity vs. non-anonymity"
\--- i.e. either anonymity is good for research or it's bad for research. Why
not just accept that in some venues there will be anonymity, and in some not,
and let them each develop according to the merits of the respective
approaches? No need to have all research anonymous or all research clearly
attributed to a public identity.

------
hyperion2010
I usually think that peer review is broken, but that is because we almost
always miss the point of what the actual purpose of peer review is for. Peer
review is NOT for laymen. A scientific paper is part of a much longer process
of discovery which includes massive amounts of debate and argument. Papers
always reflect they very edge of a field where literally the only people who
can judge whether the results are plausible are peers because they are the
ones who are also trying to make sense of the same phenomenon. Over time some
primary research papers are singled out because they really are excellent
explanations of a phenomenon most of the time however you have to go look at a
review paper. Furthermore peer review is absolutely NOT about reproducibility,
it is a prerequisite that says that it MIGHT be worth trying to reproduce this
work or work contingent on these results being valid. There are so many
reasons why a result might be wrong that it often takes fields years to figure
it out and the published literature is a record of that.

------
pizza_boy
With Publons.com ([https://publons.com](https://publons.com)) we have
different philosophy: the more transparency we can bring to the review
process, the better. At the same time we recognise that both blind and double-
blind peer review play an important role in generating quality research.

Our approach is to focus on turning review of all kinds (including both pre-
and post-publication) into a measurable research output -- something you can
add to your resume. We support both anonymous and signed review with the idea
that it will lead to greater transparency in the long run and also motivate
reviewers to contribute more.

We have a significant number of both types of review now and are starting to
look ways to measure if there are significant differences between blind and
open review.

~~~
efangs
Hi, I like the site and idea. I was wondering if you were thinking of taking
it one step further, however, and making it an actual pre-print server?

I would think that if you expect people to take the "credit for reviewing"
model seriously, then likewise they should take seriously the notion of
"credit for being reviewed" or "credit for being upvoted".

Also, there would have to be a way to verify submitters/reviewers credentials
(e.g. connection to a research institution). Obvious arXiv has a model for how
this could be done.

------
analog31
Something I've noticed is that most online articles about scientific
misconduct and invalid results revolve around the medical sciences. Granted,
medicine is probably the biggest piece of the scientific pie right now, and
other branches of science may have their own problems, but I think it misleads
the public to have a title like "scientific peer review is broken," for an
article that focuses exclusively on one branch of science.

Disclaimer: I'm a physicist. I'm sure that physics is not beyond critique, but
unless somebody is willing to take enough of an interest in the inner workings
of physics research to say something specific about it or credibly include it
in a generalization, I'd rather say something like "medical peer review is
broken."

------
vacri
_Fortunately, the First Amendment is on our side. It protects the right to
anonymous speech._

I don't understand how "The government cannot outlaw your speech" means "Your
anonymity is protected by law from private parties". How does that
interpretation come about?

~~~
dalke
[http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v....](http://en.wikipedia.org/wiki/Dendrite_International,_Inc._v._Doe_No._3)
.

> "the court must balance the defendant's First Amendment right of anonymous
> free speech against the strength of the prima facie case presented and the
> necessity for the disclosure of the anonymous defendant's identity."

~~~
hibikir
I don't think that opinion says what you think it says.

What is really being discussed there is if the government could compel Yahoo
to unmask an anonymous user. It says nothing about actual anonymity guarantees
among private parties. If Yahoo decided to hand over the account details, the
court would not have intervened at all. So when you have an account with some
service, any expectations on how the company will handle the data come from
the TOS.

In a similar fashion, a company is free to delete any content you submit to
their website. The government can't curtail your speech, barring a few
exceptions, but a company certainly can, as long as the government is not
involved.

~~~
dalke
We know that the government could compel Yahoo to unmask an anonymous user -
get a FISA order to inspect the business records and, poof, done. So I don't
see how that's the issue. Here's the breakdown:

"A prominent cancer scientist, unhappy with the attention his research papers
have received on PubPeer, is suing some of our anonymous commenters for
defamation."

That's a third-party, who is not a government organization, suing PubPeer to
de-anonyomize a commenter on PubPeer's site.

"Dendrite International, Inc., a purveyor of computer software used in the
pharmaceutical industry, brought a John Doe lawsuit against individuals who
had anonymously posted criticisms of the company on a Yahoo message board.
When Presiding Chancery Judge Kenneth MacKenzie rejected one of Dendrite's
requests to compel Yahoo to reveal the identity of an anonymous defendant,
Dendrite appealed."

That's a third party, Dendrite, who is not a government organization, suing
Yahoo to de-anonyomize a commenter on Yahoo's site.

If Yahoo wanted to, they could turn that information over. The question was,
could Dendrite compel Yahoo to do so if Yahoo didn't want to. And the answer
is that there's a First Amendment issue which must be considered, so the
answer is "sometimes."

Yahoo didn't want to. And PubPeer doesn't want to.

So I don't understand why you are bringing up first party actions (actions
that Yahoo and PubPeer are free to do) when the issue is what third party
actions may compel them to do.

------
sideshowb
Not as deep an article as I'd hoped, but it points to a difficult question. As
reviewers of scientific papers have unprecedented power over what research
gets published, _who reviews the reviewers_?

(A personal example - I have just spent a week constructing a counterargument
to a reviewer who didn't read my paper properly. Imagine having an online
discussion where your career success hangs on the response of an anonymous,
disengaged flamer to a single post of yours: can you imagine how much effort
you put into that post? Sadly this particular argument is pointless and serves
nobody; it would be better use of my time to get on with actual research, but
that's not how the system works).

------
kgarten
I don't get their stance ... peer-review is already "anonymous". There are
also a lot of issues with truly anonymous user forums (see 2chan and 4chan).
For me it always seems as if the social aspects of communication disappear
when one is truly anonymous, e.g. hate speech. The problem is not lack of
anonymity but lack of incentives for reviews (I don't get anything from doing
a thorough review of a paper and often it's hard to impossible to judge the
contribution without dataset and code). It seems peerpub and similar systems
will attract people who have the incentive to attack specific authors (as it
happened in this case).

------
Fuzzwah
The obvious problem here is with fields that are small and highly specialized,
such that anyone knowledgeable enough to comment on a topic are known to be
from a small circle of scientists.

------
HandleTheJandal
Thank you for helping to promote reproducibility of published results by
supporting anonymous peer review! Science Exchange (YC S11) is also making
great progress in the facilitation of scientific reproducibility. We just
completed independently validating select results from 50 cancer biology
papers.
[https://news.ycombinator.com/item?id=8731274](https://news.ycombinator.com/item?id=8731274)

------
return0
Pubpeer is not a peer review site, it's site with comments. I think they are
doing it wrong by promoting it as "anonymous peer review", because by
definition if its anonymous it can't be "peers". It should (continue to) be a
companion to published papers that every once in a while does the service of
spotting an error.

------
Animats
Another anonymous review system. What could possibly go wrong?

If there's anything we've learned by now about "crowdsourced" review systems,
it's that, without an elaborate way to evaluate reviewers, they fail. Badly.
Facebook "likes", Google "+1", and Yelp reviews are heavily spammed. This just
does not work.

------
daemonk
Anonymity is hard in practice when so much of science consist of small niche
fields where it can be pretty obvious who the author of a paper is by just
reading the content.

A better solution might be to enforce non-anonymous peer-reviews that can be
read by the public after the paper has been published.

~~~
tnhh
Some venues are doing this, e.g. f1000. Here's a paper that I reviewed:
[http://f1000research.com/articles/3-38/v2](http://f1000research.com/articles/3-38/v2)

------
LiweiZ
I don't think push some form of crowd intelligence would change much. I guess
the best thing we could do is to create some conditions to let new nodes
emerge and complete. I also believe better funding distribution
mechanism/system will help a lot.

------
atsaloli
Dr. Mark Burgess just blogged "Why I stopped caring about peer review, and
learned to love the work".
[http://markburgess.org/blog_peer.html](http://markburgess.org/blog_peer.html)

------
tokenadult
The submission here is an interesting article by the founders of PubPeer,
which has already been in the news quite a bit for finding examples of shoddy
science papers that have had to be withdrawn by journal editors. I learned
about PubPeer on the group blog Retraction Watch (RT), and I just bopped over
to Retraction Watch after reading the article kindly submitted here. RT
reports in detail on the defamation suit against PubPeer that is mentioned in
the parent article of this thread.[1] I hope the PubPeer experiment can
continue and thrive and promote better scientific research practices.

Some of the other comments here suggest that anonymity of reviewers is
dangerous in itself. That's why some researchers promote an open review
process. Jelte Wicherts and his co-authors put a set of general suggestions
for more open data in science research in an article in Frontiers of
Computational Neuroscience (an open-access journal).[2]

"With the emergence of online publishing, opportunities to maximize
transparency of scientific research have grown considerably. However, these
possibilities are still only marginally used. We argue for the implementation
of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and
(3) online data publication. First, peer-reviewed peer review entails a
community-wide review system in which reviews are published online and rated
by peers. This ensures accountability of reviewers, thereby increasing
academic quality of reviews. Second, reviewers who write many highly regarded
reviews may move to higher editorial positions. Third, online publication of
data ensures the possibility of independent verification of inferential claims
in published papers. This counters statistical errors and overly positive
reporting of statistical results. We illustrate the benefits of these
strategies by discussing an example in which the classical publication system
has gone awry, namely controversial IQ research. We argue that this case would
have likely been avoided using more transparent publication practices. We
argue that the proposed system leads to better reviews, meritocratic editorial
hierarchies, and a higher degree of replicability of statistical analyses."

Wicherts has published another article, "Publish (Your Data) or (Let the Data)
Perish! Why Not Publish Your Data Too?"[3] on how important it is to make data
available to other researchers. Wicherts does a lot of research on this issue
to try to reduce the number of dubious publications in his main discipline,
the psychology of human intelligence. When I see a new publication of primary
research in that discipline, I don't take it seriously at all as a description
of the facts of the world until I have read that independent researchers have
examined the first author's data and found that they check out. Often the data
are unavailable, or were misanalyzed in the first place.

[1] [http://retractionwatch.com/2014/12/10/pubpeer-files-
motion-d...](http://retractionwatch.com/2014/12/10/pubpeer-files-motion-
dismiss-sarkar-defamation-case/)

[2] Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom.
Letting the daylight in: reviewing the reviewers and other ways to maximize
transparency in science. Front. Comput. Neurosci., 03 April 2012 doi:
10.3389/fncom.2012.00020

[http://www.frontiersin.org/Computational_Neuroscience/10.338...](http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full)

[3] Wicherts, J.M. & Bakker, M. (2012). Publish (your data) or (let the data)
perish! Why not publish your data too? Intelligence,40, 73-76.

[http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf](http://wicherts.socsci.uva.nl/Wichertsbakker2012.pdf)

------
nemoniac
What the scientific peer review needs is not anonymity, but accountability.
Authors put their reputation on the line. Let reviewers put their reputation
on the line too. Good reviews and good reviewers need to be appreciated, not
anonymized.

Let valued scientific reviewers gain reputation in a similar way to how
contributors to HN, SE and other sites do.

There could even be a viable business model in this. Publishers, be creative!

