

Reddit, Google+, LinkedIn as Science Peer Review Systems - IgorCarron
http://nuit-blanche.blogspot.com/2013/02/reddit-google-linkedin-as-peer-review.html

======
lutusp
> Science is only as robust as its peer review system.

This exaggerates the influence of peer review on modern science. Modern peer
review is frequently (not always) a rubber-stamp way to catch perfect rubbish
before it gets into print, but it cannot detect intentional fraud or sloppy
work:

Source:
[http://www.plosmedicine.org/article/info:doi/10.1371/journal...](http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124)

Title: "Why Most Published Research Findings Are False"

In fact, science is only as robust as its practitioners and their motivations.
At the moment, science is in the midst of a credibility crisis because of
sometimes overwhelming pressure to produce "results" when there are none to be
had:

Source: [http://www.reuters.com/article/2012/03/28/us-science-
cancer-...](http://www.reuters.com/article/2012/03/28/us-science-cancer-
idUSBRE82R12P20120328)

Title: "In cancer science, many 'discoveries' don't hold up"

Quote: "During a decade as head of global cancer research at Amgen, C. Glenn
Begley identified 53 'landmark' publications -- papers in top journals, from
reputable labs -- for his team to reproduce. Begley sought to double-check the
findings before trying to build on them for drug development. Result: _47 of
the 53 could not be replicated_."

This is deplorable and justifies the "crisis" label. But changes in peer
review won't make any difference.

~~~
IgorCarron
You are mistaken because you are, like many others, confusing peer review with
"pre- publication peer review", the act of your peers judging a manuscript
before publication as is currently implemented.

The peer review that is allowed by the system I mention actually enables post
publication peer review. The review of papers that are already public (either
in some preprint or in some published form).

Right now the only mechanism that is allowed in terms of feedback in the pre-
publication peer review model, is writing to the journal for a corrigendum, or
even a retraction. As witnessed in retractionwatch.com blog, few are
enthusiastic about publishing corrections to papers that have gone through
their inefficient pre-publication peer review process. Not only it is
inefficient but it also directly yield examples such as the ones you,
rightfully, point to.

~~~
lutusp
> You are mistaken because you are, like many others, confusing peer review
> with "pre- publication peer review"

I invite you mimic the behavior of a scientist and uncover evidence that I
took that position anywhere. Here's what I said: "Modern peer review is
frequently (not always) a rubber-stamp way to catch perfect rubbish before it
gets into print, but it cannot detect intentional fraud or sloppy work ..." I
am well aware of post-publication peer review, but it's less likely to solve
problems that pre-publication peer review haven't solved.

> The peer review that is allowed by the system I mention actually enables
> post publication peer review.

Yes, and post-publication peer review still cannot prevent the kinds of fraud
and abuse that have led to the present credibility crisis, issues I outlined
in my original post.

~~~
IgorCarron
The position you take on the rubber stamping part of peer review is simply not
serious at best in many areas of science and engineering. Ask anybody around
you that has published in good journals or conferences. Hence I did not
address this statement because it simply is not true. Try sending papers to
ICML, Siggraph, SODA, Nature, etc, I doubt the peer reviews are performing any
sorts of rubber stamping.

But once they have gone through that process and they are published it is
simply very difficult to root out bad work given new data.

You might be aware of post publication peer review, yet, this model is not in
use except through accidental replication exercises that sometimes uncover
problems that eventually yield corrigendae or retraction. Currently it is very
adhoc.

If you understand how science work beyond press releases, you'll know that the
fraud and abuses will always be in the system. Post publication peer review is
the only way to rooting out bad work that has been published or is in the
preprint stage (I consder that if your preprint is out on arxiv or some other
medium, it is in effect published).

Right now, for published papers, it is left as an exercise for journals to
acknowledge they let some dubious work through. There is simply no economic
incentive for a speedy process. If you read retractionwath.com often you'll
notice that the current system is simply not regulating itself.

Open post publication peer review is a way to perform that function. Looking
back it is formalizing the process by which people used to trust or not older
work. It is also blurring the lines between preprints and published work since
they are now under the same scrutiny.

I agree it is also less convenient for the press or the science press to be
comfortable with this situation (post publication peer review) but Science
becomes robust when it is clearly capable of rooting out bad work through
processes like this one.

~~~
lutusp
> The position you take on the rubber stamping part of peer review is simply
> not serious at best in many areas of science and engineering.

Post your evidence, not your opinions. Here is my evidence -- one of many
papers that makes the same point I do:

<http://jrsm.rsmjournals.com/content/99/4/178.full>

Quote: "But does peer review `work' at all? A systematic review of all the
available evidence on peer review concluded that `the practice of peer review
is based on faith in its effects, rather than on facts'[1]"

[1] Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer
review: a systematic review. JAMA2002;287:2784 -6 (not available online)

If you want to continue posting to this thread, by all means try to imitate a
scientist and locate evidence for your claims, as I have for mine.

> If you understand how science work beyond press releases, you'll know that
> the fraud and abuses will always be in the system.

A non-sequitur that fails to address my point in any meaningful way.

> Science becomes robust when it is clearly capable of rooting out bad work
> through processes like this one.

I'm waiting for you to try to refute my original claim using evidence.

~~~
IgorCarron
Let me take the counterpoint: If peer review was a rubber stamping process
then most journals or conference would publish most papers. Just take the
journals and conferences I mentioned earlier and you'll see that their
rejection stats are above 60-70% if not more.

With regard to the paper you mentioned
<http://jrsm.rsmjournals.com/content/99/4/178.full>

You need to go beyond the title of the paper. Please read the whole paper you
are referencing and then please tell me how the following extract is running
opposite to what I mentioned earlier:

" Opening up peer review ...The final step was, in my mind, to open up the
whole process and conduct it in real time on the web in front of the eyes of
anybody interested. Peer review would then be transformed from a black box
into an open scientific discourse. Often I found the discourse around a study
was a lot more interesting than the study itself. Now that I have left I am
not sure if this system will be introduced...."

The simple fact is you understand peer review to mean how it is currently
employed i.e. one time process used in the pre-publication stage (you do the
review before the paper is published). That type of peer review is flawed as
some bad papers still go through the process. What is proposed is having an
on-going peer review so that over time, only the stronger papers stand. It
really is not difficult to understand.

~~~
lutusp
> Let me take the counterpoint: If peer review was a rubber stamping process
> then most journals or conference would publish most papers.

False. Rubber-stamping is a two-way street. Some deserving papers are not
published, some that are not deserving are. Examples of both kinds abound.

