
Peer Review as a Service: It's not about the journal - ngoldbaum
http://theoj.org
======
dougmccune
_For scholarly publishing, the secret sauce - the essential thing - is a
mechanism for review_

I’d venture to say that’s close but not phrased quite right. The secret sauce
of academic journal publishing is that we get young academics hired. If you
can figure out a way for a young junior faculty member to get tenure without
publishing in a traditional journal then you’re golden. But as it stands
today, the name of the journal is the currency on your CV. Peer review is part
of the mechanics of how that works, but it’s the prestige that’s the key part,
not the review mechanism itself.

Tenure committees can’t read the actual papers of all the applicants to form
an opinion on the quality of the research. They need some form of proxy. Today
that’s the impact factor and reputation of the journal title, almost
exclusively. It’s entirely possible that that will change and something else
(or most likely a variety of something elses) will take its place. There are a
lot of people working on alternatives right now. I don’t know what the winners
are going to be, but it’s not going to be number of tweets or Facebook likes.
If you want to break the existing system you have to figure out how to get a
young academic without an established reputation noticed in her field and
hired. If you can do that without the need for a journal at all then you can
put a crack in the foundation of the industry.

~~~
gioele
> Tenure committees can’t read the actual papers of all the applicants to form
> an opinion on the quality of the research.

I once discussed this with the dean of a humanities faculty in Norway. He said
they did not take h-index and similar metrics into account while debating
tenure.

So I asked: "How do you evaluate the applicants, then?". "We read their
papers" he replied. A room full of professors laughed loud.

He then went on telling us about how they shortlist the candidates and how
many paper each person in the committee has to read. The figures where
something like one paper or small book read in depth for each candidate and
five papers read more casually. The candidates choose the reading list.

To me it does not seem a great amount of work for somebody that is about to
appoint another scholar as their peer. It also seems to me as a very decent
way to treat applicants: as people that did contribute interesting knowledge,
not data points.

~~~
impendia
I am a mathematics professor who has served on several hiring committees. In
our context this would be impossible.

In particular, I was asked to help hire an algebraic geometer -- algebraic
geometry not being my specialty. For me to read and understand the highly
technical papers of each of 500 job applicants, outside my area of expertise,
would take an astronomical amount of effort -- completely out of the question.

And not only that, but distinguishing thorny unsolved problems from fairly
routine technical exercises is quite a challenge -- one reason that high-end
journals solicit multiple reviews. It is sometimes possible to fool many (but
not all) of the experts.

We rely principally on recommendation letters (generic, and submitted via a
centralized system, so candidates can get one set of letters for all jobs to
which they apply). But, yes, which journals candidates have published in is
important.

~~~
xerophtye
>high-end journals solicit multiple reviews. It is sometimes possible to fool
many (but not all) of the experts.

I thought this was a debated point. Don't a lot of people assert that the
prestigious high end journals don't really review them for correctness etc,
but instead focus on how "ground-breaking" or "news worthy" ??

~~~
duaneb
Experts are sometimes necessary to identify and translate the papers into
something that you can understand—include whether it's ground-breaking or
news-worthy.

------
ylem
I'm a physicist and a see a few problems with this. Currently, refereeing is
essentially community service. Have you talked to any editors at journals such
as the Physical Review or New Journal of Physics? I would imagine that the
largest cost of running a journal (let's say an online journal) are the costs
of the editors salaries. Let's say that you scale your referee system (which
is cute) and add in statistics on referees (do they often accept/reject? Areas
of expertise? Time to submit a review, etc.), your editor will still have to
take time to resolve the inevitable disputes between authors and referees. As
the number of papers grows, you will need to either have more editors, or
start paying editors to do this full time.

Another problem that you will run into is the question of curated content.
There is a place for having refereed content that is technically correct, but
not necessarily impactful (for example, Nature's scientific reports), but such
a journal will have a low impact and will have a hard time attracting
impactful papers (which are still valuable for career advancement). Otherwise,
your editors can start to curate the content based on impact, but this
requires even more time and expertise from your editors. I think you'll
eventually run into some of the same problems as traditional journals...

Have you thought about working with existing journals to improve the
refereeing process?

For some of the other issues that you bring up, such as attaching code--I
think that this will have to be something required by funding agencies. I have
found the "new" supplementary material sections of journals to be a source of
improvement...

------
opminion
This seems to realise the "disentangling dissemination and authentication"
suggested by the ArXiv guys themselves ([1], linked from [2])

[1]
[http://www.cs.cornell.edu/~ginsparg/physics/blurb/pg01unesco...](http://www.cs.cornell.edu/~ginsparg/physics/blurb/pg01unesco.html)

[2]
[http://www.cs.cornell.edu/~ginsparg/physics/blurb/](http://www.cs.cornell.edu/~ginsparg/physics/blurb/)

------
guynamedloren
It seems there's been quite a bit of talk about collaboration/review in
science and academia as of late. Makes sense, as there are some real unsolved
problems here.

Shameless plug: I'm also working on tackling peer review, but from a bit of a
different angle. Penflip.com
([https://www.penflip.com/](https://www.penflip.com/)) is like GitHub for non-
programmers. It hosts public and private writing projects backed by git repos,
but the interface is stripped down and simplified. Command line access is
unnecessary (though still possible) thanks to an in-browser writing interface.

While still relatively early in development, I think Penflip has big potential
in academia and science. If anybody here is interested in this space, I would
love to hear your thoughts on my project.

~~~
ekianjo
You say Penflip is for non-programmers, but your project page says stuff like:

* Markdown support

* Built on Git

and non-programmers have no idea what these are and why they should care. You
should probably do some user testing with non-programmers to figure out what
claims are relevant on your front page.

EDIT: pricing is confusing as well. It says "Plans can be paid monthly or
annually.", but the paying plans mention that they have to be "(paid
annually)" which does not make sense. Can you clarify ? So can you actually
pay per month or do you have to pay per year ?

Additionally, what is the license for public projects ? GitHub makes it clear
that they have to be open-source. How about on Penflip ? Can they be open,
while retain a copyright license ?

Also, why don't you have some intermediate plans ? Imagine I'm writing a book,
having the 8 dollars plan for 50 projects seems completely overkill, I'd
probably want a plan in there with 2 to 5 project or something like that. 50
seems like a company/organization plan.

And it's not clear what "premium support" means in the pricing, nor why we
should care about it.

This being said, it's a good project, but I see many ways you could improve on
how you communicate around it.

~~~
bonchibuji
I second this. I found out about Penflip earlier today and was interested in
using it for my fiction writing hobbies. But the 50 repos plan is an overkill
for me. I would also like to see some smaller plans, maybe for 10 repos?

------
dfc
I came here to comment that it is a little strange that a project organized
around web publishing would not have hyperlinked footnotes. I had forgotten
how distracting it was to have to scroll down to the footnote and then scroll
back up and try to get back to where you were in the paper.

Halfway through this comment I realized that I could not recall if there was
any discussion about uniform publication style/format. Would papers in the
_OJA_ all have the same format (including hyperlinked footnotes!) or would it
be a potpourri of different formatting quirks?

------
p4bl0
> if our editorial board had been paid for their work (as many are)

To my knowledge, the vast majority of editorial boards are composed of
researchers who are not paid by the publisher (they have their normal salary,
the same as if they did not take part in the editorial board), nor even have a
contract with the publisher. I don't think it's true that many editorial
boards of academic journals are getting any money out of this job.

------
chmike
In traditional journals, the outcome of reviews is a yes or no for the article
to be published. My feeling is that this is stressful and unecessarily
constraining for an open journal on internet. Why not assigning points and let
reader set their own point sum threshold when they subscribe ? The review
mechanism could then be more automatic and wouldn't need an editor.

~~~
dagw
My wife has peer-reviewed a number of articles and she doesn't just provide a
Yes or No. The final outcome is one of four choices. Publishable in current
form, Publishable after minor revisions, Not publishable in current form or
not suitable for journal. In addition to this she is expected to provide at
least half a page of critique and feedback that is (anonymously) passed on to
the authors as justification for her decision.

For example the last paper she just reviewed was the third revision of paper
she rejected the first time around. The authors had obviously taken her (and
others) comments on board and produced a better paper because of it. Had
instead the paper been published online in its original form, but with a very
low score, neither the authors or the readers would have benefited.

------
robertwalsh0
Scholastica used to have functionality that made it easy to create ArXiv
overlay journals. See an older video demonstration:
[http://vimeo.com/36720688](http://vimeo.com/36720688)

It wasn't a heavily utilized feature so we deprecated it. Maybe if the
community says there's a big need for this we'll bring it back?

------
leccine
I would like to see code review as a service too! :)

~~~
fiatjaf
I would like ideas feedback as a service, please.

------
opminion
What about peer-to-peer crytocurrency payments of peer-reviewed research,
using bounties and reviewer reputation?

~~~
wbillingsley
A colleague at NICTA was working on something similar (a market-machanism for
review, reviewer reputation, and citation) a few years ago:

[http://rickyrobinson.id.au/2009/07/24/citemine-a-new-way-
to-...](http://rickyrobinson.id.au/2009/07/24/citemine-a-new-way-to-do-peer-
review-and-publishing)

