I'm a recent PhD graduate that focused heavily on publication and collected ~20 publications. I think the situation with peer review is worse than discussed here.
Peer review:
1. dramatically reduces the pace of progress
2. exacerbates publication bias
3. creates a false sense of accomplishment
4. creates a false meritocracy
5. creates many perverse incentives
(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).
The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.
What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.
Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.
Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.
Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).
Those are all very real problems (and as you point out, they impact grant review as well). On the other hand, fully transparent peer review doesn't necessary address most of them. Reviewers who have to sign their reviews may be reluctant to anger colleagues (or try to curry favor with them).
As you presumably know a many journals have experimented with open peer review, but editors still need to police the reviews to look for bias. It solves some problems but creates others.
transparent review isn't a perfect solution. But the problem you suggest of reluctance to anger colleagues or currying favor are worse under the current system than they would be under a transparent one.
Just because other journals failed in the past doesn't mean we shouldn't try again in the future! Maybe there were some mistakes we could learn from? Also the internet is becoming a more familiar tool used more often by more people every day, maybe it just wasn't the right time previously.
> In the late 2000s, widespread debate and controversy ensued after Budden and colleagues (2008a) found that a switch to double-blind review in the journal Behavioural Ecol- ogy led to a small but notable 7.9% increase in the 2proportion of articles with female first authors
> Findings from studies of journals that have actually adopted the practice are non-conclusive. For example, Budden and colleagues show that the introduction of double-blind review by Behavioral Ecology was followed by an increase in papers with female first authors (Budden et al., 2008). However, a second paper reports that female first authorship also increased in comparable journals that do not use double blind review (Webb et al., 2008). A study by Madden and colleagues shows that the adoption of double-blind reviewing in the SIGMOD conference led to no measureable change in the proportion of accepted papers coming from “prolific” and less well-established authors (Madden and DeWitt, 2006), but a later study contests this conclusion (Tung, 2006).
I totally agree with the transparency thing. I would love to see something like open review (https://openreview.net/forum?id=ryQu7f-RZ) become more common, and even have something like this replace the traditional peer review system.
We would all have to think long and hard about how to align incentives to get the desired results.As a first pass I think hiring some amount of people to review every paper, and then allowing anyone who wants to come by and leave their comments as well might be a place to start. I think some sort of reputation system similar to stack overflow might be a motivator as well.
I've badly soured on open peer review, having had some experiences with it. I think you'll have very differential levels of review, and very differential values of how free people feel to go after authors.
Peer review:
1. dramatically reduces the pace of progress
2. exacerbates publication bias
3. creates a false sense of accomplishment
4. creates a false meritocracy
5. creates many perverse incentives
(3) and (4) unfortunately wrap into grant financing as well. The merit of your research isn't measured on its impact on the human condition, but on it's 'impact' factor (a measure derived from your publications).
The gatekeeper problem here is pernicious as well. If you become a highly cited author your ability to get/maintain financing improves. You also become a gatekeeper as a peer reviewer. Which means that you are now strongly incentivized to accept papers from people who cite your work or align themselves with you, and reject everything else.
What is absolutely amazing here, is that the peer review process is opaque. It is my belief that if you knew who reviewed which papers you would quickly discover that mild to severe abuse of peer review is the norm, not action by a handful of bad actors. This is because the entire academic reward system is wrapped into the process. Getting your name on a big paper can have lifelong ramifications on your ability to get grants, start companies, do consulting work, etc.
Peer reviewers should probably get paid for the work. If they don't get paid then their incentive to do review must come from somewhere else, vague notions of improving the field don't cut it. Peer review should obviously be transparent. Some people might be uncomfortable signing their name to a paper rejection, but its time to get over that. A small payment might help reviewers overcome this discomfort. It is bizarre that peer reviewers don't get paid. Peer review is valuable work if done right, and without payment all the reward of being a peer reviewer comes for the wrong reasons.
Finally, I agree with some other comments. Publication should not be contingent on peer review, it should come first. This would increase the pace of progress, reduce publication bias, reduce the false meritocracy, reduce the ability for bad actors to censor research, and more. The cost would be a larger number of publications, but perhaps this would help people realize that many of the publications coming out right now are of little value.
Transparency is badly needed in many facets of academic research. My company made a site that helps bring transparency into literature review (sysrev.com).