1. You're very angry at some people in the field. I get that, everyone in science shares these feelings to some degree. But I think that kind of bitterness is bad for your objectivity and [more importantly] bad your soul. You need to find a way to let it go. This isn't shade, it's genuine advice. Holding onto this resentment is terrible for your mental health, and it ruins the joy of actually doing science for its own sake.
2. Substantively, arXiv isn't "vanity press." Your use of this term again makes it seem like you are fixated on the role of publication for academic credit rather than publication as a means to communicate results. A number of fast-moving fields use preprints as their primary communication channel (ML is a big example.) Even slow-moving fields rely on preprints to exchange new ideas. There's a higher risk of incorrect results and "spam" in this area, but scientists routinely work with these results anyway because that's how we learn about new ideas quickly.
(Specifically in my field [of cryptography] new un-reviewed results are useful because I can usually determine accuracy myself, either by reading the proofs or running the code. If you try to convince me that I should ignore a novel result with a correct proof because it's "not science," well, all you're going to convince me of is that you don't understand science. I realize that for experimental work this can be more challenging, but even un-replicated results can still be useful to me -- because they may inspire me to think about other directions. Adding a slower layer of "professional replication and peer review" would be a net negative for me in nearly all cases, because replication takes a lot of time. At most it would be helpful for promotions and funding decisions which again is not why I read papers!)
3. I don't expect you to reform the incentive process at NSF, NEH, Universities, etc. These are incredibly difficult tasks. At the same time, reforming that stuff is much less ambitious than what you're proposing, which is to fix all academic publishing with the follow-on effect that your reforms will then repair all those broken incentive problems. To use an analogy: you're proposing to terraform Mars in order to fix the housing crisis, and I'm suggesting that maybe we just build more houses here on Earth. If your response is that I'm being unreasonable and that building more houses here on Earth is much too hard, then you're definitely not going to succeed at building new houses on Mars.
4. Your main proposal is to (somehow) come up with a pot of money to make peer review paid. I don't hate that idea, since I would love to be paid and have better peer-review. I am skeptical that this would dramatically increase speed, quality and availability of peer reviewing, especially when you include something as nebulous as "replication" into the goals of this new process. I am skeptical that the money exists. And I am skeptical that this will prevent "cheating" and "gaming" of the resulting systems. Most likely it will prove impossible to do at all and even if you did it, it will just cause less money to be allocated to actual research.
> I would love to be paid and have better peer-review.
Well, let's build on that common ground :-)
> fast-moving fields use preprints as their primary communication channel
Note, I'm not proposing any changes in the preprint system. Maybe you can explain why you think getting faster and better peer reviews would stop researchers from rapidly sharing ideas?
> I am skeptical that the money exists.
Francesca Gino made over $1 million a year at Harvard. Its not a question of can we afford to do this, its a question of can we afford NOT to do this??
If they would have funded a $20k replication study 15 years ago to see if Dan Ariely and Francesa Gino's paper was an actual scientific result, how much money would Harvard and all the funding agencies saved?
It would have even been better for Ariely and Gino--yeah, its no fun when your hypothesis is disproven, but that's a lot better than suffering from a career-ending fraud scandal.
I think the proposal would be more than self-funding, inasmuch as it would prevent money being wasted on frauds.
> I am skeptical that this will prevent "cheating" and "gaming" of the resulting systems.
I'm sure that we will always have "evil scientist"-types. But right now, the system actually incentivizes fraud, and punishes honest researchers.
1. You're very angry at some people in the field. I get that, everyone in science shares these feelings to some degree. But I think that kind of bitterness is bad for your objectivity and [more importantly] bad your soul. You need to find a way to let it go. This isn't shade, it's genuine advice. Holding onto this resentment is terrible for your mental health, and it ruins the joy of actually doing science for its own sake.
2. Substantively, arXiv isn't "vanity press." Your use of this term again makes it seem like you are fixated on the role of publication for academic credit rather than publication as a means to communicate results. A number of fast-moving fields use preprints as their primary communication channel (ML is a big example.) Even slow-moving fields rely on preprints to exchange new ideas. There's a higher risk of incorrect results and "spam" in this area, but scientists routinely work with these results anyway because that's how we learn about new ideas quickly.
(Specifically in my field [of cryptography] new un-reviewed results are useful because I can usually determine accuracy myself, either by reading the proofs or running the code. If you try to convince me that I should ignore a novel result with a correct proof because it's "not science," well, all you're going to convince me of is that you don't understand science. I realize that for experimental work this can be more challenging, but even un-replicated results can still be useful to me -- because they may inspire me to think about other directions. Adding a slower layer of "professional replication and peer review" would be a net negative for me in nearly all cases, because replication takes a lot of time. At most it would be helpful for promotions and funding decisions which again is not why I read papers!)
3. I don't expect you to reform the incentive process at NSF, NEH, Universities, etc. These are incredibly difficult tasks. At the same time, reforming that stuff is much less ambitious than what you're proposing, which is to fix all academic publishing with the follow-on effect that your reforms will then repair all those broken incentive problems. To use an analogy: you're proposing to terraform Mars in order to fix the housing crisis, and I'm suggesting that maybe we just build more houses here on Earth. If your response is that I'm being unreasonable and that building more houses here on Earth is much too hard, then you're definitely not going to succeed at building new houses on Mars.
4. Your main proposal is to (somehow) come up with a pot of money to make peer review paid. I don't hate that idea, since I would love to be paid and have better peer-review. I am skeptical that this would dramatically increase speed, quality and availability of peer reviewing, especially when you include something as nebulous as "replication" into the goals of this new process. I am skeptical that the money exists. And I am skeptical that this will prevent "cheating" and "gaming" of the resulting systems. Most likely it will prove impossible to do at all and even if you did it, it will just cause less money to be allocated to actual research.
But if you can make it happen, I won't object.