There are obviously some things I could usefully do there: it's often possible to identify and reject total crank books, to notice some early tendency towards overstating claims, to give general feedback on the direction of the book and whether it is likely to reach its audience, etc. But it's not the same as a journal, where you have the full paper to review and are expected to read it and give detailed feedback (which is also a more reasonable thing to ask of reviewers, because the text isn't as long!).
This reviewing approach is pretty traditional (it's how most famous philosophy books were published, for example) and relies heavily on good faith. Reviewers are supposed to judge whether this is a serious historian writing a good-faith book, but if they say yes, then you let the historian publish the book they want to publish without a lot of further rounds of vetting. It isn't expected they'll get everything right, but it's expected they'll make an earnest effort to get it right, perhaps even by employing research assistants themselves to do fact-checking. Anything they get wrong can then be corrected in the literature by someone else writing their own article or book in response.
One exception are books which are adaptations of dissertations. In that case there would have been more substantial review of the claims, by the dissertation committee, assuming nothing fishy happens between dissertation acceptance and adaptation into a book. The first author of the paper linked here (Lieberman) has a book out on the history of sex toys that's an adaptation of her dissertation, and tells a pretty different history than the one she criticizes in this paper. The book being criticized (by Maines) is not a dissertation adaptation, though.
I'd like to be a professional reviewer. The closest existing job is a patent examiner, but they incentives there aren't great either. After my PhD I intend to at least do some consulting on the side, and perhaps "professional reviewer" could be a service I could offer. Worst case is that no one wants the service.
Yeah it's always interesting as an outsider to read about settled scientific wisdom being enthusiastically challenged. I liked:
* the Gerta Keller dinosaur asteroid story -- https://www.theatlantic.com/magazine/archive/2018/09/dinosau...
* the Rebecca Fried "No Irish Need Apply" back-and-forth -- https://web.archive.org/web/20150805045521/http://intl-jsh.o...
I only skimmed around so am perhaps misinterpreting their tone, but the authors seem to emphasize the vulnerabilities of empirical research in the humanities specifically, but I’m wondering if any field of research is not vulnerable to widely propagated falsehoods, or at the very least, poorly verified findings.
"Yet the von Neumann proof, when you actually come to grips with it, falls apart in your hands!... It’s not just ﬂawed, it’s silly... You may quote me on that: The proof of von Neumann is not merely false, it is foolish!"
This is more of a mathematical physics example, but everybody in every field could stand to have a bit more humility and rigor.
is basically true 100% of the time-it is called the linearity of expectation and is a basic fact of probability.
Could you include the papers or provide a more detailed comment?
Typically at conferences you have a area chair who is selected due to some connection with organizer and then this area chair choses other people is his/her network to be reviewers for submissions they receive. So from top to bottom, the process is driven by social connections and very often used as paying back past favors or create a future favor. I have coined even term for this: favor economy.
One way to fix this would be to create a metric for reviewer like r-index in similar line to h-index which rewards reviewer for choosing high impact papers and somehow penalized them to pass on good papers. On a simplest level, r-index could simply be same as h-index for papers reviewed instead of authored. However I think there ought to be better measure.
I didn’t carbon date them, but, yeah, they existed, and they were used.
I always thought that whole "Victorian physicians used vibrators to treat female hysteria" thing seemed fishy...
We examined every source that Maines cites in support of her core claim. None of these sources actually do so.
I wonder if, when NLP is sufficiently robust, one could automate this fact-checking process en masse...