
A Failure of Academic Quality Control [pdf] - luu
http://journalofpositivesexuality.org/wp-content/uploads/2018/08/Failure-of-Academic-Quality-Control-Technology-of-Orgasm-Lieberman-Schatzberg.pdf
======
budadre75
Independent professional reviewers should exist in the academia. Currently
it's those who do research also serve as reviewers, but these people are very
busy in their own research and don't necessarily review others' papers
carefully. It's a shame that most of the funding goes to researchers who
perform novel research, not too much incentive for reproducing results and
reviewing papers. This is as if the science is built upon an unstable
foundation, and lots of man- hours and powers will be wasted from that.

~~~
_delirium
In this case the thing being criticized is a book, which is a somewhat
different case. Reputable academic presses do send out book manuscripts for
review, but it's a very different process from paper reviews, often involving
only a partial early manuscript. I don't know how every press does it, but
when I've reviewed for an academic press, I was given a draft of the first
three chapters, and asked to give a recommendation for whether this was a
serious academic book worth publishing, along with my positive and negative
comments on the partial draft.

There are obviously some things I could usefully do there: it's often possible
to identify and reject total crank books, to notice some early tendency
towards overstating claims, to give general feedback on the direction of the
book and whether it is likely to reach its audience, etc. But it's not the
same as a journal, where you have the full paper to review and are expected to
read it and give detailed feedback (which is also a more reasonable thing to
ask of reviewers, because the text isn't as long!).

This reviewing approach is pretty traditional (it's how most famous philosophy
books were published, for example) and relies heavily on good faith. Reviewers
are supposed to judge whether this is a serious historian writing a good-faith
book, but if they say yes, then you let the historian publish the book they
want to publish without a lot of further rounds of vetting. It isn't expected
they'll get everything right, but it's expected they'll make an earnest effort
to get it right, perhaps even by employing research assistants themselves to
do fact-checking. Anything they get wrong can then be corrected in the
literature by someone else writing their own article or book in response.

One exception are books which are adaptations of dissertations. In that case
there would have been more substantial review of the claims, by the
dissertation committee, assuming nothing fishy happens between dissertation
acceptance and adaptation into a book. The first author of the paper linked
here (Lieberman) has a book out on the history of sex toys that's an
adaptation of her dissertation, and tells a pretty different history than the
one she criticizes in this paper. The book being criticized (by Maines) is not
a dissertation adaptation, though.

------
rossdavidh
One is reminded of Derek Freeman's "The Fateful Hoaxing of Margaret Mead", in
that the theory seemed so good no one had the heart to challenge it. But then,
Freeman's analysis is itself still controversial, so who knows. It will be
interesting to see a response from Rachel Maines.

~~~
mherdeg
> But then, Freeman's analysis is itself still controversial, so who knows.

Yeah it's always interesting as an outsider to read about settled scientific
wisdom being enthusiastically challenged. I liked:

* the Gerta Keller dinosaur asteroid story -- [https://www.theatlantic.com/magazine/archive/2018/09/dinosau...](https://www.theatlantic.com/magazine/archive/2018/09/dinosaur-extinction-debate/565769/)

* the Rebecca Fried "No Irish Need Apply" back-and-forth -- [https://web.archive.org/web/20150805045521/http://intl-jsh.o...](https://web.archive.org/web/20150805045521/http://intl-jsh.oxfordjournals.org/content/early/2015/07/03/jsh.shv066.abstract)

~~~
rossdavidh
These are all interesting, and I hadn't heard about the NINA one, thanks! Of
course, the HIV skeptics turned out to be...just plain incorrect. So sometimes
the settled scientific wisdom is correct. But, not always...

------
ciacci1234
“The success of Technology of Orgasm serves as a cautionary tale for how
easily falsehoods can become embedded in the humanities”

I only skimmed around so am perhaps misinterpreting their tone, but the
authors seem to emphasize the vulnerabilities of empirical research in the
humanities specifically, but I’m wondering if any field of research is not
vulnerable to widely propagated falsehoods, or at the very least, poorly
verified findings.

~~~
antidesitter
Mathematics, for the most part. I wager this is due to the low material cost
of verification.

~~~
jazzkingrt
It's fairly easy for well-intentioned mathematicians to make small mistakes in
their reasearch that drasitically affect the outcome of a proof. The sorts of
falsehoods discussed in this paper are a consequence of lying about the source
material, or a severe lack of due diligence. So these fields probably have a
lower prior for a falsehood in a paper.

~~~
antidesitter
But in the vast majority of cases, these small mistakes are caught by other
mathematicians. This is because verifying a proof is easier (in terms of
resources) than, for example, replicating a randomized control trial (with the
necessary equipment and subjects).

~~~
mkl
Not necessarily. Most papers are only ever read by a few people, and even very
well cited papers can contain errors which may not be corrected in public
unless they significantly change the outcome (but which can make analogous
proofs more difficult, as I found with a 2000-citation paper in my PhD's field
(its outcome wasn't affected)).

------
sytelus
This paper has a rather lofty title but it described just but one case were
peer review failed. Sample of one is hardly generalizable. While the paper is
virtually an “academic link bait”, there is no researcher who doesn’t thinks
that review process is broken.

Typically at conferences you have a area chair who is selected due to some
connection with organizer and then this area chair choses other people is
his/her network to be reviewers for submissions they receive. So from top to
bottom, the process is driven by social connections and very often used as
paying back past favors or create a future favor. I have coined even term for
this: favor economy.

One way to fix this would be to create a metric for reviewer like r-index in
similar line to h-index which rewards reviewer for choosing high impact papers
and somehow penalized them to pass on good papers. On a simplest level,
r-index could simply be same as h-index for papers reviewed instead of
authored. However I think there ought to be better measure.

~~~
mkl
The title is literally " _A_ Failure ...". One single failure, and that's what
it describes. There are only a handful of paragraphs about generalising, and
these are knowingly subjective; two of them start with "We believe".

------
jl2718
There was a guy at DEFCON showing his collection of these antique devices, and
some literature from the era advertising their medical use.

I didn’t carbon date them, but, yeah, they existed, and they were used.

------
gone35
_[W]e could find no evidence that physicians ever used electromechanical
vibrators to induce orgasms in female patients as a medical treatment [for
hysteria]._

I always thought that whole "Victorian physicians used vibrators to treat
female hysteria" thing seemed fishy...

\--

 _We examined every source that Maines cites in support of her core claim.
None of these sources actually do so._

I wonder if, when NLP is sufficiently robust, one could automate this fact-
checking process _en masse_...

~~~
creaghpatr
Probably augment, but not automate

