
A bot crawled thousands of studies looking for simple math errors - MollyR
http://www.vox.com/science-and-health/2016/9/30/13077658/statcheck-psychology-replication
======
c3534l
> and whether online critiques of past work constitute “bullying” or shaming.
> The PubPeer comments are just a tiny part of that debate. But it can hit a
> nerve.

> Susan Fiske, a former president of the Association for Psychological
> Science, alluded to Statcheck in an interview with Business Insider, calling
> it a “gotcha algorithm.”

> The draft of the article said these online critics were engaging in
> “methodological terrorism.”

If these are attitudes typical of psychology, then I cannot say I consider
psychology to be a proper social science. There is a fundamental
misunderstanding of how knowledge is created through the scientific process if
the verification step is considered to be offensive or taboo. That anyone in
the field of psychology would even be comfortable publically espousing a non-
scientific worldview like that means that psychologists are not being properly
educated in the scientific method and should not be in the business of
producing research since they do not have a mature understanding of what
"scientific" implies.

~~~
dmd
I took a class with Susan Fiske in grad school, and the following is a quote
from one of her comments on a homework where I had called into question a
paper that was clearly not just statistically incorrectly done but was based
on urban-legend-level assumptions:

"These results are from respected investigators, so it's inappropriate to
question them."

~~~
jessaustin
Haha this reminds me of the feedback I got on a report in Comparative
Religion, when I questioned some French dude's really elaborate and fanciful
interpretation of the paintings at Lascaux. "Ummm... what supporting texts do
we have from 17,000 years ago?" "No, you can't ask that sort of question and
it's really disrespectful."

FWIW, I wasn't really upset by that feedback. Senior spring; I don't care!

~~~
V-2
It immediately reminded me of "Emperor Kennedy Legend: A New Anthropological
Debate" by Leszek Kołakowski : ) Highly recommended and very funny snce
there's much truth to it I think. I found it online in English:
[http://jennis22.weebly.com/uploads/3/0/5/3/30531242/emperor_...](http://jennis22.weebly.com/uploads/3/0/5/3/30531242/emperor_kennedy_legend.docx)

------
6stringmerc
What a clever and, dare I say it, fantastically useful experiment!

So much less harm than even "door knob twisting" type explorations - no, this
was using published works and pretty much running them through a process to
verify or not verify accuracy.

Unsolicited? So what! As a practiced writer I make unsolicited judgments on
language usage all the time. Are these people that completely write from their
own minds and don't use a spell check or grammar check program of any sort
before sending their material for editorial review? I'd strongly doubt it,
because it's a tool to make communication more accurate. Math and formulas
having a similar procedural check sounds quite constructive to me.

It's not bullying to point out errors; it's bullying to use the existence of
errors to belittle or insult a person. I don't see that happening here. Sure,
it's a little sterile or "cold" in this fashion, but I think that's for the
best if such a process / tool can gain acceptance. It just spits out results
and I think that's all it should do. Neat to read about.

~~~
cushychicken
I think "bullying" is a reductive term for what's actually happening. I think
the point trying to be conveyed is that there is a certain decline in the
author's personal status when they get holes punched in their publications.

~~~
zerocrates
Sounds like a nice incentive to double-check before publishing.

~~~
cushychicken
I mean, it's open source software, right? Nothing stopping you from
downloading it and running before publishing.

I view that as a pretty solid way to make the science better all by itself.

------
alanfalcon
I find it very disconcerting that people are trying to fend off criticism of
previously published studies by calling it "bullying" or sometimes worse. What
do feelings have to do with science?

~~~
merpnderp
Objectivity is listed as a problem with science curriculum. Welcome to the
future.

[http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&co...](http://nsuworks.nova.edu/cgi/viewcontent.cgi?article=2467&context=tqr)

~~~
Practicality
While I am not a big fan of contemporary feminism, I am pretty sure taking
issue with masculinity or "gendered" text is not the same thing as taking
issue with objectivity.

Edit: Found this quote in your source "Poststructuralism “rejects objectivity
and the notions of an absolute truth and single reality,”"

Hmm, this seems an awful lot like a religion.

~~~
zeveb
> I am pretty sure taking issue with masculinity or "gendered" text is not the
> same thing as taking issue with objectivity.

Well, since English is a gendered language, taking issue with 'gendered' text
is taking issue with the objective reality that … English is a gendered
language.

~~~
alphonsegaston
Aside from things like pronouns, English hasn't been gendered since the 14th
century:

[https://en.m.wikipedia.org/wiki/Gender_in_English](https://en.m.wikipedia.org/wiki/Gender_in_English)

~~~
zeveb
'Aside from that Mrs. Lincoln, how was the play?'

Pronouns are precisely what folks who complain about gender in English are
complaining about (well, that and words like 'mankind').

~~~
alphonsegaston
Glibness aside, the usage of gendered pronouns does not make a language
gendered. That's only true if there has to be agreement between the gender of
a noun and words relating to the noun.

And it's also not what people are arguing. Instead, they argue that gendered
terms are the product of cultural norms and prescriptive grammar that
reinforce gender roles that are oppressive to both men and women. That some
view these assignments as objective when that's untrue linguistically and
historically, I think only lends credence to their point.

------
mratzloff
Here's the GitHub page:

[https://github.com/MicheleNuijten/statcheck](https://github.com/MicheleNuijten/statcheck)

And if you're curious how it works, as I was:

 _Statcheck uses regular expressions to find statistical results in APA
format. When a statistical result deviates from APA format, statcheck will not
find it. The APA formats that statcheck uses are: t(df) = value, p = value;
F(df1,df2) = value, p = value; r(df) = value, p = value; [chi]2 (df, N =
value) = value, p = value (N is optional, delta G is also included); Z =
value, p = value. All regular expressions take into account that test
statistics and p values may be exactly (=) or inexactly ( < or >) reported.
Different spacing has also been taken into account._

~~~
snerbles
> When a statistical result deviates from APA format, statcheck will not find
> it.

Incentive for authors to obfuscate their math?

~~~
schmichael
It'd be easier for them to not obfuscate and use the checker themselves prior
to publishing.

~~~
snerbles
True, assuming no malfeasance on the part of the authors.

~~~
cortesoft
If you are trying to publish a fraudulent paper, you aren't going to do it by
including subtle math errors to fudge your data in the direction you want. You
would just falsify the data itself; way harder to find, and much more
effective.

------
munificent
> There’s a big, uncomfortable question of how to criticize past work, and
> whether online critiques of past work constitute “bullying” or shaming.

Science is fundamentally reputation-driven. One of, if not the primary
incentive that encourages scientists to do science work is the chance of
raising their prestige. Citations are one very quantifiable yardstick for
this.

If positive social sanctions are a driving force for science, then it's
entirely reasonable that negative sanctions should come into play too. If you
can well-cited paper and attract fame, then a poor paper should likewise
attract shame.

Otherwise you have a positive feedback loop where once a scientist has
attracted enough prestige, they are untouchable. We need negative feedback to
balance that out.

~~~
danharaj
Shaming becomes another power structure that allows the influential to bully
their peers. It's not so simple as you suggest.

------
thampiman
This coupled with the Automatic Statistician
([https://www.automaticstatistician.com/index/](https://www.automaticstatistician.com/index/))
will help fix a lot of biases and human errors that creep into scientific
research.

------
radarsat1
The article sadly doesn't report on the false positive rate of statcheck. I
assume the paper does?

I mean, it just uses a basic regular expression, I can see it easily
performing bad checks. I assume the authors take this into account.

------
vinchuco
A good distinction between "peer reviewed" vs "computer verified"

>“The literature is growing faster and faster, peer review is overstrained,
and we need technology to help us out,”

This is a problem in every field, not just Psychology.

I want someone to tell me the distribution (or average ratio) of papers read
to papers written.

Every thesis written is supposed to add some delta to the state of the art.
But there is no method for doing a diff between past and previous versions of
human knowledge. How to make science less redundant and more efficient?

I dream of aggregators for everything.

------
michaelt
I can certainly understand people being nervous about academic debate moving
to social media. It would be a hassle for climate scientists if every paper
got a brigade of climate change deniers criticising it and you had to respond
to those criticisms.

But this example - someone notifying you there's a mistake in your paper, when
there really is a mistake? That seems like a strong argument /for/ academic
debate via social media, not /against/ it.

------
serge2k
> some found the emails annoying from PubPeer [since PubPeer notifies authors
> of comments] if Statcheck found no potential errors

I would.

> There’s a big, uncomfortable question of how to criticize past work, and
> whether online critiques of past work constitute “bullying” or shaming.

It's facts about your work. Learn to handle it or quit pretending to be a
scientist.

> The gist of her article was that she feared too much of the criticism of
> past work had been ceded to social media, and that the criticism there is
> taking on an uncivil tone

Valid enough point. Criticism and correction can be done in a civil manner,
and in an accepted forum.

------
JBiserkov
For more context, see [http://andrewgelman.com/2016/09/21/what-has-happened-
down-he...](http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-
the-winds-have-changed/)

------
OliverJones
Remember when writers could do spell-checking and grammar-checking by "running
a program" on their text files?

Here we have numbers-checking working the same way.

I bet you this sort of feature gets built in to word processors eventually,
and puts wavy red lines under the results it flags.

We've had this sort of real-time "syntax" checking in software engineering for
half a generation. It seems wise for other disciplines to consider adopting it
too.

It's obviously got to be discretionary, just like spell-check is discretionary
in browsers.

We will get a new genre of humor, thought "statcheck fail."

------
RangerScience
Haha! What if this is actually a marketing ploy for their web-app? Stir up
some shit so everyone gets talking, and provide a service.

~~~
EdHominem
And what if? Does that negate the results?

~~~
RangerScience
Oh no not at all! The results are still solid. So, "ploy" isn't the right
word, not really. "Stunt"? "Technique"?

I was just thinking - They could have found a way to email or otherwise
contact the authors, instead of just posting comments, but posting comments
(and getting some news on it) drives more traffic due to scandal effects.

This rabbit hole can go deeper - what if private disclosure was expected to
backfire? By surprising people publicly, they're more forced to admit there's
an issue; and then by providing both the issue and a solution, there's an easy
way at hand to fix it....

~~~
EdHominem
If I use software and there's a bug, I don't want to find out years later when
the author decided to admit it, I want to find out right away so I can get it
patched. Papers are equivalent to code, if I'm developing something based on
one I want to know if the foundation is reliable or not.

> what if private disclosure was expected to backfire?

Well of course it could be expected to go badly. Look at all the "concern"
spent over their methods already.

But yes, it would have been interesting if they had contacted authors quietly
and without telling them their methods, simply to see who updated their paper
and who tried to blame the messenger... Then publish that list and let the
field really correct itself.

------
yakult
While it is definitely to the benefit of all that the bot emails authors when
it finds mistakes, emailing when it doesn't find anything is a dark pattern.
Reminds me of those bots that spam me after scraping my linkedin.

------
l0b0
Why does the article focus even for a paragraph on whether egos would be
bruised? If the result is a general improvement to the readers' understanding
that is, as far as I'm concerned, case closed. Good on them!

------
ChoHag
Will I never guess what number 7 is?

(DNR)

------
orionblastar
I wanted to make a program like that, but considered the ethics of it.

