
1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud - brahmwg
https://www.acsh.org/news/2018/10/30/1-4-statisticians-say-they-were-asked-commit-scientific-fraud-13554
======
jboggan
I was going to give a lecture across several departments about my PhD research
in bioinformatics. The night before the talk I was generating some new figures
and saw something weird, which led to me digging through source code and
discovering a bug which invalidated the last 18 months of research and all my
conclusions. I went to my advisor with this problem and he told me to present
it anyways. I refused. I suffered consequences for that, including getting
stonewalled by my advisor whenever I tried to publish a paper. I wish I had
actually been in a position to refuse.

~~~
Xcelerate
This is par for the course with computational research. I discovered a bug in
my code the _last week_ before submitting my PhD dissertation. Luckily, all of
my code was organized in a pipeline that could automatically regenerate
everything on the fly, but I needed a supercomputer. The queue was too long
for Titan, so I set up an impromptu cluster on Azure (it was the only cloud
provider with Infiniband at the time) and paid $200 out of pocket to
regenerate the correct figures.

I wouldn’t be surprised if a significant portion of published computational
research has bugs that totally invalidate the conclusions. I think we need to
push hard to require all taxpayer-funded research to make any code that
results in a journal article publicly available.

~~~
thrmsforbfast
Open source doesn't ensure quality code.

Ideally the code would be part of the peer review process, but code review is
really expensive, so who knows how that would play out.

~~~
fifnir
It's simple:

We stop publishing in papers, and instead adopt smaller chunks of our work as
the core publishing units.

Each figure should be an individually published entity which contains the
entire computational pipeline.

Figures are our observations on which we apply
logic/philosophy/whatyouwannacallit. Publishing them alongside their relevant
code makes the process transparent, reproducible and individually reviewable,
as it should be.

We can then "publish" comments, observations, conclusions etc on those Figures
as a separate thing. Now the logic of the conclusions can be reviewed
separately from the statistics and code of the figure.

~~~
jpeloquin
> Each figure should be an individually published entity which contains the
> entire computational pipeline.

I agree in principle. But, for the experimental sciences, we need better
publication infrastructure to make this practically possible.

For example, consider a figure that summarizes compares, between several
groups, the mechanical strain of tensile test specimens for a given load.
Strain is measured from digital image correlation of video of the test. Some
pain points:

1\. There is a few hundred GB of test video underlying the figure. Where
should the author put this where it will remain publicly accessible for the
useful lifetime of the paper? How long should it remain accessible, anyway?
The scientific record is ostensibly permanent, but relying on authors to
personally maintain cloud hosting accounts for data distribution will seldom
provide more than a couple years' of data availability.

2\. Open data hosts that aim for permanent archival of scientific data do
exist (e.g., the Open Science Framework), but their infrastructure is a poor
match with reproducible practices. I haven't found an open data host that both
accepts uploads via git + git annex or git + git LFS and has permissive
repository size limits. Often the provided file upload tool can't even handle
folders, requiring all files to be uploaded individually. Publishing open data
usually requires reorganizing it to according to the data host's worldview or
publishing a subset of the data, which breaks the existing computational
analysis pipeline.

3\. Proprietary software was used in the analysis pipeline. The particular
version of the software that was used is no longer sold. It's unclear how
someone without the software license would reproduce the analysis.

Finally, there's the issue of computational literacy of scientists. In most
cases, the "computational pipeline" is a grad student clicking through a GUI a
couple hundred times, and occasionally copying the results into an MS Office
document for publication. No version control. Generally, an interactive
analysis session cannot be stored and reproduced later. How do we change this?
Can we make version control (including of large binary files) user-friendly
enough that non-programmers will use it? And make it easy to update Word /
PowerPoint documents from the data analysis pipeline instead of relying on
copy & paste?

If any of these pain points are in fact solved and my information is out of
date, I would be thrilled to hear it.

~~~
fifnir
1 ans 2: I like IPFS for this, check it out

3: analysis that uses propriatory is marked appropriately as second class

> computational literacy of scientists

Welp...

------
danieltillett
The pressure to get positive results is just too high to be compatible with
good science. When you have people's whole life and career on the line all the
time (publish or perish) then people are going to do what they have to stay
employed.

Like Gresham's law bad science drives out good, because it is much faster to
do bad science than good. Those that try to maintain quality can't pump out as
many papers as the bad and so lose on the grant treadmill.

Any solution that does not address the incentives is doomed to fail. Not
fixing this problem will kill science.

~~~
zamalek
Science is a field where failure should be celebrated; see Thomas Edison and
the lightbulb.

~~~
gmueckl
... Except that Edison succeeded.

What kind of failures should we celebrate? There are two kinds that I can see:
a failure to produce a result because of a lack of skill or knowledge and a
failure to produce a result because of a demonstrable or provable reason. The
last one is actually a success because that kind of failure produces new
knowledge. But we have a culture in far too many fields where this is seen as
not publishable.

~~~
Fomite
I have always wanted a 'Journal of Cautionary Tales', for studies that were
well thought out, approached correctly, and just didn't work.

------
truantbuick
Beyond overt fraud, I feel like there's so much incentive for imprudent
analysis.

Several times when I've worked on a project at work that involves data
analysis, I get really impatient responses when I don't come to a firm
conclusion. We're not even talking about highly charged subjects. I'm not
aware if my stakeholders are biased toward one conclusion or another -- I
think they're just very upset that my analysis contains uncertainty. They
expect me to be able to massage any kind of data to derive clear and obvious
facts.

Thankfully, I can refuse this without much consequence, but it's really opened
my eyes to the potential pressures to corrupt the integrity of fact-finding in
even mundane circumstances.

~~~
lucb1e
> They expect me to be able to massage any kind of data to derive clear and
> obvious facts.

Not to be rude, but if that's in your job description, then I can understand
where they are coming from. As someone in IT, it's always my fault that the
computer is not working even if the ten year old device simply broke. That
doesn't mean they are trying to get me to do an imprudent analysis.

Sure, there is a difference between operations and trying to find correct
answers to questions (i.e. science), but they do have money to make, and if
they're not (even slightly) asking you to bias your work, I can understand if
they put a bit of pressure on getting the results clear.

Edit: As u/jeremyjh points out, I completely misread this post. If it's about
uncertainty due to not enough data, then please completely disregard what I
said. (I can't delete the post anymore because there exists a reply to it.)

~~~
jeremyjh
That isn't how statistics works. You can't remove uncertainty from a given set
of data.

~~~
lucb1e
Oh _that_ kind of uncertainty. I completely misread that post.

------
ms013
Reminds me of a project (not in medicine) I had a couple years ago where my
group was in the data science role for another group. We helped out with
experimental design, data acquisition, and analysis. After all of our work,
when we crunched the numbers, we found that the hypothesis being tested didn’t
hold. The people funding the work were not at all happy, and went so far as to
claim that “the plots must be wrong”. We showed them everything we had done,
and nobody could find any issues with the analysis. But, they were irritated
because we didn’t want to play the dishonest game of farting around with plots
and numbers to make it look like the experiment worked. So, they angrily broke
up with us. On the plus side, when I looked a year or two after, it looks like
the grumpy people haven’t made any progress and are either out of business or
something.

~~~
overkalix
At least you had the data... I've been a number of times in a situation where
I was supposed to conduct extensive, thorough analyses of (expensively
acquired) data that was evidently useless. I still had to go through the
motions, knowing full well that if somehow the analysis yielded the "desired"
results, any caveats would be ignored.

------
jlg23
"Prove our new train signaling system is as secure as the old one. If you can,
you'll get your degree, if you can't, you find someone else to sponsor your
research/degree and we find someone else to do the proof." \-- Deutsche Bahn
(government held national train company in Germany) in around 2001 to a friend
of mine (they and him found "someone else")

~~~
mr_toad
In government this is so common as to have a name.

[https://en.m.wikipedia.org/wiki/Policy-
based_evidence_making](https://en.m.wikipedia.org/wiki/Policy-
based_evidence_making)

------
2spicy_thrwaway
Even within the statistics community, there's a spectrum of quick-and-dirty vs
fully rigorous. People with the ability and inclination to be fully rigorous
often get treated as pedants and perfectionists (in the bad sense).

I often get that with business partners. "The data says <likely X, but with
caveats/nuance/uncertainty/under certain assumptions we can't justify>" to
which they respond "Can we just say X?" Or "can we get numbers on Y to support
a presentation on Z?" when Y _seems_ to support Z, but actually you can't draw
that connection, so it's misleading.

Stuff like this happens because people treat extra rigor as pedantry and are
comfortable making supporting assumptions that aren't supported by data. The
people making fraudulent requests aren't aware that they're fraudulent
(usually). In my experience, they just think they're being practical.

~~~
tomrod
I agree, and see this regularly.

"Picking battles" is one way to describe my counterpoint: caveat exactly as
much as needed so that a proper decision can be made with the risks involved.

If you want to query your whole team for a joint lunch location but coworker X
is out, it is still appropriate to say (assuming more than you and X on team)
that you asked the team and you decided on lunch place Y. It's not rigorous (X
is left out) but it's still accurate.

This is very different from, say, regulatory or securities reporting where
ambiguity is not appropriate.

------
LeonB
It’s “nearly 1 in 4” in the text, but I guess they altered the statistics
slightly to make their case look better.

------
postit
That happens, a lot. Not only statisticians.

My friend was half way over her 5 years vesting with a startup as when the CFO
asked her to help then improve their numbers due a foreseen investment round.
The idea was basically bump revenue basing it solely on the GMV masking
returns and not calculating discounts and shipping. They also wanted to hide
running costs by forcing vendors to agree on a 90+ days billing cycle, they
also pushed the CTO on turning off parts of the system during weekends and
holidays and forbid PTO until the deal was closed.

She refused doing the number masking and was asked to leave.

~~~
HillaryBriss
The story I heard related to an actuarial consultant who, after analyzing data
and suggesting a defensible year-over-year rate hike to their insurance
company client, was asked to change their analysis to support the rate hike
the insurance company wanted.

And, of course, if the consultant didn't play ball, the insurance company
could always consult with a different actuarial firm.

~~~
sk5t
That's the name of the game for certain types of consulting--support the
already-made decision. Mostly, but not exclusively, seen in management
consulting to support staff reduction, outsourcing, and the like.

------
dev_dull
The troubling part of this article (and _especially_ the comments) is how easy
it is to be labeled “anti science” today when questioning certain studies. I’d
hate to think our culture is causing us to lose the required academic rigor to
make data-based decisions.

~~~
dwaltrip
It's a tricky thing. There is a lot of high quality science. There is some
lower quality science -- outright fraud even, unfortunately. And then there is
science reporting, which is all over the map, and can easily be
misinterpreted.

Humanity has built up an enormous amount of legitimate scientific knowledge
and understanding. There has been much difficulty, confusion, and dead-ends
along the way -- and it seems this continues to be the case. But it really is
an incredible thing, how much we know at this point. It took me a long time
and a lot of self-learning to develop a rich appreciation for this.

Of course, there is still much for us to figure out. And we should keep doing
so, difficulties be damned (and hopefully mitigated over time).

------
wwhuang
Very interesting study. I'd love to see a follow up on 2 things:

1) Did the statisticians refuse or comply with the request?

2) When they refused, how did the requester react? Were the requesters
actually malicious, or just bad at statistics? If everything was fine once the
statisticians explained that removing "just an outlier" wasn't a valid option,
then this report isn't quite as concerning, and is maybe just an indication
that more researchers need to hire statisticians to help them out.

------
IronWolve
I was working on the Cingular and ATT merger, my upper management asked me to
fudge the numbers on roaming. I would gave the numbers in blue and orange, and
blue (att) had better data handoffs on 3g. It made orange look so bad, (shitty
network), they made me combine them, to which tanked ATT's network stats to
the c-level.

One of many times, stats made someone look bad, and they made me change them.

Most the time, support metrics are altered to make it look like support
contracts are hitting everything contractual.

This is daily business everywhere in tech. Mostly fudging to downright lying,
just depends on the importance of the data if some mone is tied to it.

I don't approve of this shit, but lucky, I never been asked to commit fraud,
like reporting sales that don't exist...

~~~
WrtCdEvrydy
We've had this issue before with people scraping our websites, as our software
gets better at detecting scraping, our number of page impressions went down.

Luckily our board was pretty understanding the reduction in infrastructure
strain took a nice percentage out of operating costs.

------
DrNuke
Sensitivity / ablation studies plus datasets opensourcing should be made
mandatory for every computational proposition and research asap, so that
results can be checked by peers and by the community at large. With publish-
or-perish pressure, democratisation of tools, rogue parties and
hyperspecialisation, is it getting to the point almost nothing would sustain
any stringent scrutiny? Science cannot afford opacity any more, more than that
in these blurred times.

------
DoreenMichele
_Of the less serious offenses, 55% of biostatisticians said that they received
requests to underreport non-significant results._

Sigh.

------
raincom
That's why one should big consulting companies like McKinsey, Bain, BCG, etc,
who don't need to use hardcore statistics to prove whatever the board/CEO
wanted in order to execute some policy.

~~~
casefields
You mean like good ole Arthur Andersen? Even the big boys will play dirty for
the right price.

------
phaemon
...in the USA, in medical research...

~~~
adtac
That title is too long for HN. Sometimes I wish there was a limit longer than
80 characters so that no one has to resort to slightly more sensational
titles.

~~~
cwyers
I don't know, it's actually more upsetting if it's just medical research?
Otherwise I could've told myself it happens less when it really matters.

------
BenoitEssiambre
This didn't even cover the most frequent type of fraud which is to redo the
experiment, or reanalyze the data until a sample of noise looks like a signal
and then pretend you found something (as illustrated here:
[https://www.xkcd.com/882/](https://www.xkcd.com/882/)).

I got out of academia and stopped trusting most university research because I
observed too much of this culture of fraud.

------
weberc2
I wonder how prevalent this sort of thing is in other fields. I also wonder
how prevalent fraud is generally, and the extent to which it is driving our
decreasing faith in the academy. Also, how much (if any) of this fraud is
motivated by political (instead of strictly personal) outcomes.

------
Dowwie
What is the "American Council on Science and Health"? See here:
[https://www.sourcewatch.org/index.php/American_Council_on_Sc...](https://www.sourcewatch.org/index.php/American_Council_on_Science_and_Health)

1 in 4 Statisticians weren't asked to commit scientific fraud. Within the
article:

"Researchers often make "inappropriate requests" to statisticians. And by
"inappropriate," the authors aren't referring to accidental requests for
incorrect statistical analyses; instead, they're referring to requests for
unscrupulous data manipulation or even fraud."

This isn't even remotely close to what the title of the article claims.

------
DoreenMichele
I have serious health issues. I've gradually gotten healthier, in part because
I am skeptical of a lot of studies. I know how borked so many of them are, not
a thing most people want to hear me assert. I get a metric fuck ton of flack
over my skepticism.

I kind of have a mental box where I squirrel away little tidbits that meet two
criteria: 1. They seem to come from rigorous studies and 2. They also fit with
my general understanding of how life works.

Over time, I mentally group things -- a la A and B seem related -- without
assuming that I know how they relate. I seem to have an inordinately high
tolerance for ambiguity.

Most people seem to need An Answer even if it's wrong. They have two
categories -- black and white -- and when presented with purple or pink or
blue, they force fit it to one of their existing categories and don't confuse
them with the facts.

I wasn't trying to prove anything to anyone. I was just trying to deal with my
life. But having gotten substantially healthier, I now wonder how to talk
about such. It seems a wasted opportunity for the world for me to not share,
but the world has treated me pretty horribly and done all in its power to tell
me to STFU, I'm just crazy and spouting nonsense.

So I sometimes think I should write what I understand to be true and then
carefully back it up with citations to try to support it. Then I read articles
like this, throw my hands in the air and go "Why bother?"

The way I have been treated seems particularly unfair when you learn how much
"real scientists" cook the books. Like why? It feels like pure prejudice when
I read things like this.

------
jimnotgym
1 in 4? How do we know that number is not made up?

In fact there is a 25% chance that the person involved in working that out has
been asked to commit fraud at some point.

~~~
ubitaco
>In fact there is a 25% chance that the person involved in working that out
has been asked to commit fraud at some point.

But there's only a 10% chance of that.

------
usgroup
Another statistic just in: 3/4 statisticians lie about not committing
scientific fraud

------
aluren
To be honest I'm surprised it's not higher. Relevant discussion:
[https://news.ycombinator.com/item?id=17789308](https://news.ycombinator.com/item?id=17789308)

The bottom line is that there's not much incentive for doing the boring
statistical validations (I can tell you that _no one_ likes doing statistics
apart from statisticians, and not even all of them) and verifying that
everything is reproductible, and a _huge_ incentive for, let's say, 'arrange'
a thing or two so that the paper looks better. So many people just kinda do
it, and it slips through the cracks because:

-In many fields, the reviewers are not statisticians themselves

-No one really bothers to download the data, setup the environment and libraries, and reproduce the exact steps taken to obtain the very same figures in a paper. Which is understandable given how doing all of that can be such a chore. Anyway, most of the time the exact steps aren't even described. No, jupyter notebooks aren't a solution either (it would take too long to explain why and the post is already long).

-In many cases, the results turn out to be true anyway so people don't notice they were initially put forward with fraudulent validations

-When they turn out to be false, people just shrug and move on with what's actually true. Bogus results often fail to stand the test of time and get forgotten quickly despite initial hype. No one bothers to say: "Hey, that paper from 8 years ago is bullshit and their authors are hacks!" because nobody cares.

-There's a huge psychological barrier to actually call out one of your peers and affirm that they're an impostor and their work is bogus. Especially when said impostor happens to be a big name in the field, that part of you still doesn't believe they would commit fraud, not to mention the social repercussions and backlash of doing such an accusation. We scientists aren't a very confrontational bunch.

So most of the time it kinda works and we're all bumbling along hoping to find
some modicum of truth at the end of the road with minimal harm done. But of
course sometimes you get these guys who kick off their entire career on a high
profile, much hyped fabricated result (Woo-Suk) or even a _series_ of bogus
papers (Sato), and _that_ may lead to long-term harm. The good news is that
hyped papers or very prominent figures quickly attract scrutiny from their
peers, and sooner or later reality catches up with them as labs around the
world fail to replicate their 'breakthroughs'.

All in all, I'd say we're doing fine. We're just not the ethereal source of
truth that some people hold us to be, the very same people who, after claming
that 'God is dead', are very quick to replace Him with His sillicon-based
equivalent around which we would act like priests, except with lab coats in
lieu of clerical garments.

~~~
yesenadam
Ok, so why _aren 't_ jupyter notebooks a solution?! I was gonna suggest that
also in response to another comment above suggesting figures should be
published 'alongside their relevant code'.

~~~
aluren
They are so _unwieldy_. JSON formatting means the ipynb format is a huge mess
to handle by simple means, so you _have_ to open a browser window to do
anything. Loading, editing, formatting, saving is just... ugh. You can't do
proper diffs (though some tools are trying to alleviate that). Loading,
stopping and restarting kernels is excruciatingly slow. Because cell execution
isn't necessarily done in order it can become very, very easy to lose yourself
and feel like you're back in the 70's experiencing GOTOs and the joys of
spaghetti code. And that's from _my_ point of view with a technical
background. Imagine explaining notebooks to a biologist to whom writing any
line of code is still something new and daunting. Imagine their reaction when
they happen to mess up one cell in their own execution order and alter all the
subsequent workflow, try to grok some of the magic %commands, or get some
cryptic error message due to some config file not having the proper rights
because they installed a library with 'pip --user' and not 'sudo pip'. It's
not realistic to think that an entire community of people who aren't
technically minded, many of which actually loathe or fear anything looking
like code, is going to adopt a tool that even technically minded people
struggle to use they way it's intended.

On top of that, many steps necessary to reproduce a pipeline typically need to
load enormous datasets. Terabytes of simulated protein structures, hundreds of
gigabytes of sequencing reads, phylogenetic trees, alignment files, what have
you. Once you somehow acquire that dataset, you need the appropriate tools,
many of which need to be specifically compiled for your platform, then run
them onto a powerful machine if you don't want the pipeline to take months to
complete, etc. (And that's if you didn't use any proprietary software or any
GUI based application with no command line interface.) You can't exactly load
all of that into Github+Mybinder and call it a day. You can't ask a community
of people that doesn't like coding to learn about Docker containers either.

Nevertheless, we (at our lab) _do_ use notebooks when we can because we know
they're fashionable. We can only present parts of our results, though, due to
the aforementioned constraints, but it's still pretty looking and people like
them, so we write short demos using them.

------
tomahunt
"Can you make the error bars a bit smaller please?".... "um no, I'm affraid
they are as I've calculated "

------
sjg007
Is this result statistically significant?

------
tomlock
What's odd to me is that some people seem to, even in the face of this, think
it's fair to criticize liberal arts journals for sometimes publishing bad
papers. People are just people, in every field.

------
nathan_long
"1 in 4 Statisticians Say They Were Asked to Commit Scientific Fraud"

It would be hilarious if this report were debunked on the grounds that they
inflated the number of statisticians surveyed

------
ACow_Adonis
... and 1 in 4 not prepared to answer truthfully, 1 in 4 not having worked in
the field long enough, and 1 in 4 not competent enough to know the difference.

/haha, only serious.

------
agentofoblivion
How do I know this study isn’t part of the 1/4?

------
rwilson4
Most statisticians work for people who are trying to sell something.
Statistics is not always the best way of selling something.

------
anarchop
I think that this should prompt scrutiny in fields that are highly dependent
upon statistical analysis, such as climate change. However, this very
suggestion is intolerable to many and in a way equivalent to blasphemy against
the church science. Scrutiny will not be tolerated by academe with regard to
ideas held as sacrosanct. I find this culture in academia pretty disgusting.

------
TwoBit
I wonder what the stats are in China.

~~~
brahmwg
80% of China’s clinical trial data are fraudulent, investigation finds

[https://www.bmj.com/content/355/bmj.i5396](https://www.bmj.com/content/355/bmj.i5396)

------
jiveturkey
'25%' would have been a more appealing headline ... :D

------
jamesb93
Is this fraudulent stats?

------
coldtea
The other 3 in 4 just lie about it.

~~~
munk-a
Alternatively just one statistician lied, the one conducting this study.

------
edoo
I can't help but wonder what the rate is that didn't admit to being asked to
commit scientific fraud because they did.

------
orthros
And the other 3 lied

