
A paper’s central finding was based on polling that probably never happened - remarkEon
http://fivethirtyeight.com/datalab/as-a-major-retraction-shows-were-all-vulnerable-to-faked-data/
======
Fede_V
The incentive system in science is immensely screwed up. Researchers are
rewarded not for high quality work flows, reproducible results, and solid
methodology but for neat, linear stories with a simple high impact result.

Imagine that a scientist designs a very intelligent experiment to test a
plausible hypothesis - she has an excellent plan to analyze the results (pre-
registered, to avoid p-value hacking) - and after doing the experiment, she
finds no evidence of the effect that she was looking for. Why should the
result (which is the only thing the scientist is not in control of) determine
the reward given to the scientist? Funding should be given to investigate a
particular phenomena, _not_ to find high impact results.

~~~
jacquesm
The reason funding goes to those that find 'good' results is the same reason
why funding goes to those that start successful start-ups. Even though both
parties probably have less control over the outcome than they think the
_perception_ is that they do and that betting on them again is a better bet
than betting on someone that did not succeed to score.

That's how we're wired, we simply like success stories much more than that we
like stories of failure.

~~~
digi_owl
Just world?

~~~
verbin217
False inherencies are the staple of those that wish to understand the world
such that they might exploit it. If they weren't prepared to demonstrate that
the pursuit of a solution was necessarily a fool's game then they'd have to
give an actual shit.

------
hudibras
It's a little hard to tease out from this particular article, but the amazing
thing is that the canvassing actually happened: dozens of volunteers talked
with hundreds of people at 22 minutes for each conversation. Easily thousands
of man-hours of work. But the grad student running the study didn't do the
easy part, hiring a survey company to poll and measure the before-and-after
attitudes of the people being studied.

It's one thing to get data and then fraudulently change it to fit a story; at
least then other people can go back and un-do the damage. It's a different
thing to not do anything at all, completely make up data, and waste all that
hard work.

------
caminante
Here's the scary part:

    
    
      As Peter Winker, an economist at the University of Giessen 
      in Germany who co-edited a book about poll fraud, put it in 
      an email: “If the faker would have been a bit cleverer in this 
      procedure, I doubt that the fraud would have been found this way.”

------
haberman
Even when data isn't outright fabricated, it often seems that studies like
this are looking for a predetermined outcome that will reinforce the
preconceived ideas of the authors. I feel so skeptical anytime a study is
published that purports to show the truth of some idea that people would
really _want_ to believe ("if only they just _met_ a gay person it would
change their mind.")

~~~
pedrocr
_> it often seems that studies like this are looking for a predetermined
outcome that will reinforce the preconceived ideas of the authors._

Isn't this the definition of how science _should_ work. 1) Formulate an
hypothesis, 2) design an experiment to prove it 3) and then run it rigorously
and accept whatever is the result.

This case (and many others) failed 3) but everything else seems more than
natural. Physicists smashing particles at the LHC were also trying to
"reinforce their preconceived ideas" of how particles work. They have the
standard model and they want to validate it where possible.

~~~
haberman
I think the problem is that it takes diligence to design an experiment that is
not biased towards the desired result.

I also think it's a systemic problem of positive results being far more
rewarded (in terms of funding, prestige, etc) than negative results, which
gives even more incentive to bias the result (intentionally or
unintentionally) towards the positive, satisfying outcome.

Imagine you have two social scientists, A and B. Scientist A publishes a
positive result that resonates with people; it fits the social and political
climate of the day. They give TED talks about it, and the result is reported
in newspapers. Later experiments are unable to replicate the result, maybe
because the original experiment was a little biased somehow, but nobody can
say for sure and there is no scandal because there was no fraud. Scientist B
is more rigorous in their experimentation and their work fails to demonstrate
a hypothesis. They either publish the negative result, or don't publish at
all.

Which scientist will have advanced their career more? I think that is the
problem.

~~~
pedrocr
Yeah, but then the problem is that the experiments are crappy not that the
hypothesis generation is at fault. You used social scientists and I used
physicists and that is telling. The social "sciences" are often far from
scientific. If a physicist suddenly goes bonkers and claims that space is
filled with ether he can't go very far because he can't generate a spurious
positive result in favor of his crazy hypothesis.

~~~
haberman
I used social scientists because that is what this article was about and what
my comment was about. I agree that in physics (the hardest of hard sciences)
it is harder to fall prey to this kind of false positive result.

------
KurtMueller
There should be a Github for Academia where scientists open source their data
& formulas the use to draw conclusions - people could comment and annotate on
specific parts, fork & create pull requests, and file any issues they might
have.

~~~
randomsearch
There is widespread provision of data and software on computing science papers
in academia. It's currently a piecemeal approach, which is a problem, but
here's an example of how things are changing:

[http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+...](http://conf.researchr.org/track/pldi2015/PLDI+2015+Artifact+Evaluation)

Universities all over the world are working on repositories to store
electronic resources such as data and code for the long-term. This is non-
trivial, as data may need to be available for decades or even hundreds of
years. Github, by contrast, has been around for 7 years.

Regarding the idea of organising research in a more change-control fashion,
that's a more radical change. Academia will move slowly, and rightly so, as
research practices evolve and persists for hundreds of years; to adopt an idea
based on a (relatively, very) new method of software development is hugely
risky.

------
thaumasiotes
> LaCour said he couldn’t find files containing the survey data and indicated
> that he would write a retraction; when he didn’t, Green sent off his.
> Science hasn’t yet ruled on whether it will retract the original paper.

Sure, the paper's author may want you to know that it was all faked... but
_Science_ stands by what they printed, _no matter what_.

~~~
aetherson
You're misunderstanding. There are two papers here: LaCour's, which is the
original paper, and Green's, which is based on LaCour's.

Green has retracted his paper by way of essentially accusing LaCour of
falsifying data. His paper is retracted, fin.

Science has not yet decided whether it will on its own recognizance retract
LaCour's paper.

~~~
thaumasiotes
I really wish I could downvote a direct reply. There is one paper. Green, the
primary author, retracted it when LaCour, the grad student, failed to do so.
But just because the author says "this paper is full of lies" doesn't mean
Science will admit it.

------
ylem
The actual report showing the debunking is quite good!
[http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...](http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf)

This was a relatively high profile paper. I wonder how much slips through in
lower impact journals...

