
Science, Now Under Scrutiny Itself - seliopou
http://www.nytimes.com/2015/06/16/science/retractions-coming-out-from-under-science-rug.html
======
hitchhiker999
The rise of 'scientism' \- until a few years ago I considered myself a
'science type' (excuse the phrase), immersed in logic all my life.

Recently the depth of negative influence, corrupt findings, blind faith and
religious-style zeal within the scientific community and pseudo scientific
communities (like Reddit) has become absurd and embarrassing.

Science is a method, not a conclusion. I hope it becomes 'open', and fast.

~~~
steve19
Science has not changed, nor has bad science. There gave always been
incentives to produce bad science, just like there has always been a moral
incentive to produce good science.

The religious zeal I think comes from fan boys/girls not from scientists
themselves.

~~~
kijin
True, there's always been bad science.

The increase in retractions can mean either that (1) there's more bad science
now than there was before, or (2) we've got better at spotting bad science.
Without further evidence, it's difficult to tell which one is the case.
Perhaps it's a mix of both.

~~~
eli_gottlieb
I'd go with (1), caused by funding pressure. If your career can't survive when
you do good science, you'll do bad science.

------
MarcScott
I wonder whether a 'GitHub' for Science wouldn't help things along, just as
it's helped the open source software community.

Scientist could have repos with full version control, so you could go back and
look at their experiments from day one. All their data would be accessible, as
well as their statistical analysis, and you could see the evolution of their
written paper.

The repos could start off private, and then opened on the day of publishing,
so others couldn't take credit for their work.

Just a thought.

~~~
beloch
Very little of what is actually needed to reproduce an experiment goes into
academic papers. What you'd really want is a more formal, digital version of
the "lab book". Experimentalists are taught, from their earliest days as an
undergrad, to keep notes on what they're doing. The idea is to put everything
useful in there. Got a new piece of gear with a long startup procedure? Put it
in your book. Designing an experiment? Put the design in there. Ordered parts?
Make a note of it. Running an experiment and finally found a way to do things
that works? Write down the procedure. Got some results from your experiment?
Print them out and paste them in your book. Saw something odd? Write it down.
The result is a linear log that tracks a lot of stuff, both useful and
useless. With experience, you learn to make use of tables of contents and
margin notes, but searching through them for something dimly recollected can
sometimes be time consuming. Lab books are where the gory details of how to
actually do things is stored. The procedures, diagrams, and details in them
are simply not conveyed in papers because there's no space for them, and many
are viewed to be "common knowledge", although this kind of knowledge is
probably only common to a small number of labs in the world.

The problem with these lab books is that they're often the personal property
of the scientists and, as such, almost a sort of diary. They're written with
yourself as the intended audience and can sometimes be impenetrable for anyone
else. If you're working in a commercial lab, as opposed to research in
academia, these books might be considered the property of your employer. In
this situation you might try to make them more friendly to other human beings,
but that often isn't a priority.

Putting things into a digital form would, in general, take a lot longer. Even
with good proficiency in LaTeX, complex equations are quicker to just write on
paper. Hand-drawn diagrams are also very common in lab books, and the
software/hardware to draw easily in a digital form is not yet ubiquitous.
Finally, digital storage occasionally dies. A lab book is a precious
possession whose loss can cause months or years of difficulty. Physical hard-
copy is easier to store long-term. Some people have undergraduate lab books
from half a century ago sitting on their office bookshelves.

There is nothing that is technologically insurmountable to moving to a digital
lab book format. Microsoft's surface tablets make digital drawing pretty
painless. Cloud storage could be maintained long-term. Etc. The benefits of
digital lab books could be considerable. They'd be easier to search and
distribute. You could include copies in a digital appendix to an experiment
(currently, most journals discourage large attachments because they don't like
to pay for storage). To my knowledge, nobody has actually put everything
that's needed together into a single cohesive program. You'd want a note-
taking program with full LaTeX and freehand drawing support. You'd need to be
able to paste in tables, graphs, and data files easily. It would need to have
a long-term archiving solution and a really good interface. It would need to
be as fast or faster than writing on paper for pretty much everything. It
would also probably need to be open source, as scientists would probably not
trust proprietary software that might make their notes difficult to access at
some point in the future. Finally, the hardware to run it on would need to be
ubiquitous.

~~~
jkimmel
>The problem with these lab books is that they're often the personal property
of the scientists and, as such, almost a sort of diary

Further complicating things, individual researchers don't actually own the
books. Often, an institution or the lab owns the rights to the book. This can
make individuals who are otherwise keen on "open science" hesitant to place
copies of this information in a publicly accessible place.

See page 4 of this manual from the NIH (National Institutes of Health) about
the ownership of notebooks.

[https://www.training.nih.gov/assets/Lab_Notebook_508_%28new%...](https://www.training.nih.gov/assets/Lab_Notebook_508_%28new%29.pdf)

~~~
beloch
Everything on that page is stuff you're going to find in a grad student's lab
books, which are their property. This document is basically a primer for
people transitioning into paid research where the lab-books become lab
property. A huge proportion of research, especially in less commercializable
areas of fundamental science research, is done by students (because they're
cheap). This means the kind of lab-book described in your link is probably
less common that you might expect!

------
hackuser
While I think increased accountability is a good thing for the world of
science, I'm concerned this will fan the anti-intellectual, anti-science
rhetoric.

Science, like every human institution, is flawed. That's not a reason to
reject everything about it.

~~~
ploxiln
It is annoying to see a retort "scientists always end up being wrong". But I
don't think science should sink to the level of fake-confidence "everything
going according to plan" of non-science, I think it should embrace its honesty
about its mistakes.

Over time people can't help but depend on science's results, and new
generations take it to heart. It's actually similar to how people can't help
but depend on open source tools/platforms, as much as they hated "free-tards"
and said "you get what you pay for" in the past.

~~~
kijin
Well, the whole point of science is to falsify past theories and produce
better theories. So of course the majority of scientific theories will end up
being wrong at some point.

So it's pointless to try to rebuke the "scientists always end up being wrong"
like of argument. If anything, I think scientists should just bite the bullet.
Theory proven wrong? That's proof that science works!

------
aaron695
The internet is eating the world. People perhaps aren't realising that yet.
Everything has changed.

Just a Google search can reveal plagiarism (by accident even).

A bot can find issues with made up numbers (Benford's law for starters)

It's all pretty cool stuff for science.

------
jugad
The outstanding line from the article (quoting Mina Bissell)...

> But it is sometimes much easier not to replicate than to replicate studies

I wonder in what cases its easier to replicate studies. Probably when the
study involves doing nothing...

~~~
Al-Khwarizmi
In computer science, the case where the easier thing to do is to replicate
something does happen sometimes.

For example if you create a natural language parser that you claim is more
accurate than another, you may need to replicate the other in order to compare
accuracies. Sometimes you can just take the authors' reported accuracy, but
often it's not that easy (maybe they just tested on English and you're
interested in performance in other languages, maybe you want a different
metric, maybe you want to compare on some standard set of features, etc.)

------
petegrif
This all seems to be 'as it should be.'Science is after all a human endeavor
with all the frailties that implies. Errors and fraud are to be expected. What
matters is that the process is reasonably efficient at catching such things.

And the push for open data repositories and open science to increase
accountability has to be a good thing.

------
hackuser
Many of those journals charge very high prices, 4 figures per year IIRC.
Perhaps they could earn some of that by verifying and vouching for the
integrity of what they sell.

~~~
nycticorax
I think those journals charge high prices for the same reason that niche
software vendors charge high prices. Namely, the product takes a goodly number
of highly educated (and thus highly compensated) people to produce, and they
don't have that many customers. So if the business is going to work at all,
the prices have to be high. Furthermore, they _are_ taking steps to verify the
integrity of what they sell. And of course, the more steps they take, the more
the journals are going to cost...

~~~
hackuser
I don't know about that industry, but generally prices don't depend on the
sellers' costs. Prices are set to maximize profit, that is to maximize (volume
* net profit). Sometimes that's a huge markup (e.g., the $5 soda at the movie
theater), sometimes that works out to a loss (e.g., the 2-year-old smartphone
on clearance). There are many complexities of course, such as loss-leaders,
regulated markets, public goods, exploitative pricing, etc.

I've read that the journals realized that they had a captive audience, i.e.,
that college libraries had no choice but to subscribe in order to support
their faculty, and the publishers raised prices dramatically. There was a
backlash from some schools at the time.

~~~
LyndsySimon
These days, scientific publishers are bundling everything together into one
package, commonly known as the "Big Deal". Institutions pay a single fee for
access to hundreds or thousands of journals.

Here's a paper that explores the idea and its impacts -
[http://www.vanderbilt.edu/econ/faculty/Wooders/APET/Pet2004/...](http://www.vanderbilt.edu/econ/faculty/Wooders/APET/Pet2004/Papers/Exclusion%20or%20efficient%20pricing.pdf)

~~~
hackuser
Like cable TV

------
Tycho
Is there any chance that there will be some sort of google-books-style
centralization of experimental datasets from researchers which will then be
subject to scrutiny from machine learning models trained to suss out faked
data, at which point we'll uncover lots of shenanigans from our scientific
past?

------
stefantalpalaru
> It also includes moving the goal posts: that is, mining the data for results
> first, and then writing the paper as if the experiment had been an attempt
> to find just those effects. “You have exploratory findings, and you’re
> pitching them as ‘I knew this all along,’ as confirmatory,” Dr. Nosek said.

Why is this a problem? If the experiment's design is not in conflict with the
new findings, why complain?

~~~
imh
Let's say I flip a million different fair coins 20 times each. Then I analyze
my findings and see that coin number 54 of them was heads all 20 times. I
present my results to my peers saying, "Coin #54 is defective! The chances of
this happening by chance are on in a million!" But I'd expect one of them to
be heads every time flipping that many coins. It just happened to be coin #54.

The problem is we're not matching up our statistical analysis to our testing
when we look for findings then naively test hypotheses based on those very
findings.

~~~
ikeboy
Would using Bayesian statistics fix that? If the prior for a coin being
defective to produce only heads is more than one in a million, then 54 is
probably defective. If it's less than that, then your analysis doesn't
conclude it's defective anyways.

Bayes can be hacked by ommitting info, but not by things like this, I believe.

~~~
imh
It's not a bayesian versus NHST thing, it's just about doing the right test.
You can do this kind of thing easily enough in a NHST kind of way. But yeah I
think doing it in a bayesian manner is much easier to interpret and explain,
and so requires less training, which is a HUGE boon for the crisis at hand.

~~~
ikeboy
You can do the test with NHST only if you know what the researcher had in mind
when experimenting. That can lead to absurd results at times, such as your
example. With Bayes, as long as no lies are told, your inference doesn't
depend on the researcher's private thoughts.

~~~
imh
You only need to know how the experiment was performed, same as you do for
bayes. If they present the whole dataset, nothing is changed. If they
cherrypick and only show the data for the coin that happened to be all heads
saying that was their whole experiment, no amount of stats will help you,
bayesian or otherwise.

~~~
ikeboy
No, if the coins are independent, the Bayesian is not fooled even when seeing
only that coin. The argument was in my post above.

The inference in Bayes only depends on the data, and the flips of other coins
doesn't make a difference if independent. Frequentist testing can depend on
things like stopping rules and hypothesis tested, which aren't correlated to
the actual truth and therefore should have zero effect on inference.

They can cherry pick and only show some flips of that coin, but then they
really need to be outright lying or you'll ask why only some flips were
reported.

------
imh
Love the article, hate the headline :( What the is wrong with these editors?

~~~
sctb
We updated the title to a less bait-ey phrase from the article, and we're open
to suggestions for a better one.

