
We Should Not Accept Scientific Results That Have Not Been Repeated - dnetesn
http://nautil.us/blog/we-should-not-accept-scientific-results-that-have-not-been-repeated
======
misnome
Define repetition.

It's not as simple as that, for all sciences - once again an article on
repeatability seems to have focused on medicinal drug research (it's usually
that or psychology), and labelled the entire "Scientific community" as
'rampant' with " statistical, technical, and psychological biases".

How about, Physics?

The LHC has only been built once - it is the only accelerator we have that has
seen the Higgs boson. The confirmation between ATLAS and CMS could be
interpreted as merely internal cross-referencing - it is still using the same
acceleration source. But everyone believes the results, and believes that they
represent the Higgs. This isn't observed once in the experiment, it is
observed many, many times, and very large amounts of scientists time are spent
imagining, looking for, and measuring, any possible effect that could cause a
distortion or bias to the data. When it costs billions to construct your
experiment, sometimes reproducing the exact same thing can be hard.

The same lengths are gone to in order to find alternate explanations or
interpretations of the result data. If they don't, they know that some very
hard questions are going to be asked - and there will be hard questions asked
anyway, especially for extraordinary claims - look at e.g. DAMA/LIBRA which
for years has observed what looks like indirect evidence for dark matter, but
very few people actually believe it - the results remain unexplained whilst
other experiments probe the same regions in different ways.

Repetition is good, of course, but isn't a replacement for good science in the
first place.

~~~
return0
We don't need to 'define repetition', we need to foster a culture that a)
accepts repetition and b) does not accept something for a fact just because
it's in a journal. Right now, (a) is not even acceptable; nobody will publish
a replication study. Ofc, LHC is impossible to replicate, but the vast
majority of life science studies are.

I should think this is mostly needed in life sciences. Other, more 'exact'
sciences seem to not have this problem.

~~~
throwaway729
_> we need to foster a culture that a) accepts repetition_

Do we?

I don't think we do. I think we need to foster a culture of honesty and rigor.
Of good science. Which is _decidedly_ different from fostering a culture of
"repetition" for its own sake.

Paying for the cost of mountains upon mountains of lab techs and materials
that it would require to replicate every study published in a major journal
just _isn 't_ a good use of ever-dwindling science dollars. Replicate where
it's not far off the critical path. Replicate where the study is going to have
a profound effect on the direction of research in several labs. But don't just
replicate because "science!"

In fact, one could argue that the increased strain on funding sources
introduced by the huge cost of reproducing a bunch of stuff would increase the
cut-throat culture of science and thereby decrease the scientist's natural
proclivity toward honesty.

 _> and b) does not accept something for a fact just because it's in a
journal_

Again, it's entirely unclear what you mean here.

It's impossible to re-verify every single paper you read (I've read three
since breakfast). That would be like re-writing every single line of code of
every dependency you pull into a project.

And I'm pretty sure literally no scientist takes a paper's own description of
its results at face value without reading through methods and looking at (at
least) a summary of the data.

Taking papers at face value is really only a problem in science _reporting_
and at (very) sub-par institutions/venues.

I don't care about the latter, and neither should you.

WRT the former, science reporters often grossly misunderstand the paper
anyways. All the good reproducible science in the world is of zero help if
science reporters are going to bastardize the results beyond recognition
anyways...

~~~
devishard
> I don't think we do. I think we need to foster a culture of honesty and
> rigor. Of good science. Which is decidedly different from fostering a
> culture of "repetition" for its own sake.

No one is proposing repetition for its own sake. The point of repetition is to
create rigor, and you _can 't do rigorous science without repetition_.

> Paying for the cost of mountains upon mountains of lab techs and materials
> that it would require to replicate every study published in a major journal
> just isn't a good use of ever-dwindling science dollars. Replicate where
> it's not far off the critical path. Replicate where the study is going to
> have a profound effect on the direction of research in several labs. But
> don't just replicate because "science!"

I could see a valid argument for only doing science that will be worth
replicating, because if you don't bother to replicate you aren't really
proving anything.

~~~
throwaway729
_I could see a valid argument for only doing science that will be worth
replicating, because if you don 't bother to replicate you aren't really
proving anything._

Exactly. A lot of the science _I 've_ done should not be replicated. If
someone told me they wanted to replicate it, I would urge them not to. Not
because I have something to hide. But because some other lab did something
strictly superior that should be replicated instead. Or because the experiment
asked the wrong questions. Or because the experiment itself could be pretty
easily re-designed to avoid some pretty major threats.

The problem is that is that hindsight really is 20/20\. It's kind of
impossible to _ONLY_ do good science. So it's important to have the facility
to recognize when science (including your own) isn't good -- or is good but
not as good as something else -- and is therefore not worth replicating.

I guess the two key insights are:

1\. Not all science is worth replicating (either because it's too expensive or
for some other reason).

2\. Replication doesn't necessarily reduce doubt (particularly in the case of
poorly designed experiments, or when the experiment asks the wrong questions).

~~~
adrianm
This is a really good post which contributes to the conversation. Why make it
on a throwaway account? We need more of this here!

------
mmierz
I see a lot of people commenting here that there's no incentive to repeat
previous research because it's not useful for getting grants, etc. This kind
of true but I think it misses something important.

At least in life sciences (can't comment on other fields), it's not that
scientists _don 't_ repeat each other's results. After all, if you're going to
invest a significant fraction of your tiny lab budget on a research project,
you need to make sure that the basic premise is sound, so it's not uncommon
that the first step is to confirm the previous published result before
continuing. And if the replication fails, it's obviously not a wise idea to
proceed with a project that relies on the prior result. But that work never
makes it into a paper.

If the replication succeeds, great! Proceed with the project. But it's time-
consuming and expensive to make the reproduction publication worthy, so it
will probably get buried in a data supplement if it's published at all.

If the replication fails, it's even more time-consuming and expensive to
convincingly demonstrate the negative result. Moreover, the work is being done
by an ambitious student or postdoc who is staring down a horrible job market
and needs novel results and interesting publications in order to have a future
in science. Why would someone like that spend a year attacking the work of an
established scientist over an uninteresting and possibly wrong negative
result, and getting a crappy paper and an enemy out of it in the end, instead
of planning for their own future?

If enough people fail to replicate a result, it becomes "common knowledge" in
the field that the result is wrong, and it kind of fades away. But it's not
really in anyone's interest to write an explicit rebuttal, so it never
happens.

~~~
odbol_
This is why I'm always skeptical of people zealously claiming "all GMOs are
perfectly safe! It's been _proven_!"

Yeah, proven by expensive studies that were funded by the company making the
GMO. Who is going to pay for another study to try to disprove it?

There's a reason we sprayed DET all over our vegetables for years before it
was banned: there was no scientific studies proving that it was harmful, even
though it clearly was harmful in hindsight.

Science is not instant, and there's no way someone can claim that some brand-
new GMO is "perfectly safe", without any long-term studies on its effects over
10, 20, 30 years of exposure. That's just not possible. And yet you try to
explain it to these science zealots and they just brush you off as being
"anti-science".

~~~
Obi_Juan_Kenobi
Do you mean DDT?

Anyway, beyond the 'philosophy of science' issue of whether you can prove
something, there is good affirmative evidence that existing GMOs are safe for
numerous reasons.

First, there's no mechanistic reason to think they would be dangerous. T-DNAs
are not somehow magically toxic to humans; everything you eat is riddled with
millennia of t-dnas, old viruses, transposon blooms, etc. etc.

The technologies themselves should be safe as well. BT is well understood and
embraced by the organic community as an applied natural pesticide, so you
would need to find evidence that the localization somehow makes it toxic.
Glyphosate resistance is also unlikely to have any effect _a priori_ because
it affects a metabolic pathway completely absent in animals.

Argue all you like about how nothing can be 'perfectly safe', sure, but
there's no reason to think that GMOs are dangerous, and people have looked
quite hard. ______

Finally, just look at the Seralini pile-of-bullshit for evidence that there's
plenty of incentive to publish anything critical of GMOs. No one is sitting on
career-making evidence.

[https://en.wikipedia.org/wiki/S%C3%A9ralini_affair](https://en.wikipedia.org/wiki/S%C3%A9ralini_affair)

~~~
odbol_
> First, there's no mechanistic reason to think they would be dangerous.

That's like saying "there's no reason to think peanuts would be dangerous"
since humans eat them all the time. And yet, they are deadly to some humans.
No one knows why.

But they do know that food allergies are far more prevalent in the U.S. than
in other countries. And now, suddenly, people in Africa and China are starting
to exhibit food allergies that the U.S. has had for a while. So what have we
started shipping over to them that's causing these allergies? Who will fund
that study?

~~~
philovivero
What you've hinted at here is fascinating to me.

Do you have some references?

In case it's not clear, I'd like to read articles about food allergies that
have been common in USA for some time now becoming common in countries that
are coming out of 2nd world status and into 1st world.

------
undergroundOps
I'm a physician and I've been suggesting this to my colleagues for a few
years, only to be met with alienated stares and labeled cynical.

The doctors and doctors-in-training I work with have altruistic motives, but
place too much stock in major medical studies. They also frequently apply
single-study findings to patient care, even to patients that would've been
excluded from that study (saw this a lot with the recent SPRINT blood pressure
trial).

And don't even get me started on the pulmonary embolism treatment studies.
What a clinical mess that is.

It's frustrating.

~~~
appleflaxen
but in medicine, what is the practical alternative?

how do _you_ incorporate these findings? ignore them?

if so, it's probably bad for your patients. the only thing worse than a
single-study finding is a zero-study finding.

~~~
undergroundOps
I was simply suggesting what we all learn in medical school and residency: to
appropriately evaluate clinical studies. Just don't think most doctors do.

Let me give you an example of how I approach things. The guidelines for acute
pancreatitis recommend using a fluid called LR instead of NS for volume
resuscitation. This is based on an single study that included 10 patients and
simply noted slightly better lab numbers; there was no difference in clinical
outcome. Lots of problems with that study, right (small, underpowered,
confounders, validity issues, etc)? However, there's no major disadvantage for
using LR in those patients (unless hyperkalemia is a concern), so I use it
since it might have a benefit.

This is a very simple example. It gets much more complicated than that.

"Probably" is one of favorite words in medicine, btw :).

~~~
mablap

        "Probably" is one of favorite words in medicine, btw :).
    

Right as it should. If somebody answers my question by "It depends", then I
know I'm in good company!

------
ythl
> Nowadays there's a certain danger of the same thing happening (not repeating
> experiments), even in the famous field of physics. I was shocked to hear of
> an experiment done at the big accelerator at the National Accelerator
> Laboratory, where a person used deuterium. In order to compare his heavy
> hydrogen results to what might happen with light hydrogen, he had to use
> data from someone else's experiment on light, hydrogen, which was done on
> different apparatus. When asked why, he said it was because he couldn't get
> time on the program (because there's so little time and it's such expensive
> apparatus) to do the experiment with light hydrogen on this apparatus
> because there wouldn't be any new result. And so the men in charge of
> programs at NAL are so anxious for new results, in order to get more money
> to keep the thing going for public relations purposes, they are destroying--
> possibly--the value of the experiments themselves, which is the whole
> purpose of the thing. It is often hard for the experimenters there to
> complete their work as their scientific integrity demands.

\-- Richard Feynman, "Surely You're Joking, Mr. Feynman", pp. 225-226

~~~
gohrt
Current top comment
[https://news.ycombinator.com/item?id=12186295](https://news.ycombinator.com/item?id=12186295)
, specifically rebutted 30 years before the comment was written.

~~~
throwaway729
This _isn 't_ a rebuttal of the linked comment.

The linked comment doesn't state that it would be a waste of time to replicate
on a hypothetical LHC clone.

Rather, the linked comment states that we can accept the Higgs result with
reasonable confidence _even though_ it's currently infeasible to replicate
that experiment.

Feynman's issue was also qualitiatively different -- the scientist was
_comparing_ results from two different instruments. The people in charge of
one of the instruments wouldn't allow the scientist to run both experiments on
a single instrument. In fact, from context, it's not even clear to me Feynmann
would have insisted on re-running the original experiment if the scientist
were not using a different accelerator for the second one. Anyways, in the
Higgs case, there's no potential for a "comparing readings from instrument A
to readings from instrument B" type bug.

More to the point, and FWIW, I somehow doubt Feynman would insist on building
a second LHC for the sole purpose of replicating the Higgs measurement. But I
guess we have to leave that to pure speculation.

------
cs702
My first reaction to this headline was "duh." _Of course_ we should hold off
on accepting scientific claims (i.e., predictions about the natural world)
that to date have been verified only by the same person making those claims!

My next reaction was, "wow, it's a sad state of affairs when a postdoctoral
research fellow at Harvard Medical School feels he has to spell this out in a
blog post." It implies that even at the prestigious institution in which he
works, he is coming across people who treat science like religion.

~~~
nxzero
Of course you as a reader of said claims confirm at the very least that
they've been independently reproduced, right?

(If so, this shouldn't be news.)

~~~
lrem
Unfortunately, there is no easy way to do it. Confirmation studies are not
easily accepted by impactful journals/conferences, thus nearly nobody bothers
to do them. Even if there is one, it can be surprisingly hard to find it.

As a point of anecdata: my wife's master thesis was a confirmation study of
using LLDA for face recognition. I remember seeing it included in some book by
the university press. I gave up Googling for it after 5 minutes.

~~~
randall
There needs to be better scientific protocol. More linking through data
instead of annoying cites. I think anyway.

------
jbb555
We shouldn't "accept" or "reject" results at all.

It's not a binary option. One poor experiment might give us some evidence
something is true. A single well reviewed experiment gives us more confidence.
Repeating the results similarly does. As does the reputation of the person
conducting the experiment and the way in which it was conducted.

It's not a binary thing where we decide something is accepted or rejected, we
gather evidence and treat it accordingly.

~~~
bbctol
So many scientists I talk to don't have a basic understanding of philosophy of
science. I don't necessarily blame them--I understand why "philosophy" as an
academic field is seen as a soft, speculative, and pretentious field compared
to the rigor of science, but as Daniel Dennett said, “There is no such thing
as philosophy-free science; there is only science whose philosophical baggage
is taken on board without examination."

These days, if you ask a scientist "So how do we prove something is true using
science?" they'll be able to recite Popper's falsificationism as if it's a
fundamental truth, not a particular way of looking at the world. But the huge
gap between the particular theory that people get taught in undergrad--that
science can't actually prove anything true, just disprove things to approach
better hypotheses--and the real-world process of running an experiment,
analyzing data, and publishing a paper is unaddressed. The idea that there's a
particular bar that must be passed before we accept something as true is
exactly what got us into this mess in the first place! There's a naive
implicit assumption in scientific publishing that a p-value < 0.05 means
something is true, or at least likely true; this author is just suggesting
that true things are those which yield a p-value under 0.05 twice!

What's needed, in my opinion at least, is a more existential, practically-
grounded view of science, in which we are more agnostic about the "truth" of
our models with a closer eye to what we should actually _do_ given the data.
Instead of worrying about whether or not a particular model is "true" or
"false," and thus whether we should "accept" or "reject" an experiment, focus
on the predictions that can be made from the total data given, and the way we
should actually live based on the datapoints collected. Instead, we have
situations like the terrible state of debate on global warming, because any
decent scientist knows they shouldn't say they're absolutely sure it's
happening, or a replication crisis caused by experiments focused on propping
up a larger model, instead of standing on their own.

------
mydpy
I agree in principle. There are a few concerns:

1\. How should we receive costly research that took special equipment and lots
of time to develop and cultivate? I.e., CERN?

2\. A lot of research is published, ignored, and then rediscovered. In this
case, we may want to accept the research until it cannot be repeated (i.e., in
another journal publication).

3\. Reviewers of academic publications probably are not qualified or have the
time to recreate all scientific research.

4\. Isn't the academic system at its core kinda... broken?

~~~
thefastlane
"Isn't the academic system at its core kinda... broken?"

can your elaborate on what you mean?

~~~
hdra
I gather its things like misaligned incentives. Like funding sources dictating
the "desirable" result, reluctance to publish negative results, pursue of
(vanity) metrics that leads to quota system by universities, etc.

------
gnuvince
Maybe conferences should have a "reproducibility" track for that purpose?
Also, I don't know about other fields, but I'm pretty sure that in CS, if you
just took a paper and tried to reproduce the results, you'll get rejected on
the ground that you offer no original contribution; no original contribution
=> no publication => no funding.

~~~
studentrob
For CS, reproducing should be easy given the code and input data right? Other
sciences' input isn't so easily shared

~~~
dagw
Ideally you should re-implement the algorithm based on the description in the
paper to verify that the description of the algorithm is correct. You should
also test with your own data to make sure that the algorithm works on all
reasonable data and not only on some provided cherry picked data. If you can't
get the expected results with your own implementation and your own data then
the results aren't reproduced.

~~~
eru
Yes. So being able to rerun with the same code and same inputs to get the same
outputs is a lower bar. Many papers don't meet even that bar.

(Mostly because they don't publish code nor data; and academic code is often a
horrible mess, and the code was mucked around with between different stages of
running.)

------
fhood
Many people have mentioned that replicating an experiment can be expensive,
but I don't think anybody has really brought up just how expensive this can
be.

Not all science is done in a lab. Replicating an experiment is obviously
feasible for a short term psychology experiment, but in earth sciences
(oceanography for instance.) it is far less often possible to reproduce an
experiment for the following reasons. N.B. This is all from my personal
experience of one field of science.

1.) Cost. If you got funding to take an ice-breaker to Antarctica to "do
science" it required several million dollars to fund. It is difficult enough
to secure funding for anything these days, none the less prohibitively
expensive attempts to reproduce results. (honestly any serious research vessel
will run costs into the millions, regardless of destination.)

2.) Time. Say you are on a research vessel taking measurements of the Amazon
river basin. This is a trip that takes months to years to plan and execute. If
you return to duplicate your experiment 2 years later, the ecology of the area
you were taking measurements of may have changed completely.

3.) Politics. Earth sciences often require cooperation from foreign entities,
many of which are not particularly stable, or whom may be engaging in
political machinations that run counter to your nationality's presence in the
country, or both. Iran and China are two good examples. Both are home to some
excellent oceanographers, and both of which can be very difficult to Science
in when your team includes non Iranian/Chinese nationalities.

------
bontoJR
The big issue right now is funding of replicated research, who wants to fund a
research to prove someone else was right? Most of these funds are granted
based on potential outcome of the new discovery like: potential business,
patents, licenses, etc... not being the first one would probably wipe most of
these benefits, cutting down to a small probably getting funded...

Now, straight to the point, who's going to pay for the repeated research to
prove the first one?

~~~
wodenokoto
On a low level, I think it should be mandatory for masters students to do a
pre-thesis project, which is replicating findings in a published paper.

It would do something about low hanging fruit in terms of testing
reproduceability and since there is a published paper, the student has access
to guidelines for setting up and reporting on a large project, which will help
them learn how to do their own, original thesis.

~~~
tnhh
I had my Masters students do this as part of my wireless networking class this
year. It was very instructive for me and the students seemed to enjoy it, so
I'll definitely keep it in the syllabus.

------
guaka
Totally agree. I'd go even further and make free licenses on scientific source
and datasets mandatory. Research that is funded by public money should lead to
public code and data.

~~~
jeremysmyth
What about defense research?

~~~
amboar
Well, there's the GPL-styled approach: anyone with access to the results must
also have access to the associated data. This doesn't mean it is mandatory to
make it public, though you'd have to restrict the redistribution freedom.

~~~
wodenokoto
I recently used a large dataset of tweets in a research project. As far as I
know, I do not have the rights to distribute these.

I also used a dataset consisting of newspaper articles. It cost me $1.000 to
get access to, and I definitely do not have the rights to redistribute it.

~~~
dagw
As long as you provide a detailed enough description of the source of your
dataset that I can reproduce it myself then that is fine. So in your first
case tell me what criteria you used to select your tweets and in the second
tell me where to send my $1000 and what to ask for.

~~~
tnhh
Unfortunately not everyone reports this information. Here is a study that we
did of over 500 papers using online social network data:
[http://tnhh.org/research/pubs/tetc2015.pdf](http://tnhh.org/research/pubs/tetc2015.pdf)
While most authors would report high-level characteristics (e.g., which social
network they measured), fewer authors reported how they sampled the network or
collected data, and very few people reported on how they handled ethics,
privacy and so forth.

------
Gatsky
Lots of scientific results are repeated but not published. If it doesn't work
then people just move on. The problem is journals. There is no way to publish
your attempts to repeat an experiment, unless you put it into another paper.

The other issue, especially in the life sciences, is inaquedate statistical
input. If someone performs an underpowered, confounded experiment and gets a
positive result, then someone else performs the same underpowered confounded
experiment and gets a negative result, what have we learned except that the
experiment is underpowered?

------
unabst
With science, the profession and the product are distinctly different, and we
are failing to hold the profession to the standards of the product. Science,
the profession, is political, incentive driven, and circumstantial. Scientists
need to get paid. Science, the product, is apolitical, existential, and
universal. So those who love and believe in the products of science may wish
upon themselves to be these things also. I know I do. Except, sometimes it
just ins't practical, or even possible.

But repeatability actually matters more professionally. Scientifically
speaking, if the science is bad it just won't work when others try to make use
of it. All bad science will be identified or corrected as we try and make use
of it and convert it into new technology. Technology mandates repeatability.
So those scientists who fail to produce repeatable science, regardless of how
professionally successful they may be, will inevitably fail to produce any new
technology or medicine, and vice versa.

------
jmilloy
Obviously I agree that scientific results must be reproducible. But I also
realize that it's simply infeasible to repeat the entirety of every study, and
much less to also go to the effort to write and peer-review those repeated
results.

What I think is overlooked in this discussion as that a lot of confirmation
work already happens. Most (all?) scientific results are incremental progress
built on a heap of previous work. In the course of normal research, you
reproduce existing results as necessary before altering conditions for your
own study. If you can't confirm the results, well then perhaps you have a
paper (though it can be politically challenging to get it published, and
that's a separate problem). But if you do, then you don't waste time
publishing that, you get on with the new stuff.

Ultimately, I don't think scientists do accept results _in their field_ that
they have not repeated.

------
nonbel
Cue all the people justifying their pseudoscientific behavior. If it is too
expensive to fund twice, it shouldn't be funded once. If that means the LHC
and LIGO wouldn't get done, then we should have only funded one of them. We
need to remain skeptical of those results until replicated by a new team. Even
one replication is pretty weak...

Independent replications of experiment (and the corresponding independent
reports of observations) are a crucial part of the scientific method, no
matter how much you wish it wasn't. Nature doesn't care if it is inconvenient
for you to discover her secrets, or that it is more difficult for you to hype
up your findings to the unsuspecting public.

~~~
daveguy
You do realize that scientists who work on the LHC have the highest
repeatability standards of any science profession, right?

~~~
jomamaxx
The LHC experiment is not the issue here.

There is a lot of transparency there, a lot of well meaning people with a lot
of oversight.

I suggest most would admit 'there could be a problem' there, but it's out in
the open if there is.

The problem of lack of repeatability I think has to do with subconscious bias
on the part of the experimenters which will be less pronounced when there are
5000 people working on it.

------
cube00
Having wasted time trying to replicate someone else's results who 'lost' their
code, I agree! Maybe repeating the experiment should be part of the peer
review.

~~~
closed
So frustrating. Lost = "I didn't think anyone would hold me accountable for
it."

------
hudathun
Looking good is, sadly, better rewarded than doing good in many areas of life.
It's doubly sad that this affects our body of scientific knowledge. Even
claims that are reproduced can suffer from funding bias and confirmation bias.
The truth hopefully comes out in the end, but I'm sad for the harm that's
caused in the interim.

------
jernfrost
I don't get why this is not top on the agenda for the scientific community and
the government. Huge amounts of research money is lost in repeating stuff that
doesn't work. Huge amounts of money is lost chasing broken science.

I blame this on the neo-liberal ideology. This intense focus on getting
money's worth, on tying grants to specific goals, counting publications etc.
Driving research exclusively on a very narrowly defined money incentive has
driven us further into this sort of mess. The money grabbing journals which
has prevented any significant innovation in how science is shared.

I think what science needs is a model closer to that of open source. With open
projects anybody can contribute to but where verification happens through
personal forged relationships. The Linux kernel code quality is verified by a
hierarchy of people trusting each other and knowing something about each
others quality of work. Work should be shared like Linux source code in a
transparent fashion and not behind some antiquated paywall.

I don't think the grant system can entirely away, but perhaps it should be
deemphasized and instead pay a higher minimum amount of money to scientists
for doing what they want. Fundamental science breakthrough doesn't happen
because people had a clear money incentive. Neither Einstein, Nils Bohr, Isaac
Newton or Darwin pursued their scientific breakthroughs with an aim of getting
rich. Few people become scientists to get rich. Why not try to tap into
people's natural desire to discover?

------
framebit
This problem, like many in modern day science, can in large part be traced
back to unstable funding. On the Maslow's-style hierarchy of research lab
needs, the need for funding is a lot lower on the scale than the aspiration
for scientific purity, just as a human's need for food is lower on the scale
than their desire for self-actualization.

If competition for research dollars ceases to be so cutthroat, it will go a
long way towards solving this and many other seemingly entrenched cultural
problems.

------
habitue
A big distinction here is that different fields have different levels of
dependence on prior results. In fields like psychology etc, you don't need the
previous results to work in order to run your own experiment. In other words,
if you cite a well-known paper saying "people seem to work faster near the
color red" and your paper runs an experiment to see if they work faster near
the color yellow, if the red paper is later unreplicable, it doesn't change
the outcome of your experiment in any way.

In contrast, if you are in machine learning and you are extending an existing
architecture you are very directly dependent on that original technique being
useful. If it doesn't "replicate" the effectiveness of the original paper,
you're going to find out quickly. Same for algorithms research. Some other
comments here have mentioned life sciences being the same.

So I think there's a qualitative difference between sciences where we
understand things in a mostly statistical way (sociology, psychology, medical
studies) where the mechanism is unknown (because it's very very complicated),
but we use the process of science mechanistically to convince ourselves of
effectiveness. e.g. I don't know why this color makes people work faster/ this
drug increases rat longevity / complex human interactions adhere to this
simple equation, but the p value is right, so we think it's true. Versus
sciences where we have a good grasp of the underlying model and that model is
backed up by many papers with evidence behind it, and we can make very
specific predictions from that model and be confident of correctness.

------
kayhi
In the world of chemistry, biochemistry and microbiology a huge step forward
would be for journals to require a complete list of products used. The
publication should also include the certification of analysis for each item as
they vary over time.

For example, here are two product specifications for a dye called Sirius Red,
the first by Sigma-Aldrich[1] and the second by Chem-Impex[2]. The Sigma-
Aldrich product contains 25% dye while the Chem-Impex contains equal or
greater than 21%. These two dyes could be quickly assessed with a
spectrophotometer in order to determine an equivalency, however you need both
dyes on hand which doesn't seems like a good use of funding. Also this touches
on another problem in replication which is, what is in the other 75%+ of the
bottle?

[1]
[http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPE...](http://www.sigmaaldrich.com/Graphics/COfAInfo/SigmaSAPQM/SPEC/36/365548/365548-BULK_______SIAL_____.pdf)
[2]
[http://www.chemimpex.com/MSDSDoc/22913.pdf](http://www.chemimpex.com/MSDSDoc/22913.pdf)

------
Mendenhall
Look at research done on many political hot button topics. They love results
that have not been repeated. I see all sorts of posts even on HN that
reference such "science" as well. The root problem, people who are pushing an
agenda.

------
yiyus
> The inconvenient truth is that scientists can achieve fame and advance their
> careers through accomplishments that do not prioritize the quality of their
> work

An even more inconvenient truth is that scientists cannot even keep their jobs
if they prioritize the quality of their work. The pressure to publish novel
results is too strong and it is almost impossible to get any support for
confirming previous ones.

------
bshanks
I agree with the main point of this article but in terms of its analysis and
prescriptions I think it gets two things backwards. (1) Most scientists seek
fame as a means to the end of getting tenure and funding, not the other way
around; if you gave them tenure (and the ability to move their tenure to
somewhere else if they wanted to move) and perpetual funding and told them
they could choose to be anonymous, I think many would choose that option. (2)
Replication is not done/published enough because the incentive to do so
(measured in: increase in probability of getting tenure per hour spent) is not
high enough, not because people are overly willing to accept unreplicated
work.

In order for a lot more replication to get published, what would be needed
would be for people who spent their careers replicating others' results (at
the expense of not producing any important novel results of their own) to get
tenure at top institutions (outcompeting others who had important novel
results but not enough published replications).

------
doug1001
"repeated" in this context is not incorrect, but i think "replicated" is
perhaps a better choice.

That aside, i think _repeatability_ is a much more useful goal (rather than
"has been repeated"). For one thing, meaningful replication must be done by
someone else; for another, it's difficult and time consuming; the original
investigator has no control over whether and when another in the community
chooses to attempt replication of his result. What is within their control is
an explanation of the methodology they relied on to produce their scientific
result in sufficient detail to enable efficient repetition by the relevant
community. To me that satisfies the competence threshold; good science isn't
infallible science, and attempts to replicate it might fail, but some baseline
frequency for ought to be acceptable.

------
VikingCoder
This is wrong-headed in the extreme.

What we should demand is scientific results that have FAILED.

When we see a p=0.05, but we don't know that this SAME EXACT EXPERIMENT has
been run 20 times before, we're really screwing ourselves over.

Relevant: [https://xkcd.com/882/](https://xkcd.com/882/)

------
pc2g4d
Replication isn't enough. It's also necessary to know how many non-
replications have occurred but got swept under the rug. It's not the existence
of replications that matter---it's the rate of replication relative to number
of replication attempts.

So I agree with the title "We Should Not Accept Scientific Results That Have
Not Been Repeated". But I would add to it "We Should Not Accept Scientific
Results from Studies That Weren't Preregistered". Registration of studies
forces negative results to be made public, allowing for the positive result
rate / replication rate to be calculated.

Otherwise the existence of a "positive" result is more a function of the
trendiness of a research area than it is of the properties of the underlying
system being studied.

------
aminorex
More pragmatically, we should not accept scientific _publications_ and
_conferences_ which do not publish negative results and disconfirmations.

------
dalke
I disagree.

One part of science is observation. Including observations which cannot be, or
at least have not been, repeated. For example, consider a rare event in
astronomy which has only been detected once. Is that science? I say it is. But
it's surely not repeatable. (Even if something like it is detected in the
future, is it really a "repeat"?)

Some experiments are immoral to repeat. For example, in a drug trial you may
find that 95% survive with a given treatment, while only 5% survive with the
placebo. (Think to the first uses of penicillin as as real-world example.)

Who among you is going to argue that someone else needs to repeat that
experiment before we regard it as a proper scientific result?

~~~
ebbv
> One part of science is observation. Including observations which cannot be,
> or at least have not been, repeated. For example, consider a rare event in
> astronomy which has only been detected once. Is that science? I say it is.
> But it's surely not repeatable.

First off, you can accept the observation at face value as an observation, but
conclusions drawn from the claims which have no other support or means of
verification should not be accepted and would not be accepted. Fortunately,
most of the time even if something is initially sparked by a very rare
occurrence, it will have some kind of implications that are verifiable by some
other means other than just waiting for something to happen in space.

But even something that is rare and relies on observation, like gravitational
waves, we have already been able to identify more than one occurrence.

> Some experiments are immoral to repeat. For example, in a drug trial you may
> find that 95% survive with a given treatment, while only 5% survive with the
> placebo.

What's more immoral, releasing a drug that's only had one test, even a
striking one, on the public as a miracle cure that you have not truly verified
or performing another test to actually be sure of your claims before you
release it?

> Who among you is going to argue that someone else needs to repeat that
> experiment before we regard it as a proper scientific result?

That's how science works. If something is not independently repeatable and
verifiable then science breaks down. Look at the recent EM drive. Most
scientists in the field were skeptical of it, and once it was finally
attempted to be independently verified the problems were found.

Independent verification is the cornerstone of science and what makes it
different from bogus claims by charlatans.

~~~
dalke
> conclusions drawn from the claims which have no other support or means of
> verification should not be accepted and would not be accepted

I disagree. In _all cases_ , even with repeated experiments, the claims are
only tentatively accepted. The confirmation by others of Blondlot's N-rays
didn't mean they were real, only that stronger evidence would be needed to
disprove the conclusions of the earlier observations.

Astronomy papers make conclusions based on rare or even singular observations.
Take SN1987a as an example, where observations from a neutrino detector were
used to put an upper limit on the neutrino mass, and establish other results.

> "or performing another test"

This question is all about _repeating_ an experiment. Repeating the experiment
would be immoral.

There are certainly other tests which can confirm the effectiveness, without
repeating the original experiment and without being immoral. For the signal
strength I gave, we can compare the treated population to the untreated
population using epidemiological studies.

But under current medical practices, if a drug trial saw this sort of
effectiveness, the trial would be stopped and _everyone_ in the trial offered
the treatment. To do otherwise is immoral. As would repeating the same trial.

~~~
hx87
> But under current medical practices, if a drug trial saw this sort of
> effectiveness, the trial would be stopped and everyone in the trial offered
> the treatment. To do otherwise is immoral. As would repeating the same
> trial.

Then perhaps current medical practices should change. The benefits to those
who were previously given the placebo should be balanced against the
probability that the observed outcomes may not occur in other circumstances.

~~~
dalke
Are you for real? You would sacrifice people upon the alter of
reproducibility?

Down that path lies atrocities. The system was put into place to prevent
repeats of horrors like the "Tuskegee Study of Untreated Syphilis in the Negro
Male".

~~~
hx87
I'd rather not sacrifice people on the altar of a single study, no matter how
significant the results. Down that path lies atrocities, too, albeit of a
quieter sort.

~~~
dalke
As I said earlier, there are alternatives which are both moral and can verify
effectiveness without having to repeat the original experiment.

You chose to not verify, and insist upon repeating, thus likely consigning
people to unneeded pain and even death.

I'll give a real-world example to be more clear cut about modern ethics and
science. Ever hear of TGN1412?
[https://en.wikipedia.org/wiki/TGN1412](https://en.wikipedia.org/wiki/TGN1412)

It went into early human trials, and very quickly caused a reaction. "After
very first infusion of a dose 500 times smaller than that found safe in animal
studies, all six human volunteers faced life-threatening conditions involving
multiorgan failure for which they were moved to intensive care unit."
([http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964774/)
)

Here's a publication of the effects:
[http://www.nejm.org/doi/full/10.1056/NEJMoa063842](http://www.nejm.org/doi/full/10.1056/NEJMoa063842)
.

Is it moral to reproduce that experiment? I say it is not moral, and must not
be repeated even though it is possible to do so.

Can a publication about the effects still be good science even though medical
ethics prevent us from repeating the experiment? Absolutely.

What say you?

------
webosdude
Didn't John Oliver say the same thing few months ago in his episode on
scientific studies,
[https://www.youtube.com/watch?v=0Rnq1NpHdmw](https://www.youtube.com/watch?v=0Rnq1NpHdmw)

------
leecarraher
It's not just money that prevents people from repeating experiments, it's
recognition.

The general idea for research to be accepted is that it makes some novel,
albeit small, impact on the field, acceptable for publication in a peer
reviewed journal or proceeding. Repeating someone else's experiments wont get
you that, so in general it wont help you graduate or move you toward a higher
position at a university or in your profession, meaning there is very little
motivation for researchers to pursue such endeavors.

So instead of just throwing money at the problem, we may need to entirely
revamp how we recognize the pursuits of researchers.

------
ehnto
We learned the importance of this in high school science and it baffles me
that it's not already the case.

~~~
wccrawford
We have some kind of weird hero-worship of scientists where the general public
just believes what they say, even if they never even attempt to replicate
their results. They do an experiment (which may or may not be scientifically
sound to start with) and then publish results, and the public eats it up.

And then people have the nerve to say, "Last week chocolate was bad for me,
now it's good? Make up you mind!" No, _stop_ listening to un-replicated
studies! Jeez.

~~~
hudathun
Good point about the public involvement. The public and the news systems are
part of the problem.

I've lost count of how many 'battery breakthrough' articles I've come across,
but they seem to pass the newsworthy test.

~~~
eru
Wasn't the problem with battery breakthroughs that they don't commercialise
well, rather than that the science doesn't repeat?

------
ramblenode
I have an alternative proposal: do a study right the first time.

That means:

A) Pre-registering the study design, including the statistical analysis.
Otherwise, attaching a big label "Exploratory! Additional confirmation
needed!"

B) Properly powering the study. That means gathering a sample large enough
that the chances of a false negative aren't just a coin flip.

C) Making the data and analysis (scripts, etc.) publicly available where
possible. It's truly astounding that this is not a best practice _everywhere_.

D) Making the analysis reproducible without black magic. That includes C) as
well as a more complete methods section and more automation of the analysis
(one can call it automation but I see it more as reproducibility).

Replication of the entire study is great, but it's also inefficient in the
case of a perfect replication (the goal). Two identical and independent
experiments will have both a higher false negative and false positive rate
than a single experiment with twice the sample size. Additionally, it's
unclear how to evaluate them in the case of conflicting results (unless one
does a proper meta-analysis--but then why not just have a bigger single
experiment?).

~~~
physicalist
Your proposal is comparable to saying that checks and balances are not needed
in a democracy, politicians just need to govern "right". This is about
incentivising scientists to do the right thing instead of merely demanding it,
like you do.

~~~
ramblenode
How is advocating for a new set of best practices any more "demanding" or
wishful than a regime of obligatory replication? And how is this categorically
different from current practices such as peer review, disclosing conflicts of
interest, an IRB, etc.?

------
csydas
I think with the increased visibility of scientific research to the general
public, it's less that science needs to stop accepting unrepeated results, but
instead the paper process needs to be updated to reflect the new level of
availability, and journal databases need better relationship views between
papers and repeated tests.

As an outsider looking in on the Scientific process, I am not really sure how
applicable my opinions are, but I see these as useful changes.

Basically, in reverse order, my suggestions for science to adopt are as
follows:

Papers in databases need to have fields related to reproduction studies, and
it needs to start becoming a prideful part of the scientific process; just as
there is a lot of pride and money, researchers should start to thump their
chest based on the reproducibility of their work, actively seeking out
contemporaries and requesting a reproduction study as part of the pubilshing
process, and subsequently updating.

The papers published themselves should take a moment (perhaps no more than a
paragraph) to include a "for media" section that outlines the "do's and
don't's" on reporting on the research. For example, cancer research should
clearly state examples of acceptable understandings in lay person terms as a
sort of catch for sloppy reporting. Something like "Do not write "cure for
cancer found" or "Effective treatment", instead write "progress made, etc".
Basically put a sucker punch to outlandish headlines and reporting right in
the paper itself, and let journalists who want to be sensationalist embarrass
themselves.

This seems like two very simple acts that could raise the bar for science a
bit.

~~~
ebbv
Those are both good but the key here is the media needs to understand that
scientific papers that have not been independently verified are in a "maybe"
state.

Of course, they probably do know this and just choose to ignore it because
"Unverified Study that MIGHT Point to M&M's Being Good For You" won't get as
many clicks as "M&M's Are Good For You Says New Study!"

~~~
csydas
This is sort of why I think having it stated explicitly within the paper, not
just an aside but part of the actual process. It's to pit less scrupulous
journalists against one another, in an "honor among thieves" sort of way I
guess. If someone wants to go ahead and write clickbait, they can, but it
leaves them open to someone else looking to discredit them going "well, did
you even read the paper? they told you not to write that."

it's not so much checking for the public purpose, it's for others.

------
munificent
Most disciplines where correctness is important seem to end up having some
adversarial component. It is explicitly how the justice system in the US works
[1]. Many software companies have separate QA departments that are
deliberately kept at a remove from the engineers to encourage some rivalry
between them. Security issues are almost always managed in an adversarial way
(though here, you could argue that's because it reflects how the system itself
is [mis-]used). Markets are intended to allow fair competition between
producers to find an optimal price and product for consumers.

Peer review is supposed to do this, but the fact that peer reviewers are often
colleagues leads to collusion, whether intended or not.

Maybe we need a separate body of scientists whose sole job—and whose entire
prestige—derives from taking down and retracting bad science.

[1]:
[https://en.wikipedia.org/wiki/Adversarial_system](https://en.wikipedia.org/wiki/Adversarial_system)

------
dahart
It's unfortunate that the suggestions at the end don't seem to offer a
realistic attack vector.

> First, scientists would need to be incentivized to perform replication
> studies, through recognition and career advancement. Second, a database of
> replication studies would need to be curated by the scientific community.
> Third, mathematical derivations of replication-based metrics would need to
> be developed and tested. Fourth, the new metrics would need to be integrated
> into the scientific process without disrupting its flow.

Yes, absolutely those things need to happen, but the problem is how to get
this funded, how to get people to not see reproducing results as career
suicide, right? Items 2-4 will fall out as soon as item #1 happens.

How do we make item #1 happen? What things could be done to make reproducing
results actually an attractive activity to scientists?

~~~
dragandj
The problem is that, if you put mere reproduction as a goal, many scientists
would see that as low hanging fruit to beef up the resume, so we'd get
countless unnecessary "experiments".

I'd say the goal that gets credited should not be merely reproducing the
results, but finding errors in the previous research. That would count as
novel, and is something that is presently recognized as contribution. The only
problem is that journals or conferences treat it as unattractive, so good luck
publishing something of the kind...

~~~
dahart
> The problem is that, if you put mere reproduction as a goal, many scientists
> would see that as low hanging fruit to beef up the resume, so we'd get
> countless unnecessary "experiments".

Only if you assume the incentives for the 2nd, 3rd, 4th, etc. reproduction
experiments remain the same, right? I wouldn't assume that, both because the
first reproduction is the most valuable, and for the reasons Ahmed discussed
in the article - that scientists are motivated by their perceived ability to
do something novel. So first reproduction might be novel, but the fifth would
certainly be less valuable, so I wouldn't personally assume we'd get a flood
of useless experiments.

> I'd say the goal that gets credited should not be merely reproducing the
> results, but finding errors in the previous research

Reproducing an experiment is meant to, without prejudice, either confirm or
deny the previous research. It's not meant to confirm the previous results, it
is meant to ask whether there could be errors in the research, but without
assuming there are errors.

It _is_ novel to validate a result the first time, whether it's positive or
negative, and for this incentive system to work, it has to appeal to people
who might not find something dramatic or contradictory. It _must_ be appealing
to do the work, regardless of the outcome, or it's not an incentive at all.

------
middleman90
I thought this was in the definition of "scientific"

~~~
nxzero
Peer review is how most science is defined as science and peer review does not
require reproduction of the work.

~~~
seanmcdirmid
Much of what we peer review is not real science, at least in its definition of
applying the scientific method.

For example, much of computer "science" is not. Math maybe, engineering
probably, design sometimes, but "science" is rarely done. BUT the science envy
is there, especially post 1990s, and it is as confusing as heck when multiple
definitions of "science" collide in a conference culture.

Yes I'm a researcher, no I'm not a scientist.

------
lutorm
Define reproduced? Do we mean "conduct the same experiment multiple times so
we can assess the variance on the outcome"? Or do we mean "conduct the same
experiment multiple times to figure out if the first result is a screw-up"?

Those two aren't the same, and I think far too many think that the point is
the latter when, imho, it's actually the former. Pure screwups will likely get
found out, just like glaring bugs are usually found. It's when your result
actually has a huge variance but you're looking at only one (or a few) samples
and draw conclusions from it that's insidious, like the fact that it's the
bugs that just change the output by a tiny bit that are the hardest to notice.

------
typhonic
I've always been amazed by how widely the Stanford Prison Experiment results
are accepted when a) the experiment has not been repeated and b) the
experiment didn't even get completed. It was stopped when the researchers had
made up their minds about the results.

------
westurner
So, we should have a structured way to represent that one study reproduces
another? (e.g. that, with similar controls, the relation between the
independent and dependent variables was sufficiently similar)

\- RDF is the best way to do this. RDF can be represented as RDFa (RDF in
HTML) and as JSON-LD (JSON LinkedData).

... " #LinkedReproducibility "

[https://twitter.com/search?q=%23LinkedReproducibility](https://twitter.com/search?q=%23LinkedReproducibility)

It isn't/wouldn't be sufficient to, with one triple, say (example.org/studyX,
'reproduces', example.org/studyY); there is a reified relation (an EdgeClass)
containing metadata like _who_ asserts that studyX reproduces studyY, _when_
they assert that, and _why_ (similar controls, similar outcome).

Today, we have to compare PDFs of studies and dig through them for links to
the actual datasets from which the summary statistics were derived; so
specifying _who_ is asserting that studyX reproduces studyY is very relevant.

Ideally, it should be possible to publish a study with structured premises
which lead to a conclusion (probably with formats like RDFa and JSON-LD, and a
comprehensive schema for logical argumentation which does not yet exist).
("#StructuredPremises")

Most simply, we should be able to say "the study control type URIs match",
"the tabular column URIs match", "the samples were representative", and the
identified relations were sufficiently within tolerances to say that studyX
reproduces studyY.

Doing so in prosaic, parenthetical two-column PDFs is wasteful and
shortsighted.

An individual researcher then, builds a set of beliefs about relations between
factors in the world from a graph of studies ("#StudyGraph") with various
quantitative and qualitative metadata attributes.

As fields, we would then expect our aggregate #StudyGraphs to indicate which
relations between dependent and independent variables are relevant to
prediction and actionable decision making (e.g. policy, research funding).

------
dschiptsov
According to old-school philosophy of science truth could be discovered only
by removing all the nonsense, as a remainder, not by pilling up nonsense on
top of nonsense out of math and probabilities.

Probabilities, for example, are not applicable to partially observed, guessed
and modeled phenomena. It should be a type-error.

As for math - existence of a concept as a mathematical abstraction does not
imply its existence outside the realms of so-called collective consciousness.
Projecting mathematical concepts onto physical phenomena which could not be
observed is a way to create chimeras and to get lost in them.

Read some Hegel to see how it works.)

------
triangleman
Ironically, one of the reasons Semmelweis's colleagues rejected his "hand-
washing" hypothesis was that it did not have a good enough
empirical/statistical basis.

[http://www.methodquarterly.com/2014/11/handwashing/](http://www.methodquarterly.com/2014/11/handwashing/)
[https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz...](https://en.wikipedia.org/wiki/Contemporary_reaction_to_Ignaz_Semmelweis)

------
macspoofing
Or at least, the media shouldn't report on results until they have been
repeated. This would cut down on the daily "X causes cancer / X doesn't cause
cancer" media spam.

------
aficionado
The solution is easy and it applies to most sciences: all research articles
should include a pointer to download the dataset that was used and an annex
with the details on how it was collected.

------
michaelbuddy
Agreed, which means 50% of social science at least is disqualified and should
not be making into future publications or become part of curriculum.

------
VlijmenFileer
Like climate science, right? Let's set up a statistical meaningful set of
equivalent earths, and start doing some serious peer review.

------
tudorw
this increasingly includes code that needs to run in the future, and citations
within code, see this group working in that field
[https://www.force11.org/sites/default/files/shared-
documents...](https://www.force11.org/sites/default/files/shared-
documents/software-citation-principles.pdf)

------
collyw
Just to play devils advocate, won't there be a self correcting mechanism?

If results are genuinely useful, then people will want to build upon that
work, and will have to repeat the science. On the other hand if it can't be
repeated, then it will not get further work done and fade into obscurity.
Curious what other peoples opinion on this are?

------
bane
I think a better way of thinking about what we want than "repetition" is
"independent corroboration".

------
grashalm
Sometimes in cs if your research is embedded in a huge ecosystem, it can
become quite expensive to reproduce results. I mean proper reproduction, not
just rerunning the Benchmarks. If you are dealing with complicated stuff, the
reproducer might also just not be able to do the same thing technically.

~~~
osivertsson
Maybe, maybe not. Do you have something specific in mind here?

I hope researchers and scientists don't considers others not capable enough,
and therefore withhold info on how to reproduce.

Even if the experiment is crazy expensive and complex right now it might be
considered much more tractable in 10 years, or someone builds upon your work
and invents a simpler method to show the same thing.

~~~
grashalm
I am thinking of huge endeavors like building an asic or huge complex systems
like virtual machines. Not always a comparable system for Repetition is
available and must be built from scratch. Affording such rebuilds require huge
sums.

Of course nobody does consider others not capable enough. Its just that there
are not so many people experienced enough to build certain systems in a decent
amount of time.

------
chucky_z
In for instance, a bioscience lab, I don't believe that results should even be
accepted unless they're repeated with similar reagents. Some reagents are so
specific they only prove something.... for that one single thing, which could
be unique on this planet.

------
vonnik
In that case, macro-economics is simply disqualified from being scientific.
It's almost impossible to repeat large-scale events, controlling for all
variables. Have to say I'm not particularly impressed with the quality of
Nautilus's analysis.

~~~
T-A
> In that case, macro-economics is simply disqualified from being scientific.

Well, duh? (In other words: of course it's not a science.)

------
twoslide
One problem is that there is no incentive to replicate. From the PhD onwards,
academia creates incentives for original research. Replications, particularly
those that confirmed existing research, would not benefit the researcher much.

------
tedks
From the authors of "Why Science Needs Metaphysics" this rings a little
hollow.

Nautilus is just a slightly less vitriolic version of the Aeon-class anti-
science postmodernist blog. Like Aeon, it's garbage.

------
Zenst
Independently verified and repeated I would add.

After all any scientific test that fails when somebody else repeats it is not
science but the domain of magic and religion, so clearly not science.

------
colinprince
I searched for the word "tenure" in the article, but didn't find it.

The drive to get tenure is a big reason that scientists publish so much, funny
that was not mentioned.

------
mankash666
What we need is accountable statistics - something that cannot be manipulated.

One idea is to enforce storing or indemnifying a time-stamped data base of the
raw data on the block chain

------
cm2187
How do you that in the medical field? Studies are often based on a small
number of patients affected by a particular condition.

------
jheriko
wait! we should use the scientific method for science?

its a radical suggestion. sad that it is, but true... ;)

seriously though... you can't have falsifiable results if you don't constantly
try to falsify them. then it just becomes a result, which means the conclusion
you can draw is close to nothing.... not quite nothing, but exceptionally
close. :)

------
vorotato
There is no science without repetition.

------
sevenless
We also should not accept historical claims that have not been repeated :)

------
return0
We now have the tools to do it , and we should be doing it. The fate of
scientific findings is not to publish papers, they belong to open and
continuous scrutiny. And someone should build a github of scientific facts.

~~~
ipstone2014
let's try it, if interested, send me a tweet at @isaacpei, seriously thinking
about creating something for this

~~~
return0
There is no lack of efforts to get scientists discussing (shameless related
plug [http://sciboards.com](http://sciboards.com)), but unfortunately there is
a disincentive for scientists to do so (politics).

------
Eerie
UGH. Scientists are not getting funds to repeat result.

------
known
Lottery < Statistics < Science

