
We Should Not Accept Scientific Results That Have Not Been Repeated - schrofer
http://nautil.us/blog/-we-should-not-accept-scientific-results-that-have-not-been-repeated
======
eesmith
I did not see a discussion about science based on non-repeatable observations.

Are the neutrino observations of SN 1987A not science because it's not
repeatable?

"To the surprise of everyone involved, they were able to replicate only six of
those papers—approximately 11 percent."

Oddly, that publication is not repeatable. More specifically, I think they
described only 3 of the 53 experiments they tried to replicate.

Under the author's own definition, we should not accept the results of that
paper.

~~~
oldandtired
The neutrino observations are a datum. Along with much other data and
reproducible experiments, this can be helpful in furthering our understanding
of the universe around us.

But it is not a reproducible experiment, it is simply an observation. It can
be the basis for creating reproducible experiments that will further our
knowledge.

But if it is a one-off observation then it can be given no more significance
than that.

I think the upshot of what the author of the article is highlighting is that
there is little incentive to do the replication experiments as this does not
give any significant recognition or other funding opportunities.

Replication required having all (or most) of the information about the
original experiments and being able to confirm the original findings. It has
been commented on before by others in various science fields that this
information is not available for many experimental reports.

It is also known that if anomalous results are obtained that contradict
prevailing theories then it takes a brave and hardy experimenter to continue
pushing those results.

When it boils down to it, scientists are no more honourable or upright than
any other segment of society. They are people too and as such, like all the
rest of us, have foibles, follies, biases, etc. that can get in the road of
furthering our knowledge.

~~~
eesmith
I agree that my example is not a reproducible experiment. My question is, why
should we reject it as a scientific result?

I believe that the author has an incorrect understanding of what science
means, is biased by being a researcher in a field ("stromal-tumor interactions
in pancreatic cancer") where it is easier to carry out reproducible
experiments.

Astronomy is not based on reproducible experiments, but it sure feels like
science to me.

I believe that the author's acceptance of the un-reproducible Amgen results
highlights how the author hasn't fully integrated the consequences of only
accepting reproducible results.

~~~
oldandtired
The answer is that it is simply a datum. Data can be results, but you would
usually have it as the result of something, an experiment or an equation, etc.

The definition of what "science" is or means, is one of those "mirky" kind of
definitions. It depends on who you are talking to and the context in which the
discussion is taking place. I have yet to see a universally agreed upon and
well defined definition.

People seem to act as though it is one of those "I know it when I see it" kind
of things.

You could start with a Popper's definition, yet there are fields in which this
definition is not excepted (various theoretical physics areas are some).

Astronomy is an interesting field. There is a whole lot of data that is
collected and systematically so. Some of this data is used to support the
various theories and models that have been developed over time. Yet, there are
quite distinct disputes that have arisen as to what some of the various
anomalous data means.

Questions that relate to red-shifts or galactic movements that are too fast,
etc.

science has, I think, a number of possible features that data collection and
analysis, hypothesis testing, theory and model development. The thing I see
across a lot of fields is that there are specific ideas that are accepted by
the majority but have difficulties with the various anomalous data points that
are found.

How to solve this in the short or long term is not going to be easy, simply
because all scientists are people and like all people have biases in what they
believe is true. So you get various "experts" in their field having disputes
with one another over the "truthieness" of their specific models and theories.

So collecting the data is essential but doesn't necessarily lead to any
understanding.

~~~
eesmith
I'm not asking you to give a good definition of science.

I'm pointing out that the author of the piece make an assertion that only
repeatable experiments count as scientific results.

I disagree.

I want to know why we should not accept results which are not repeatable.

You response seems to be more aligned with mine views than that of the author.

For something down to earth, is the Amgen publication a scientific result?

Even if we accept that the author is correct, what it it mean to repeat, say,
an epidemiological study? We can't repeat the exact conditions. All we can do
is be "close".

~~~
oldandtired
Now that I am back from cleaning a pool and collecting wood for winter, I
would say that determining if a datum from a non-repeatable event is
"scientific" or not, requires a clear definition of what "science" is.

I don't have a problem with data from non-repeatable events being used. But it
would depend on how it was collected and how it is used. Too often, we see
non-repeatable data being used to defend a specific point of view, when that
data is only data and could be used to defend any number of points of view.

If one cannot develop repeatable experiments to validate your model and
theory, then non-repeatable data cannot be used as a proxy to defend your
model and theory. It can be used as additional support but that's about as far
as it goes.

Of course, different people in different fields can and will have different
views about such data. The thing to remember is that data is data and not the
theory itself. It may support a theory but it cannot ever prove a theory. In
every field of study, much data has been collected but there is always
anomalous data found. This can arise from the simplest of things like the
measuring equipment was set up wrong to the theory or model being developed
being wrong.

~~~
eesmith
Again, I think we are in general agreement.

The author of the original essay doesn't cover any of these issues, which is
my complaint.

BTW, there are experiments which could at one time be repeated but can no
longer, either because the required conditions no longer exist (East Germany,
passenger pigeons) or because the experiments are no longer ethical or
warranted (long term progress of untreated syphilis, the most effective way to
firebomb a city).

The Trinity test, even without being repeated, did a good job of proving the
theory that setting off an atomic bomb does not ignite the atmosphere.

