
Highlight negative results to improve science - headalgorithm
https://www.nature.com/articles/d41586-019-02960-3
======
6gvONxR4sf7o
There's another issue at play here that's tightly connected to this one.
Underpowered studies. Here's the process. I look elsewhere (e.g. similar
studies) to make an educated guess about the size of effect I'm going to look
for. Let's say I land on 2 units. Then I ask how many samples I need to detect
a 2 unit effect with some low false positive procedure. I want to detect my 2
unit effect with high probability if it's there.

So you end up with three numbers. The probability you find something given
that there's nothing, often 5%. The probability you find nothing given that
there's something (something being a 2 unit effect), often 20%. The size of
effect you're using in these calculations (2 units).

The smaller effect size you're looking for, the more data you need. The lower
error rates you're looking to get, the more data you need. So you make
sacrifices and accept higher error rates. You make optimistic effect size
estimates (making your true error rates even higher).

Using p < 5% as a positive result publishing threshold is bad enough. Using
power > 80% as a negative result publishing threshold is even crazier. That's
a 20% error rate for publishing negatives. Higher if you used optimistic
effect estimates.

What this all means is yes we should publish negatives, but a negative worth
listening to is going to be expensive. In a bayesian sense, it's the
difference between me spending little money to learn that the effect of coffee
on mood is probably between -1 and 1 (a bad negative result) vs spending a lot
of money to learn that it's probably between -0.001 and 0.001 (a more useful
negative result).

~~~
SubiculumCode
Thank you for summarizing this. Overly powered studies are rare, and this is
the primary reason that null results are not given the status as positive
results do. They should go into a journal of negative results and boldly state
a power analysis.

------
dekhn
Reason #37 that I am no longer a scientist: most of my time as a scientist was
spent attempting to replicate the positive results of other scientists and
failing, because the work was not replicable.

I actually trust negative results way more than positive results now- it's a
proxy that you can trust the investigator. I wish we had a Journal of Negative
Results.

~~~
rjyoungling
Such a fascinating take. When I was young I used to think that scientists and
the scientific process were perfectly objective. As you get older you learn
that human beings have human being problems. Those don't magically disappear
once you become a scientist.

You see this even in highly objective fields such as physics, where much
progress comes from the previous generation simply dying.

Our own ego's, as well as the factors mentioned in the article, create
problems much more than you might expect when you're young and naive.

~~~
dekhn
Is physics truly objective? I see people still arguing about quantum
mechanics, and it looks like the safest conclusion is that we don't actually
live in a straightforward, temporally causal world (prediction:
[https://en.wikipedia.org/wiki/Wheeler%27s_delayed-
choice_exp...](https://en.wikipedia.org/wiki/Wheeler%27s_delayed-
choice_experiment), experimental support:
[https://en.wikipedia.org/wiki/Delayed-
choice_quantum_eraser](https://en.wikipedia.org/wiki/Delayed-
choice_quantum_eraser))

Nobody's making progress because people are dying (we still see many vestiges
of the Cophenhagen interpretation!)

~~~
rjyoungling
Yeah, I see your POV.

Think it depends where you're coming from. Physics is objective if by
objective you mean we can predict how the wavefunction will evolve according
to the Schrodinger equation.

(Depending on who you ask) it might be less objective if you ask: ''But what's
REALLY going on?'' Then you start getting into the interpretations of qm and
different scientists will have different views.

I think it's at least likely that we'll be able to answer that question in a
few decades (or at least move the ball forward). The view of Everettians, for
example, is falsifiable and there are experiments going on as we speak trying
to achieve that.

------
hannob
So how many negative results has Nature published lately?

------
Alex3917
If studies with negative results are as valuable as studies with positive
results, do the sales curves for reprints look similar for the 100 best-
selling studies with positive results and the 100 best-selling studies for
negative results?

~~~
celeritascelery
It’s not monetarily as valuable, but scientifically.

------
bryanmgreen
Applicable across mostly everything, right? Relationships, business, Amazon
product reviews.....

------
slowhand09
relevant
[https://news.ycombinator.com/item?id=20997978](https://news.ycombinator.com/item?id=20997978)

