
Too many medical trials move the goalposts. A new initiative aims to change that - annapowellsmith
http://www.economist.com/news/science-and-technology/21695381-too-many-medical-trials-move-their-goalposts-halfway-through-new-initiative
======
nonbel
We will never be able to enforce this and do productive research. It is all
due to mathematicians' fantasy about how research works and researchers'
fantasy about how stats work.

Something _always_ needs to be adjusted that is figured out during the study.
Focus on testing precise predictions and observing similar results in
different studies rather than significant differences. Despite it being much
his "fault", Fisher pointed this out nearly 60 years ago:

Fisher, R N (1958). "The Nature of Probability". Centennial Review 2: 261–274

~~~
napoleoncomplex
The standard that COMPare is aiming to achieve, as far as I understand it, is
the following:

1\. Outcomes that will be measured for clinical trials need to be registered
and public ahead of the trial (this is already achieved with recent
regulation, as mentioned in the article)

2\. If planned outcomes change, the change needs to be reported and explained,
with new outcomes being again clearly stated.

I really struggle to see how enforcing this will prevent productive research.
This is basic scientific hygiene, not fantasy. No one is arguing that studies
shouldn't be adjusted, just that the adjustments should be clearly stated.

Completely changing the goals of a study after failing on your initial goals,
and then claiming success without mentioning any of the failures along the way
is poor science, because you're masking a lot of potentially significant
findings along the way that others can benefit from as well.

Not to mention that these standards have been agreed to by all the largest
journals decades ago, with big PR claims of how this is needed etc., yet are
never followed in practice. The responses of some journals (NEJM, JAMA that I
recall) have been shameful, and contradictory to standards they claim to
follow.

~~~
nonbel
Misrepresenting your methods is just fraud, so you will get no argument from
me there.

However, think about the difference between A) setting a point prediction of
your model as the null hypothesis vs B) "two groups are the same and samples
were independent, etc" as the null hypothesis.

In the first case there is a very small range of plausible outcomes consistent
with the researcher's theory. In the second case there is a very large range
(usually 50%). In the case of scenario A, messing up the experiment (either on
purpose, or accident) makes it more difficult to claim evidence consistent
with your model. In the case of scenario B, the same thing makes this easier.

So researchers using scenario A are incentivized to be as careful and account
for as many different sources of error as possible. Under scenario B, the
incentive is the opposite, the sloppier the study the easier it is to get
results consistent with the researcher's theory.

------
StavrosK
Isn't what AllTrials[1] is doing a much better way of fixing medical science?

[1]: [http://www.alltrials.net/find-out-more/all-
trials/](http://www.alltrials.net/find-out-more/all-trials/)

~~~
hannob
These are different goals.

All trials is about not hiding when you have done a trial that doesn't produce
your desired result. (Publication Bias) COMPare is about being transparent if
you change the purpose of your trial while it's happening. (Outcome switching)

Both problems can be countered by good preregistering practice. And both are
important issues that need to be fixed. (And by the way: not only in medicine.
This is a problem that is widely a problem in science.)

~~~
StavrosK
Oh hmm, I thought you had to pre-register your purpose with AllTrials as well.
Too bad, they could have killed two birds with one stone.

