

"Why do Phase III Trials of Promising Heart Failure Drugs Often Fail?" - gwern
https://dl.dropboxusercontent.com/u/182368464/2003-krum.pdf

======
gwern
Some relevant excerpts:

"There has been considerable recent disappointment with the failure of a
number of major new pharmacological strategies for the treatment of chronic
heart failure. In turn, there has been much speculation as to why trials of
these therapies have not shown benefit. Among a number of plausible and
scientifically valid reasons, consideration should be afforded to the
potential contribution of "regression to the truth." Regression to the truth
derives from the biological concept of regression to the mean, whereby random
fluctuations in a biological variable occur over time, such that the true
value of the variable is approached with repeated measurements. This same
concept can be applied to clinical trial programs for new drugs for heart
failure. Because only strongly positive trials generally go on to phase III
testing, and some of these early phase studies are positive by chance alone,
on retesting in phase III the results are very likely not be as strongly
positive. Numerous examples of regression to the truth apply for trials of
heart failure therapies, as well as in other areas.

To understand regression to the truth we must first consider the concept of
regression to the mean. This concept derives from the random fluctuations that
can occur in a variable over time. As a consequence, a single measurement of
that variable more often yields a value removed from the mean, and the "true"
value of the variable is approached with repeated measurements. As a
corollary, in population studies, a single measurement of the dependent
variable - for example, cholesterol - can lead to an underestimate of the
strength of its association with an outcome such as coronary heart disease
death ("regression distribution bias"). Consider a theoretical drug (drug x)
being studied to determine its benefit in heart failure, as assessed by a
surrogate measure, lowering of plasma norepinephrine (Fig. 1). The left panel
shows that there is really no difference in plasma norepinephrine levels
before and after drug x. However, the investigators went on to perform a
subgroup analysis of those patients with norepinephrine levels above the mean
(middle panel), and that subgroup demonstrated a significant reduction in
norepinephrine levels with drug x. The investigators might therefore claim
that drug x is effective in lowering plasma norepinephrine in patients with
high norepinephrine levels. Furthermore, it is these patients (ie, patients
with high levels) who are those that are particularly in need of a drug that
will lower such elevated levels. Although it is possible that drug x does
indeed lower elevated plasma norepinephrine levels, it is equally plausible
(if not more so) that the high plasma levels were "captured" as being falsely
or atypically high (for the individual patient) at baseline and then when the
same patients were remeasured at a later time point, levels were not as high
(ie, classic regression to the mean). This concept is well-understood for a
biologic variable, but how can this concept be applied to that of a clinical
trial program for a new drug for heart failure? This is conceptually
illustrated in Fig. 2, which depicts early phase trials conducted in the
assessment of a variety of potential new drug therapies for heart failure.
Each dot represents a trial of a certain drug. As can be seen, some early
phase studies will be strongly negative, some strongly positive, but most will
cluster around neutrality and, therefore, one can construct a standard bell-
shaped curve. We know that many trials of new chemical entities are conducted
in the setting of heart failure. Because of the large number of studies
conducted, some will be positive by chance and indeed some will be strongly
positive by chance. Does this matter? Yes, it does. It is highly likely that
only drugs associated with strongly positive trials (ie, those to the right of
the vertical dotted line) will go on to phase III testing. Because some of
these studies that are positive by chance alone will be among these, then when
retested in phase III trials, the results will no longer be strongly positive.
This is analogous to the high plasma cholesterol or norepinephrine being
retested in the earlier examples. This concept is true, not just of heart
failure trials, but of any drug therapy for any specific indication. What
exacerbates the problem in the setting of chronic heart failure is the low
percentage strike rate in the development of successful pharmacologic
therapies for this condition. Only renin-angiotensin and β-adrenoceptor
blocking agents have come to the market over the last 30 years or so.
Therefore, very few promising drugs in early phase would be positive in phase
III (if tested) and thus registrable for a heart failure indication. This is
illustrated by the open circles below the curved line, interposed on the
totality of early phase trials in Fig. 2. This line is curved because, of
course, a strongly positive early phase study will make it more likely (but
possibly still with low probability) of positive findings in phase III
studies. Nevertheless, this still leaves a large number of trials strongly
positive in early phase by chance alone (circled cluster) "regressing to the
truth.""

