
Eyes wide shut: pseudo-placebo effects (2012) - gwern
http://mungowitzend.blogspot.com/2012/03/eyes-wide-shut.html
======
thaumasiotes
> The paper illustrates the point by undertaking two different RCTs on cowpea
> seeds in Tanzania. One is a traditional study where the control group knows
> they are getting traditional seeds and the treatment group knows they are
> getting modern seeds. The second is double blind; neither group knows what
> seed it is getting.

> The traditional RCT shows a significant over 20% increase in yields from the
> modern seed. But the double blind RCT shows that virtually all of that
> improvement comes from changed behavior

> Specifically, the average treatment effect in the double blind RCT was zero!
> And when the harvests in the control groups across the two RCTs were
> compared, the blind control group showed a significant over 20% increase
> over the traditional RCT control group which knew they were getting the
> traditional seeds.

> In other words, the significant effect found in the traditional RCT was not
> due to better seeds, it was due to actions taken by the farmers who thought
> they were getting better seeds (they planted them in larger plots with more
> space between the plants on better quality land). These farmers'
> expectations were wrong (in post experiment surveys, over 60% of them said
> they were disappointed in the yields)

I don't like this reporting. The way things are phrased, it looks like
Tanzanian farmers have depressed yields 17% just by having bad farming
practices. But it goes on to specifically say that the farmers were
disappointed in their 20% increased yields.

The most sense I can make of this is that these experiments both measured
yield per seed rather than something useful like yield per acre -- despite the
fact that the farmers being studied clearly considered this irrelevant. Are
seeds really such a major cost in farming? Was the study just terribly
designed?

------
xx101010
The general argument is true, of course.

"And, as a fascinating new paper points out, this knowledge can produce
"pseudo-placebo effects" in RCTs. That is to say, the expectation of receiving
the treatment can cause people to modify their behaviors in a way that
produces a significant "average treatment effect" even if the actual
intervention in not particularly effective."

It is barely even "pseudo". This falls into my area of specialty, where
suggestions and expectations are 'how you actually get anything done'. The
more implicit and indirect, nearly, the more likely the suggestion is to take
effect. If one discounts the all important 'confidence' and 'expectation'
angle.

In a clinical trial there is going to be very heavy implications, however
towards confidence. They invested these resources to do the testing, so that
implies confidence and expectations.

------
gumby
This is a serious problem in clinical trials (anomalously high placebo
response) even without self-reported results.

