This isn't correct. The statistical power of n=5 humans is quite low.
It is, however, a good example of the "law of small numbers" of Tversky and Kahneman, a cognitive bias in which people believe that the law of large numbers applies to small numbers as well.
See Tversky and Kahneman 1971, or Kahneman's fantastic recent book Thinking, Fast and Slow which is an excellent guide to how our cognitive biases can wrongly influence our thinking.
> This isn't correct. The statistical power of n=5 humans is quite low.
A few points are important to consider.
First, I was only talking about experimental studies searching causal relationships. There are other possible designs, for example field studies (e.g. "school district A gets the new math curriculum, school district B the old one. Which one fares better?") or simple population observations ("people playing golf live longer than the average population."). Each design has advantages and disadvantages regarding generality of the statement one can make, and for each one different statistical considerations apply.
Second, the statistical power does not rely on a high population alone, as that (more or less) only affects the significance tests. Much more important is the effect size. If you can measure a large effect (as this study did), it's pretty hard not to reach significance anyway.
Third, from a statistical point of view, the population isn't 5, but much higher.
Let me explain:
There are certain kinds of treatments whose effect is reversable. Caffeine intake is an good example: Once you stop taking caffeine, the effect recedes. While designing the study, you can use that property. One common way is an ABAB design, where A is a phase with treatment and B is a phase without. You can chain as much AB pairs as time permits, and additionally you can measure multiple times per phase. Statistically, the population now is real_humans x number_of_phases x measure_points_per_phase.
It is, however, a good example of the "law of small numbers" of Tversky and Kahneman, a cognitive bias in which people believe that the law of large numbers applies to small numbers as well.
See Tversky and Kahneman 1971, or Kahneman's fantastic recent book Thinking, Fast and Slow which is an excellent guide to how our cognitive biases can wrongly influence our thinking.