Hacker News new | past | comments | ask | show | jobs | submit login

In this case, the problem is more subtle & interesting than the usual correlation!=causation issues, and I owe it to the psychologist Paul Meehl: matching methods have the problem that when you match groups on some observed variables but different outcomes, you're guaranteeing they will be very weird in unknown ways.

His example was kids, schooling, and IQ; to 'control for' IQ you might try to compare kids with bad grades & high IQ with some other kids with good grades & high IQ, in order to find the effect of schooling on some outcome like crime. Except... kids with high IQ should have good grades; so why does that first group have bad grades? Why are they weird like that? What's going on? Do they have some sort of severe home problem? Emerging mental health issues? Systematic personality differences? You may well find that having 'controlled for' IQ, the kids with bad grades are more likely to be jailed and conclude that schooling helps prevent crime... but what have you really found?

In this case, what the researchers have found is that twins are powerfully locked together in terms of health outcomes, body types, diets, and exercise tolerance/preference, so locked together that they have to scour millions of people to turn up a handful of twin pairs. These twin pairs are 'matched' on genetics/upbringing (IQ), but not matched on exercise (grades), and so one gets health outcomes (jailing) 'controlling for' genetics/upbringing... so one concludes exercise helps prevent worse health... but what have you really found?

(At least, I think this is what Paul Meehl was talking about in http://www.mcps.umn.edu/assets/pdf/4.13_Meehl.pdf "Nuisance Variables and the Ex Post Facto Design", 1969, in _Minnesota studies in the philosophy of science: Vol. IV. Analyses of theories and methods of physics and psychology_ - that paper is unusually opaque for Meehl, and I'm not sure I understand it.)




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: