If it was so simple, everyone would be doing it correctly. Even you don't give a simple account of what needs to be done. What you have done is talk down to someone else. All in all, I think your entire argument would be a lot more sympathetic if you made it by giving a concrete example of your own beliefs (what do? Is that really simple?), otherwise you will have to get used to folks not listening to you.
> Even you don't give a simple account of what needs to be done.
Fair enough - though I did mention a few basic things like A/B testing and survival bias (trying to being aware of one's own cognitive biases is the first step in making better decisions; to be aware means you first have to know them).
And yes, it is not trivial at all and dependent on the context.
The best and simplest starting point would be to make a list of potential confounding variables (I mentioned a few of them) and then just looking at the R^2 statistic.
That already should point him in the right direction.
I was less bothered that his methodology wasn't 100% correct, but rather by casually making a statistically unsound decision which has a big effect on people's lives, so getting it right is important - and then when called out dismissing it.
I have a lot of sympathy for people honestly trying to make decisions rigorously and getting it wrong; but you have to try.
