

A/A Testing: How I increased conversions 300% by doing absolutely nothing - aidanf
http://kadavy.net/blog/posts/aa-testing/

======
geoelectric
If nothing else, this succinctly describes how to vet decision processes based
on statistical tests.

I ran into a situation within my own organization where costly decisions were
being made on variations in statistical test results that I suspected were
local minima/maxima in natural variance. Replaying the decision process
against the same target multiple times provided strong evidence towards
changing how we did that.

It wasn't enough to just calculate the estimated population variance, etc.,
because explaining the significance of that was essentially a TL;DR problem to
anyone whose opinion counted. It also required significantly better stats
skills than I had, since adequately explaining the issue generally requires
more expertise than recognizing it in the first place.

However, empirically demonstrating that the decision process was flipflopping
on the same available information was extremely compelling.

