

A/A Testing: How I increased conversions 300% by doing absolutely nothing - taylorling
http://kadavy.net/blog/posts/aa-testing/

======
thanatropism
Which is why scientific tests have a placebo option.

Very loosely, what hypothesis testing is telling you is whether your results
are so aberrant that they can't just be explained by model variance (coins
flipped at random). The hypothesis that can't really be tested is whether your
model explain most of the variance in observed data.

If I take a random sample of people, ask them whether they're male or female
and what's their yearly earnings, I'll find statistically significant
differences in averages; however, that will explain only a very small portion
of the total variance in wages, given that these also depend on education,
skills, experience...

It could be that the main source of variance in the poster's experiment is the
time of the day emails are sent. How would he know? When doing crude A/B
testing one hopes that all the things in the world one _isn't_ controlling (is
it raining? have local sports team won trophies?) are uncorrelated with the
control variable. But here it might not, there might be something very
different between the batches he doesn't know about.

And he's proud of that!

------
robodale
He makes some great arguments. Especially for the solopreneur (like me),
making maximum use of time and energy available to me during the day is
critical. There is risk in spending time on A/B testing, with the results of
much work put towards this coming back "Inconclusive".

I could see extreme cases where you need to test a major area of the business,
testing an entire market segment, etc...but then you could argue just pick up
the damn phone and call people in that market segment.

