

Why we were wrong about why we were wrong. - StavrosK
http://blog.historio.us/why-we-were-wrong-about-why-we-were-wrong

======
ced
_Decide your sample size beforehand, and only stop the test after you've
reached it._

That may be the gospel among A/B testers, but from a Bayesian perspective,
this is definitely not the best you can do (though it won't necessarily be
_wrong_ ). Beware, Bayesian methods are provably optimal.

The details are here for the interested (chap. 4 & 6):

<http://www-biba.inrialpes.fr/Jaynes/prob.html>

It's not casual reading... I wish there was something shorter available.

~~~
StavrosK
Thank you, I downloaded the Bayesian method paper linked in the article and
was going to read it, I'm very interested in a probably optimal stopping
method (obviously). I'm only afraid it'll fly over my head, despite my having
a bit of stats background.

Thanks again!

~~~
ced
You mean this paper?

Berry, Donald A. “Bayesian Statistics and the Efficiency and Ethics of
Clinical Trials,” Statistical Science, Vol. 19, No. 1 (Feb., 2004), pp.
175-187

I haven't heard of it. But if you're seriously motivated to do things right
and you're "mathematically mature", I would advise you to read Jaynes' book
linked above. For me it was a huge eye opener. The screed of the Bayesians is
"You don't _need_ a separate statistics field! You just need the laws of
probabilities, and that's enough to answer any question you want."

Let me know if you need help, my email is in my profile.

~~~
StavrosK
Yep, that's the paper. I'll try to read the book when I find some time
(thankfully, "get more data" is good enough for now).

Thanks again for your help.

------
carbocation
StavrosK, I'm delighted to see you post this; I was hoping you'd do so!

If people are interested in calculating the sample size before running an A/B
test, I'd recommend tools such as the following (to which I have no
connection):

<http://www.danielsoper.com/statcalc/calc01.aspx>

~~~
StavrosK
Thank you very much, I looked for a calculator like this but they were all too
complicated. I'll update the article to include this.

