
ABBA: A/B Test (Split Test) Calculator - rgbrgb
https://www.thumbtack.com/labs/abba/
======
connoredel
This is in my opinion the best simple A/B test significance calculator. I like
it for two reasons:

1) The multiple testing correction, which many people ignore when testing > 1
variants vs. the control

2) They display not only the p-value, but the confidence interval around the
_difference_ between the variants (i.e. the effect size). It's tempting to
observe an increase of say 50% that is statistically significant and think
that you can be confident about the 50%. This is not the case! You chose your
sample size to be confident that the actual effect size is > 0 -- not that you
would have a tight confidence interval around whatever difference you
observed. Depending on your sample, the lower bound could still be close to 0.

------
ascotan
Ah A/B testing. Looks like someone needs to watch groundhog day.
[https://blog.codinghorror.com/groundhog-day-or-the-
problem-w...](https://blog.codinghorror.com/groundhog-day-or-the-problem-with-
ab-testing/)

~~~
danenania
What that post describes is simply a bad approach to ab test design. Measuring
impact is orthogonal to creativity and good strategy. It's not the test's
fault if your design is uninspired or your strategy piecemeal, and likewise
declining to accurately measure the impact of an 'authentic' and outside-the-
box type feature doesn't mean your design is actually achieving its goals--
you've only found an excuse to put your head in the sand and avoid
accountability.

------
bhattisatish
I am not sure what kind of service this site provides, but it is region
locked! When I try to access the site, (any part of the URL, even the
homepage, I get to see the following error:

"Sorry, this request is coming from a country which we currently do not
support. If you feel this is in error, please contact our support team by
emailing support@thumbtack.com."

I guess even blogs and articles are now coming under region protection.

~~~
birken
That is pretty funny, I wrote the code for that blacklist like 5-6 years ago
when Thumbtack was being hammered with spam requests from China and India. I
thought it only blocked POST requests, though maybe my memory is wrong or they
have made the block more expansive in the meantime.

Anyways there is a version or ABBA usable on github or you can always get the
source and run it locally:

Source: [https://github.com/thumbtack/abba](https://github.com/thumbtack/abba)

Demo:
[http://thumbtack.github.io/abba/demo/abba.html](http://thumbtack.github.io/abba/demo/abba.html)

