

A/Bingo split testing now on App Engine, built for Khan Academy - dmnd
http://bjk5.com/post/10171483254/a-bingo-split-testing-now-on-app-engine-built-for-khan

======
patio11
Hearing about this made my week.

~~~
wiradikusuma
Hearing about this make me sad because my App Engine is Java-based :(

~~~
patio11
Some options:

1) The code is available under MIT and the docs are CCed and specifically
designed to make it easy to clone this. At least three people have
successfully done so, in a variety of programming languages, web frameworks,
and technology stacks. If programming Java is not difficult for you, block off
about a week or two and _bam_ problem solved.

2) If you have access to vast amounts of cash but your programmers are just
too busy delighting people to write an A/B testing framework, you could hire
me to rewrite the whole codebase in Java. This would only cost you low five
figures.

3) If you don't want to program and don't have tens of thousands lying around
but do have tens lying around, Visual Website Optimizer is a great option for
$CHEAP.

4) If you have neither programming skills nor tens of dollars then you may
want to reevaluate whether A/B testing is a huge need at this point in your
life.

------
cavedave
I do not think you can just stop the test when you find something you think is
significant

<http://www.evanmiller.org/how-not-to-run-an-ab-test.html> 'If instead of
deciding ahead of time, “this experiment will collect exactly 1,000
observations,” you say, “we’ll run it until we see a significant difference,”
all the reported significance levels become meaningless. '

Reading the stats.py it looks like you might be doing this.

------
jedc
Wow... this is awesome! I was literally looking around this week for exactly
this kind of framework. Fantastic work, and definitely plan on using it.

Thanks to Ben and Khan Academy!

------
euroclydon
Troy Goode wrote a port for ASP.NET MVC, ABsolutelyMaybe. I've had good luck
with it. It wasn't too difficult to migrate from the default storage engine,
which is serialization.

<https://github.com/TroyGoode/ABsoluteMaybe>

