

Genetic Algorithms in Multivariate Email Optimization - erickerr
https://tech.bellycard.com/blog/genetic-algorithms-in-multivariate-email-optimization/

======
dlss
> The minimum threshold odds [to stop showing a variation] are calculated by
> 40% / number enabled variations

Uh... How did you choose those cutoffs? Looks like you have a significant
chance of making the wrong choice.

Also:

> Once enough data is collected to start making conclusions (1000 sends per
> variation)

You should check out the Bayesian solution to the Multi-Armed Bandit problem.
It's very close to what you are doing, but makes decisions much faster than
you do because it isn't deciding to turn off a variation, merely to scale it
down.

~~~
erickerr
Thanks a lot for the feedback. This is a first pass implementation, but I
agree that more thought should be put into the cutoff threshold, specifically
for when there are only initially 2 (or maybe 3) variations.

We considered a weighted decision approach but 1) were turned off by posts
like [http://visualwebsiteoptimizer.com/split-testing-
blog/multi-a...](http://visualwebsiteoptimizer.com/split-testing-blog/multi-
armed-bandit-algorithm/) and 2) wanted to keep moving parts to a minimum for
V1.

Any thoughts?

~~~
andrewryno
You can calculate the minimum sample size needed to come up with that
threshold rather easily. See: [http://vuurr.com/split-testing-determine-
sample-size/](http://vuurr.com/split-testing-determine-sample-size/)

~~~
Homunculiheaded
Except in that example the author is choosing E based on the observed
difference between two means (which is the exactly the unknown you're trying
to determine, so it makes no sense to use it as a constant in a formula),
rather than the threshold for the minimum distance you care about.

If you're going the classical statistics route the entire point is that you
need to determine your sample size before you peek at the data. In that post
you would need to replace E with a threshold of difference that you care
about, then calculate n before you start the test and not look at the results
until you had reached n observations.

------
btilly
With email optimization in particular be warned that people will respond to
changes simply because it is different. But then acclimatize.

What that means is that the changed version tends to win the test, but then
may or may not perform well. The flip side of this is that if you have a
choice, have multiple variations on the same email that you rotate between so
that people don't get too used to your emails.

------
TrainedMonkey
Nice seeing GA applied to more things. People often do not realize how many
problems with feedback mechanism can be solved with GA framework. Here is
small tutorial that got me started with GAs long long time ago:
[http://www.ai-junkie.com/ga/intro/gat1.html](http://www.ai-
junkie.com/ga/intro/gat1.html)

Neural network tutorial of the same site is pretty cool too.

