

Ask HN: Is anyone using machine learning for A/B tests? - pmtarantino

I am starting to apply machine learning in different ways, and I would like to know if someone is using machine learning for A/B tests.<p>For example, is is saving data and then showing A or B based on that data and improving it with each new case. I am not just talking about if A or B converted (that would be only success %), but more data like date, time, gender. For example:<p>I just came with the next idea: Maybe on the night, your visitors are tired for the day, so if you put something like "Register now. Only takes 2 minutes", has more conversions because has a better effect on tired people.<p>I am sorry if I am just divagating (and sorry for my English)
======
patio11
There existed at least one project to do this, but you essentially can't
justify it unless you are an ad network with hundreds of millions of
impressions. The statistical confidence math is otherwise prohibitive. Ask if
you need elaboration.

~~~
rfergie
When you say "hundreds of millions of impressions" do you mean that literally
or are you meaning "a lot of traffic"?

If it is the former then I would like some elaboration; my intuition is
surprised that the amount of data required would be that large.

~~~
barry-cotter
He is talking about Google. Probably Adwords but maybe other Google properties
too.

~~~
rfergie
I'm sure that Google and other large ad networks do this.

But I also see small/medium size ecommerce retailers doing "People like you
also bought" type stuff. They don't do it as well as Amazon, but sometimes it
seems relevant.

So I'm curious about whether hundred of millions of impressions are necessary
for this type of optimisation (and that the good recommendations I've seen are
just a coincidence - quite possible) or if saying "hundreds of millions"
actually just means "quite a lot"

------
e-dard
Well, Myna <https://mynaweb.com/> uses multi-armed bandit algorithms for A/B
testing (or A/B/../N testing if you prefer).

An n-armed bandit problem is a type of reinforcement learning problem, and the
associated algorithms for solving these problems are considered machine
learning.

