
Why A/B testing will loose against advanced AI, all the time - freemachines
https://medium.com/@ustoll/why-a-b-testing-will-loose-against-advanced-ai-all-the-time-c14fc99cf78c#.6ujpgdbpg
======
gus_massa
Lot of words, not a singe real use case. Do they have a working prototype? Is
the AI better than humans?

The idea of A/B testing is that humans are not good judging which version is
better, so the only solution is to run an experiment, so in many cases you
will get better results with an A/B test.

And there are also variations where the split is not constant, like the
mutiarmed bandit. If A/B test is better or worse than multiarmed bandit is
also a good flamewar topic.

~~~
olavolav
Regarding that last bit: You will clearly get better results by using A/B
tests than by having humans take a guess as to what works best – but as you
already pointed out, those are not the only two possibilities. Different
varieties of mutli-armed/contextual/etc. bandits, and then the new wave of
reinforcement learning tricks (i.e. AI) are much better suited to the problem.

(Full disclosure: I didn't write the article, but I am the CTO of the company
that the article is about.)

------
olavolav
I would argue that A/B testing is close to it's end for 99% of it's current
business applications. Right now there is a very limitied range of AI products
that aim to replace A/B testing, but that will change soon enough.

