
Simulating A/B tests offline with counterfactual inference - Arnie0426
http://abhadury.com/articles/2019-05/simulating-ab-tests
======
mooneater
> To mitigate presentation bias, we can always pick a small fraction of users
> that are always shown uniformly random results

Would it be equivalent to displaying a small number of random (exploration)
items to each user in addition to algo recommended (exploitation) items, which
seems more palatable?

~~~
Arnie0426
Thanks for your question!

As long as you can compute the probability of each item being shown
(regardless of whether it's uniform random or through your algorithm) to each
user, the method still applies. While it does seem more palatable to just show
a maximum of X% uniformly random items to all users, the computation of p(y|x)
can get a bit hairy for any item y, because now it is a product of a uniform
distribution and your stochastic ranking algorithm. If you _can_ compute that,
you'd be fine.

------
Arnie0426
As long as you know which items are random, and you can compute the
probability of each item (regardless of whether it's uniform random or through
your algorithm) for each user, the method still applies.

While it does seem more palatable to just show a maximum of X% uniformly
random items to all users, the computation of p(y|x) can get a bit hairy for
any item y, because now it is a product of a uniform distribution and your
stochastic ranking algorithm. If you _can_ compute that, you'd be fine.

