Hacker News new | past | comments | ask | show | jobs | submit login
Simulating A/B tests offline with counterfactual inference (abhadury.com)
57 points by Arnie0426 25 days ago | hide | past | web | favorite | 3 comments



> To mitigate presentation bias, we can always pick a small fraction of users that are always shown uniformly random results

Would it be equivalent to displaying a small number of random (exploration) items to each user in addition to algo recommended (exploitation) items, which seems more palatable?


Thanks for your question!

As long as you can compute the probability of each item being shown (regardless of whether it's uniform random or through your algorithm) to each user, the method still applies. While it does seem more palatable to just show a maximum of X% uniformly random items to all users, the computation of p(y|x) can get a bit hairy for any item y, because now it is a product of a uniform distribution and your stochastic ranking algorithm. If you _can_ compute that, you'd be fine.


As long as you know which items are random, and you can compute the probability of each item (regardless of whether it's uniform random or through your algorithm) for each user, the method still applies.

While it does seem more palatable to just show a maximum of X% uniformly random items to all users, the computation of p(y|x) can get a bit hairy for any item y, because now it is a product of a uniform distribution and your stochastic ranking algorithm. If you _can_ compute that, you'd be fine.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: