
Ask HN: How do you test your Machine Learning models? - dmonn
Hey all!<p>I&#x27;m looking to get some inputs from professional engineers in Machine Learning about how they test their models that <i>might</i> enter production.<p>I ran a survey in my own network before through a Twitter poll, and found out that over 70% of respondents were not fully confident in the abilities of their models, with 33% of all respondents saying that they are not confident in the abilities of their models at all.<p>Now I&#x27;d like to find out why and how to solve this. I prepared this survey and it would mean a lot if you could take 2 minutes to fill this one out.<p>I&#x27;m looking to write an article about ML testing methods based on those, and will be sure to share that here (and to anyone who requests it).<p>https:&#x2F;&#x2F;dominicmonn.typeform.com&#x2F;to&#x2F;n2AbR7
======
chudi
For production we usually just use some form of a/b testing, as we found out
that the offline metrics are often misleading. When the offline metrics of
your model Like f1 or whatever are good enough we just promote the model to
production competing we the other models in a multiarmed bandit way.

------
ThePhysicist
I'm interested to share our experience. Do you have a link to your Twitter
poll? You can find my contact information in the profile if you don't want to
share infos here, happy to chat!

~~~
dmonn
The survey is here:
[https://dominicmonn.typeform.com/to/n2AbR7](https://dominicmonn.typeform.com/to/n2AbR7)

This is the poll that sprouted this all:
[https://twitter.com/dqmonn/status/1139092798521712641](https://twitter.com/dqmonn/status/1139092798521712641)

