Rule #1: Don’t be afraid to launch a product without machine learning.
Just because you can use machine learning doesn’t mean you should.
(AI has some very advanced techniques in it and require attention to detail and background knowledge to use them correctly)
It's easy for data scientists and machine learning engineers see P/R, AUC, or whatever as the goal, especially if there isn't much support in the organization for measuring product performance. It's often not the end goal. Measurements of a model's performance in this context indicate some measure of statistical performance with respect to training and test data. Real, live measurements from "in the wild" application are the true fitness test.