One can commit to a performance metric threshold prior to A/B testing. For example, a rule that states "New, nonmaintenance features must increase revenue for the product by at least 1.0%".
An objective rule like that neatly resolves the problem of biasing analysis after the test, and doesn't have to necessarily involve any complicated statistics.
An objective rule like that neatly resolves the problem of biasing analysis after the test, and doesn't have to necessarily involve any complicated statistics.