He did say they "don't do a ton," so they are definitely doing some, maybe that's where?
That said, I do appreciate the sentiment here. This is a different context entirely, but in my ecomm past a/b testing was often employed by those seeking the politically safest growth path. I considered it proactive CYA where Product Managers would prefer premature optimization over having an opinion on what's best for the user.
I witnessed what I considered an unhealthy amount of a/b testing resources spent on micro adjustments eg moving a buy button a few pixels or changing the color slightly. There was very little calculated risk taking, which can be enabled with A/B testing employed differently. There's absolutely a place for those kinds of optimizations but I think they come much later in product maturity than when many organizations actually begin using them.
Then there's an entire other discussion about understanding the math behind building a relevant test and interpreting the results.
You're right. We do some A/B testing when we're running promotions (test headlines, CTAs, etc.) but that's about it. There is a lot that goes into correctly running split tests on the website and interpreting those results. We decided that once something is working good enough we'd focus on other areas of the business. However, now that the team is larger and we have resources to dedicate to testing and optimizing we will likely start doing this by the end of the year.
Like you said, moving buttons and changing colors is really just a waste of time. You have to spend the time to do the analytics research, get customer feedback, etc. to first understand the problem so you can come up with a solution. Changing the color of a button doesn't solve a problem. Rather than wasting resources on doing conversion optimization the wrong way, we've decided to hold off until we can do it the right way.
That said, I do appreciate the sentiment here. This is a different context entirely, but in my ecomm past a/b testing was often employed by those seeking the politically safest growth path. I considered it proactive CYA where Product Managers would prefer premature optimization over having an opinion on what's best for the user.
I witnessed what I considered an unhealthy amount of a/b testing resources spent on micro adjustments eg moving a buy button a few pixels or changing the color slightly. There was very little calculated risk taking, which can be enabled with A/B testing employed differently. There's absolutely a place for those kinds of optimizations but I think they come much later in product maturity than when many organizations actually begin using them.
Then there's an entire other discussion about understanding the math behind building a relevant test and interpreting the results.