Interesting article and very contrasting opinion. It's hard for me to agree with most of it though, but I might simply be too closed minded.
A few notes:
>I should admit that we still A/B test some ideas. Rather than adhering to a strict A/B testing process, we let our team to A/B test ideas they're curious about.
The idea is that you don't test everything in the first place, only test things where your thinking is "My hypothesis is that changing or adding feature X will improve certain metrics" -- and in those cases, you a/b test. Else you aren't really proving that you're right or wrong. If it doesn't matter if you're right or wrong, why implement feature X at all? If it's a qualitative improvement, it doesn't make sense to a/b test it.
>3) Performance
>Related to lost conversions, A/B testing tools make your site slower and this also reduces your conversion rate--even for the control. The additional conversions you lose because of this performance hit are another, often ignored, cost of A/B testing.
You should probably be using a different tool if it makes your site slower.
>5) Speed
>A/B testing slows down your organization's decision making. Some ideas are obviously good and adhering to a strict A/B testing process reduces your time to go live. Time is finite and the number of improvements you implement per year has a major impact on your growth trajectory.
It's impressive that growth is simply a factor of time within this company. This makes sense only if their average production decision making has a positive impact on growth/conversion rates as a factor of time.
>You can monitor your long term conversion trend rather easily via Google Analytics to gauge if you're making good decisions.
Yes, but that would be very unactionable. If the growth is going up, you pat yourself on the back and say "I'm clever" and if it isn't you say "I suck". But since you don't know which decisions or what the real reason is for conversion going up or down, you don't know what the fundamental metrics are that drive your growth up or down. You basically say "I don't care why things are going well" or even lie to yourself and say "Things are going well because I make good decisions."
Growth is not just about growing, but about knowing why you are growing and using that knowledge to find new ways to grow. If you know the fundamentals about what drives your growth, you also know which things you can or cannot change without affecting growth.
> You should probably be using a different tool if it makes your site slower.
Any recommendations? I see a lot of tools out there that work by basically rewriting the page on the client side, which will tend to give you suboptimal performance. And if your app's already a JS heavy SPA thing, your over optimistic byte budget is probably already running a deficit.
To take another contrasting opinion. Who cares if you know exactly why you're going. We know in a general sense why we're growing: we constantly improve the experience of our service and make customers happier.
It's quite hard to maintain aggressive year over year growth targets. If you're pulling it off it really doesn't matter if you figured out why via a/b testing or not.
Obviously, we'd prefer stronger YOY growth without knowing the exact causes than weaker YOY growth because we wasted resources on a/b testing to get precise answers to why we're succeeding.
Beyond that, I wasn't referring to trending growth via GA to see if you're making good decisions. You can trend conversion rate by channel to see if you're getting better over time. Obviously, if conversion is going up and to the right for most channels you're making good decisions.
A few notes:
>I should admit that we still A/B test some ideas. Rather than adhering to a strict A/B testing process, we let our team to A/B test ideas they're curious about.
The idea is that you don't test everything in the first place, only test things where your thinking is "My hypothesis is that changing or adding feature X will improve certain metrics" -- and in those cases, you a/b test. Else you aren't really proving that you're right or wrong. If it doesn't matter if you're right or wrong, why implement feature X at all? If it's a qualitative improvement, it doesn't make sense to a/b test it.
>3) Performance >Related to lost conversions, A/B testing tools make your site slower and this also reduces your conversion rate--even for the control. The additional conversions you lose because of this performance hit are another, often ignored, cost of A/B testing.
You should probably be using a different tool if it makes your site slower.
>5) Speed >A/B testing slows down your organization's decision making. Some ideas are obviously good and adhering to a strict A/B testing process reduces your time to go live. Time is finite and the number of improvements you implement per year has a major impact on your growth trajectory.
It's impressive that growth is simply a factor of time within this company. This makes sense only if their average production decision making has a positive impact on growth/conversion rates as a factor of time.
>You can monitor your long term conversion trend rather easily via Google Analytics to gauge if you're making good decisions.
Yes, but that would be very unactionable. If the growth is going up, you pat yourself on the back and say "I'm clever" and if it isn't you say "I suck". But since you don't know which decisions or what the real reason is for conversion going up or down, you don't know what the fundamental metrics are that drive your growth up or down. You basically say "I don't care why things are going well" or even lie to yourself and say "Things are going well because I make good decisions."
Growth is not just about growing, but about knowing why you are growing and using that knowledge to find new ways to grow. If you know the fundamentals about what drives your growth, you also know which things you can or cannot change without affecting growth.