Maybe. Without outing myself, I have work on a choke point widget on the front page of a top 10 USA website. We had 30 engineers running these tests all day long. Was common to get 10 million+ increases to the bottom line on very minor changes. This was over a sample size of 100 million page views.
So a single data point? I didn't see it claimed that there is not a single small thing that can have a big effect. It's about large numbers. I think individual examples obfuscate more than they help us understand, when it is given in lieu of wider view instead of just as a support for one.
In this context, I would also like to see considerations of a larger picture. Sure, it helps any single company make more money. What about beyond that? Because money is a proximate goal, even if it's the ultimate one for a business, from a human society point of view that introduced this tool for a specific reason and not for its own sake.
For example, what does it mean to have some ultra-optimized large companies dominating their space? Would society really lose something without some micro-optimizations, even if they raise profit? The people who end up not making a purchase on a website because they are not as much caught in the optimized patterns, is it actually bad for them, or maybe they are actually better off and the optimizations tricked them into making an actually unfavorable (even if slightly) purchase? I think there is more to consider than just the view from the respective company or even the department. Are the means used to win actually good from a wider point of view that is not centered on that winner alone?
If someone does not make a purchase because something is slightly off, are they actually worse off, neutral or better in the end? It can't have been all that important to them to make that transaction, no?
Did you ever test doing nothing? I assume it's been studied, but I can imagine ways for 10% to actually be noise or seasonal/weekly/whatever variations.
