
Local maxima and the perils of data-driven design - andybak
http://www.90percentofeverything.com/2011/01/06/local-maxima-and-the-perils-of-data-driven-design/
======
jerf
There does not exist a single methodology that can always avoid local
minima/maxima, short of somehow brute forcing the entire design space. (If
evaluating the design space and locating the best design is somehow not an NP-
hard problem, it is close enough for human purposes.) Accusing one methodology
of producing local minima/maxima is therefore a noop. One must demonstrate
that it is somehow more _prone_ to such a result.

Good luck with that. I would despair of even getting the definitions correct,
let alone demonstrating a result with even minimal rigor.

My intuitive feel is that if you feel like testing is getting you stuck in a
local optima, this is more likely to be because your testing is actually
giving enough sense of how your design fares that you can actually feel you
are in a local optima, rather than the usual situation where you are just
clueless, happy about it, and not even in a local optima, let alone the global
one. You can't blame the equipment that allows you to detect that your are in
a local optimum for getting you stuck in it; that's textbook observation bias.

~~~
mbateman
There doesn't exist a single _data-driven_ methodology that can avoid local
maxima without brute force the entire design space. Conceptualizing things and
being creative is what avoids local maxima.

~~~
jerf
Nope. Being a human is not the magic algorithm for finding global optima. We
aren't anywhere near that smart. There's no methodology for finding the global
maxima, period.

~~~
andybak
No-one is talking about finding the global maxima. We are simply discussing
ways to avoid local maxima. "Better" not "Best"...

------
paraschopra
My counter-point (written earlier): In defense of A/B testing
[http://www.smashingmagazine.com/2010/08/26/in-defense-of-
a-b...](http://www.smashingmagazine.com/2010/08/26/in-defense-of-a-b-testing/)

~~~
harrybr
I don't think the facebook case studies in the article are _against_ AB
testing per se, but more against the fact that AB testing is often used to
focus on minutae. It seems to be a psychological temptation to use it to focus
on minutiae, rather than something that's inherent in the research method
itself.

