
How Much Optimization Is The Right Amount? - dennisgorelik
https://training.kalzumeus.com/newsletters/archive/premature_optimization_postmature_optimization
======
cperciva
Patrick: Sorry to hear you've been sick!

Everybody else: There's another reason to "not optimize yet" in many cases: If
your traffic doesn't support A/B testing. Taking the example of 100
visitors/day of whom 2.5/day sign up for a free trial -- you're going to have
anywhere from 0 to 5 signups/day on a regular basis. If could increase your
yield from 2.5% to 3.5%, you should be thrilled -- but an A/B test would take
over a month to distinguish that from a random fluctuation.

~~~
timr
That's true, but there's still a valid reason to test things on small traffic:
you don't necessarily need to have statistical certainty on every decision. A
lot of the time, it's sufficient to know that when comparing two dramatically
different options, you can't _quickly and easily distinguish between them_.
That's a strong hint that you should put your future efforts elsewhere. It
makes no sense to dedicate lots of effort to a project if the impact is
statistically indistinguishable from noise.

Moreover, the changes that matter early on are the 10x changes, and they'll
often be subjectively visible to you, even if the math shrugs and tells you
that it isn't confident. So while it probably isn't worth testing button copy
or images on 100 visitors a day, it might be worth testing complete redesigns,
or entirely different value propositions.

~~~
teleclimber
10x changes are hard to come by, even at an early stage. That's like going
from 3% to 30% conversion. If your site is mostly broken and one change takes
you to 30% conversion, then you probably didn't need an a-b test to figure it
out.

~~~
timr
More realistically, when you're an early stage startup, is that 10x means
going from 0.1% to 1% conversion. That kind of change can be achieved by
changing pricing models, changing product descriptions, changing the purchase
flow, etc. And you'll _notice it_ \-- 1% means getting one buyer about every
other day on 50 uniques; 0.1% means getting a buyer every three weeks. You
won't need statistical validation to tell you that things have changed.

But even "smaller" changes -- 10%, 20%, 30% -- are usually noticeable, even
with a tiny stream of traffic. The point is, the kinds of winners that you
need at an early stage are usually big enough to feel. Rarely do you get to
the win in single-percentage increments.

------
jmduke
As a point of context, I'm guessing this blog post was based a little bit on
the "20 days as a growth hacker" submission (and Patrick's subsequent
comment):
[https://news.ycombinator.com/item?id=6883357](https://news.ycombinator.com/item?id=6883357)

~~~
patio11
That did make me tie a string around my finger, but there's no particular line
in the article that you should read as me referring to him.

The editorial calendar basically dictates that I have to talk about A/B
testing for the time being (product launch and all), and I generally prefer to
say new things rather than saying old things. I've written an awful lot about
A/B testing from different angles, but have rarely said "Actually, hold up,
don't do it."

I previously had, if anything, a little too much "All I have is a
HammerFactory so all problems must implement INail or they will be recast
until they do" going on due to my own business history. I've talked to many
people who have that to varying degrees over the years.

So "Hmm, as I come to be a little more mature in my understanding of this,
there's actually some times where I wouldn't reach for scalable, metrics-
driven, quantitative conversion optimization as the first solution" seemed
like a pretty good skeleton of a piece.

------
dennisgorelik
Finally Patrick seriously addressed the topic of when A/B testing is NOT
needed.

------
ufmace
Good article. I especially liked the section on how to get your first user for
a more serious business app and how you should treat them. That fleshed out a
lot of what I was already thinking about how to go about launching the app
that I'm working on.

------
EGreg
Simple: optimize the amount of optimization, and you're good to go!

------
abyx
Lots of good points there, one of my favorites is the Collison Close: “Open up
your laptop. I'll get you an account and code the integration right now.”

------
mebreuer
Title reads: How to optimize your optimization.

Now what I'd really like to know is, how much optimization of my optimization
is the right amount?

Jokes asides, good read!

~~~
rcthompson
Premature optimization of your optimization is the root of the root of all
evil.

~~~
gk1
You're doing it all wrong! You're only optimizing for the local maxima of your
optimization's optimization!

------
pstuart
As someone whose goal it is to have viable startup next year, this is pure
gold. Thank you Patrick!

~~~
patio11
Thanks, glad to have helped.

------
exo_duz
As always your stuff is great to read and very insightful! Thanks again
Patrick!

------
entendre
relevant xkcd: [http://xkcd.com/1205/](http://xkcd.com/1205/)

