
Good Experiment, Bad Experiment - xtacy
https://www.reforge.com/blog/good-experiment-bad-experiment
======
mwexler
This is philosophically interesting, but little of this really is about
_running_ a good experiment. Perhaps it's about choosing which ones to run?

Kohavi's book will probably provide much more value than this kind of abstract
post. See [https://experimentguide.com/](https://experimentguide.com/) for
more details.

~~~
tantalor
This bit is crucial for running a good experiment:

> Good experiments use tight exposure groups

Not all your users will have the conditions necessary for the experimental
treatment.

You should only look at changes in behavior where the treatment condition was
true. Otherwise your experimental effect is diluted.

But your comparison between experiment and control groups must be neutral on
the condition, that is, users in the control group would have seen the
treatment if they were in the experiment group (counterfactual).

------
daveFNbuck
> Good experiments define success up-front.

This is absolutely critical. If you're not defining success up-front, you're
not running an experiment. You're just doing a staged rollout. You can use the
data to craft whatever story you want for most changes.

------
Nzen
tl;dr former Slack director of product outlines twelve principles for a/b type
experiments with features, products, whatever. (ex Good experiments involve
analyzing the results as opposed to 'ship/kill?') Each principle has a couple
of sentences of elaboration and calls out the opposite style to avoid.

