
The what and why of product experimentation at Twitter - squarecog
https://blog.twitter.com/2015/the-what-and-why-of-product-experimentation-at-twitter-0
======
crabasa
This is an odd blog post for a company people consider to be utterly paralyzed
and incapable of shipping new features.

------
fideloper
The only thing Twitter is likely considering for new features is "Can we sell
ads on this?"

------
illicium
Sounds like analysis paralysis to me.

------
yabatopia
Apparently there's a framework for experimentation at Twitter, but it looks
like few experiments actually succeed to turn into new features. What goes
wrong there? Or is Twitter really a prime example of analysis paralysis, as
others suggest here?

~~~
squarecog
Hi, author of the post here.

A few brief points:

1\. The post talks about most experiments not moving core metrics in a large
way. This does not mean the experiments don't ship, it just means core metrics
are very hard to move. So "few experiments actually succeed to turn into new
features" is not entirely correct -- quite a few of them do ship because they
improve a particular feature, or because the team believes they are the right
thing to do as long as they don't do any harm, or even an acceptably small
amount of harm.

2\. If the majority of the experiments worked, we wouldn't need to experiment,
would we? :-).

3\. Core metrics being hard to move is not a Twitter-specific thing. If you
find them easy to (positively) move, you are either so tiny that any change is
a large change, or you are measuring something incorrectly. LinkedIn, Bing,
and Amazon experimenters contributed to a paper, linked from the post, which
states as "rule of thumb #2" that "changes rarely have a big positive result
on key metrics." Anecdotally, the same is true for Google, EBay, and
Pinterest.

4\. Analysis paralysis, according to wikipedia, is "an anti-pattern, the state
of over-analyzing (or over-thinking) a situation so that a decision or action
is never taken, in effect paralyzing the outcome". Deciding to not ship
something is not analysis paralysis. It's a valid outcome. If everything you
try works, you are probably not trying hard enough. Now, if you ran an
experiment and kept saying there is not enough data to make a decision, and
kept analyzing the data and collecting more inputs -- that would be analysis
paralysis.

------
wslh
Give me an API and I will move the world... I can't find API in that page.

------
meatysnapper
hahahaha oh wow. yeah, no.

EDIT: It is the game of thrones version of product development, except no
dwarf or sexy people.

