

Mixpanel (YC S09) releases macro-level A/B testing tool - trefn
http://blog.mixpanel.com/releasing-mixpanel-tests-a-way-to-ab-test-you

======
patio11
I love you guys, but there is something to be said for using words like other
people in the industry use the words.

The first time I noticed this was wrapping my head around funnels, which is a
subject I'm fairly deeply experienced with. The three magic words in that
documentation are funnel, step, and goal. In every other analytics tool I've
ever encountered, the word "goal" is synonymous with conversion: somebody went
through the funnel and reached a result which was positive for the business.
In Mixpanel, goal does not mean goal. Goal apparently means "step name": for
example, the first goal of the seven step funnel that represents the core
interaction for my website is the Dashboard goal. The second goal is the new
card goal. The third goal is... you get the general picture. This breaks my
brain every time I directly touch the API or docs. I wrapped it in my own API
and try not to think about it too much.

That is the prelude to me saying: your A/B tests are something potentially
wonderful, but _they're not A/B tests._ You're going to suffer an impedance
mismatch when explaining them to anyone who knows what they're doing in
testing because when you say the words "A/B test" we're going to think of
something else.

An A/B test involves:

1) A user interacting with an element which is subject to the test. An element
can be almost anything: a design feature, a bit of copy, an entire checkout
workflow, whatever.

2) The user being randomly assigned into one of two groups and shown the
appropriate variation on the element.

3) The user does some stuff.

4) Hopefully, the user converts.

5) You compare the propensity for conversion among the populations of users in
each test group to see which performed better.

The solution you have delivered is different than A/B tests in many ways:

1) It does not randomly partition users into groups. That is, presumably,
something your customers have to do prior to firing an event with the property
(or super-property -- I forget what you call it) testWhatever: "alternativeA".

2) A/B tests are cool not because A and B are cool letters but because they
make statistical significance testing really easy, so that if alternative A
has 20% more conversions than alternative B you can quickly tell whether that
is solid evidence that A is indeed better or merely a possible artifact of
random variation. Your testing feature does not implement significance
testing.

3) There would normally be some halfhearted bow in the general direction of
independence at this point (we mostly gloss over that and hope people forget
stats 101 since, in practice, just wishing this requirement away tends to
actually work acceptably) but the entire point of your stats feature is, as
far as I can tell, taking the independence notion and beating it black or blue
or purple and then seeing how that choice of color interacts with six other
things.

Again: I like the feature. I think that given an hour or two I will probably
even be able to do something with it that will eventually make me money. But
if you try to communicate this as A/B testing you're going to cause a lot of
unnecessary confusion.

------
mogston
I've bee tracking their progress recently - This is just the start. Good to
see that Posterous are already using their solution.

------
ALee
this was holy crap awesome, presented another perspective to us at Jamlegend.

------
jfong
MP is the way to go!

------
adora
nice I like it

------
vinhboy
I am gonna get neg to hell for saying this, but MP is garbage. You can't even
delete a funnel. Seriously. Try it.

~~~
adriand
Really? Garbage?

Other hackers: I was about to install Google Website Optimizer for
multivariate testing on a startup I've been working on. Should I try Mixpanel
instead? Or does Google still have the edge?

~~~
vinhboy
Ok let me explain myself. MP is very easy and reliable to use, but the problem
is when you get into their dashboard. It's hard to analyze your data since
there are no reporting tools. In Analytics you can create really detailed
reports to segment your data and measure against certain metrics, in MP you
just get basically a click logger. Also my gripe with the funnel setup is that
you can create an unlimited amount of funnels, but you cannot delete them. You
can only view your funnel by week-duration and no other time frame. There are
also no built-in baseline metrics. If you want to track user's browser, OS, or
whatever, you have to log those yourself. Some might consider that a feature,
I didn't enjoy it.

~~~
ApolloRising
Sounds like you are a better fit for Google Analytics and Optimizer.

