

A/B Testing to Increase User Engagement - trefn
http://blog.mixpanel.com/ab-testing-to-increase-user-engagement

======
btilly
The post has good questions for people to ask, but not specific advice about
how to analyze results and decide when you have a conclusion. So I added a
link to my tutorial in the comments section. ;-)

------
richcollins
You guys suggest using Mixpanel to measure user engagement via split tests,
but Mixpanel doesn't have a good way to split test!

I've resorted to hacking something together using properties. I use the
property name to represent a segment/test name and the values to represent the
splits. This leads to having a ton of properties, which the UI isn't designed
for.

ex:

    
    
      {"event":"signup","properties":{"techcrunchreaders/signupbutton":"blue"}}
      {"event":"signup","properties":{"techcrunchreaders/signupbutton":"red"}}

~~~
trefn
An A/B testing tool is on the roadmap, but we just haven't had time yet.

The best way to do tests right now is by utilizing funnel properties. You can
test the whole length of your signup funnel and just add a 'button-color'
property.

This lets you see the difference in conversion rates between your two
treatments pretty easily.

~~~
richcollins
I would also like to do this for one off events. The important thing for us is
the ability to view the event metric across property values (intersecting
segments):

{"google/organic":"widgets"}

vs

{"google/organic":"sprockets"}

------
mey
How do you stub out a community?

If the community is engaging, people will be more likely to comment (like HN),
but if you are a site, you really don't have two separate communities to swap
in/out in the user's mind. Additionally, even if you did optimize your site
design/functionality to increase comments, is that really desirable without an
engaging community in the first place? Would everything end up being nothing
that adds to the conversation? (see Youtube)

