

Ask HN: What is the best way to track user fidelity? - marcelocamanho

Hi.<p>I develop a price comparison website, and we track user conversions (pay by click model).<p>What is the best way you guys suggest to track user fidelity? That is, if a given feature is making users use the website more? This could be interesting if used together with A&#x2F;B testing.<p>Any ideas? Don&#x27;t want to use any invasive user tracking stuff though.
======
patio11
Cohort testing. For users signing up between T+0 and T+2 weeks (or whatever),
you randomly assign them to N test groups. Each of the N groups has some
combinations of features turned on or off, controlled via feature flags. You
collect aggregate statistics over time of the conversions in each test group.
Presumably, if you want to track outgoing clicks, you have the page intercept
clicks and fire a Javascript event with an anonymous ID number which your
backend will score as a click for the appropriate test group. The technical
implementation of this will take you between 2 and 5 days or less.

The math gets hairy and you need a metric truckload of time/traffic to do
anything useful with this, especially as you get to many types of conversions
of interest or many different variables under consideration, as the nature of
statistics means you'll always be able to find _some_ correlation if you look
in enough places. "Look doing Signup Form B and disabling private messaging
decreases friend requests by 18% in month 3 with 99.999% probability!" [+]

\+ This is not actually what the math is telling you.

~~~
marcelocamanho
I will look into it, thanks!

