

How to measure the cost of an A/B test? - mistercheese

At my past few companies, there has been heavy investment into live A/B tests on the customer population. However, for bigger companies that have more exposure, I have a worry that we may not be able to have the proper ways to measure the real cost of an A/B test that may 1) Have negative PR or branding effects 2) Conflict with the objectives of other features.&#60;p&#62;Sure we can measure customer drop off or engagement, but given the number of tests and projects at any given time, there are too many confounding factors to isolate those measures to a particular test. I'd love to hear from Hackernews suggestions! Thanks!
======
darkxanthos
For #1 if customers are assigned to groups that get a certain set of variants
for each test and you have enough groups to run the full combination of tests
that are active, you should be able to correlate drop off with certain tests.

For #2: That's much harder. How do you handle this for larger products that
you don't A/B test? I'd bet this isn't a unique issue.

~~~
mistercheese
For #1, the issue is actually that even the awareness of Test A might effect
Test B. I'm curious if there's anyway to capture that then.

------
karolisd
How do you measure PR and branding effects?

If you aren't controlling for variables, it isn't very scientific. It's
possible to run multiple tests at the same time and test different things, but
you need to be able to isolate what you're measuring and be able to measure
something that's statistically significant.

~~~
mistercheese
Pretty much every test we run has the potential to effect user engagement, so
I would suspect unless isolated it's a lost cause. What's more interesting to
me is how to measure perhaps how bad branding has lost consumer confidence and
the effect on sales thereafter.

It's easy to measure drop off in a funnel, but harder to measure customers who
refuse to even enter the funnel due to bad brand. Is there some way of
capturing that?

