

Ask HN: A/B/C testing - something bothers me about this one. - tdenkinger

Say you have a piece of digital content (a song or book), and you sell it on iTunes, Amazon and on your own website.  You're interesting in finding out which of those sales outlets your customers prefer.<p>So, you devise a test: you'll create three splash pages, each has some marketing copy and a single link to one of those sales outlets.  Randomly, one third of your visitors will see the iTunes link, one third the Amazon link and the remaining third will see a link to your own website's sale page.  The visitor is identifiable, so the same link is presented on subsequent visits.<p>I don't think this actually tells you about what your customer's like.  But I'm having a hard time saying why. If I'm presented with iTunes but I don't like iTunes and never click through, does that really say anything useful about whether I like the alternatives?<p>That we're even doing the test is a nice change of pace, but I hate the idea of doing a test and claiming the results mean something that they may not. I'm happy to hear that my misgivings are wrong.
======
DanBC
> _I don't think this actually tells you about what your customer's like. But
> I'm having a hard time saying why. If I'm presented with iTunes but I don't
> like iTunes and never click through, does that really say anything useful
> about whether I like the alternatives?_

You're performing the test on one large group of people, not on lots of
individual people. (Does that make sense?)

It doesn't tell you what Bob as an individual likes, but it might tell you
what a collection of people prefer.

It's quite important with A/B testing to set a sample size at the beginning of
the experiment and stick to it. A statistician will be able to explain why.

Because you're not testing individual people it'll be tricky to disentangle
the reasons for preference from the data.

------
chris_dcosta
The answer to your question is in the the question - which means you almost
get it.

You are adding an additional variable: for example "liking iTunes or not" that
can't be tested like this, so you are confusing yourself as to the meaning of
the results.

You are also trying to imply meaning on the "not doing something" aspect of it
"does that say anything about the alternatives". This can only be true if the
alternatives are available at the point of selection. Therefore, you are right
this says nothing about the alternatives.

If you really wanted to see which was the preference from all three then all
three choices need to be available to choose equally. At the moment by only
giving one albeit random choice you are not answering anything.

To understand this process better think of the classic "google blue" test,
where the colour of the links in the search results was subtly changed for
random users to find out which blue worked best.

------
redguava
I agree this won't show what your customers prefer, because you aren't giving
your customers a choice. This might show you which of these your customers
will buy from if they have no choice though.

If you want to see which they prefer, why don't you just have all three
options on the page and see which one gets the most sales? Surely you can just
see how much revenue is coming in from the different outlets?

It seems to me like this isn't a great use case for a/b testing. If you have
all three options and one of them never gets used, stop showing that option.

~~~
pixelcort
If anything, you could A/B test the order in which the options are shown.

------
markerdmann
Isn't the conversion rate what you're interested in? This experiment will
definitely tell you whether one link has a higher conversion rate.

