Ask HN: A/B/C testing - something bothers me about this one.

[tdenkinger]

Say you have a piece of digital content (a song or book), and you sell it on iTunes, Amazon and on your own website. You're interesting in finding out which of those sales outlets your customers prefer.

So, you devise a test: you'll create three splash pages, each has some marketing copy and a single link to one of those sales outlets. Randomly, one third of your visitors will see the iTunes link, one third the Amazon link and the remaining third will see a link to your own website's sale page. The visitor is identifiable, so the same link is presented on subsequent visits.

I don't think this actually tells you about what your customer's like. But I'm having a hard time saying why. If I'm presented with iTunes but I don't like iTunes and never click through, does that really say anything useful about whether I like the alternatives?

That we're even doing the test is a nice change of pace, but I hate the idea of doing a test and claiming the results mean something that they may not. I'm happy to hear that my misgivings are wrong.

[DanBC]

> I don't think this actually tells you about what your customer's like. But I'm having a hard time saying why. If I'm presented with iTunes but I don't like iTunes and never click through, does that really say anything useful about whether I like the alternatives?

You're performing the test on one large group of people, not on lots of individual people. (Does that make sense?)

It doesn't tell you what Bob as an individual likes, but it might tell you what a collection of people prefer.

It's quite important with A/B testing to set a sample size at the beginning of the experiment and stick to it. A statistician will be able to explain why.

Because you're not testing individual people it'll be tricky to disentangle the reasons for preference from the data.

[chris_dcosta]

The answer to your question is in the the question - which means you almost get it.

You are adding an additional variable: for example "liking iTunes or not" that can't be tested like this, so you are confusing yourself as to the meaning of the results.

You are also trying to imply meaning on the "not doing something" aspect of it "does that say anything about the alternatives". This can only be true if the alternatives are available at the point of selection. Therefore, you are right this says nothing about the alternatives.

If you really wanted to see which was the preference from all three then all three choices need to be available to choose equally. At the moment by only giving one albeit random choice you are not answering anything.

To understand this process better think of the classic "google blue" test, where the colour of the links in the search results was subtly changed for random users to find out which blue worked best.

[redguava]

I agree this won't show what your customers prefer, because you aren't giving your customers a choice. This might show you which of these your customers will buy from if they have no choice though.

If you want to see which they prefer, why don't you just have all three options on the page and see which one gets the most sales? Surely you can just see how much revenue is coming in from the different outlets?

It seems to me like this isn't a great use case for a/b testing. If you have all three options and one of them never gets used, stop showing that option.

[pixelcort → redguava]

If anything, you could A/B test the order in which the options are shown.

[markerdmann]

Isn't the conversion rate what you're interested in? This experiment will definitely tell you whether one link has a higher conversion rate.