
Results from Candy Japan box design A/B test - NickSharp
https://www.candyjapan.com/results-from-box-design-ab-test
======
aresant
In a past life I founded/operated a conversion rate optimization company.

One of our all time most successful tests was related to this story and
happened almost by accident.

The client had just rolled out new packaging and wanted help validating the
change statistically, very similar to this situation.

Their metric was measuring upsells to a premium product after a purchase, and
like Candy Japan, found statistically insignificant results.

But the new packaging was beautiful and we wanted to test it on their landing
page.

We tried just replacing the old product image on landing page. Meh results, no
real bump.

We tried increasing the size of the image + circling with "Improved" (no
joke). 10% bump. Interesting.

We hired a professional model to hold the product and smile. 15% bump. !?

We hired a group of professional models to hold the product and smile. 40%
bump in sales. !@%K$!?

We re-hired the same group of people to hold product, look SUPER excited
(jumping up and down like their team just won the superbowl). 50%+ bump in
sales.

No joke.

If Candy Japan is reading and you'd like some help recreating I would not be
shocked to see something similarly group oriented, visceral, and beautiful
bump yeh.

~~~
eru
I wonder if you could have gotten a similar result from the same models
holding the old box?

~~~
frozenport
Alternatively, what if you already had the models and wanted to improve?

------
akx
How about a plain box in some relevant color (sakura blossom pink? :) ), hand-
stamped with the Candy Japan logo?

In my mind's eye, that'd remain classy enough not to be
creepy/cringy/embarrassing (as
[https://news.ycombinator.com/item?id=11641715](https://news.ycombinator.com/item?id=11641715)
complains), but would also have sort of a personal touch to it.

Having a couple rubber stamps made and stamping each box during packaging
wouldn't probably cost an awful lot...

~~~
bemmu
Seems like a good alternative. There has been a lot of valuable feedback in
these comments. I hadn't considered how these might be perceived at a
workplace, or that people even receive packages at work.

------
keiferski
Have you considered that the new box is too "generic" or "corporate?" (For
lack of a better word)

In other words, the old box, while plain, adds a certain intimacy. It doesn't
look like a huge multinational candy company sent it. The new box, though,
looks basically like any other piece of Japanese media.

~~~
bemmu
I can see how that might be the case. Current subscribers might know me
personally or through my blog. That would then mean that as I start to reach
out to people who don't know me personally, these results could change.

~~~
stordoff
As a current subscriber (who has been receiving the old packaging), the new
packaging is off-putting:

* It looks childish. If you are familiar with the Japanese style of advertising etc., probably not an issue, but as a box to be sent anywhere and to be received in a variety of settings (workplaces etc.) it seems like a poor choice.

* Anime may have poor connotations (unfair IMO) as pointed out elsewhere in the thread, which will reflect on the receiver.

* It feels corporate / like advertising. It may seem silly, but I'm more likely to continue subscribing to a small company if I'm on the fence about canceling.

* It feels like a waste of money. The current packaging is utilitarian, which signals that money is being spent only where it needs to be. The new packaging feels like an unnecessary indulgence, which would make me think I'm getting less value out of the subscription.

------
mrmondo
I much prefer the plain packaging because:

\- The anime picture is embarrassing to receive in the mail / at work.

\- The picture itself is kind of creepy, someone walking past my desk once
even described it as 'rapey'.

\- I can reuse the box.

\- I am not 6.

~~~
mc32
It depends on the audience.

Someone who admires Japanese culture will likely not have the same reaction as
someone with little exposure to Japanese culture.

The picture is not even on the same planet as 'rapey' Whoever that was either
said it to get a rise or they have some distinct issues.

One solution, perhaps, is avail both packaging options to buyers.

The typeface and font, I think, could be improved. It kind of looks like an
afterthought. I don't see "branding". It's not Lobster or Comic Sans... but...

~~~
morgante
> It depends on the audience.

One problem is that even if the person receiving the box is exposed to
Japanese culture, their coworkers probably aren't.

~~~
mc32
So I think the main issue is how the buyer might perceive the packaging. If a
coworker has issues... I think the coworker has issues. It's like someone
finding Barbie offensive (for other than body image issues). I can't see it.
Are these people who whisper and shuffle when they buy condoms or feminine
products at the checkout counter? Do they wrap their wine or beer in brown
bags? It's hard to imagine this person.

~~~
morgante
Humans are social animals. When I buy things, I absolutely consider how that
purchase will look to other people. it's one of the primary reasons why brands
have value.

You might not like that the world works that way, but as a business it's wise
to not deny reality.

~~~
mc32
Right, but this nettlesome coworker you imagine is, I can only imagine, the
lone outsider who can possibly find offense in the packaging. I think most
would be like, hey, what's that? Oh, candy from _Japan_? Can I try one> Wow,
Oishi!

~~~
morgante
I can guarantee you that nettlesome coworker is not alone in perceiving creepy
links between Japanese cartoon characters and hentai.

~~~
mc32
That's the kind of person you do not want to work with. This the conspiracy
theorist who misinterprets benign things and paints them in whatever twisted
way they want for whatever odd reason they have.

_That_ person is the problem in the office, not the person who bought sweets
online and probably shared them with some of the office.

If anything, the packaging helped reveal an oddball at the office.

------
xlance
I run a subscription business through mail myself, and I love your posts.

Have you tried to run introduction offers? 50% off for first shipment for
example? If I were you I would try to put the countdown for the next shipment
on top of your site. Thats a really powerful psychological factor.

Your CTA is also a little unclear, why do I need to first get an email, then a
link? The buy process should be straight forward, not through my email. And if
you have a really good reason for doing it this way, you should atleast tell
me why I need to leave my email.

I really like the cartoon that explains the process. Very good.

Regarding packaging design, we did a huge remake in 2013. We increased our
prices with 30%, and redesigned packaging completely. Doubled our LTV, insane
gains. Just keep testing, but try to change provider for carton. We pay $0,1
USD for bigger cartons then yours, in one of the most expensive countries in
the world. (With plastic window for address field) We chose to put our three
USP on the back of the shipment box, to remind your customers every time they
receive a shipment why they subscribed in the first place.

Keep testing!

~~~
Agustus
* Could you give me more information on how you sourced your cartons?

* What does LTV mean?

* What does USP mean?

~~~
NathanKP
LTV = Lifetime Value = How much money you make off your subscriber over the
lifetime of their subscription

USP = Unique Selling Point = Why your subscription gives the subscriber
something they can't get anywhere else

------
manav
I don't really think A/B testing something like this, with this size of a
sample, on a subscription based niche-product can give you any real insights.

Why not poll your subscribers or (assuming you can get the costs to work out)
have an option for anime art. Personally I wouldn't want the anime style box,
something custom and simpler would be fine.. but thats my personal opinion.

------
Orangeair
> In a world where it makes no difference which package is sent, you would get
> a result as significant as this 80% of the time.

I suddenly get p-values now. Why has no one ever explained it like this
before?

~~~
bemmu
Thank Marek Gibney of
[http://www.productchart.com](http://www.productchart.com), he contributed
that line.

------
bemmu
Just a small update, but I just ran a survey for customers to see which
package they would rather have.

Which package design would you rather receive?

The white one with illustration 79%

The plain dark one 21%

Total votes: 52

~~~
NathanKP
I didn't get in on the survey and got the old box in the A/B test, but I would
100% prefer the white one with the illustration for my subscription. I order
my candy to the office to share and people usually see the box from Japan and
are curious so they ask me about it and I tell them about the candy
subscription.

The white box would be better advertising for you in my case :)

------
zacwest
I like my subscription. My biggest issue has been shipments that end up in my
mailbox melted. It would probably not be possible for new packaging to fix
this, but this would be an area of improvement worth higher cost.

~~~
NKCSS
a light-colored Vs a dark colored box might help in that regard as well with
exposure to sunlight.

------
INTPenis
I think candyjapan should add a new form of subscription, one-time shipment,
and I believe they would see it being ordered by about 40% of their customers.
;)

------
tokensimian
One consideration is the user experience is not connectable to a purchase.

If I went to a friends house and saw this, I would never remember the name. If
you could combine "making the purchaser look cool" with timing-relevancy, I
think your packaging improvement would pay dividends.

Put another way, I believe your A/B ratio test is actually X*Y/Z where X/Z and
Y/Z are both < 1\. Synergy.

Terrible idea, (possibly) useful for analogy: Instead of optimizing the
unboxing experience alone, if you also included cards with QR codes, where
every QR code immediately triggers a free candy bar (with user data entry on a
mobile app). It should also provide social value to the host in realtime;
maybe they also receive an immediate notification that their guest scanned
their card plus some other bonus I can't think of.

The point is to get closer to the conversion/churn decision, or provide more
value to your consumer.

------
mistahdarcy
I signed up for CandyJapan several years ago when they were just getting
things started. I canceled after probably 3 shipments. Each shipment I was
only hoping that the next would contain better products and actually be worth
the value in which I was paying. Unfortunately that never happened. While a
cool concept and very eye catching, that's all it is. Importing candy is
expensive no matter how you go about it. Most people just want a poke to see
how it is, and then move on to something else.

~~~
bemmu
Thanks. Yes it costs much more to send candy in individual boxes from here, so
if you want more candy per dollar it is definitely better to visit a Japanese
market if there is one near you.

------
rajacombinator
Both packages look pretty ugly. Hire a better designer...

------
a_bonobo
Could the customs sticker have caused problems?

I used to receive Candy Japan in the "plain" box to Australia, and the customs
sticker declaring the value & contents took up most of the space on the box.

------
melvinmt
Just a FYI: The pictures stopped working here:
[https://www.candyjapan.com/mini-assort](https://www.candyjapan.com/mini-
assort)

~~~
bemmu
Had those pictures in the app engine blobstore, not sure what happened.

------
lukeholder
Why not allow the customer to choose the box type and default to the plain
box. Add $1 for the designer box, see what people choose?

------
Keyframe
Now you have to try out different designs to see if it matters (it does and it
can).

------
Medea
Just out of curiosity, which statistical test should be used in this case?

~~~
jupiter90000
Some decent answers on Quora:

[https://www.quora.com/What-statistical-test-should-be-
used-f...](https://www.quora.com/What-statistical-test-should-be-used-for-an-
A-B-test-on-conversion-rates)

~~~
gwern
Those answers are bizarre. Use a t-test or z-test?! That is both less
interpretable* and less efficient than doing a test of difference in
proportions (based on a chi-squared) or better yet, doing a generalized linear
regression with a binomial link which lets you test multiple interventions and
include covariates and other things. These are little harder than using a
t-test and work much better.

(Of course, even better would be a Bayesian decision analysis which takes into
account the full posterior distribution, costs, and benefits, allowing one to
decide whether the evidence is enough to justify the more expensive box or
whether one should extend the experiment further to collect more data.)

* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.

~~~
jupiter90000
To be fair, a couple answers there mentioned the Chi-squared approach, and a
z-test could also be appropriate for testing a difference in proportions. I
think your Bayesian approach would make the most sense for this kind of stuff,
to examine the posterior especially. There's other things that could be done
than what you mentioned too to assess a difference. I'm not convinced the
parametric assumptions made by the standard tests mentioned (including
difference in proportions, whether a z-test or Chi-square test) would hold up
well for this kind of data, it would take further examination. I think a
z-test or Chi-square test are a decent starting point, and could be fine to
use if the assumptions made for those tests are met (to defend some of the
Quora answers). The issue is often tests are used without folks either knowing
how to check or ignoring the underlying assumptions, but that isn't an issue
with the test itself I'd argue.

~~~
gwern
> I'm not convinced the parametric assumptions made by the standard tests
> mentioned (including difference in proportions, whether a z-test or Chi-
> square test) would hold up well for this kind of data, it would take further
> examination.

I'm not sure what you mean. If all you have is binary data, there's not much
you can do _but_ a chi-squared test or binomial regression. As I understand
it, a chi-squared is considered nonparametric:
[https://en.wikipedia.org/wiki/Chi-
squared_test](https://en.wikipedia.org/wiki/Chi-squared_test)
[https://en.wikipedia.org/wiki/Fisher's_exact_test](https://en.wikipedia.org/wiki/Fisher's_exact_test)

~~~
jupiter90000
Yes, you're right that chi-squared is non-parametric, my mistake. However,
even non-parametric tests can have assumptions that should probably be checked
into at least for a given data set. It may be robust to the violations the
data collected may exhibit, I'm not sure. This article lists the assumptions
for chi-squared test:
[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/)

------
RA_Fisher
That 30 day shipping period is a killer. Candy is an impulsive treat. :)

------
Aelinsaar
Honestly, I think it would have been really bizarre if something as unrelated
to the central product as the design on the box, changed something as
fundamental as retention of customers.

~~~
huddo121
That works under the assumption that humans are purely rational beings, which
we are not. Though I don't necessarily think the design chosen was a great
choice based on other comments here. Maybe something a bit more generic, yet
still colourful (like the candies in games like candy crush) would have been a
better choice.

~~~
Aelinsaar
Maybe, and if they were selling something for display, like figurines, or even
a quasi-collectible I would be right there with you. I know people who like
that kind of thing, and they would like a funky box too. This is JUST candy
though, it's just people buying something they want to eat. It's a different
mindset, and I think, less likely to be engaged by the aesthetics of the
packaging (as in the box). They're interested in the packaging of the actual
candy, and the candy itself.

All of this is "I think" and "IMO" of course.

