
A/B testing original design vs. random template bought from ThemeForest - Sujan
https://www.candyjapan.com/behind-the-scenes/ab-testing-landing-page-template
======
no_gravity

        During this time there were a total of 67 new
        subscriptions. Of these 58% (39) came from the
        new design and 42% (28) came from the old design.
        Looks like the new one is a clear winner.
    

Is it? This seems a small population to settle on a clear winner.

Using R's prop test, I get a p value of 0.22. (Type "prop.test(39,67)" to
calculate it).

I think this means that in a world where it makes no difference which design
is used, you would get a result as significant as this 22% of the time.

An alternative is the Adjusted Wald method. You can try it online here:

[https://measuringu.com/wald/](https://measuringu.com/wald/)

Which gives some confidence intervals which also range from "could be better"
to "could be worse". Even when you reduce the confidence level from the
typical 95% to 90%.

    
    
        a quick check with an A/B testing calculator
        even says that this result has significance
        (~90% likely)
    

Which calculator was that?

~~~
bemmu
I used [http://www.abtestcalculator.com/](http://www.abtestcalculator.com/)
and entered 3000 participants -> 28 conversions and 3000 participants -> 39
conversions.

I neglected to record how many views each version had, but should be at least
3000 since the conversion ratio is about 0.5 - 1%.

~~~
no_gravity
The statement I see on that page is "There is a 91% chance that Variation A
has a higher conversion rate".

I am not sure how to interpret that. We would have to dive into the GitHub
repo and figure out which test it performs I guess.

~~~
popinman322
It looks like the difference of two beta distributions based on the
visualization.

So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions
the difference between the two distributions is greater than zero 91% of the
time. It's only guaranteed to be above zero at about the 80% credible
interval, and since we started with an uninformed prior that'd be about p=.2?

I'm open to correction here.

------
nls
It's definitely not just the design itself. The CTA (your email submit) of the
new landing page is better positioned throughout. You have also added urgency
("time until next shipment", "subscribe now before time runs out"). You can
also see up to 3 previous candy boxes now, whereas before, you'd have to click
through. All of those contribute to the conversion rate of your new landing
page.

Your old landing page actually contains some great elements that are missing
from the new one, like the reviews, the explanation anime video and the "new
tastes only available in Japan", which seems a great feature to me. I'd
definitely try and add those back in, and split test that new version against
the current winner.

~~~
ericcholis
This. Design != Content. Often times, simplified CTA and Value Proposition can
have the same effect as a full blown redesign.

------
iamben
Both pages would probably convert better if there was an explanation as to
what happens when I put in my email address. There's also a confusing use over
the word 'mailbox' \- which I'd use for email, but also physical mail. The
page asks for an email, immediately after saying "candy surprises in your
mailbox" \- if you assume no previous knowledge of what this site is going to
offer, it's pretty confusing. And if it's confusing, you're less likely to get
people to sign up.

Without having a proper look at and understanding the site, I'm not sure
whether that email is a leadgen form, or start of the sign up process, but
either way you could probably increase conversion with some better explanatory
copy.

Interesting read though! Thanks for sharing!

~~~
bemmu
I've always struggled to find good wording to separate physical mailboxes from
electronic ones. Maybe "shipped to you twice a month" would be less vague than
"in your mailbox twice a month"?

~~~
dom96
Perhaps "delivered to you twice a month" would be even better

~~~
lunchables
Seems ambiguous - delivered where? My e-mail?

------
tylercubell
I think the actual product images are more tangible to the customer compared
to the illustrations. My gut feeling is that the new design and repositioning
of the elements on the page don't make a huge difference aside from drawing
attention to the product images.

------
Theodores
I don't think this is necessarily the right conclusion.

Imagine a friend with a cafe, they get the opportunity to get another cafe,
same area, same clientele. With the new cafe they decide to just get some
franchise, a Subway type of thing. People are familiar with it and flock there
in relative droves. The original 'ma and pa' cafe is therefore deemed to be
not as good as a 'template'.

The other option to the franchise could have been to have gone with the 'ma
and pa' offering, same deal.

But that is not all of the choices. You could actually create something new,
not hold with what you have or chuck it in and go with the 'franchise'.

With website design it is very easy for people to chuck it in and go with a
Themeforest effort, Shopify, Squarespace and the like.

But a lot of 'new ingredients' have came along with CSS Grid, semantic markup
and much else that the themes, 'serviced websites' and the like are not up to
speed with yet and show no signs of wanting to implement.

It is also possible to build out more than just a landing page from scratch in
two weeks with the new tools. This involves learning instead of botching
someone else's floats, divs and margin hacks. We have Cargo Cult programming
and in my opinion this story is just another example of this.

I would recommend anyone else in this position to start content first, not
buying a theme and then taking the photos needed to push in the theme because
the boxes are there already. This is a backwards, 'design led' process and
there are plenty of good reasons to go content first, then structure it
properly, spend a day or two with CSS grid, then put the existing branding on
there. A static HTML page is a good start, a working prototype instead of a
PDF mockup.

Then, instead of the A/B testing, ask an honest person who knows a thing or
two and won't bullshit you. They will ask 'why have you done that' questions
which will get the content in shape. Thereafter, once you have a go at doing
it you can spot what you like elsewhere and learn from it rather than cargo-
cult-copy-paste it to never be confident of anything.

~~~
fbemployee1234
How many successful businesses have you launched doing things “the right way”?

The point is there are many paths to success. Maybe candy japan didn’t do
things the way you think they should be done (it’s frankly rude to refer to
template hacking as cargo cult programming; we’re talking design here, not
programming). But despite their “primitive approach”, they’ve been successful
in multiple ways.

OP has taught me quite a bit with their experience, whereas your negative tone
has taught me nothing except you believe you know how things should be done.

Honestly, you sound like a know it all.

~~~
Theodores
I get your point, but have you ever found a theme that hasn't came along with
an extraordinary amount of bloat, technical debt and featuritis?

Have you ever worked for an agency where a 'theme from Theme Forest' was
deemed professionally acceptable?

The fundamentals have changed for web development but nobody gets off the
hamster wheel to learn the fundamentals. The web doesn't have to be an
obstacle course of hacks, polyfills, libraries, frameworks and other crutches
any more. It also doesn't have to be full of fancy build tools, CSS compilers
and other things that make web development accessible for the first time in 20
years.

I preferred to do my own homework when I was at school, not copy what everyone
else thought the answers were, changing details around.

There is an intimidating thing going on with web development and the
increasing specialisation. No one can hope of being able to create a web page
due to this atmosphere. But so many people - a decade or two ago - got started
writing actual HTML, not assuming it is all too hard and you have to just hack
someone else's work. The industry is lacking these people now and is becoming
less diverse.

Frontend development isn't a creative medium if people are just using
frameworks from yesteryear and taking on technical debt from themes whilst
busying themselves with the latest buzzword bingo. Things like content,
accessibility and document structure matter, it is no good just going for a
visual design and working backwards to the inevitable 'div soup'. Something
has to change.

~~~
fbemployee1234
You make a principled point that I respect. I’m generally trying to get shit
done. They aren’t always at odds with each other, but often are.

Personally, I hate web design and layout, so I want to do as little of it as
possible. But I know I need a good looking site to be taken seriously.
Templates have helped me out and generally last long enough that I can afford
to hire a “real” designer.

Also, templates are $50 for a beautiful one. There are 2-3 OOM difference in
cost.

------
ajharrison
Willing to bet that with some improvements to the original design the results
would've been even better.

For example:

– Your navigation at the top is illegible. Theres very little contrast and
it's hard to read. Make that more clear, higher contrast.

\- Your main CTA ("Japanese candy surprises in your mailbox twice a month."
[Email]) is also hard to read because of the orange and the custom font. Swap
out the orange for a more vibrant color (maybe the orange color of the button)
and use a nicer font that's easier to read.

\- MAKE THE FONT SIZE BIGGER

\- Add in some of the elements you added in the new design: When the next box
is going out, etc.)

Your original design had an air of authenticity and honesty to it. The bought
template seems like a copycat.

Keep (and improve!) the original!

------
chiefalchemist
Small sample size aside, they used the wrong KPI (imho). This needs to be
tested over time. That is, which one retains better? Focusing on conversion's
is a fool's errand. That is, you convert better but then if churn is higher
you could end up at net loss. That's not ideal, obviously.

I think also, you'd have to look at referrals (if you have them). Perhaps a
lower subscription rate actually led to more sales because (for some reason)
those subscribers like to tell their friends.

The analysis here is too shallow.

------
cphoover
I always love this person's posts. Great informative and fun reads... Keep it
up.

------
gketuma
Japanese websites amaze me. The colors and density always seem so extreme.

------
huangc10
Can you A/B test how many of your blog readers prefer 0 memes, 1-2 memes, 5-10
memes, and 10+ memes? After the first few I stopped reading...

------
pabl0rg
Are you using woocommerce? Care to share how you pinned users to either the
ild or new template?

------
arthurofbabylon
The author lost credibility by describing statistical significance as
unecessary for making informed decisions.

In this context, that’s just not correct.

~~~
gwern
No, he's correct. This is a decision theory problem. Controlling false
positives or false negatives at some arbitrary significance level like 0.05 is
completely and utterly irrelevant. All of the debates above about which
proportion test to use are irrelevant because none of them answer the question
about which _decision_ (keep current template or revert) maximizes _utility_
(revenue). The hypothetical long-run rejection rate of a particular test in a
world in which there is no difference between templates (which is a priori
known to be false), has little or nothing to do with that.

Here, since he has no strong prior probability of the new version being bad,
the switching costs are already sunk, and the evidence increases the posterior
probability of the new version being better, and experimentation involves a
high opportunity cost while being unlikely to yield a result indicating the
new version is noticeably worse, keeping the new template probably is the
correct decision.

