

How two magical words increased signups by 28% - spif
http://visualwebsiteoptimizer.com/split-testing-blog/ab-test-case-study-how-two-magical-words-increased-conversion-rate-by-28/

======
patio11
One of the things on my list is to try calls to action which are not Sign Up,
precisely because folks think that that involves either making a commitment or
paying money. (I get at least a half dozen emails a month asking me to cancel
a free trial, from folks who grew up in a generation where anyone giving you
something free is going to send you a bill 30 days later.)

~~~
paraschopra
I'm curious. Do you have any suggestions for the alternatives?

~~~
patio11
"Sign Up" solves a problem for the site owner. Write something which solves a
problem _for the customer_.

I'm going to try things like, e.g., Start Making Bingo Cards.

I actually had a few dozen buttons made in a particular design to test
different calls to action against each other earlier this year, but the design
itself flopped -- hugely -- against my pre-existing buttons, so much so that I
didn't think color or copy changes on it would recover the loss. Curses on
having my testing bandwidth limited by graphical expertise...

~~~
paraschopra
Though actual result will depend on A/B test, my experience suggests that this
could be a double-edged sword. The call to action on my homepage said 'Start
Optimizing Your Website Now' and some of the savvy users complained they
couldn't find how to sign up. Some users simply wanted to sign up and my call
to action looked like they had to go through additional pages before they get
what they were looking for.

~~~
eagleal
You can use a different version for people who uses Chrome/Iron (I think most
chrome users are tech savvy). For other browsers you can use some other filter
mechanism (like referrer).

~~~
vdm
If Chrome users are like me, a lot of Chrome users were set up by other Chrome
users who fixed their computer, deleted the IE shortcuts from Desktop and
Quick Launch bars, and renamed the Chrome shortcuts to "Internet".

~~~
eagleal
I said "I think" and "most". However you can a history miner (CSS) to refine
your demographics.

------
ajg1977
"There is no reason why 'It’s free' should work better than 'Sign up for
free'"

Actually I disagree. The phrase "sign up for free" contains some hint of a
potential "but", similar to "free to join" or "download trial for free", or
other carefully worded phrases that litter anything-but-free internet offers.

"it's free" is far more emphatic and open.

~~~
nreece
Maybe. But "Free Sign Up" sounds/works just fine.

Having said that, a hint of a potential "but" (i.e. trial-ware) is good at
times so that the users don't take the free part for granted and they are
aware of an upgrade option.

------
jacquesm
hey Spif,

Effectiveness of this kind of test should not just be done at the spot where
you make the change but end-to-end.

It's very well possible that by placing the 'it's free!' there the signups
shift to a much larger number of people that will never convert to paying
members (for a freemium site).

If you only test at the point of making the change it is very well possible to
actually increase free sign-ups but to _decrease_ your conversion rate to
paying members.

So, this test is about an intermediary goal, the real conversion is the rate
between new visitors and paying members.

I sure hope that your overall conversion rate to paying members shows a
similar increase, but make sure that really is the case.

~~~
paraschopra
Actually, spif mentions the following in the post:

>To be honest we aren’t sure yet [of what impact this makes to the actual
signups]. We wonder how these tests measure up to the goal funnels in Google
Analytics and compared to actual conversions into paying customers

~~~
jacquesm
Ah, cool, I missed that. Thanks. It really is the crux though, any kind of
partial measurement is essentially meaningless without the overall picture.

~~~
spif
This went live before we launched our premium account so don't have those
stats to track.

We measure 3 related goals in GA, the homepage -> signup form; homepage ->
signed up user; and new visit -> paying user (using the goals but also
e-commerce values).

Looking back and learning from this exercise, in the next tests that we're
doing we'll be adding different UTM campaigns to the different variations (in
VWO) and be able to track conversions to paying customers further down the
track. I cannot stress enough how much valuable information this gives.

In any case we CAN see the increase in homepage -> signup form:
<http://cl.ly/503bf3943cc390e4f715>

This did increase the actual signup conversion as well (homepage -> signed up
user): <http://cl.ly/e6403e90307c58c8a6e7>

~~~
paraschopra
Fantastic info. Should have included in the case study ;)

~~~
spif
Only just looked up those numbers as per Jacques question. ;-)

------
spif
Although the two words weren't revolutionary we were quite surprised by the
increase. Mind you we had about 6 other variations as well, including "It's
free up to 250 contacts" and an orange version of the "Signup now!" button -
both we thought would have higher conversion.

~~~
jacquesm
I'm very curious about your end-to-end conversion rates, from raw unique
visitors to paying customers.

~~~
spif
Right now it's about 1% of our raw unique daily visitors.

The tricky thing is to drill down and see which users these are and where are
they coming from and how are they using Soocial. Any tips are welcome! ;-)

~~~
jacquesm
One thing I've found that can make a big difference is to adapt your landing
page to the language of your visitors.

~~~
spif
Yeah, true. We are looking into l10n and I18n. Right now just under 50% of our
users are from the US. There is a seeming correlation between high conversion
rates and english speaking countries and this would confirm your point.

In order of highest conversion rates are: New Zealand (1.16%), United States
(0.98%), Brazil, Australia, Ireland , UK (all around (0.79%), Singapore, South
Africa (around 0.6%) then it levels out.

------
bdickason
Just wanted to point out that it seems like you guys are following the
feedback in this thread: <http://news.ycombinator.com/item?id=1351460>

Which is totally rad :)

Now if only you could modify the video on your site or post a blog post to
explain... how does my site get loaded into your 'visual editor?' Does all my
css stay there? etc. The video makes it look like the site has magically been
loaded into the editor and everything's intact :P

~~~
paraschopra
Yes, it is really magical.

Go, try the live demo
[http://dev.visualwebsiteoptimizer.com/first_time_ab.php?demo...](http://dev.visualwebsiteoptimizer.com/first_time_ab.php?demo=1)

Do you think we should blog about the fact that you don't need to do any thing
special to load a site into our system?

~~~
bdickason
To me, this is the #1. I would go so far as to place that 'first_time_ab.php'
box directly at the top of the page. Kind of an 'enter your webpage and get
started NOW' type call to action.

After seeing this link, it's totally clear to me and i will definitely sign up
when we're ready.

~~~
NEPatriot
I agree strongly. If I can paste my url into a demo site, and and get a taste
for how easy this would be for me to a/b test my next question is what do I
have to do to make this happen? A blurb telling me all it takes is copying and
pasting one line of code takes care of that. Maybe show a video showing how
easy it is - how long would it take? under a minute?

~~~
paraschopra
Yup, it will take under a minute. Should do an end-to-end screencast.

------
timcederman
The importance of calling out that something is free cannot be understated for
conversion.

------
almost
EDIT: Just confirm, I think I was incorrect in the message below.

I'm not completely sure I'm correct (please correct me if I'm not!) here but
as I understand it the article does not support the claim in the headline. The
headline claims that signups increases by 28% in the changed version and that
this was _all attributable to the change_.

It's the second bit that isn't supported, they say that the result was
statistical significant but what I understand them as saying was that it was
statistically significant that the new variation was better than the control.
But it could be better by an amount more or less than 28%, all we know from
that is it's almost certainly (95%) at least a little better. We would need to
know the number of trails to be able to get a certainty for the amount of
improvement.

Could someone with a slightly better understanding of statistics chip in
maybe? I could use some more information in my own A/B tests, sometimes I know
a change is going to be a pain to maintain so I want to know not just if it is
better, but by how much.

~~~
paraschopra
Since I am the one who wrote the case study, perhaps I can explain. Loosely
saying, what the article actually means is that the observed difference of 28%
or more may actually be true in 95% of cases. However, there is a 5% chance
that that difference is a lesser than 28%.

If you have any specific question, please feel free to ask.

~~~
almost
I must be have been wrong then, my apologies.

How are you using to arrive at that conclusion though? When I do a A/B testing
I (or rather, the software I use) use the chi-squared test to give a
confidence value expressed as a percentage. When I get over 95% I know I have
statistical significance and I end the test. At that point I also have an
"improved by" number but I would have to collect much more data for that to be
statistically significant as far as I understand.

~~~
RiderOfGiraffes
I'm not an expert in statistics, but I'm careful about what I do, and I know
the limitations of my knowledge.

However ...

I believe it is flawed methodolgy to run a test until you get a significant
result. I'm pretty sure I read something lunk to from HN that discussed this
at some length. It's possible - if you simply run your test until you get
significance and then stop early - that you will get significance because of
random fluctuations in the middle of your trial and stop early when you
shouldn't.

As I recall, you should decide on the length of your trial at the beginning,
then run your stats at the end.

I'll try to find the article in question, but my Google-Fu is pretty poor
today for some reason.

EDIT: It's here:

<http://news.ycombinator.com/item?id=1277004>

------
whyenot
The results could really benefit from some inferential statistics. The
percentages, while interesting aren't particularly meaningful on their own. I
realize you may not want to share how large your sample sizes are, but
something like a Pearson's chi square test would be very useful here.

------
siong1987
I read somewhere saying that "<strike>2.99</strike> Free!" works better that
just saying that "Free!" because it's hard for users to understand the real
value of the latter.

~~~
pmjordan
At least in Europe, I'm fairly sure you'd run afoul of unfair competition laws
with that - you're not allowed to run "infinite sales". If you show a non-sale
price, you _must_ have charged that much prior to the sale, and your sale must
have an end date.

~~~
pclark
how would they ever enforce/police that?

~~~
pmjordan
Not how. Who: competitors. And yes, this sort of stuff is enforced in
practice. Initially you might only get a cease & desist notice, but you'll
probably be reported to the authorities if you keep doing it, or you're doing
it especially egregiously.

------
joshu
Can we not talk about percent changes of percentages?

~~~
pdx
I agree. When somebody uses this sort of second order percentage, it is
confusing.

When a car's top speed increases from 105mph to 120mph, after you modified the
engine, you say it increased 15mph ... or, perhaps you could say it increased
14.3%.

Using this paradigm, when a conversion rate increases from 14.5% to 18.6%, you
would say it increased by 4.1% ... or, perhaps you could say it increased
28.28%

Woops! We have a problem. We just expressed the same increase, in the same
units, using two very different numbers.

This is obviously a problem. The reader doesn't know what you're talking
about.

It's perfectly valid to say the car increased by 15mph, so it must be
perfectly valid to say the conversion rate increased by 4.1%. In fact, I think
that when most people read something like this, this is what they assume, that
the delta between the old conversion and the new conversion is that
percentage. At first blush, saying the conversion rate increased by 28.28%
should be OK, but it's not OK, because the whole point is to convey
information, and you have failed to do so. Now you have to use context, to
figure out what the hell they meant. This is unacceptable, especially when
talking about mathematics, we should not need context to sort out what's going
on.

What's the answer? I don't know. But there's definitely a problem here.

~~~
paraschopra
>Using this paradigm, when a conversion rate increases from 14.5% to 18.6%,
you would say it increased by 4.1%

No you cannot say it increased by 4.1% and that's because you are talking
about per cent. I don't know why you would ever use 4.1% in this context. It
is simply wrong.

May be you can say 4.1 percentage points but I don't see a need for it when
you can say (and be right) that it increased by 28%.

~~~
pdx
I believe your logic is backwards.

To be sure, the fact that we are talking about percent changes something,
which is what I spent several paragraphs exploring.

To say that subtraction of percentages is against the rules, (while perfectly
valid and often used in other units) but percentages of percentages is not
against the rules, requires a bit more than "It is simply wrong", don't you
think?

Sure, you're "right", when you say it increased by 28%. The point is, perhaps
you shouldn't be "right", and what is "right", anyway, if nobody knows what
the hell you're really saying?

We see this when talking about tax changes. If somebody wants to minimize the
sticker shock of a tax increase, they use the delta. So, if the rate rises
from 3% to 4.5%, they say they increased the rate 1.5%. If you want to
maximize the sticker shock, you say that the tax rate was increased 50%.
Either side can say they are doing the math "right".

Those who care about actually conveying the information, rather than produce
sticker shock of either sort, always have to spell it out carefully, by
actually saying, "the tax rose from a rate of 3% to a rate of 4.5%" This is a
bit longer, but it's necessary, given this weird unit.

------
einarvollset
I think you'll find it is three words.

------
Charuru
Maybe <http://visualwebsiteoptimizer.com/> should take a look at the button on
their own homepage.

Edit: It doesn't work.

~~~
chaosmachine
You're right, clicking "sign up" does nothing in Firefox or Safari.

That can't be good for conversions.

~~~
paraschopra
Oh, no. Mistake! Thanks, fixed it.

