
How Split-Testing Our Opt-In Form Increased Our Conversion Rate by 102.2% - tswicegood
http://diythemes.com/thesis/increase-conversions-split-testing/
======
wayneyeager
I'm wondering if merely displaying a number is really "social proof" in the
Cialdinian sense.

Possibly at one time, but now I think the "proof" part of social proof must be
demonstrably not BS for it to work. In the same way that testimonials by
"J.S., Ohio" have no real meaning anymore, but "Jared Sanderson, CEO of
Whatever.com" would probably be taken seriously.

An interesting experiment would be to get permission from some of your better-
known users to say something like "Join Bill Gates, Dalai Lama, and 14,152
others...". Possibly even better to use little avatars or pics of a bunch of
people, ala Facebook Like or the old BlogLog widget. Who knows? I feel like
that'd be a better thumbs up/down on social proof than just the number.

------
swalberg
I wish I knew more about statistics. Punching just the two variants that made
it to the end into a Chi square calculator gave me a 90% confidence, not the
95% he cited.

I also wonder about stopping the variation early, and even the conditions
under which he stopped the test. <http://www.evanmiller.org/how-not-to-run-an-
ab-test.html> talks about pitfalls when "testing until you reach confidence".

Are there any better references for these types of questions?

~~~
Datasta
I cited 95% confidence, which was calculated by Visual Website Optimizer. I
wonder how they calculate their confidence level.

~~~
bermanoid
Never used Visual Website Optimizer, but chances are, whatever they do leaves
you vulnerable to the flaw mentioned in the article that swalberg linked.

Basically, every time you peek at the results of a split test to see if the
confidence level is high enough, you make your split test more prone to error.

There are ways to adjust for this which allow you to continuously monitor the
test and stop if it hits a threshold, but it's much easier just to set a known
number of samples to take and accept the confidence level that comes out then.

~~~
Datasta
Actually, I had planned to test 2,000 views from the outset. It went a little
over because I didn't stop it fast enough. I do agree that you are vulnerable
to influencing results if you stop the test as soon as you hit your confidence
level

------
alain94040
Not sure if it's significant yet. What he really got was: 9 people click on
the default form, 5 people on variation one, and 14 people on variation two.
These are very low numbers no matter what way you look at them.

~~~
Datasta
Yes, those numbers are "lowish" but remember, it was only on 2,068 uniques. We
plan on running the test again soon with another 2,000 uniques to see if
timing had anything to do with it.

------
pilom
Not only is it a distraction. I see that and think "You are going to send me
email that will be sent to 15,000 other people? That would just be spam for
me! Why should I sign up if you are going to spam me?"

------
Godhammer
The devil's in the details, but the angels are there, too.

Amazing how much impact these "minor" changes can have on the bottom line.

Thanks for the reminder to split test all things great and small.

~~~
Datasta
It's funny how that works. I suspect social proof was a distraction on that
optin form.

------
michael_dorfman
I found it very amusing that after the long post demonstrating that the
"social proof" line was cutting his sign-ups in half, there was a box asking
me to sign up which included the social proof.

~~~
Datasta
That's because I haven't split-tested that form yet. I'm currently preparing
to run that one right now.

------
WillyF
It looks to me as though they should try the original headline without the
social proof. Since the Control beat Variation 1, but Variation 2 beat both,
that would seem to be a logical next step.

~~~
slimier
Agreed. I feel the majority of the science behind this is simply keep things
clean and minimal. I would hypothesise 'Get Email Updates' would do better
than 'Get Free Email Updates', too.

I don't believe a statistic offers any significant social proof. Lots of
faceless people i don't know have signed up. So what?

