
What is A/B Testing? - paraschopra
https://visualwebsiteoptimizer.com/what-is-ab-testing/
======
modernerd
What motivates web designers to tell stories by repurposing the scroll
mechanism instead of, say, using video? This type of "scrollmation" feels
incredibly clunky to me. I have never got to the end of one.

If you need that much animation on a page to tell your story, isn't it a good
sign that video might be more fit for purpose? Video was designed for
storytelling, it will work on most devices, and it has well-established UI and
conventions, such as a play button and progress bar.

If you're going to provide a play button in your scrollmation to automatically
advance the action anyway, you've already recognised how irritating it is to
manually page through an animated story by scrolling or swiping. So why then
go to the trouble of creating a scroll-based animation that repurposes a 20+
year old convention and attempts to reinvent a tiny proportion of what video
has to offer?

Is it just for the novelty value?

Is it because it makes you feel smarter?

How is it better for users trying to understand your story?

I'd love to hear from anyone who's considered both video and scrollmation and
chosen the latter.

~~~
huhtenberg
There's a good reason. Two of them, actually.

This particular animation in a video format would take 10 seconds of running
time. It would also take few seconds to load and initialize whatever video
playing widget it'd be using. That would make it for an oddly short video and
one of those "What? That's it?" situations. So it simply wouldn't work as the
_only_ content on the page. OTOH, if you present it as an interactive
animation, it will take users longer to work their way through it and so it
would _feel_ more substantial.

Secondary reason is that it makes the page more memorable and helps it stick
in more people's heads than some yet another embedded video. The goal of the
page is to engage people. This is _the_ reason why you'd want to have things
_presented_ rather than just thrown on a page in black and white Times New
Roman / 12px. You want your page to engage and clicking on Play Video is not
engagement, it's a reflex. So by forcing visitors to do something out of their
routine the page, effectively, makes them to pay more attention than usual.

~~~
mhb
Neither of those reasons is compelling. A short video is a feature and a long
video is a disincentive to viewing it.

Not everything that's memorable is good. I was happy to see the "play" button
alternative to the scrolling interaction. I used the play button, enjoyed the
video and believe I will remember it without experiencing the annoyance of
having to manually play it using the scroll bar.

------
petenixey
I love it that you've produced this and I think the simplicity of it is
wonderful but I fear the story could leave someone a bit confused.

It's confusing first of all because you've chosen a product which people don't
generally sell online - fish. If you'd chosen bikes or clothes it would have
made sense.

The other thing is on the "play" version - it takes about half the animation
before you even get to the online bit and when you do, the illustration of how
A/B testing actually works only takes up a tiny bit of real estate.

For my money I'm not convinced that the offline analogy really adds to it. You
know your market better than me (or anyone else here except Pete and Dan) but
I feel like you could jump straight in with a story about an online store.

"Mike sells bikes online but he's not sure which ones to put on the front
page. Will he get the most sales from showing someone bikes on the frontpage
or showing them chainsets? He doesn't know and he finds it hard to read his
analytics to figure out which one is best. VisualWebsiteOptimiser does all
this for him..."

~~~
speeder
Beside the stupidity of selling fish online...

You CAN do A/B testing of sorts on the physical store too. This might break
the attention of people that know that.

I guess the intention were good, and the drawings are of good quality, but
otherwise the execution is just terrible.

~~~
CGamesPlay
Agree with this. When I saw "no solution" it immediately removed the faith I
had in the site's ability to follow the scientific method.

------
mtrimpe
Really love the animation, but I can't help but wonder if A/B testing couldn't
be explained more powerfully.

Within the example of the store for example, the real pain A/B testing
addresses is cyclical variation leading to incorrect conclusions.

Working from that you could've shown the story of:

* the store owner rearranging his display and seeing sales go up

* but then the rest of the year sales actually going _way_ down

* and when he changes things back to the _old_ layout sales went _waaay_ up again! How can that be?

* Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all!

* Now ... on his website he can send visitors to different versions at the _same_ time!

* So he'll never have a year of bad sales again!

~~~
siddharthdeswal
Those are good points, but we were gunning for the marketer who isn't familiar
with A/B testing. This because we see that a lot of our potential customers
have usually heard the term somewhere but need a basic, easy-to-understand
explanation.

Do you think it achieves that purpose?

~~~
e12e
Perhaps you should deploy an alternate version, and meassure conversion rates?
;-)

Seriously, I think you make a great point of selling your service. I also
think you gloss over things like cyclic variations -- but I'm not sure it's
your job to educate people in the subtle art of fact oriented market research
(or what one should call it).

For those that don't have the time to learn, changing it probably wouldn't
matter. For those that do -- chances are they probably will find it out from a
more comprehensive resource (like, say, a good book).

~~~
siddharthdeswal
Exactly. For someone who's just starting with A/B testing, this serves as a
good/interesting introduction. But for someone who really wants to understand
the detailed behind-the-scenes, this will obviously fall short.

Coming to your suggestion, do you have a good book/website in mind?

------
ad93611
The animation is very good. Could you explain a bit about how this was done?
Did you have to write the code or is there a tool that can generate a page
like this?

~~~
_kushagra
Thanks! I used skrollr.js [1] to create the parallax animation and some JS for
scroll handling. I will write a blog post [2] on how I made it in detail soon.

[1]
[https://github.com/Prinzhorn/skrollr](https://github.com/Prinzhorn/skrollr)
[2] [http://kushagra.me/blog/](http://kushagra.me/blog/)

~~~
nbouscal
One problem for you to be aware of: something about the scroll handling breaks
the two-finger swipe Back functionality on my Macbook. It's a minor thing, but
very annoying.

------
huhtenberg
Stars of David coming out of the "A/B-Testing" explosion - I bet it's some
sort of clever innuendo, but it just seems to escape me :)

~~~
goatcurious
Certainly not a clever innuendo. Looks more like an oversight. There isn't
much awareness about Star of David in India.

It is the easiest star to sketch -- even my teacher in elementary school used
to use the same star in my grading.

~~~
paraschopra
Hmm. We weren't (and aren't) aware of Star of David.

~~~
winter_blue
FYI it's the flag of Israel...

------
zachlatta
Very neat. Hate the loading screen though. Having to display a loading screen
is almost as bad as using flash. Maybe load the assets in the background?

~~~
paraschopra
Actually, we load first 4 screens at first go. Rest is lazily loaded.

------
rajington
Beyond the comments over the presentation style, the metaphor, and browser
compatibility I want to see more about your specific app. I like how you did
the VWO interface on top of his website, you could easily change the line
graph section to show the "Intuitive Reports" feature. Line graphs are easy
enough to understand so by including more of your value add features it will
still stay simple enough.

I know it isn't specifically A/B testing related, but throwing another
shoutout to your app at the end like "heatmaps too!" might be nice. It seems
people that would look at this slideshow might not be familiar with any web
analytics services, and it seems you guys offer many services.

------
paulodeon
Scrolls Down...Meet Bob...Goodbye Bob...Clicks back

------
IanCal
When say "Bob knows which version brings him more sales", I'm curious about
how you extract the relevant information from someone to know if it's a
statistically significant change in a user friendly way. Do you find people
understand this well?

What about multi-variate testing? How do you explain the increase in the
number of visitors needed to make a decision?

Finally, how do you reconcile giving up-to-date information without
compromising on the results? (keeping an experiment going until it's
statistically significant is something I can see users doing)

~~~
siddharthdeswal
The tool has that built-in. As soon as significance is achieved, we show a
"winner", so to answer your question, instead talking about statistical
significance we focus on the more positive aspect of an A/B test.

Those not initiated in Statistics find this a little difficult to understand.
I personally go about it like this:

Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with
certainty that the coin is loaded?" Them: "Umm, no not really"

Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah
maybe"

Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded"

And that's how I explain the basic concept of statistical significance, but
without the math. More data means it becomes more 'obvious'.

To take it a notch further, I ask "If your life depended on it, would you
still bet that the coin is loaded?" People usually become unsure after this
and I joke that their previous decision was based on 95% significance, but if
they really want to be sure, they should wait for 99% significance.

~~~
Ovid
_As soon as significance is achieved, we show a "winner"_

If you've described that accurately, you're hurting some of your customers.
It's not well-known (though it should be obvious, in retrospect), but a test
can produce a false positive when, in fact, it's a negative.

The problem is that there is always noise in customer data and sometimes
you'll randomly have enough conversions such that one of your tests will show
a winner when it's not.

Matts Einersen has some simulation code to show this:
[http://www.einarsen.no/is-your-ab-testing-effort-just-
chasin...](http://www.einarsen.no/is-your-ab-testing-effort-just-chasing-
statistical-ghosts/)

Evan Miller has a more in-depth (but harder to follow) discussion here:
[http://www.evanmiller.org/how-not-to-run-an-ab-
test.html](http://www.evanmiller.org/how-not-to-run-an-ab-test.html)

If, in fact, you show the winner as soon as significance is achieved, it's
then trivial to demonstrate that some of your customers will lose money as a
result. However, if you take this knowledge and then find a _good_ way to
incorporate this into your tool, you may have another great selling point that
competitors do not (that being said, the idea is complicated enough that some
customers won't understand it).

~~~
siddharthdeswal
Trying to do exactly that at [http://visualwebsiteoptimizer.com/ab-split-test-
duration/](http://visualwebsiteoptimizer.com/ab-split-test-duration/) and
[http://visualwebsiteoptimizer.com/ab-split-significance-
calc...](http://visualwebsiteoptimizer.com/ab-split-significance-calculator/)
but yeah, we need a better way to present this.

~~~
Ovid
Have you considered adding a "recommend days" field to your product? Given the
number of people per experiment and the number of conversions, you could
potentially calculate a minimum time to run the experiment and simply explain
that sometimes you are more likely to get false positives prior to the minimum
time?

Also, it makes me very happy to know that you're aware of this problem and are
trying to tackle it. It makes me more likely to recommend it to clients.

------
nixarn
A bit off topic. But I've had problems with the visualwebsiteoptimizer, not as
a customer but as a user on websites using their system. At two different
occasions websites have had problems to render, no matter how many times I try
to re-load them and wait. I've then checked the webdevtools to check what's
wrong and it's been the websiteoptimizer's script that's been blocking the
whole site.

~~~
paraschopra
I'm sorry to hear your feedback. Was it a recent problem? Did you contact our
support? In fact, we're the only A/B testing tool out there with asynchronous
code snippet that NEVER blocks the website. Here are more details:
[http://visualwebsiteoptimizer.com/split-testing-
blog/asynchr...](http://visualwebsiteoptimizer.com/split-testing-
blog/asynchronous-code/)

~~~
nixarn
It happend a while back, maybe 3-4 months ago. I didn't contact support. If if
happens again on some site, I'll do it. But the script was blocking, it was
waiting for the js file to load, and blocking the site while doing so.

------
mute
It's similar to animation at
[http://icelandingpagedesign.com](http://icelandingpagedesign.com)

~~~
siddharthdeswal
That's a beautiful landing page. Assuming you work there, how much time did it
take to build?

~~~
mute
Around 6 months. It's inhouse project and I spent as much time it was required
to make it good. No idea how many actual hours has been spent.

------
philfrasty
I personally find it „too much effort“ to scroll down all the way. Nice story,
but make it somehow shorter and choose a storyline that is familiar to people
so they only have to concentrate on the A/B-aspect because otherwise the story
is distracting. Would be great to get some feedback how the site is doing
(compared to previous versions).

~~~
paraschopra
We do have statistics that 60% of people scroll till the last slide.

------
retube
IE8: "There is a problem with this website's security certificate".

Also doesn't render at all well...

~~~
orik
>IE8: "There is a problem"

couldn't have said it better myself

~~~
_kushagra
IE8 is a problem.

------
alexcroox
Nice work! We did something on a smaller scale at the bottom of our homepage
(with live stats on users choices)

[http://sidigital.co/](http://sidigital.co/)

(you'll have to wait for the liquid to reach there to see the effect of your
choice)

~~~
BuddhaSource
Very neat. It kept me glued till the end.

------
TamDenholm
I hate to be "that guy" but you've got a spelling mistake. "Fishes sell
volumes". The plural of fish, is fish. "Fishes" would be the possessive.

~~~
siddharthdeswal
Multiple species of fish are usually called "fishes". We used this explanation
to verify [http://grammarist.com/usage/fish-
fishes/](http://grammarist.com/usage/fish-fishes/)

Not sure if it is correct though. Do you have a link to a better explanation?

~~~
ahoy
This is correct. You'll also notice the word 'peoples' used when referring to
multiple groups of people.

~~~
mortov
Such as in The Peoples Front Of Judea and The Judean Peoples Front

------
zekenie
Are there jewish stars that appear when AB testing is introduced? No issue
with this, it just seems a little odd. It reminds me of Hannuka at my parents
house! :)

------
dbg31415
Sorry, why does this warrant an animated deck? Sheesh, next up, "How to use
gravity to fall over."

------
alxbrun
Just when you thought Powerpoint was finally dead at last, it comes back
disguised in scrolling.. Damned !

------
fny
While I appreciate the responsiveness, the page is obnoxious to navigate on a
mobile device.

------
Illniyar
Great, another example of a site that is vendor locked to the Chrome browser.
Considering your site doesn't work on the majority of browsers (Neither
Firefox nor IE), I'm not sure what "testing" you've actually done.

~~~
ankneo
Working fine on Firefox 22

~~~
Illniyar
Strange. I was working with Firefox 21, and it didn't work. I've updated to 22
and it did.

~~~
nish1500
Works fine on my 21

~~~
e12e
Works fine on iceweasel 21 (Debian ff rebrand).

------
gordondevoe
This was awesome. Great work!

------
mindslight
What is A/B testing? Local optimization with an assumption that every user is
stateless and identical.

------
sanju
awesome animation :) love this way of explanation compare to a video cast

------
mkhalil
Thanks for this.

