Hacker News new | past | comments | ask | show | jobs | submit login
What is A/B Testing? (visualwebsiteoptimizer.com)
220 points by paraschopra on July 5, 2013 | hide | past | favorite | 109 comments

What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video? This type of "scrollmation" feels incredibly clunky to me. I have never got to the end of one.

If you need that much animation on a page to tell your story, isn't it a good sign that video might be more fit for purpose? Video was designed for storytelling, it will work on most devices, and it has well-established UI and conventions, such as a play button and progress bar.

If you're going to provide a play button in your scrollmation to automatically advance the action anyway, you've already recognised how irritating it is to manually page through an animated story by scrolling or swiping. So why then go to the trouble of creating a scroll-based animation that repurposes a 20+ year old convention and attempts to reinvent a tiny proportion of what video has to offer?

Is it just for the novelty value?

Is it because it makes you feel smarter?

How is it better for users trying to understand your story?

I'd love to hear from anyone who's considered both video and scrollmation and chosen the latter.

There's a good reason. Two of them, actually.

This particular animation in a video format would take 10 seconds of running time. It would also take few seconds to load and initialize whatever video playing widget it'd be using. That would make it for an oddly short video and one of those "What? That's it?" situations. So it simply wouldn't work as the only content on the page. OTOH, if you present it as an interactive animation, it will take users longer to work their way through it and so it would feel more substantial.

Secondary reason is that it makes the page more memorable and helps it stick in more people's heads than some yet another embedded video. The goal of the page is to engage people. This is the reason why you'd want to have things presented rather than just thrown on a page in black and white Times New Roman / 12px. You want your page to engage and clicking on Play Video is not engagement, it's a reflex. So by forcing visitors to do something out of their routine the page, effectively, makes them to pay more attention than usual.

I've consulted for a video production company and seen them do these, there's no way this would be a 10 sec video.

Try speaking it clearly and slowly as a voiceover artist would as you go through the slides. 'Meet Bob. Bob is a fisherman. He runs his own store and has an online store [boom, there's your 10 secs]'.

Memorable isn't good if they don't get to the sales ptch, which most people wouldn't.

Personally I completely agree with modernerd, this would have been a waste of time and money if it wasn't novel enough to get featured on HN or /r/programming or stuff for a few extra sales. For its stated purpose, it's useless.

Neither of those reasons is compelling. A short video is a feature and a long video is a disincentive to viewing it.

Not everything that's memorable is good. I was happy to see the "play" button alternative to the scrolling interaction. I used the play button, enjoyed the video and believe I will remember it without experiencing the annoyance of having to manually play it using the scroll bar.

I've closed websites with nothing but a video on the landing page because I didn't want to wait for a video to load. This landing page, on the other hand, is the best product presentation I've ever seen. It's simple and clear and doesn't have boring narration to waste my time.

Fair enough, but I closed that page after 3 clicks. For me it was wasting my time.

Not saying I'm right and you are wrong, just that people have different irritants, as it were.

You might not be the target audience for this page. I think this is more about educating people who don't know what A/B testing is. (edit: if you are on HN, you probably don't need something this juvenile to explain the concept)

A lot of people will need something thats this simple to digest on their own speed, though. I will be sending this link out to a few past and potential future SMB clients.

The amount of people like "Bob" who are running their own website is astounding. They need simple things like this that help introduce them to the idea of A/B testing.

One would hope that at some point previous to this they had a video on a separate version of the page and did a bit of A/B testing themselves.

Glad you liked it :)

Does anyone else just want a written description of things? Make it look pretty. Create beautiful pictures/characters throughout the page. I don't care. I just want a good explanation of what a service offers.

- It's user-friendly.

- Compatible with everything.

- I can read as much or as little as I want.

- If I want to re-read something, I know exactly how to do that and it doesn't involve clicking a play head trying to find the place it was in.

I feel like people keep trying to bring back reincarnations of flash interactivity and since it uses html5 and other buzzwords, it's ok. Well, it's not.

Every time someone "fixes" the experience of what should just be reading online, they make it much more cumbersome and annoying. One day everyone will look back at what people did with html5 and laugh the way we all look back and laugh at flash intros and playing music on load.

Video forces me to go at someone else's pace -- the creator or expected 'average viewer' -- which is often too slow.

It also makes skipping or rewinding a matter of finely-navigated clicks into tiny target areas, whereas the best hardware scrolling controls offer giant, forgiving action-areas yet that still enable nearly pixel-level resolution and fine steps of speed/acceleration.

Depending on your device, scrolling in many directions might be very, very easy. Large trackpads, the Apple 'Magic Mouse', and most touch screens can make scrolling far more easy, adaptive, and intuitive than even 'forward' and 'back' buttons.

I tend to think people most peeved by scrolling as a major navigational technique (in both space and time) are still using tiny trackpads, mice with nubby scroll wheels, or even (shudder) ThinkPad-like joysticks or keyboard arrows. Stone knives and bearskins!

Does this apply to you?

I honestly never watch demo videos.

They go too slow, they're usually painfully cheesy, and I hate that I have to listen to them. I read much faster than the narrator talks.

A video for this demo probably would have lasted 1:30 or so. I was able to scroll through in about fifteen seconds, didn't have to find headphones to do so, and got all of the same information.

I think scrolling is better in the way that it does not aquire audio and user can controll the speed of information. This part is boring and I get it, lets skip to the next part. Video has no way for that.

> I think scrolling is better in the way that it does not aquire audio

That's a good point. There is a general expectation that playing a video also means playing audio, and some environments – like offices – aren't great for that.

> ...user can control the speed of information. ... Video has no way for that.

I frequently skim through video by dragging the head in the progress bar, or skip three or four minutes ahead in longer videos. With scrollmation, the default scroll speed is normally much slower than the equivalent video speed of 30fps, it results in jerky movement, and the speed of the video is directly proportional to how hard I am prepared to work to make it go faster. Good UX designers don't make users do shit work.

+1. I think it is definitely valuable to allow the user to control the speed of information.

I'd usually agree with the scrolling mechanism comment, but this time it's very well done. The initial page clearly explains that you can scroll down for more content, or click play, or click a button to get to the next page. The story and graphics are fun and keeps me interested.

What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video?

I can only speak for myself, but I rarely, if ever, watch website videos, especially at work.

I made it through this entire thing by scrolling.

While I understand video is sometimes the best way to convey an idea, I fricken hate it. It would need to be something very, very intriguing for me to watch. 99% of websites selling things don't fall into that category.

Strongly agree. It's getting more and more irritating having to watch videos to get information instead of being allowed to read at my own pace.

Maybe there should be an A/B testing version. One with video and one with animation :-)

Personally, I liked the animation.

I'd say it is mostly novelty. Video could do the same job, maybe even slightly better. We have considered adapting it into a video as well, but then everyone does that. Second important factor after novelty was integration of signup form and other interactive elements within the story. With video, that's not possible.

Perhaps you should do a split test.

I've seen great results from the signup forms that http://wistia.com allows you to embed anywhere in a video. I'm not affiliated, but am impressed with the product; check out their "Call to action" and "Turnstile email capture" features in the embed builder on this page: http://wistia.com/product/superembeds

I've made this sort of stuff before at request, and I can't stand it either. It quickly degrades the one thing that users can expect to do; control the vertical position of the page.

With stuff like this I always refer to how my mom uses computers. (Note - This a person who thought when you copied text on her iPhone that the text magically went into her finger, only to have to click paste with another finger so the text didn't get lost in her finger...)

I agree. There is no way she would know to scroll properly in this interface and in most cases would get confused and give up. The objective of the presentation is to make it easy for someone (like my mom) to understand A/B testing. Make it a video, or at least provide a link to a video version.

EDIT: This akin to Amazon's usability - http://www.nngroup.com/articles/amazon-no-e-commerce-role-mo... i.e. once some behavior is establish on your site, keep it that way, but doesn't mean you need to emulate it.

Lots of personal preferences in this thread. As an A/B testing company - they should have A/B tested a scrolling site against video and then been transparent about their results.

I'm not a "UI" guy, but I loved it. For me, it feels like a presentation, but the transition from slide to slide make it feel like a video. My 20 cents.

A back/forward button would be enough for me. The scrollimations always work confusingly on platforms with smooth, momentum based scrolling (like OSX).

These are questions worth A/B testing though. I'm on team video, but if anyone has a link to the research, let me know.

> What motivates web designers to tell stories by repurposing the scroll mechanism instead of, say, using video?

Control for the users.

If they had used video, they would have lost me in 5 seconds.

Also this scrollmation pages load very slowly.

I love it that you've produced this and I think the simplicity of it is wonderful but I fear the story could leave someone a bit confused.

It's confusing first of all because you've chosen a product which people don't generally sell online - fish. If you'd chosen bikes or clothes it would have made sense.

The other thing is on the "play" version - it takes about half the animation before you even get to the online bit and when you do, the illustration of how A/B testing actually works only takes up a tiny bit of real estate.

For my money I'm not convinced that the offline analogy really adds to it. You know your market better than me (or anyone else here except Pete and Dan) but I feel like you could jump straight in with a story about an online store.

"Mike sells bikes online but he's not sure which ones to put on the front page. Will he get the most sales from showing someone bikes on the frontpage or showing them chainsets? He doesn't know and he finds it hard to read his analytics to figure out which one is best. VisualWebsiteOptimiser does all this for him..."

Thanks for the feedback. Yep, we had debated whether fish is the right product to be sold online. We concluded that probably the choice of what product is sold is not as important as demonstrating the actual confusions during the sales process (both online and offline). We considered selling books, cakes and some other products but with fish, the options were limited and very clearly distinct.

Beside the stupidity of selling fish online...

You CAN do A/B testing of sorts on the physical store too. This might break the attention of people that know that.

I guess the intention were good, and the drawings are of good quality, but otherwise the execution is just terrible.

Agree with this. When I saw "no solution" it immediately removed the faith I had in the site's ability to follow the scientific method.


First off, I was distracted by that slideshow because I kept thinking, "Who the f sells fishes online?"

Second, I felt like there were two points in the video because of the whole offline analogy. 1) Selling online is better than offline 2) A/B testing makes you more money

Apparently you can buy fish online.


Really love the animation, but I can't help but wonder if A/B testing couldn't be explained more powerfully.

Within the example of the store for example, the real pain A/B testing addresses is cyclical variation leading to incorrect conclusions.

Working from that you could've shown the story of:

* the store owner rearranging his display and seeing sales go up

* but then the rest of the year sales actually going way down

* and when he changes things back to the old layout sales went waaay up again! How can that be?

* Cue image of cyclical charts and the facepalm of realizing it wasn't his redesign at all!

* Now ... on his website he can send visitors to different versions at the same time!

* So he'll never have a year of bad sales again!

Those are good points, but we were gunning for the marketer who isn't familiar with A/B testing. This because we see that a lot of our potential customers have usually heard the term somewhere but need a basic, easy-to-understand explanation.

Do you think it achieves that purpose?

Yeah, you could interpret my feedback as saying that you're addressing the what but not the why.

Looking at it as sales/marketing copy though I'd say that your biggest competitor by far isn't people not understanding what it is but people thinking "Why don't I just change my site and see what happens?"

In that context I'd guess this story telling format is perfect for taking a concept like 'prevent acceptance of false hypotheses by controlling for external variation through randomized trials' and turning it into a visceral "Oh fuck. That's painful!" moment your target market can relate to.

Yes, we indeed missed explaining why A/B simultaneously is how it should be done v/s A after B. Thanks for the inputs.

Perhaps you should deploy an alternate version, and meassure conversion rates? ;-)

Seriously, I think you make a great point of selling your service. I also think you gloss over things like cyclic variations -- but I'm not sure it's your job to educate people in the subtle art of fact oriented market research (or what one should call it).

For those that don't have the time to learn, changing it probably wouldn't matter. For those that do -- chances are they probably will find it out from a more comprehensive resource (like, say, a good book).

Exactly. For someone who's just starting with A/B testing, this serves as a good/interesting introduction. But for someone who really wants to understand the detailed behind-the-scenes, this will obviously fall short.

Coming to your suggestion, do you have a good book/website in mind?

I'm afraid I don't have any books to recommend -- maybe someone else has?

> (like, say, a good book).

Can you suggest one?

Yes I agree that it feels that the animation is not powerful enough.

The animation is very good. Could you explain a bit about how this was done? Did you have to write the code or is there a tool that can generate a page like this?

Thanks! I used skrollr.js [1] to create the parallax animation and some JS for scroll handling. I will write a blog post [2] on how I made it in detail soon.

[1] https://github.com/Prinzhorn/skrollr [2] http://kushagra.me/blog/

One problem for you to be aware of: something about the scroll handling breaks the two-finger swipe Back functionality on my Macbook. It's a minor thing, but very annoying.

Really nice work Kushagra - I'm looking forward to your blog post.

Great job Kushagra!

Heya, this was done by one of our talented interns Kushagra A. (https://twitter.com/kushsolitary). We did not use any parallax page making tools, it was written using HTML5, CSS3 and some js library.

Stars of David coming out of the "A/B-Testing" explosion - I bet it's some sort of clever innuendo, but it just seems to escape me :)

Certainly not a clever innuendo. Looks more like an oversight. There isn't much awareness about Star of David in India.

It is the easiest star to sketch -- even my teacher in elementary school used to use the same star in my grading.

Hmm. We weren't (and aren't) aware of Star of David.

FYI it's the flag of Israel...

Likewise, I found it distracting and off-putting.

I'm well aware of it and didn't even notice it the first time. Why would it be off-putting, though?

Maybe the shop owner sells... gefilte fish. :)

Very neat. Hate the loading screen though. Having to display a loading screen is almost as bad as using flash. Maybe load the assets in the background?

Actually, we load first 4 screens at first go. Rest is lazily loaded.

Beyond the comments over the presentation style, the metaphor, and browser compatibility I want to see more about your specific app. I like how you did the VWO interface on top of his website, you could easily change the line graph section to show the "Intuitive Reports" feature. Line graphs are easy enough to understand so by including more of your value add features it will still stay simple enough.

I know it isn't specifically A/B testing related, but throwing another shoutout to your app at the end like "heatmaps too!" might be nice. It seems people that would look at this slideshow might not be familiar with any web analytics services, and it seems you guys offer many services.

Scrolls Down...Meet Bob...Goodbye Bob...Clicks back

When say "Bob knows which version brings him more sales", I'm curious about how you extract the relevant information from someone to know if it's a statistically significant change in a user friendly way. Do you find people understand this well?

What about multi-variate testing? How do you explain the increase in the number of visitors needed to make a decision?

Finally, how do you reconcile giving up-to-date information without compromising on the results? (keeping an experiment going until it's statistically significant is something I can see users doing)

The tool has that built-in. As soon as significance is achieved, we show a "winner", so to answer your question, instead talking about statistical significance we focus on the more positive aspect of an A/B test.

Those not initiated in Statistics find this a little difficult to understand. I personally go about it like this:

Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with certainty that the coin is loaded?" Them: "Umm, no not really"

Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah maybe"

Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded"

And that's how I explain the basic concept of statistical significance, but without the math. More data means it becomes more 'obvious'.

To take it a notch further, I ask "If your life depended on it, would you still bet that the coin is loaded?" People usually become unsure after this and I joke that their previous decision was based on 95% significance, but if they really want to be sure, they should wait for 99% significance.

As soon as significance is achieved, we show a "winner"

If you've described that accurately, you're hurting some of your customers. It's not well-known (though it should be obvious, in retrospect), but a test can produce a false positive when, in fact, it's a negative.

The problem is that there is always noise in customer data and sometimes you'll randomly have enough conversions such that one of your tests will show a winner when it's not.

Matts Einersen has some simulation code to show this: http://www.einarsen.no/is-your-ab-testing-effort-just-chasin...

Evan Miller has a more in-depth (but harder to follow) discussion here: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

If, in fact, you show the winner as soon as significance is achieved, it's then trivial to demonstrate that some of your customers will lose money as a result. However, if you take this knowledge and then find a good way to incorporate this into your tool, you may have another great selling point that competitors do not (that being said, the idea is complicated enough that some customers won't understand it).

Trying to do exactly that at http://visualwebsiteoptimizer.com/ab-split-test-duration/ and http://visualwebsiteoptimizer.com/ab-split-significance-calc... but yeah, we need a better way to present this.

Have you considered adding a "recommend days" field to your product? Given the number of people per experiment and the number of conversions, you could potentially calculate a minimum time to run the experiment and simply explain that sometimes you are more likely to get false positives prior to the minimum time?

Also, it makes me very happy to know that you're aware of this problem and are trying to tackle it. It makes me more likely to recommend it to clients.

Nice. I'm impressed with your pages telling people how long to run their experiments for as well. I think early stopping can be a problem, but there's a difficult gap there. Good experiments are difficult to set up, and may be too much for a site owner who just wants to know if he should make the damn button green or red. The more you force someone to set up a complex trial, the more they'll want to go to your competitors who give them a nice flashy message saying "GREEN BUTTON WINS BY 1.4%" without all the hassle.

It looks like you've got a good balance there :)

A bit off topic. But I've had problems with the visualwebsiteoptimizer, not as a customer but as a user on websites using their system. At two different occasions websites have had problems to render, no matter how many times I try to re-load them and wait. I've then checked the webdevtools to check what's wrong and it's been the websiteoptimizer's script that's been blocking the whole site.

I'm sorry to hear your feedback. Was it a recent problem? Did you contact our support? In fact, we're the only A/B testing tool out there with asynchronous code snippet that NEVER blocks the website. Here are more details: http://visualwebsiteoptimizer.com/split-testing-blog/asynchr...

It happend a while back, maybe 3-4 months ago. I didn't contact support. If if happens again on some site, I'll do it. But the script was blocking, it was waiting for the js file to load, and blocking the site while doing so.

It's similar to animation at http://icelandingpagedesign.com

That's a beautiful landing page. Assuming you work there, how much time did it take to build?

Around 6 months. It's inhouse project and I spent as much time it was required to make it good. No idea how many actual hours has been spent.

Yeah..this is just more with images.

I personally find it „too much effort“ to scroll down all the way. Nice story, but make it somehow shorter and choose a storyline that is familiar to people so they only have to concentrate on the A/B-aspect because otherwise the story is distracting. Would be great to get some feedback how the site is doing (compared to previous versions).

We do have statistics that 60% of people scroll till the last slide.


Thanks for the feedback. I'm the CEO and I'm sorry to hear your feedback. Could you please detail on your feedback? It will help us a lot.

I think you should re-read what he said.

IE8: "There is a problem with this website's security certificate".

Also doesn't render at all well...

>IE8: "There is a problem"

couldn't have said it better myself

IE8 is a problem.

Nice work! We did something on a smaller scale at the bottom of our homepage (with live stats on users choices)


(you'll have to wait for the liquid to reach there to see the effect of your choice)

Very neat. It kept me glued till the end.

checked this a while ago. Clean work!

I hate to be "that guy" but you've got a spelling mistake. "Fishes sell volumes". The plural of fish, is fish. "Fishes" would be the possessive.

Multiple species of fish are usually called "fishes". We used this explanation to verify http://grammarist.com/usage/fish-fishes/

Not sure if it is correct though. Do you have a link to a better explanation?

Well consider me corrected. I wasn't aware of that. Learn something new every day.

Delivery I think was spelled wrong in one of the slides. I can't be bothered to go back and verify though, but may want to check it out.

This is correct. You'll also notice the word 'peoples' used when referring to multiple groups of people.

Such as in The Peoples Front Of Judea and The Judean Peoples Front

Are there jewish stars that appear when AB testing is introduced? No issue with this, it just seems a little odd. It reminds me of Hannuka at my parents house! :)

Sorry, why does this warrant an animated deck? Sheesh, next up, "How to use gravity to fall over."

Just when you thought Powerpoint was finally dead at last, it comes back disguised in scrolling.. Damned !

While I appreciate the responsiveness, the page is obnoxious to navigate on a mobile device.

Great, another example of a site that is vendor locked to the Chrome browser. Considering your site doesn't work on the majority of browsers (Neither Firefox nor IE), I'm not sure what "testing" you've actually done.

Working fine on Firefox 22

Strange. I was working with Firefox 21, and it didn't work. I've updated to 22 and it did.

Works fine on my 21

Works fine on iceweasel 21 (Debian ff rebrand).

Working fine for me in Firefox 17 (ESR)

Fails when Ad Block Plus is running. Don't have time to figure out which bit Ad Block is blocking though I'm afraid.

Its working with AdBlock plus 1.4.1

It's gonna depend on what filters you're subscribed to I imagine. The anti-social list that blocks facebook spying etc. seems to break pages most out of the lists I sub to.

It also works fine in Safari, although that's likely because of the -webkit- CSS prefixes.

This was awesome. Great work!

What is A/B testing? Local optimization with an assumption that every user is stateless and identical.

awesome animation :) love this way of explanation compare to a video cast

Thanks for this.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact