However, before everybody goes out and puts puppies on their homepages, they need to realize that there are a bunch of things being tested.
Image vs. no image: Is it possible that having any image at all improves the conversion. You should test with other pictures: perhaps some animals, people, nature, and see if the puppy is what makes it work.
Call to action: The 'puppy' version also features a more succinct call to action in "Sign up now" rather than "Start your 30 days free trial." Perhaps this also contributes some of the difference.
Button size: The button size in the 'puppy' version is smaller. Perhaps this has some effect as well.
Length of text: The 'puppy' version has more description of what is involved in the free trial. It says "Pick a plan & sign up in 60 seconds. Upgrade, downgrade cancel at any time." vs. the no puppy version that says "Start you 30 days free trial."
Vertical vs. Horizontal Layout: The 'puppy' version has a vertical layout of the text and button, where they are stacked on top of each other rather than left or right.
So there are at least five different changes made between these two designs. Clearly the second design wins on conversions, but it's not entirely clear to me why it wins.
Even when possible to isolate and remove ancillary changes to improve split test purity, it's often not beneficial. If there's a significant number of changes, achieving statistical significance across the full matrix probably isn't even possible.
But that's ok, because limiting changes to a single test queue restricts your ability to move fast and try lots of stuff, which is beneficial. So when you test, try cheap multivariate methods (there's a bunch!) to quickly understand how interactions between multiple changes affect results.
For example, when I was living in Japan around 1996, we needed a kerosene heater for our apartment. We went to the big department store, and they had a lot of different ones. Each of them had a sticker affixed describing the various features and advantages, but they were mind-numbingly similar. But one of the stickers had a photo of Mai-chan, the cheerful and engaging child star of a cooking show that we had enjoyed on Japanese TV a couple years earlier.
My wife pointed it out to me, and we ended up getting that heater. Not because it was any better, but because we liked Mai-chan, and her photo was on it. Of course Mai-chan has absolutely nothing to do with heaters, and we knew it. But we didn't care.
We always called it the "Mai-chan" heater, and even though we left it behind when we returned to the states, I guarantee if I mention the Mai-chan heater to my wife today, nearly 20 years later, she'll smile and know exactly what I'm talking about.
The puppy screenshot in the original post has a similar effect on me. A part of me wants to reach out and pat the cute dog on the head. But I can't. But I can click the button, which is what he obviously wants me to do. Sure it's not at all rational. But it motivates people. I suggest keeping the dog and adding it all your advertising. Make it your mascot. You can't go very wrong with a puppy!
That's exactly what I imagined when I saw the title; a startup testing/comparing the conversion rates of a dog mascot/logo. It goes hand-in-hand with establishing a memorable brand. I'd be really curious to see how the conversion rates are affected by a complete website/brand overhaul containing a dog mascot.
There's a known few years old meme now, featuring a panda and a number "3". It's a rebus that should be read as "panda 3", and understood as "pan da 3", which is a kind of impolite way of saying "sir, please give me a 3 (D grade)".
"Start your 30 days free trial" text
"*Sign up now* 30 days free rial. Pick a plan & sign up in 60 seconds. Upgrade, downgrade, cancel at any time." with a puppy image.
It is a great post and just goes to show how testing can really help boost conversion. It's often a lot cheaper to grow your business by improving conversion rates than by increasing traffic.
"But we can still be fairly confident that the image was the main factor since this test had a much higher change in click through than what we typically see when we just test text vs text."
How can you be fairly confident? They broke the cardinal rule of the scientific method.
Come on. It's a puppy. A puppy!
The best way may be with the puppy or without. From this you have no idea.
- babies (various poses, ages)
- tools (wrenches, screwdrivers, axes, etc)
- auto parts
- aliens (as in fake grey aliens)
- people in weird situations
- women crying (this one works well near saint valentines)
- an animal with a cast (better yet if its a small bird)
- toilet paper (a new roll. not used)
- random charts (pie, bar, etc.)(The charts doesnt even need to make sense)
- a lock (big master lock)
- and one time I ran an alpaca. Weirdly, it converted better than anything else. Don't ask.
I wonder how well real aliens would do.
Coincidently I now work at a company with currently 4 dogs in the office.
Jesus, and I complain about how hard it is to concentrate in a noisy open plan office. A bunch of non-human animals roaming about might just tip me over the edge.
Good call. Aren't black dogs the least likely to be adopted from shelters?
(to elaborate for the downvote-debbies: there is an entire industry designed around getting users to use your product based on ideas such as this, and this link is one place to learn more about that)
What's next someone telling us they increased conversions because they used photo's of smiling people?
- In-joke-ish (octopus merges, the internet is for cats)
The most popular color? Blue! IBM is blue, now Windows Blue, Mac OS X aqua blue... you name it.
You added an image, period.
You also changed the CTA button style.
The split-test did not test an independent variable.
Having said that, it's good that your conversion went up.
It could just have been that you described what signing up entailed. Plus you included 'cancel at any time', which might help some people decide.
People used to game Google Adsense by putting their own images next to the adsense blocks in order to get more clicks. MFA sites did this so much, Google banned the practice of using images next to their ads. This is the same result.
Question: Is a skillfully designed bug-free easy to use CMS package more important than a picture of a puppy?
Answer: The picture of the puppy is clearly more important.
Just the fact you used a graphic can make it superior to the text version, not necessarily because it's a dog.