

Lesson from Madlibs Signup Fad: Do Your Own Tests - soundsop
http://www.kalzumeus.com/2010/02/27/lesson-from-madlibs-signup-fad-do-your-own-tests/

======
jrockway
Doing your own tests is good, but I think the username/password is too simple
to benefit from the madlibs style. I also thought the wording was unusual,
"I'd like to use this word ...... as my password." It could be that the
strange wording was what caused the drop-off.

~~~
andymism
I thought the username and password was a good choice for this test. There
really wasn't that much of a difference in phrasing between Patrick's form and
the Huffduffer form. The most likely explanation for the drop off is the
difference in the audience these sites are made for.

As far as design decisions like these go, the data doesn't say anything on an
absolute scale about the merits of one form versus the other, but it does tell
us which is more effective for a particular audience in a particular context.

------
dpcan
Well, he did it all wrong.

When you are done reading the Madlibs, it should read like a sentence. What he
did was surround regular text fields with strange sentences. When you are done
filling it out, you can't just read it like a regular sentence.

It should have been:

I would like to make bingo cards! Please save them for me under the name [
Your Name ], and and my email address is [ Email address] in case you need to
contact me. But please [do / don't] contact me twice with hints, and [do /
don't] send me a newsletter.

See, when you are done, you can read it like a sentence.

All he did was replace "Username" with longer wording. It didn't seem "fun"
the way you did it, just odd and off-putting.

~~~
micrypt
True. Made a few adjustments and I think it looks a bit better already:
<http://twitpic.com/15t9cc/full>

~~~
Herring
A bit, yeah. But the pixels are still really off. I don't even know why, I
just see they're wrong.

~~~
alttab
I think its possible its the wide difference in font. Because the madlib text
looks very different (as opposed to bold input labels) it looks frail and
slapped together. Larger fonts, bolder text, and more space could probably
help it more visually.

------
arnorhs
Your version of the madlibs form also looks much worse than your usual signup
form. I don't know how much that affects user signup percentage, but
personally I'd have an easier time trusting your standard form because if it's
design, not because of the "style"

~~~
patio11
Can all designers exit the room for a moment?

I am skeptical that "looks pretty" drives conversions. I think customers
pretty much don't care, based on "ugly" versions winning in many, many of my
tests. Remember: the customer _doesn't get to see_ the pretty alternative laid
out side by side with the ugly alternative.

Example: form submit buttons in your Internet browser of choice look fairly
pretty, right? (Or at least comfortingly familiar.) I would not describe
<http://www.bingocardcreator.com/images/bcc.net/next-step.png> as pretty -- it
is what I could throw together in ten minutes. But it _clocks_ "pretty"
buttons with the same text on it.

You're welcome to try it yourself if you think pixel-perfect alignment will
make a difference. Please post results.

~~~
ggruschow
Ugly/scary drives _me_ away from converting.

~~~
ssp
It's remarkably difficult to evaluate your own reasons for making decisions.
Just consider advertising. It works very well, but nobody thinks it works on
them.

------
rauljara
Running tests is definitely good advice. But execution definitely matters in
these things. Just looking at the two forms tested side by side, the original
looks more well spaced out and balanced. The madlibs has the two text fields
actually touching. Compare that to the original madlibs form
(<http://www.lukew.com/ff/entry.asp?1007>) and you'll see that the original
madlibs form just looks a lot better.

------
patio11
Crikey you guys are fast on the submit buttons. I hit the post button, try to
answer one customer email before HNing it, and it is already here.

~~~
soundsop
I was expecting the submission to be an upvote. Looks like my RSS feed updated
while you were answering that email.

------
mechanical_fish
While everyone is piling on with the untested hypotheses, here's mine:

The madlibs form is just too wordy. Whereas the username/password field pair
is not only less wordy, it's an _idiom_ : I've seen those fields, in that
arrangement and that order, thousands of times over the last decade. I barely
even have to read the captions at all.

Not that this hypothesis, or any of the others, matters. The point of this
article isn't to design a better form. It's to show _us_ how to design a
better form.

------
Batsu
As an intently web savvy group, we want this simple "fix" to work for us.

You provided us with reason to believe this (albeit simple) change will not
provide us with results.

Please, don't do this.

~~~
patio11
If I were to play pop psychologist, I think we have some fairly simple fixes
that we _want_ to work because they're cool/sexy/etc, and some simple fixes
which we _don't want_ to work because they're "stupid" or sound marketing-y.

For example, in the original madlibs test, where seven things changed, we
_want_ to attribute the change to the madlib element (which was sexy and
creative) rather than to e.g. the text on the conversion button changing. I've
gotten 20% lifts out of "stupid little things" like that before, and seen good
writeups of the same from other people. But e.g. designers do not want to hear
that the thing they spend 98% of their time on is essentially meaningless and
the elements of the page which they scrutinize least carefully -- like
microcopy or button text or (in one memorable test) which way the stock photo
model is facing dominate the measurable results. This is like how doctors do
not want to hear that the difference in clinical outcomes due to professional
experience is totally dwarfed by the difference in clinical outcomes due to
following checklists. (On the plus side, designers don't kill anybody for
their professional pride.)

In general, I think the point of A/B testing is that you should ruthlessly
soulcrush what you want with what the data actually tells you. I respect HN
folks enormously, but I'm not hearing "I tested it and it worked" -- I'm
hearing "I've got a great theory on why your data does not conform to reality.
Reality is, of course, what I predict it to be."

------
ramchip
Isn't it a bit early to call it a fad? I only saw one website try this (the
one you gave the link to). Perhaps I missed some, but still...

------
paraschopra
Yeah, it fantastically proves the point: do your own A/B tests. I am guessing
the implementation can create a huge difference. Maybe you can ask for help
from some of the designers here on HN and see if a beautiful implementation
makes a difference. Plus your two checkboxes can also go into the "malibs"
form.

------
maxklein
Seeing as we have two totally different results from different people, my
conclusion is that even though the madlibs style can improve your conversion
rate, it's too easy to get it wrong and actually reduce your conversion rate.

The majority of websites have used the standard sign up form, and it's known
and recongized by most users. It's risky to to go with something like this
that could have a strong negative impact if you do it wrong.

------
ams6110
Just as a personal observation, I much prefer the traditional style. All these
"I want..." and "Give me..." style dialogs make me want to vomit.

------
tkiley
The madlibs form has a "sign in as guest" link, while the original sign-in
form does not. The article doesn't specify whether a guest sign-in is counted
as a conversion or not.

~~~
patio11
_P.S. If you have good eyes you’ll spot the other A/B test ongoing on this
page. I’m using the traditional way of mitigating cross-test interaction…
ignoring the possibility of it_

Both forms have a 50/50 chance of having the guest link on them, and guests
don't count as trial conversions (because I have several months of stats that
suggests they give me money infrequently and they cause more support issues
than all trialers and paying customers put together). The thrust of that A/B
test is to see whether aggregate sales go down if the guests option is taken
away. If they don't, I will drop it.

~~~
nitrogen
I noticed something in sprites2.png referenced in your A/Bingo page: the
bottom row of pixels on the third blue background image is duplicated from the
second blue background image.

