
Writing Decisions: Headline tests on the Highrise signup page - brm
http://www.37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page
======
teej
I'm glad that Jason has decided to start throwing data behind his design
decisions. It's certainly easy to go overboard on data collection, but a
little good data is better than no data at all.

Quick story - a company I used to work for decided to A/B test their signup
page. Everyone in the company could submit their "version" of the page. In the
end, the best performer was the secretary, while the worst performer was the
marketing guy.

~~~
staunch
Not surprising. To do sales well you really have to be able to step back and
think like someone who knows _nothing_ about what you're selling. Hard for
someone who's been buried deep in a project for a long time.

One of the reasons I love trade shows. By the end of a few days talking to
potential customers at a booth you'll have a rock solid pitch.

------
aneesh
One caveat in A/B tests -- you have to be careful in deciding what metric you
use to pick the winner. They used the number of immediate conversions. But
maybe they just increased signups of freeloaders who will abandon the account
within 30 days. And maybe not. Another, possibly better, metric would be the
number of paying customers after 30 days, rather the number of conversions
today.

------
arockwell
That's a huge difference in sales from subtle changes in wording and font
sizes. Definitely shows the importance of A/B testing.

------
dpifke
We played with multivariate testing at the last company I was at, using an
expensive platform built by Optimost (now part of Interwoven):

[http://www.interwoven.com/components/pagenext.jsp?topic=PROD...](http://www.interwoven.com/components/pagenext.jsp?topic=PRODUCT::OPTIMOST)

(We also paid a company that did optimization via Taguchi methods, and found
them to be a bit of a scam.)

Besides this and Google Website Optimizer, I'd be curious if anyone on HN has
used other AB or multivariate testing tools? I would absolutely love something
open source that I could plug into Django templates on the server side, rather
than depend on external Javascript code to rejigger the page.

------
vColin
Interesting – I wonder what the variance in the “kind” of sign-up might be.
Perhaps those designs that highlight the “free” aspects will encourage a lot
of try-outs that then bail after the 30 days; while the designs that don’t
will see more longer term sign-ups.

The results of a follow-up after 30 days to see how many stay could also be
enlightening.

------
auston
Here is similar article related to design - etc
([http://www.boalt.com/blog/2008/10/great-design-is-a-
bottom-l...](http://www.boalt.com/blog/2008/10/great-design-is-a-bottom-line-
business/))

------
diN0bot
the moral of the story is that advertising in place of information gets
consumers/users to act.

~~~
jwesley
Advertising is a medium of information, not a substitute for it. I think the
moral is that people love free trials and it's essential to test your landing
pages.

