
 The Tim Ferris way of testing ideas and how I did it. - dawie
http://www.docleyblog.com/2009/10/the-tim-ferris-way-of-testing-ideas-and-how-i-did-it/
======
fookyong
There is an ocean of difference between 14% of visitors committing their email
address to something, and that same figure actually pulling out their credit
card.

A more prudent assumption would be 10% _of that 14%_ will actually pay for
your product when it is launched.

I learned this first hand. Your mileage may vary of course, but it's
incredibly easy to drive traffic and stimulate interest, enough that people
will give you their email address or whatever other personal information you
want. The moment you ask them to pay though, you're in another world in terms
of conversions :) This MVP-type google ad testing doesn't really translate
that well into real world data like purchase conversions. I don't think that's
the point anyway.

It's good for seeing what people might be interested in, which is often
something that startups aren't sure of :)

~~~
dawie
I agree with you. Maybe a good idea would be to ask for a credit card, but not
to charge it. I think this borders on being illegal though.

I am still pretty happy with 10%, since this was my cut-off for going ahead
and building the product.

~~~
andymism
Instead of asking for the credit card information (which does sound illegal),
I've heard of others who have placed a "Buy Now"-type button on the landing
page or fill out form fields just up to the credit card fields (say, for a
multistep payment process). A user who clicks on "Buy now" or fills out the
first part of the payment form is a pretty convincing datapoint.

I haven't tried any of this myself since it raises a big issue for me: Doing
this makes a promise that you don't intend to keep (at that moment). If trust
is what gets users to buy, isn't it counter-productive to violate a user's act
of trust?

I'd like to get over this because I can see how experiments like this can be
useful and I hope you'll keep us all posted with your success rate. Some
things I'm curious to know are:

* How much time passed between your AdWords test and your MVP launch? * Of the emails captured, how many participated in the beta and eventually became paying users?

Best of luck!

~~~
netsp
What would it take to make you feel comfortable about it?

~~~
andymism
That's a good question. My reservations mostly have to do with this:

 _Of the emails captured, how many participated in the beta and eventually
became paying users?_

Which is really asking: Will those users really come back after you've broken
that initial promise?

Other success stories say yes. And I speculate that there's a certain window
of time that you need to deliver in to make it work.

There'll always be another question though, so I guess it can't hurt to just
give it a shot.

~~~
netsp
About a month ago, I filled out an application/request with an NGO (Acumen
Fund) about opening a local chapter where I live (Melbourne). My understanding
that if they get enough requests, they'll try to get a local chapter going.

Now I got a response that they want to extend a 'special invitation' to use
their 'to register on our brand new community site' where I can get involved
and maybe new chapters will materialise.

The timing seems pretty convenient. I suspect that the request/suggestion form
was really a list building excercise for the community site. It's a Ning.
Those things need to cross a substantial chicken-egg hurdle before they are
any use to anyone.

I admit, there was no blatant lie. It isn't really exactly the same thing. But
it is sending out feelers & building email lists. They could have also put up
a big 'join our community' button on the site that led to an 'sorry, under
construction' page after you "sign up." Maybe they did. It was a little bit
sneaky. I still joined & even started a group. I'm not angry with them. I'll
even mention it to others.

BTW: <http://community.acumenfund.org/>

------
tocomment
That seems hard to believe. I did the same testing strategy with my Job
applier software (<http://fastjobapplier.weebly.com/>) and first of all I only
got around 30 ad clicks total, and no one filled out the form. (I was
thoroughly disappointed.)

Granted maybe I just suck at sales copy, or perhaps there just isn't demand
out there for software that helps you apply for jobs.

Still though, a 10%+ conversion rates seems too high.

Do you find you're still getting a similar conversion rate now that you're
actually selling the product?

~~~
utnick
yea 10% seems pretty high, I think it might have to do with saying the 1st
month is free and you can cancel any time.

Therefore, the 10% rate isn't really a 10% sales rate but more of a 10%
willing to try your demo. But 10% is still pretty good for that I think.

~~~
dawie
I agree that 1st month free and cancel anytime might have something to do with
it. If I do a test like this again, I will probably leave that out.

I will have these terms in my live product though...

~~~
teej
The core issue isn't that you offer a month free, the issue is that your form
doesn't measure purchase intent. You have no idea if 1% or 10% of your leads
will convert into sales.

------
teej
I've been itching to try this for some time. I would appreciate any war
stories HNers could share.

~~~
patio11
Email addresses don't covert on a 1 to 1 basis to sales. (If. Only.)

After the autoresponder runs out (6 days in my case) they've only got only a
moderately higher chance of conversion than my average trial user. (~5% vs.
2.8%)

I really, really don't like the Put Up Fake AdWords Ad, See Who Gets Suckered
minimum viable product. Aside from misleading customers and hurting the
AdWords ecosystem (they don't actually get offered what they clicked to get
offered, which Google will come down on you like a ton of bricks for if they
detect, incidentally), it _doesn't test your product_. It tests your ability
_to write AdWords copy_.

I could get 14% conversion to stabbing toothpicks under your fingernails, but
what does that tell you? The world has weird people in it, and I'm good at
qualifying non-weird people out with my ad text and keyword selection? Whee?

Similarly, if I delivered an AdWords campaign that was _awful_ (low CTRs, low
conversions), would that tell you the product is awful? Because I've done that
before, too -- the first draft of my AdWords campaign for Bingo Card Creator
was _terrible_. It took me almost a year before I figured out how to write
AdWords decently. That was not an indication that the product was terrible (I
sell it quite successfully), it was just an indication that I needed to adjust
to a new way of copywriting.

If it can't tell you if an idea is good, and you can't trust it if it says the
idea is bad -- what is the point, again?

~~~
sgk284
I think you've hit on some good points, but missed a big one. Running ad words
drives traffic to a site that currently has no traffic, better yet that
traffic has shown some interest in the product you're claiming to have. If
some of those people try to buy (or express interest in) this non-existent
product, that says something.

You're not measuring how much traffic ad words brings in. You're using ad
words as a traffic generator (ideally generating interested traffic), and of
this traffic that comes to your site, you can determine if people are clicking
through.

For ~$100 you can determine if there is _any_ interest at all. This can be
powerful for people who really have no idea what the market will do. It's not
perfect, but not useless.

~~~
patio11
_This can be powerful for people who really have no idea what the market will
do._

These people should take their hands off the keyboard and not put it back on
until they have spoken to at least one Real Person who has the problem that
they are trying to solve. After they have done this, they will know if there
is any interest at all. This produces _actionable insights_ : if they tell you
they have a problem, then at least some people have the problem. If they tell
you that the industry's favorite way for solving the problem costs $1,000,
that immediately establishes that there is a viable market for solving the
problem.

Compare this to the AdWords campaign: can you tell me what numbers you can get
that say "This is a good idea -- do this" and what numbers you can get that
say "This is not a good idea -- don't do this"? No -- subjectively "good"
results don't imply purchasing intent, subjectively "bad" results don't imply
lack of it. You'll get _noise_ \-- very precise noise, noise you can measure
down to the click, but still noise.

~~~
sgk284
Suppose I came up with an idea that I absolutely knew was brilliant, let's say
edible shoelaces. Unfortunately these edible shoelaces aren't cheap to
produce. I'm a one man shop and have almost no money.

I make a fully functional website to sell this product, only I don't have the
product yet. When people browse the site and have gone through the pricing
page, etc... and finally click "buy" I simply tell them that we're out of
stock.

Now I want to see if people are actually trying to buy this thing, so I start
marketing but have a limited budget. I throw $1000 at ad words. If I get 100
people trying to buy my edible shoelaces at $20 each then I know that maybe
it's worth investing in manufacturing the product.

How is this not useful data? Instead of going all out and spending money on
manufacturing and a website, you only pay for the website and hold off on the
manufacturing expense until you're confident it'll pay off.

~~~
revorad
Except that we are talking document management here, not edible shoelaces. Or
anything that requires "manufacturing" in the made-in-a-factory sense of the
word.

A fake ad survey for a document management system from someone who "works in
the electronic Document Management and Records Management industry" is
shouting out loud, "I'm too scared to actually sit and code this thing because
it's hard and I might take too long to do it or completely fail to do it."

And I don't know why people like throwing money away. With $100, I'd rather
buy 5 months of Linode server time than 12 email addresses. When the best
entrepreneurs are saying be cheap, I think we ought to listen and _be cheap_.
And with 2 weeks of time, I'd rather sit and code the most basic functionality
of my app and show it to people I know who might need it.

------
spokey
Here's another (cheaper, although possibly less realistic) approach to this: I
recently did some market research using Amazon's Mechanical Turk, and added a
lead collection form as a bonus.

I put up a simple survey asking users about relevant background information
and their experience with the problem my application is trying to solve.
Included in this survey was a field to collect the user's email address. This
field was very clearly marked as optional (it even appeared after the
confirmation code that allowed turkers to complete the "HIT") and the label
included something like "Your answers to this survey will help us to evolve
the website to better meet your needs. If you'd like to be notified when the
new site launches, please enter your email address here." along with an
indication that users would receive exactly one email from us due to this
form.

I was pretty pleased with the results of the survey--I received a lot of
actionable information very quickly and cheaply (at about $0.10 per response)
and I was pleasantly surprised that a little more than 10% of the respondants
entered what looks like a valid email address.

I don't expect many of those 10% to convert (and lead generation wasn't the
point of the excercise anyway) but I was very happy with the ROI on this
survey.

Seeing who clicks on ads (and at with what copy) and later "soft" converts may
be a more realistic test, but the MTurk approach is an order of magnitude less
expensive.

------
abalashov
Actually, the figures here may be pessimistic. It takes a certain kind of
person to submit their e-mail address when there is quite obviously no
deliverable being explicitly promised to them for immediate purchase &
download. If there were one, there might have been higher conversion.

------
joubert
I don't get it. So you "sell a product", including a 30 day trial, but once
the user signs up there's no product?

