
How To Validate Your Business Idea By Testing A Hypothesis - alex_ka
http://hatchery.io/how-to-validate-your-business-idea-by-testing-a-hypothesis.html
======
davidroetzel
This kind of advice pops up on HN from time to time and while I believe it to
be valuable, I guess it is at least incomplete. Here are two stories from my
failed ventures to augment this advice:

1) Years ago I started a small side-business with a few colleagues. It was a
web-based service for a very specific group of clients. Best of all: The idea
had already been validated. We had our first paying customer before we even
built the product. And we quickly found a few other customers. But after a few
sign-ups things were not looking so good anymore. Even though our early
adopters were well known in the industry, other companies would not follow.
Everyone we talked to loved the product, but as we had to learn the hard way
nobody was willing to pay for it. Some even went so far as to implement
possibly illegal solutions instead of paying us a few bucks.

Lesson learned: Even if you talk to a few possible customers, question hard if
those really are representative of your target audience. Also, liking a
product/idea is not the same as being willing (or able) to pay for it.

2) A while back I wanted to find out if there would be any commercial interest
in an app I had already built. So I set up a landing page and collected email
addresses. Since I had already built the app I could easily hand out demo
accounts for every interested party. I quickly collected around 60 addresses
and set them all up with a demo account. Only a single person ever logged in
at all! I sent out reminder emails but did not get a single response. To this
day I do not fully understand what went wrong.

Lesson learned: Newsletter- or Beta-signups from your landing page is NOT the
same as validation for your idea.

~~~
mason55
_> Also, liking a product/idea is not the same as being willing (or able) to
pay for it._

A great thing to do is when you're validating products you ask the prospect
how much they'll pay. Then say "Ok, so if I put a contract in front of you
that says you'll pay $x if I deliver this product you will sign it?"

You'll find out very quickly whether someone is just being nice or whether
they are serious.

~~~
wpietri
Indeed, rather than just asking _if_ they'll sign a contract, you can pull out
a Letter of Intent [1] and ask them to actually sign it. You can explain that
they're not legally binding if you need to keep people from freaking out. But
there's something about signing an official-looking document that a) makes
people take it seriously, and b) makes people more likely to really sign up
when you come back to them.

Depending on context, you can also ask for a credit card number and then not
charge it. People are very careful about giving out credit card info, so them
giving it to you is a great sign that they're really serious.

[1]
[http://en.wikipedia.org/wiki/Letter_of_intent](http://en.wikipedia.org/wiki/Letter_of_intent)

~~~
RogerL
Please, no.

As soon as someone starts trying to maneuver me to take money out of my
wallet, I'm __gone __. It 's not _my_ problem that you don't know if your
product is good or not. There is no way on earth I'm signing anything, giving
you a credit card number, or what have you. I'm utterly shocked that this is
considered a good way to gauge interest in a product.

Even if I really am interested, so what? I have a budget, my appetite exceeds
my grasp, just because your product might solve a pain point doesn't mean I
will buy it. I need to consider the needs of my entire business, weigh pros
and cons, and so on. If you are pulling out papers to sign after a
conversation, I'm so gone.

I'm reminded of some 3rd party home alarm installers that came around to my
house a few months ago. We were thinking of re-activating the alarm in the
house, so we talked to them. Didn't take long for the papers to come out,
etc., and it didn't take much longer for us to tell them to pound sand.

I don't give emails out (because on average the spamming becomes relentless)
to landing pages, now I'm supposed to give you a CC number just so you can do
marketing research?

No.

~~~
philbarr
So in which situations _would_ you buy something?

Not trying to be snarky or anything, I know yours is a common opinion so I'm
just wondering what would be a less offensive way to test the waters?

Perhaps - do the conversation, leave some contact details, give them the
papers but just say, "look, no pressure, I'll leave this with you and let you
think about it." Then see how many people get back to you?

~~~
silverbax88
Coin. (onlycoin.com)

They showed a video of what the card would do and how it would work. I
actually have no idea if it will work, or does work, but it looks like it
works in the video. It looks like they have something awesome. And it won't be
$2000 to buy one.

They said, it should be ready within a year and you can reserve one now for
50% of the sale price (regular price $100).

So I bought one, even though I am aware that the product might never come to
fruition and that my $50 might disappear. A lot of other people bought, too. I
have gotten updates and I have no reason to believe I won't get my product and
I'm excited that I got in on it.

But...

What did I buy, _really_?

I saw a guy on a video using Coin, showing how it would work, selecting
different options on his card, answering all my potential questions. Man, it
sure looks sweet and I can't wait to have one...

except...

did Coin really exist or did they just do a video to show how it _could_
exist? Does it matter? I have pretty good video equipment and coding
skills...and I've even hired actors for videos and done pretty well...what
would stop me from hiring an actor, shooting a commercial for something like
Coin, and then collecting thousands of dollars to fund the actual creation of
the product?

Nothing. I just didn't think of it and didn't act on it, and they did. I'm not
saying that's what they did at all, I'm just saying that _could_ be done.

That's market validation. They sold thousands of Coin based on their idea. I
have no idea if they even had working code or product, or if they were just
validating whether it would sell.

------
gchokov
While I do see the value of validation, this is not a silver bullet. People
try to simplify things, find patterns, but this cannot always work. There are
so many reasons why I would _not_ want to validate things: \- Sometimes,
people will not tell you what they want, because often they do not know what
they want, until you show it to them. Someone smarter than me said this
before. \- People judge from their experience. You can have the perfect group
to validate with.. but their experience will trick you.

Building a product is an art. There way too many things one should consider
based on his experience and intelligence, not everything can be quantified and
calculated (well, it can be, but still with assumptions, and once you
assume...). As one of my favorite photographers says, "There are no rules for
good photographs, there are just good photographs". Same can be said when
building a product..

~~~
vdaniuk
"There are no rules for good photographs, there are just good photographs". I
disagree with this and think that it is a self-defeating, irrational attitude.

Sure, validation is not a silver bullet. However, lots of startups make it or
break due to luck and no-one(yet) can't eliminate randomness in a
stochastic(debatable) system like technology entrepreneurship ecosystem.
Validation helps to decrease the chance of failing, though. The famous quote
by George Box reads "Remember that all models are wrong; the practical
question is how wrong do they have to be to not be useful."

Also validation is not listening to what people tell, it's also observing
actions of users when they interact with the MVP.

~~~
gchokov
It's funny that I mentioned randomness in my second comment as well :)

I tend to agree with you to some extent, even about the quote for the comment.
But, startups as an example fails to me, because as we all know, most of the
startups fail anyway.

p.s. there are rules for good photographs but following the rules can only put
you on bar with other people. Learns the rules so that you can break them.
Learn to validate so that you know when _NOT_ to validate.

~~~
vdaniuk
Hm, I have an interesting thought. If "rules" is a consensus about best
practices it is a standard model of the particular domain. Conditions of
narrative manipulation and tragedy of the commons apply.

With that in mind, one should break the rules only if they have a high degree
of confidence in their ability to discover a model that is better than
standard model or understanding that the entire modelling approach is tainted.
However, Dunning-Kruger effect hypothesis states that less competent are
irrationally sure of themselves and vice versa. Are we in a situation where
most actors on the market are trying to break the rules to their own
detriment?

------
poseid
"After all, it took Thomas Edison hundreds of attempts before he invented the
light bulb"

-> It still not easy, people were used to candles at that time. Edison did not study the candle making process or the problems of people with candles. His genius was in putting together the solution.

Apart from this, the article is nice indeed.

~~~
joeyspn
> It still not easy, people were used to candles at that time. Edison did not
> study the candle making process or the problems of people with candles. His
> genius was in putting together the solution.

^This, and this [http://www.helpscout.net/blog/why-steve-jobs-never-
listened-...](http://www.helpscout.net/blog/why-steve-jobs-never-listened-to-
his-customers/)

------
coldcode
A startup I know had a great idea, great product name, validation from some
actual people in the industry that it was something everyone would want. They
built it and launched and discovered that the industry as a whole was
technology phobic; the problem was that the supposedly representative people
they talked to where among the few that understood the benefits of the
technology (in this case mobile). It's like a poll I read about in the 30's
where pollers called a lot of people to ask about phone usage. The problem was
that they only talked to people with phones so the answers were useless.

~~~
zacharycohn
So... they didn't have validation, then?

------
maninalift
I usually skip the start-up advice posts on HN but this was worth reading for
the first point along "focus on the problem not the solution". Good advice for
hackers as well as entrepreneurs.

------
jlcummings
Nitpicking, but removing "business" from the title would fit the article body
a bit better. Questioning the viability of potential products and services is
far different than evaluation of "business" modes or models. Good products
will not save or propel a bad business, but good businesses can thrive on
marginal and even terrible products.

------
shanecleveland
I'm trying an idea for a small project that I am sure I stole from Hacker
News.

I run a football pool at my office, and I was manually creating each week's
list of games. So I scratched my own itch and automated this as much as
possible (some manual work). I've made the results available on
footballpickempool.net. It simply offers weekly pool sheets for the full
season, and results will also be posted. I know many people could find this
useful, but not necessarily a paying audience for such a simple offering.

The validation idea I'd seen before was to offer a link to a "feature," and
see how many people click on it. So I have a "Start a League" link. The page
let's them know that this feature is not available yet. And I also have a
Google form on the page asking if anyone would pay a small amount for this
feature.

Hopefully I see some results as the upcoming season starts.

------
WhitneyLand
Look at it from reverse - of all the big/successful startups out there how
many used any formality or rigor in validating their ideas?

Even if they did validate early on what made them successful often changed
drastically based on what was known when the company was started.

In otherwords, what percentage of the time does this payoff?

~~~
w1ntermute
Out of the four $10b+ companies founded since 2007, Dropbox and Airbnb
definitely used Lean techniques[0], and Uber probably did[1]. That only leaves
WhatsApp, which there isn't much information about out there. But considering
how little funding they took, it's safe to assume that they utilized similar
techniques.

0: [http://www.hult.edu/en/news/hult-labs/2013/august/the-
lean-s...](http://www.hult.edu/en/news/hult-labs/2013/august/the-lean-startup-
methodology--a-movement-transforming-business-development/)

1: [http://www.quora.com/Lean-Startups/Did-Uber-use-any-lean-
sta...](http://www.quora.com/Lean-Startups/Did-Uber-use-any-lean-startup-
methodologies-or-perform-customer-validation)

------
dsplatonov
It's quite difficult to validate an idea, after FFF approved it and when you
start development. In our startup we found it quite difficult to find the
resources, where our target audience is located.

------
mccolin
"Fall in love with the problem, not the solution" great advice

------
bane
I can give these two piece of advice.

1) Don't delude yourself into thinking you're "changing" the world (for some
ridiculous definition of changing). It's most likely that you aren't. Focus on
the business fundamentals.

There's an awful lot of startups making very ho-hum products -- small
incremental improvements to existing things or dead end paths that nobody
would ever possibly want. They've convinced themselves and their VCs that if
only the entire world starts using their chat app or appointment reminder,
that world peace would break out and we'd live in a Bill and Ted future where
Rock music unites us all. They then build a business model that _actually_
requires some percentage of humanity to get on board with their app to become
profitable.

They also forget that to the VCs, they don't care about changing the world,
the company _is_ the product and in the end the company will be sold for parts
to the highest bidder. The business needs to optimize for this case because
_this_ is what the end-game looks like. Not a neon colored future where
everybody is playing air guitar and peace and tranquility because they used
your app.

2) If you're bringing lots of technology into the startup, it's not
necessarily an advantage. Technological debt can absolutely cripple you. And
by the time you've realized it, you're completely out of money and left with
old technology that nobody wants.

I know of a more than a few startups that get kicked off with a million lines
of existing software. They think it's the matter of some source clean up and
some new paint on the front end and it'll be easy street. Inevitably, they
spend most of their VC investment bringing this legacy to market and getting
customer #1. Feedback from early customers is positive, but with one caveat,
"this software needs to do x, where x requires a fundamental rewrite of the
entire product. By the time a decision is made to start a ground-up rewrite to
support x the company is a couple years in and has lost all of the momentum a
startup needs when coming out of stealth mode.

rewrite = reboot of the company.

You accomplish the rewrite, pitching it as the next version, but shifting
entire technology stacks doesn't _feel_ like a version bump to the customers
(it's a huge investment for them as well). All the sales activity you've done
over the last year is also junk, unless you feel like supporting the old and
new versions simultaneously, and you probably don't have the resources to do
that well.

By the end, you're out of VC, coasting on what your sales team is pulling in,
but not moving revenue in enough of a positive direction to exit or raise
another round. You try to sell it on the cheap and recoup the VC, but buyers
are wary of the now _very_ old technological debt you've accumulated and the
small customer base you're bringing to the deal. Along the way you pick up
some very onerous, non-friendly software licenses for some of your components
-- %of gross revenue or something even more onerous like GPLv3 which makes
your entire technological investment worthless. Buyers doing due diligence
decide not to buy. Your company is a zombie at this point.

------
dammitcoetzee
How is the scientific method not used for this always? The scientific method
and way of thinking is probably the most useful thing you can ever master.
Science allows you to set up a question and then set up an experiment in such
a way that you cannot lie to yourself about the result. For example, let's ask
the question: Is my business doing well?

A layman's approach will almost certainly be evaluated on a
subjective/emotional basis. i.e. "We are doing something great!" "Everyone is
happy." "I have so many customers" "Funding is in!" "I got a great review on
tech crunch three months ago"

A more scientific approach is to first define the question in such a way that
it can be measured. So let's say a business that does well does three things:

Does not lose money, Has happy customers(low support calls/emails, high
reviews,product is used), Has happy employees (low turnover, gets quality work
done (measurable by issue tracking)

Now you can say: Does my business meet the requirements for a successful
business. If no, write down those numbers. That is now your control and
baseline.

It is time to experiment. It's best not to change more than one thing at a
time in science so you know exactly what the effect is vs the control.

Lets say you realize you're losing money and your customers aren't happy. You
do some research and come to believe (not know) that it's because the login
screen is incredibly awkward. Google analytics shows that the bounce rate for
the front page is huge so there's some data to back this up. It's time to form
your first hypothesis.

My business isn't doing well because the login page is so incredibly awkward.
If I fix the login page the business will do better.

Remember that both the "awkward" and "do better" while subjective declarations
have actual metrics to back them up: bounce rate on analytics and the
previously defined metrics for a good business.

Now you do the experiment defined earlier: fix the login page.

Now you collect data. If your bounce rate goes down, your customers are
happier and start paying the monthly fee. Then you have solved your problem
and you know with a very high degree of certainty how you have solved your
problem.

Now let's say you looked at the data at the end of this experiment and noticed
a decrease from the control you set a while back. Well, obviously that was not
the problem (or at least not 100% the problem, part of science is knowing how
certain to be about things) Maybe next you can define the problem as: the site
is buggy, issues aren't resolved quickly. If I observe my employees I can see
that they spend a lot of time staring at the 98inch projector screen I set up
in my 'cool office' instead of coding. Now you can do a trial run of turning
off the projector on some days vs others and seeing if the work goes up, stays
the same, or goes down. Or you can discuss the problem with your employees and
see if they have input. Then define the experiment using that; just don't
forget to actually measure something objective at the end of the day.

This way of thinking is invaluable. You can drill down exactly to the problem
core. Waste less resources. Argue less. Most importantly you can actually sit
down at the end of the day and prove that you have solved a problem with
actual data that cannot be easily disputed, and if you've really done it right
the experiment can be repeated and have the same result each time.

~~~
jjoonathan
> This way of thinking is invaluable.

Is it the way of thinking or the things you assumed the business already had
that were invaluable? You assumed

* The existence of a large enough steady stream of paying customers to quickly gather statistics on. "Steady" is especially atypical in many environments.

* Enough runway that you can afford to tackle issues one-by-one (i.e. using a control) rather than scrambling and guessing at what is going to work. If users come in "bursts" due to marketing campaigns or whatever then this issue is compounded because each burst can cost a significant fraction of your runway.

* The business is at steady state (e.g. you aren't planning a "grab the market" phase that runs at a loss and then a "make money" phase that leverages the brand recognition, network effects, etc that you've built)

~~~
dammitcoetzee
I assumed all those things in my hypothetical example application of the
scientific method to a decision. I'm not really certain what you're getting
at?

The article linked above already goes into some detail as to how to use this
to simply evaluate an idea. I was just expounding on another possible use just
to show how invaluable this is. Heck you can even use it in relationships. You
can tell if someone is lying. You can use it to tell if people like you. You
can even use it to lose weight. I'm using it right now to see if I need to
drink more water daily or if something else is wrong. It's basically a super
power.

~~~
simoncarter
That the assumptions you made are not necessarily feasible/possible/correct in
a real world, startup scenario.

