

What We Learnt From a Failed MVP - skmurphy
http://arg0s.in/what-we-learnt-from-a-failed-mvp.html

======
rmason
While lean is generally a great step forward implementation can be quite
tricky.

Saw a video last night by Jeff Lawson of Twilio about a lean test he conducted
on an email sending API. It's starts at the 9:30 mark:

[http://vimeo.com/16306094](http://vimeo.com/16306094)

He did a very thorough test using Google to drive traffic to his test website
along with various A/B tests on pricing.

His conclusion was there wasn't a viable market. Tell that to SendGrid and
their various competitors ;<). I've done the very same thing with several
sites and was always left wondering was it the test or my idea?

My conclusion is that to do lean successfully you also need to get off the web
and get out and talk with potential customers face to face. Sometimes if you
listen carefully they will hand you an even better idea.

~~~
wpietri
Yes! That's part of why "get out of the building" is such a popular Lean
Startup mantra. More generally, I tell novices to cycle between qualitative
and quantitative learning.

For example, I recently had an idea, so I did a bunch of interviews with
potential users. That gave me enough information to form a quantitative
hypothesis: if I offer users app X, then Y% will use it regularly over a two
week period.

Whether that fails or succeeds, my next step will be to talk with users. If it
succeeds, the interviews will tell me where to take the product next. If it
fails, interviews will help me distinguish between a fatal failure (e.g.,
nobody will ever use this) or a fixable one (e.g., minor UI issues make
friction too high for frequent repeat usage).

------
skmurphy
This is an excellent lessons learned write-up in a framework that is very
clear. He outlines the initial vision, key assumptions (labelled "big IFs) and
then a sequence of five experiments with a clear definition of Hypothesis,
Audience, Acquisition, Validation, Execution, Result, Learning. The author is
candid and offers a conclusion that I find very insightful:

    
    
       "However, one thing I noticed after I moved back to India from the 
       Valley was that we often hesitate to talk about failures, and 
       what we learnt from them. People are much more interested in 
       learning from a success story. However, if you ask me, I’m more 
       interested in the 51 attempts of Rovio than just the one that made it 
       big. It’s the failure to learn that’s a failure."

~~~
arg0s
Thanks for sharing this on HN, I was surprised when I suddenly saw a spike in
traffic and tweets. ;)

I certainly think the Valley is more open to discussing failures and what
worked and what didn't. In my experience, it's been hard to have honest, clear
discussions in India with other entrepreneurs about challenges being faced and
what people are doing to address them. There's just a lot of noise in the
startup ecosystem. That said, all this is maturing, and I'm sure it'll get
better soon.

------
JacobJans
I don't think this had to be a failure. A huge part of the job of building a
success is being willing to change what you're doing based on the results that
you're getting.

The one thing that I think is missing from "what we could have done better" is
this:

Be willing to CHANGE your ideas, big time.

You had people signing up. They wanted something. Instead of sticking with
your first idea (video), why not try a bunch of stuff until something worked.

You have to be creative, flexible, and determined in order to make this type
of business work. You had SUCCESS with many of the tests -- don't throw those
successes away just because one test failed.

People didn't want video. Give them something else. Does it really matter what
it is? Does it even have to be charades? Be willing to CHANGE on the spot in a
BIG way.

Think about this:

1\. People sign up and say "yes, I want to play charades".

2\. They land on a page tht asks them to play charades. How the heck can you
immediately engage them? By asking them to turn on video? Or...???

If you're giving up because people don't want to turn on video -- then you may
as well not try. Think about this: If I want to have "online boxing", should I
give up if people don't literally punch themselves in the face, in order to
simulate real punches? Sure, it sounds silly, but it's not that different than
asking people to turn on video. Online video can literally be harmful to
people, especially when it involves strangers, and even when it involves
friends and acquaintances. People don't want video charades. That doesn't mean
they don't want charades.

Don't get stuck on the wrong thing!

~~~
nmridul
You said exactly what I wanted to say...

An alternative improvement that I would feel comfortable -

1) Instead of email id, just ask "enter a nickname". More people would feel
comfortable this way.

2) Once I'm in - "Welcome to your private charade room. Invite your friends to
join your private charade room by sending this url".

Provide a private url to that room that they can send to their friends. Let
them send it by mail. And don't ask me to enter my friend's email id or link
to my facebook / twitter accounts. Many have already fallen for this trick
from Linkedin etc and paid heavily for giving them access to the friend's ids.

Now make it easy for me also to enter this room using the url. If my friends
enter the room, we are ready to play.

Allow me to create multiple rooms. So I could play with my family, my friends
or colleagues at different time. And all through the day I could watch which
room has member and start playing ...

3) And now provide an option for them to join a public room... Here mention
clearly that your video is now visible to the worlds and could even be
recorded ....

------
paul_f
This lesson shows the difficulty in testing of any game or pop song or any
other product that is really "fashion." They do not follow the same rules as a
typical business model would. You're not solving a problem or getting a job
done. You're providing entertainment. You have to try it and see if it works.
You need the vision of an artist and quite a bit of luck.

------
wpietri
Nicely explained.

One thing I have to tell novice experimenters a lot: A failed experiment _is
not_ when you get a result different than the one you wanted. A failed
experiment is one where you don't learn what you set out to learn. Which can
also be fine as long as you learn how your next experiment can be better.

This was a successful experiment.

~~~
arg0s
Agree completely. I left the misleading title in there since not everyone is
familiar yet with some of the lean terminology.

I did include a para further down in the article explaining this further. It's
the failure to learn that is failure.

[http://arg0s.in/what-we-learnt-from-a-failed-mvp.html#was-
th...](http://arg0s.in/what-we-learnt-from-a-failed-mvp.html#was-this-a-
failed-mvp)

------
midas007
"Failed MVP" is a contradiction in terms. (The V.)

Sustainability implies pivoting into something people will pay for, you can
build and convince them to use before cash runs out. If the team is highly
motivated, they will not disband on a single failure but will find something
else that people want.

------
jlukanta
I find it difficult to come up with the right validation criteria. Would you
share your thought process? How did you come up with specific numbers?

    
    
      Of a sampling set of people searching for Charades online, at least 25% will sign up
      to check out the game.

~~~
arg0s
It's fairly subjective at this stage, and is more of a judgment call than a
science. I went with the following assumptions while putting down the 25%
metric: * Conversion rates on vanilla sign up pages (with not a lot on the
page) are generally not that high. I'd read that 8-10% was common. * Users
already searching for charades online would be motivated to check out the ad
(in my mind, this justified looking at a higher rate) * Since this was SCM
based, it was not granular from a demographic perspective - likely had a mix
of early adopters, mainstream users, etc.

Hope this helps. I'd love to hear how others are doing this.

------
marcamillion
This is definitely a good exercise to do.

I just went through a similar process and did a write-up of my own -
[https://medium.com/how-to-succeed/86e70e2c33c1](https://medium.com/how-to-
succeed/86e70e2c33c1)

------
danso
I like the writeup, but one question I would've liked answered is: _does the
OP enjoy playing charades?_

To me, it seems like the best products are the ones that served the creators'
wants and needs. Github was created because the founders wanted to share code
more easily. Facebook was created as a better way to (eventually) get laid as
a college student.

If you really love playing charades, and more importantly, love playing
charades so much that you'll do it online, even with strangers, then the path
to a viable MVP seems clear. As it currently reads, the OP's analysis seems
sound, but without much of the passion that seems needed to drive a product.

~~~
swanson
In some sense it is almost better if the founder is not too attached to the
idea. I don't see how one user (the founder) being super excited translates to
a viable MVP - but I could see how it could lead them to ignore the results of
their experiments and the actual data.

~~~
raverbashing
"I don't see how one user (the founder) being super excited translates to a
viable MVP "

Less excited people simply won't bother building it.

It's that simple.

~~~
carrotleads
Less excited people won't know what makes some problems acute and some not so
much.

