
“Scientific Method for Startups”, and AMA with Michael Seibel - craigcannon
http://blog.ycombinator.com/the-scientific-method-for-startups/
======
ProxCoques
I think there's clear danger of having a local maximum problem here. This is
because the role of hypothesis in the process seems either misunderstood or
misplaced.

The Tim Robinson example in the article recounting the removing of links on
the checkout flow illustrates this. He observed something interesting in the
data (people clicking on links), but he then skipped straight to a test which
removed links and found conversion went up. It's implied that people were
being distracted from the main task by the links.

But it might equally be that they wanted to click the links to find some
missing information about the product. Or even some other reason. Without any
qualitative analysis to establish a reason _why_ people were clicking on them,
you don't know if he might have raised conversion even higher had he kept the
links and addressed a need people had. This is particularly important in a
checkout flow where the numbers of people are comparatively low and
behavioural elasticity is also comparatively low.

So I'm reading between the lines here that because qualitative analysis is
hard, people don't do it. Maybe that's fair enough, but that doesn't prevent
you from kicking the tyres on hypotheses. Quant analysis and A/B testing will
only tell you _what_ happened, not why did.

------
mwseibel
I always love the debate on whether great products are imagined or are
discovered through experimentation (certainly a bit of both). This post
details the advice I usually give to startups on this topic and I'm happy to
answer questions and join the discussion.

~~~
dpandey
In the creativity bucket, there's also a fair amount of inspiration from:

1\. Studying other products and reasoning about them.

2\. Conceptually dissecting multiple existing products and looking at
different combinations of some subset of their components.

Any thoughts?

PS: Loved your essay

~~~
mwseibel
Yes, one of our YC partner Dalton Caldwell hammered this idea in my mind. He
told me that when he sees a founder trying to attack a product that other have
either failed or succeeded at - he looks for what they've learned from those
folks. If the founder dismisses previous attempts to solve this problem or
similar ones - that is a very bad sign.

~~~
dpandey
Really enjoyed your piece on product dev cycle fundamentals:
[http://www.michaelseibel.com/blog/product-development-
cycle-...](http://www.michaelseibel.com/blog/product-development-cycle-
fundamentals)

Just curious growth wise, did Socialcam have a lot of word of mouth, or was
some growth hacking involved as well. 16 million downloads in 3 months is
incredible.

------
beat
This was a better article than I expected. Finding ways to measure your
assumptions is great, because if you can measure the behavior, you can measure
the impact of changes.

Generally, I'm not a fan of "scientific method" as a phrase. It's too often
used by people dismissing intuition in a very non-scientific way.

~~~
mwseibel
Thanks - I agree that balance is really important. Also, in many cases -
especially where the founder is a user - intuition can be a key early driver
of success.

~~~
beat
Absolutely agree. I'm a "scratching my own itch" founder, and building a
product based on years of doing the job of my target users. "Scientific
method" isn't terribly useful for me as I build the initial product.

On the other hand, I don't necessarily know what this technology is going to
mean to end users. They're going to find problems to solve that I didn't even
imagine. As we're closing in on a beta, I'm doing demo/interviews as often as
I can (a couple of times a week at this point), and seeing where their minds
run. As William Gibson said, the street finds its own uses for things.

Once we get the product in the hands of real customers, it'll be far easier to
start taking measurements and figuring out what to work on from there. But for
now, intuition is the only tool I have.

~~~
mwseibel
This is one of the reasons why YC pushes people to launch MVPs so quickly -
the speed of learning is much faster when the product is in customer's hands.

------
soneca
I actually always translated the lean startup model as the scientific method
for startups in my head and while explaining to others.

My question: how do you apply that when you have tens of visitors/users? Not
nearly enough to have a reliable ab test.

Edit: so far he answered only one question. Not much of a AMA, right? Maybe
remove it from the title.

~~~
wefarrell
It's not as scientific, but you can prepare a/b tests for paper prototypes and
ask your users to go through them as they would go through the site.

Quantitative methods aren't reliable with such a small dataset so you will
have to rely on assumptions and guesses. Unfortunately this makes your
observations prone to bias from your hypothesis, so it's especially important
to be mindful of this and keep them separate.

Edit: Didn't realize this was an AMA but hopefully my answer contributes to
the discussion.

~~~
soneca
Sure, it does! I just added it to register that is weird to publish as an AMA
and not answer many questions.

I think your approach is right. One thing that helps me is to use hypothesis
from users/visitors/customer as much as possible. Talk to customers and try to
distill what are their hypothesis about a specific problem. Talking to
customer goes a long way.

On the topic, I think that famous quote atributed to Ford "If I asked my
customer what they wanted, I would be trying to create faster horses". I think
is BS (and probably bogus). That ad absurdum scenario would only happens if he
asked the wrong questions and/or followed the most superficial answers.

~~~
mwseibel
Its funny - I think the Ford quote is right. I always tell folks to not ask a
customer what they want but ask them what their problem is. Its the customer
task to describe and demonstrate their problem - its the founder's task to
think up and test solutions for that problem.

~~~
soneca
Thanks for answering below!

About the quote, it kinds of depends on what do you mean by "want", but I
understand and agree of your approach (while still disagreeing with the quote
if that is possible).

BTW the quote is probably bogus, as the first Ford cars were slower than
horses. 45km/h vs 88km/h (world record)

[https://en.wikipedia.org/wiki/Ford_Model_A_(1903%E2%80%9304)](https://en.wikipedia.org/wiki/Ford_Model_A_\(1903%E2%80%9304\))
[https://en.wikipedia.org/wiki/Horse_gait](https://en.wikipedia.org/wiki/Horse_gait)

~~~
joekim
According to Patrick Vlaskovits and others, there's no evidence that Henry
Ford actually made the quote. So, I would agree that it's probably bogus.

[https://hbr.org/2011/08/henry-ford-never-said-the-
fast](https://hbr.org/2011/08/henry-ford-never-said-the-fast)

[https://www.quora.com/Did-Henry-Ford-actually-
say-%E2%80%9CI...](https://www.quora.com/Did-Henry-Ford-actually-
say-%E2%80%9CIf-I%E2%80%99d-asked-my-customers-what-they-wanted-
they%E2%80%99d-have-said-a-faster-horse-%E2%80%9D)

------
marciojmo
Hi Michael! What problem do you think a game company is solving? What would be
reasonable key advantages at a company level? I.e.: What do you know/have that
Zynga (or even smaller studios) don't? lol

~~~
mwseibel
I think gaming companies are solving the problem of consumer entertainment and
I think in that world a/b testing is super important.

~~~
marciojmo
Yeah! I agree! I asked that because I think it's hard to distinguish yourself
by solving this general consumer entertainment problem. Seems that all game
companies do that, even when thinking about small studios, it is hard to find
differentiating factors that does not rely on the game itself.

I also agree that A/B testing is super important! We are following exactly
what you have described here [http://www.michaelseibel.com/blog/product-
development-cycle-...](http://www.michaelseibel.com/blog/product-development-
cycle-fundamentals) but for a game, and the results are pretty interesting!
Specially because we don't need a GDD (a document describing the game), all
members of the team are involved in the creation process (and they feel more
motivated to work - and believe me, this is big problem in indie studios) and
is easier to propose changes and improvements to the game, since them just
need to bring us closer to our goal with the game (for example: 20k ad
visualizations / day). I'm anxious to get into this A/B testing phase! We
haven't launched yet.

Thank you so much for sharing the knowledge! These posts were really valuable
and helpful for me! And I think they are complementary!

------
LukeFitzpatrick
I wish the article had "screenshots" to show what was being changed, and why.
It would help people to visualize and apply the steps better to their
startups. Excellent read!

------
mwseibel
Hey folks - btw - I'll be in and out of the thread all day and I'll try to
answer every question - so keep them coming and check back in :)

------
yoshyosh
Thanks for doing this Michael! Any advice on getting people to pay for
something before it's built? I think a lot of people will say they will pay a
certain price for something but when it comes to signing up once they can
enter their credit card, that doesn't end up being the case. Are LOI's the
main way, and does that work for smaller projects < $100/mo?

------
qwrusz
Thanks for taking the time to do this.

My Q: When one a/b tests there is generally a winner, but a better option
might be possible if the team spends a bit more time brainstorming and
hypothesizing. Do you have any thoughts on when to go with the results of the
a/b test vs when it's worth going back to the drawing board to try to
brainstorm a better idea?

------
mtrombetti
Thanks Mike! Great post.

What people often don't realize is the amount of inspiration you get by
discovering your hypothesis was wrong.

------
CptJamesCook
Michael,

How much racism have you faced in Silicon Valley? Do you ever face racism from
companies going through the YC program?

~~~
mwseibel
In operating my startup or working at YC I haven't faced any overt racism (in
the real world just living life that is a different story). Have I faced any
covert racism? - very hard for me to tell because so much of startups is
facing failure and there are often many probable causes of that failure. Not
sure what you mean by the second question.

------
ObsoleteMailMan
Hey Michael, thanks for the post. It seems like most big companies started out
of projects people were making for themselves in the early days.

Why do some projects evolve into behemoths, and when do you advise people
building projects to start talking to users?

~~~
mwseibel
I always tell folks to start talking to users from day 1 :)

------
montbonnot
A shorter and cheaper version of this would be to create an emotional
connection between your users and your product. You need to become a user and
figure out if you feel accomplished after using your own product.

------
revicon
What are some best practices for evaluating ideas before building anything.
You mentioned talking to users which is good advice but assumes that an
industry has been identified already.

~~~
mwseibel
The first thing I ask a founder is how often does the target customer have the
problem they are looking to solve and how much money are they willing to spend
to solve it. The more often the customer has your problem and the more money
they are willing to spend - the easier it will be (all other things being held
equal) to get customers and iterate. Sequoia calls these problems hair on fire
problems. The second thing I ask is who else is trying to solve the same
problem and what do you know that they don't.

~~~
adentranter
This is great advice and advice I wish I taken onboard earlier. As a technical
founder Ive found myself making a large set of assumptions which are usually
proved to be incorrect when going to the market.

Going into the market and talking to your customers early on so you can answer
those two questions will really help tell you if you have a viable idea.

------
LifeOfRio
Hi Michael, how does one conduct customer research? What questions do you ask?
How do you know you're doing it right? Are there any good books on customer
research?

~~~
mwseibel
Different answer depending on whether you have a product built or not. Also
different if you have a consumer product. My best personal experience with
customer research is for live consumer products. In that case we would recruit
potential customers off of craigslist (offering $50 for an hour of their
time). We would bring them into the office and sit them at the table. The
entire team (small) would be in the room. We would give them a scenario and
ask them to complete a task in our app in front of us. For example, you saw a
really good video on Facebook made with this app and now you want to download
that same app and make a video for yourself. Then we would ask them to
complete that task asking us any questions they had and sharing their inner
monolog out loud. Some questions we would answer but most we would ask them to
figure it out. It was very important that everyone involved with building and
designing the product was in the room experiencing the user test live. We
would take notes on issues the customer faced and conduct about 5 interviews
in an afternoon. Anything that came up 4-5 times we would attempt to fix. We
would rince and repeat monthly.

~~~
jonwachob91
>>> It was very important that everyone involved with building and designing
the product was in the room experiencing the user test live.

Was there ever a point when you had too many cooks in the kitchen?

------
muratsu
Hey Michael, apart from custom internal analytics tools and solutions similar
to GA which other tools would you suggest using?

~~~
mwseibel
I always loved Mixpanel - but its been a couple years since I've been active
on the product side and there are now tons of great tools. In the end of the
day - use the events based analytics product that your team will enjoy looking
at everyday.

~~~
kumarski
snowplowanalytics.com - What do you think of this one? It's open-
source...pretty crazy.

------
taanayya
We are new to the startup scenerio.We have a great customer following.But we r
looking for our seed fund /guidance to start it up;Its a platform to connect
,facilitate and empower moms.What would be ur suggestion for us to march ahead
,its a real world problem we r trying to solve .We dnt have an app and an mvp
to show as our business model is slightly different.what would be ur
suggestion to march ahead?We want to launch asap.

------
vrstartup
Is there a legal risk of using freelancers when building product version 1.0?
What is the correct way?

~~~
mwseibel
Yes - recruit a technical co-founder :)

~~~
vrstartup
We are a technical team. But still need extra body to speed things up :) What
do you do in such cases?

