Yup. The author made a poll of fellow FB employees. As if that is even remotely representative of the overall market of the 25 to 44 year olds. Just consider the average salary of the polled population vs the general population.
Startups are street racers: you never know who you’re racing next to. They might have bigger sponsors than their sleeper suggests. A NASCAR analogy would therefore apply to traditional VC centers like SF, NYC, etc. It’s hard to pass someone before a turn at 200 MPH, but if you don’t do it, plenty of other people can and will.
Practically anyone can build a racecar, or a startup. The race puts proof to the lie: not everyone can compete, and not everyone wins. The first issue is systemic; the second: hard work, capital, and dumb luck.
Life, like business, hopefully doesn’t have a checkered flag, but it’s a fact of life all the same. All anyone can do is race their best to win, for whatever your own personal winning conditions are. Choose your racing team wisely!
But... you are not competing with Lewis Hamilton if you stay at the local go kart track! Plenty of room for small fish. He’d love to come and beat you but he can not afford to, his time would cost millions.
And yet I bumped into a childhood hero of mine, Leo Laporte of tech broadcasting fame, at an impromptu meetup he did at a bar in SF I heard about on his Twitter. So as I lived in SF at the time (~2012 I think?), I just went and met one of my role models.
Sure, you don’t know if you don’t go. But you also don’t know if you don’t know to go. Can going be taught? Isn’t that what startups do?
I did the thing. I’m still in the race. Am I winning? I know I am because I never raced you. Or maybe I am right now? If I lost, would I even notice? Should I care to win? At what cost?
The post was factual, as are all of my posts, in my own overwrought efforts to be satirical, witty, clever, or argumentative, in the spirit of debate of course.
My point is, I guess if I could even find a common thread, is this. Don’t stop racing. It’s what makes you want to be who you’re not. If that doesn’t inspire you, go meet your heroes and their friends and associates and nearby hackers, and get back to me.
The trick isn’t to build a faster car. The trick is to build something more fun across the street so going to the race is less interesting than your thing.
It's not that the author is affiliated with facebook.
The point is that for most startups, you validate your ideas from your kitchen table, fighting to get a handful of visitors to visit your site.
When facebook launches a product at blasts an infinite firehose of users at the product.
So the facebook experience is utterly different to the startup experience and really its kind of absurd to be sitting in facebook advising startups how to do what they do at facebook, they're such disconnected experiences.
Maybe the topic of the article should have been "how to validate a new product coming out of one of the biggest tech companies that ever existed, advice for other product managers from FAANG".
I just can't believe you're talking about Facebook like it's some restaurant franchise.
Never mind that your math doesn't even make sense. Even if only 1% of users retain after a year, 1% of 100m is 1m more than your 90% retention of 0 users.
Being inside FB (or any other large company) has way more advantages than users entering the door.
Just the metrics and analytics that FB can gather on experiments is an insanely huge hill a regular startup can't climb. FB can gather and analise data at a scale, on so many markets and devices and demographics that none other plan tier at 3rd party integrations can hope to match. The SRE support you get, a 30 minute conversation with other great product managers, engineers, designers one meeting invite away, a history of what others have tried and what worked and what didn't, and so much more.
And if you get a VP to support you, even for a bit, we're basically talking about a torrent of money and resources being thrown at your project while you still drawn a 6 figure salary (plus benefits and bonus and stocks!). It's really apples to oranges here.
A number of the comments are saying this is bad because from Facebook, numbers are unrealistic ($100mm?!), signs tell you nothing.
Personally, I've done startups longer than I want to admit (10+ yrs). If I was to start another one, the play book would look basically like this article.
Idea/Thesis, Hypothesis, Target, Test and some where later - build.
The problem I had when I started and why so many founders get it wrong is the steps usually look like - Idea => Build or just ... Build.
There is a lot of good advice here. Sure the numbers might appear insane (just replace, $1m, $300k, $300), but don't let that stop this from being a good framework. The goal is to remove as many assumptions about what you're supposed to build as fast as possible because chances are, more than not, the assumptions are wrong from Idea => Build. This is a good way to validate a number of assumptions and it's only just the beginning
The only thing I would have added is - ask for Money.
Ideas will iterate very quickly (or die) once money is involved.
Idea/Thesis, Hypothesis, Target, Test and some where later - build.
This is a common strategy if you just want to 'do a startup' but it misses the need. Every successful business I can think of started with a founder needing a product to exist. All the startups I've been involved in have been products I wanted to exist because I needed them myself, or I had a close relationship with someone who needed them. Turning them in to something that could make money was a key goal, but I'd have built some form of them anyway even if it wasn't.
If you're just thinking up things people might want and throwing up a landing page to see if people claim they're interested I suspect you're going to get bored quickly, and you'll move on to the next even better idea very quickly.
That's not to say it can't work if you're making something just because 300 people signed up to your Launchrock page, but I think the motivation to succeed won't weather the storms that your startup will face if that's the only reason you have to build something.
Everything I've built on my own has been something I wanted. In some cases they were more complicated than I thought so I never got to the customer validation stage, but that wasn't quite the disaster it could have been, because cash wasn't the (only) motivation.
Except...when it was, and yeah, my bad. So on the one hand I look back and kick myself for not first probing the market, but then I also remember that I did probe it, found there wasn't one, and didn't really care.
It can be a fine line between building a "me-too" product and something that no one wants.
the whole Lean Startup methodology is snake oil IMO. it is used primarily by consultants to lure big corps into "innovation methodologies".* While its certainly an improvement over status quo, results were mostly meh anyway. you just cant force good ideas
the best startups are the ones mentioned by OP. visionary founder who needs that stuff himself.
* i was director at a small 7mm/yr consultancy and did precisely that
The main point of lean startup isn't to have better ideas, it is to identify and discard bad ones faster, and focus your limited resources on the wheat among the chaff.
I'd caveat this with the note that many unicorns, including a couple I've worked with, didn't solve any problem that a solution wasn't available for - they instead solved it better/more cheaply or in a new way. This is especially true in the B2B SaaS market where any large addressable market likely has incumbents you'll be fighting against, even if only tangentially.
There's a lot to be said for working on solutions to problems you have yourself. But I think that also leaves a lot of very fertile ground untouched - the problems that startup founders tend not to have, but many other people do.
I don't know that this advice is so realistic anymore, or at least the target click thru rates should be adjusted. Ads on FB now have very low click-thru, and even then, I'm not sure how much that ultimately tells you about your product or market. You can spend a lot of money going through all kinds of different customer segments on FB, or you could recognize that there are other ways to attract customers than social media ads.
Of course none of these can tell you the 'why' -- it might be something simple that you would never have thought of, but you end up pivoting too early. Or perhaps your marketing isn't polished. Or you're way early/late to a field. If you knew the 'why,' that can be worth far more than blowing a huge cash load on FB. I feel like techies flock to such techniques because it's easy and fits into the scale techno-narrative. It means that many will be mislead.
"Test" doesn't necessarily mean spend on FB/G Ads. It means devise a test feedback signal is quick, low cost and yields a meaningful/measurable metric.
An idea selling a more efficient ship engine to ship owners is never going to do well in Ads, but there is a test that is more cost effective than build engine => sell engine.
> Of course none of these can tell you the 'why' -- it might be something simple that you would never have thought of, but you end up pivoting too early.
This is a founder dilemma and the thing founders need-to/should worry about. Don't build until you've got at least a decent, objective signal the idea is worth the cost of building, but how you get there is a doozy.
Maybe it is because I'm older, but whenever I see this advice to build a fake landing page and see if people sign up... I recall that selling vaporware was reviled in the 90s, but somehow now it is considered a wise business tool. It still feels scammy to me. If you have a market in mind, but don't have people in that market to talk to directly to validate your idea, you are aiming at the wrong market. Otherwise all these techniques prove is that you can collect emails, not that those emails will actually become paying customers.
You're not alone. This has always felt scammy. It's a little more gray-area than some of the traditional vapor-ware which was more about driving competition out of business by large players, but still something I try to avoid.
I'm older too. Maybe the issue is if the central premise is vapourware or components/add-on features So maybe if the author automated the metadata collection, ran some filtering on it and then had a fitness pro sanity check the results? I agree that this oft-recommended "idea validation" process sounds a lot like this scene from Silicon Valley, without the clarification at the end:
Just clarify your perspective - Kickstarter, Indiegogo, and other crowdsourcing platforms are even more scammy, because they go the next step and ask for money, not just an email?
Yes, exactly. There is nothing wrong with honest communication, stating that you are thinking about building a product. The problem is when people write pages acting like the product already exists, collecting emails under false pretenses.
Even if slightly unrealistic, I actually think the nicest part is the back of the napkin math and derivation of the numbers to give you context for how big a market is and what you’d need to attack to meet your goals.
I do however have an issue with the wizard of oz method of validating your product. It’s not because of the perceived ethics and I totally understand too many people today build something before ever knowing anything about whether anyone will use or pay for it. Hopefully I didn’t miss it but nowhere does the author discuss the amount of time/people/money it would take to build a working MVP, not to mention how much of your time will be tied up from working on said MVP if you’re oz (i.e. the man or woman behind the curtain until the MVP is built). This is important since it drives how you (or at least I) would move forward: Do you need to quit a full time job and spend 3-6 months 60+ hr/week building your MVP, can you build an MVP in a few weekends, would you need outside help/money to build an MVP, etc etc.
The example in this post mentions a recommendation engine of sorts for individualized fitness programs. This sounds like a fairly high amount of expertise in the field of fitness, some possible AI with trained models sprinkled in and a good amount of curated fitness routine info or I guess individual exercises based on the equipment the user enters that they have. So a lot of time getting with experts (or becoming an expert) AND building a new machine learned model possibly with a nice, usable UI. This sounds like a full time job for 5+ people for 6+ months at least to get any sort of real prototype.
I started reading the post but the moment I saw big numbers in hypothesis I felt like the author has no clue how hard reality is. Facebook is a great cocoon to try random numbers. Outside that founders take months to even find the right set of people who fit your need space.
Just talk, generally, about pains you are trying to solve. Everything else is quite unnecessary.
Did you know that even founder of Zapier had to get on a call to explain their first adopter how the product actually worked? Yeah these are early days. Listen to their podcast of IndieHackers.
I'm trying to understand your feedback. Perhaps I'm missing something.
The point of the big numbers in my post is not to look at them in absolute terms. But to look at the ratio A/B. And then apply it to a much smaller audience. Because that tells helps validate that your idea will meet your expectations (whatever they may be).
I read the comments first, and then eagerly read your post - because I hate Facebook and wanted to scorn you, but having read it I see that it's not really Facebook-centric at all, containing only a glancing reference.
That said, I do think there are problems with your use of statistics and you kind of do need to consider the numbers in absolute terms. If I check my idea out with ten people I know in my target demographic, and five of them like it, I probably shouldn't expect 50% of the target market will be interested in my idea. My sample size is too small and also not really representative of the demographic because there is a selection bias at play - they're people I know, so maybe I know the only five people in the world who have the same problem I'm trying to sovle.
Likewise, in your example, you talk about showing a concept to people on a Facebook social group, but people on a Facebook social group aren't really representative of your target demographic either. "Only 20 bucks a month? Sounds great, I love interacting with apps!" may be something a Facebook engineer would say, but that may be less likely from the mouth of the average 25-44 year old.
I do think testing your ideas out and seeing how users respond to them is a good practice but there is inherently danger to doing tests and coming up with statistics if you believe your own tests. You might run the experiments you describe and feel good about your numbers and invest time and money to go to a wider audience only to find out that a lot of people think 20 dollars a month is too expensive for an app that gives workout advice (which, I predict, I could find on the current app store for free) or that they don't like working out at home or from an app, etc.
If you know you don't know what's ahead you'll proceed cautiously. If you falsely believe you know what is ahead due to biased statistics or insufficient sample sizes, you may blunder painfully.
I also caught a few references to Thiel's "Zero to one" in there, but I feel the approach you're describing, at least the hypothetical example, is somewhat contradictory to the advice in that book. I'd characterize your approach as:
1. figure out the market - (Big number, everyone interested in fitness)
2. guess what piece you can serve - (small number - only 2% need to try!)
3. figure out if you can serve them - (small number - maybe 10% retention)
In other words, "We just need a small piece of this big market!"
From Zero to One:
>Now I think the opposite version of this is always where you have super big markets and there is so many different things that went wrong with all the clean tech companies in the last decade. But the one theme than ran through almost all of them is that they all started with massive markets. Every clean tech powerpoint presentation that one saw in the years 2005 to 2008, which is the clean tech bubble in Silicon Valley, started with where the energy market, where the market was measured in hundreds and billions of trillions of dollars. And then once you're sort of like a minnow in a vast ocean, that is not a good place to be. That means you have tons of competitors and you don’t even know who all the competitors are.
>You want to be a one of a kind company. You want to be the only player in a small ecosystem. You don’t want to be the fourth online pet food company. You don’t want to be the tenth solar panel company. You don’t want to be the hundredth restaurant in Palo Alto. Your restaurant industry is a trillion dollar industry. So if you do a market size analysis, you conclude restaurants are fantastic business to go into. And often large existing markets typically means that you have tons of competitions so it's very very hard to differentiate. The first very counterintuitive idea is to go after small markets, markets that are so small people often don't even think that they make sense. That's where you get a foothold and then if those markets are able expand, you can scale into a big monopoly business.
In other words, if I looked at "2% of people who saw my Facebook post left their email" I'd probably think the same thing I'd think prior to doing any experiment, which is "There is a lot of fitness stuff, including apps and gyms, this probably won't work". But, I'm also a pessimist, so maybe that's just me.
Anywho - sorry if I've been too negative. I did enjoy reading your post, and I hope you hit the target numbers you want on your substack subscriptions from this post! :)
I thought the same thing. These kind of analysis will point you towards already flooded markets (how many fitness apps there are already? My local gym has one..), when you’d be way more interested in going after markets unknown to most people.
So, if your goal is the fitness app market, maybe start with a more specialized segment, like parents of small children (because pushing a stroller increases the effort) and expanding from there (say, to seniors with mobility issues).
This is a neat idea, and there may be some enlightening bits of feedback, but it won't tell you what matters, and that is if people, in a specific audience will like the product.
Sign ups and likes are very noisy singals.
There's just nothing that can take the place of market validation, so the 'new standard' model of 'release early' is probably the best way.
Sometimes it takes a while to find the right user base and to get fit.
I’ve done ok with a link to a calendly on the email marketing campaign - you’d be surprised how many people we up for a chat! And as a bonus it solves the awkward timezone issues for me and im not tempted to take 2am calls to be nice!
The last step where you manually do what the product is meant to do really works in 1% of products. Think of all the successful products today: Docusign, Snowflake, Wix, yeah.. not going to happen.
It’s 2020. All the low hanging fruit ideas are gone. Any meaningful product idea requires a bit more complexity in implementing it.
New technologies expose new low hanging fruit. The act as what Taleb calls "half-invented" in AntiFragile, a working implementation applied to the wrong problem. The approach of "service first" or concierge or Wizard-of-Oz is still viable and widely used. I agree it does not work for all product types, but it's still a very useful way to defer a lot of implementation expense but still determine product desirability.
I know several people who would pay good money to have lengthy and repetitive forms (school permission slips for field trips) filled in 'automatically' and doing this manually should be a cinch!
Really? Paying for someone to fill in forms? If it’s trivial like a field trip, I don’t see anyone paying for that. If it’s important like a legal document, you dont want someone to fill it in for you.
I thought the same, but I saw the size of the form for the field trip and the detail they were asking for (most liability issues to absolve the school of responsibilities) - though, I don't think it would bother me personally enough to want to do it.
And important legal documents... Wouldn't you hire a lawyer or a solicitor to deal with them?
I think even a will - I would probably pay a nominal fee to have one generated for me, on the assumption that I can modify the final version before I personally sign it off on it.
Though the point wasn't really the forms, the point was more that there is a near infinite level of low hanging fruit, it's just not always apparent based on your own situation (and to an extent country!) and set of tasks/frustrations.
Another example that came to mind would be automatically updating all my stuff when I change address (banking details, car registration, driving license etc etc) sure there is a huge trust issue with this but with care, probably not totally insurmountable.
Not sure I can add anything more to this discussion than any more that has already been said, but the real problem about validating ones idea is,” you start out with an extreme bias” towards your idea. Conversely, if you are validating an idea, you the only true and tried and tested way is to Mom test it. It’s not enough you run a statistical poll and use that to run a product development program. You have to actually talk to people. In this case, OP needs to actually talk to the people in his demographics and validate the idea that they would like to get a personalized email from his trainer with a custom plan. If not it’s all shadow dancing with no real validation. Mom test is an excellent book to drive this point home.
Was on the phone and am not good with touch typing. But, I actually did what I mentioned above. Being stuck at home like the OP, I actually had a lot of time on hand. So, I started calling Coffee shops that had recently pivoted to Shopify model(that change is happening in a massive scale) to try and see if they would be interested in a subscription model for their coffee beans and grinds. The outcome is for a different day, but the moral is "Always try to Invalidate your idea". Your ideas are innately biased. And that's why Starting with a clearly defined pinpoint is way easier than to apply Lean Startup methodology to Your "ideas". Hope that helps. Cant recommend Ryan Fitzpatrick's Mom Test enough. Guy even has a YT channel where he expands on some of the things he talks about in his book.
Something I am starting to learn about myself: I don't care if nobody wants it, I still want to build it. Not a great trait for making money as a startup, but it's an itch I just have to keep scratching...
I have the same itch and keep scratching it almost daily. I can say scratching this itch over the last several years has absolutely been +EV overall. Maybe not in a “I made $XXXX in sales over the last month/year/whatever” kind of way but in that I’ve learned an incredible amount, have been able to use a lot of the things I’ve learned at my job and have definitely improved my overall market value as an engineer (validated with data I should say, not just an opinionated, arrogant rant).
That said, I would keep scratching that itch if you still have the motivation and drive to keep learning/building without seeing immediate material value like money right away.
I've done this before, it also appears clearly described in the "Start small stay small book" which I recommend a lot. However after running it I found myself a little lost, conversion was not as expected and it's very hard to realize what failed. Is it the idea? or the design? or the speed of the website? or the wrong target audience? or the wrong keywords?.
More importantly should I jump to my next idea? or should I invest more time pivoting this one and pushing until I get where I want to be.
What happens when you're building an improved version of something that already exist? Would you skip this kind of validation assuming that if people pay for the big product they would pay for a small one? Specially if you're not aiming for a venture backed monster but just a bootstrapped small business.
Agreed this is a problem that your product needs to solve either way. This idea validation playbook allows you to face the problem early on as opposed to later.
RE: What happens when you're building an improved version?
I'd ask you why you think yours is an improved version? Is that a sure shot fact or up for the market to decide. If it's sure shot, they yeah skip the idea validation step. Sounds like your idea has more execution risk than market risk.
But if you think its subjective, then that's probably something you could find a creative way of solving using the Painted Door test or something similar.
Nice article. Most people make this mistake of building products first and then finding customers for it. By following this approach they build something that nobody wants. Ideally one should validate the idea and market fit first. One shouldn't be building something for which there are no customers at all.
Follow this approach:
-> demo
- sell
- build
And not:
-> build
- demo
- sell
I bought this e-book recently on Gumroad and loved how it is right on point about how to strategically validate your business and generate sales before you even build it.
Maybe this is good advice for mass marketed consumer apps, but I don't believe putting up a cheap SquareSpace template and faking automation instead of building the product is a good strategy for getting initial enterprise customers.
If the author ends a statement with a question mark, I find it very hard to take them seriously. How can they be insightful but not thorough enough to check a title? Does that make me a bad person? I mean just that, on its own?
On the contrary, I have come through some articles / blogs that have delivered insightful answers to the question that led you to it in the first place. Doesn't make you a bad person, but you should definitely clean your judgmental glasses.
When we say 'startup idea', it adds too much noise to the initial validation process as the industry(Successful Entrepreneurs, Investors) almost unanimously agree that the first step is solving the right problem[1].
So shouldn't the first step always be 'How to validate the problem you are trying to solve?'
In my early life as an entrepreneur, I've failed multiple times since I didn't understand this difference between Startup ideas vs Problems. Understanding the importance of solving the right problem i.e. the problem with enough need gap that people are willing to pay for getting it solved and of course the problem which we want ourselves to be solved played a huge role in me building successful products later on.
When I started coaching entrepreneurs a year back, I found this again to be the common theme for failure among early stage startups i.e. building something just because they can and hoping everyone needs it. It became a professional necessity for me to come up with a way to validate problems/need gap.
So, I started building needgap[1] a problem validation platform where problems are the first class citizens. Posts are created by the consumers(incl. entrepreneurs) who have a problem, want to find a solution for their problem, check if others share the same problem, how ever small the problem might be[3]. Builders can post their products in the comments if it solves that particular problem or else discuss with others to create a product to solve that problem(There are couple of products built/being built for problems from needgap).
I'm not saying I've solved the issue of validating problems, far from it. I think I now have a direction towards understanding the 'language, grammar of problems and startup ideas' which might one day result in making validation easier.
This looks similar to the logline pitch suggested in Save The Cat by Blake Snyder, or really most books on scriptwriting. Hell, it could even be said for writing your thesis. Always focus hard on your idea, or you logline, as it were, or the premise of your thesis. Once clearly defined, worded and outlined, it can both keep you on track and lead your project, and it may serve as a basis to get early feedback on your idea from the real world.
A painted door test lets you gauge interest in a new feature, product, page, or piece of functionality … before you’ve built it. It involves updating your site with a link or callout that promises the new feature, then measuring clicks to see how many visitors would try it if it existed.A painted door test lets you gauge interest in a new feature, product, page, or piece of functionality … before you’ve built it. It involves updating your site with a link or callout that promises the new feature, then measuring clicks to see how many visitors would try it if it existed.A painted door test lets you gauge interest in a new feature, product, page, or piece of functionality … before you’ve built it. It involves updating your site with a link or callout that promises the new feature, then measuring clicks to see how many visitors would try it if it existed.
Whatever thing you are building at Facebook starts out with 10 hurricanes of wind behind it.
Startups begin with all the wind of a baby blowing a raspberry.