Never hire a product manager off of "spidey-sense product instincts and creativity", because you won't be able to tell a candidate who's got some skill from a candidate who's been having some luck.
A product manager shouldn't just be able to tell you good ideas - a product manager needs to be able to tell you how they come up with good ideas, and how they figure out if those ideas are right before they blow a bunch of resources on things.
If a product manager is trying to get hired on instincts, no matter how good their ideas sound, the appropriate thing to say is 'Okay, let's say my instincts 100% disagree with yours, and I happen to be your boss. Now what?' And if they can't articulate exactly how they're talking to customers, determining market needs, validating their assumptions, and doing whatever else they can to prove they're not just some charismatic lunatic advocating a fever dream, do not let them in the door - because they are a potential time-bomb.
Some people do have good product instincts. But if you've accidentally hired a product manager of the lucky variety, their performance is likely to revert to the mean - after they've persuaded your organization to sink a huge bunch of time and money into their dog of an idea that sounded good at the time.
I've hired 11 PMs (from Associate to Senior level) at my startup, and a lot of this rings true. The one thing I'll add is the most important step in my hiring process:
After phone screen, I have the recruiter send them a case study that asks them to use my product (a social game, this works better for consumer internet, obviously) and identify the most important features for monetization and to suggest improvements to the game.
1) Because the prompt is somewhat open ended and they generally have at least a week to work on it, how much effort they put into it is a great indicator of how much they want to work at my company, which to me is one of the most important things I'm interviewing for once basic intellectual horsepower is sufficient. Responses typically range from "typed some stuff into the Word doc" to "50 page PDF with screenshots, charts and graphs."
2) That also gives a concrete basis for discussing product ideas. In addition to the obvious questions of "why do you think we implemented this feature" etc, I also often take suggestions they have for the product, and ask why they think we HAVEN'T implemented it, particularly if it's a idea we've considered in the past.
Judging candidates based on their effort for an exercise that focuses on your existing product is a touchy subject. Many candidates (especially good ones) will view this as spec work - if you want their expertise for a limited time, you should pay their consultant fee. Otherwise, the effort they put in often shares more about process than free product ideas.
But that's speaking broadly. Anecdotally, I know a PM who put her heart into an exercise like this for a company she really wanted to work for, which chose not to hire her, and a few months later released a product update that matched her ideas. I don't know if they already had the product in the pipe or if it differed significantly in execution, but it did invite an easily-avoidable dispute. Even if not a legal issue, it could have been a PR one.
Of course you want to gauge candidates' understanding of and engagement with your real business, so don't throw all those conversations out the window. But when it comes to early vetting exercises, focusing on competitor products or similar products by companies in other categories can still tell you a lot about how the candidate approaches and understands real challenges/opportunities/constraints while skirting some gray areas.
We've probably brought in 20-30 candidates for interviews, none of them brought that up as a concern (though maybe it was for the people who half assed it). Realistically, the chance of them coming up with a brilliant product idea in a few days that the full time teams on the products hasn't considered is pretty small -- in fact I don't think I've ever heard a materially unique idea. Granted the game/product I use for the case is a pretty mature one that has been actively worked on since 2011, so a lot of the solution space has been covered.
We often discuss competitor products in the interviews as well -- I consider it a strong positive signal if they can already meaningfully benchmark against competitors. But the reason I don't make that the core case is because we have so much more insight into how our product works and why it is the way it is, so it's easier to validate their instincts/thought processes against reality.
> Note: I wrote this in 2005 when I was at JotSpot. Google acquired JotSpot in 2006. Since then, I’ve had the opportunity to work with some marvelous PMs and have performed 200+ PM interviews. I’m sure that my opinions have evolved, but the intervening years have only further reinforced the characteristics of great PMs. I occasionally set out to update this essay but I always decide to leave it as is. (—Ken, February 2013)
>Lots of people were hired at Yahoo! that probably wouldn’t have been
>appropriate at a startup. I recall a lot of post-interview conversations that
>went something like this – “well, I’m not sure they’re the perfect candidate,
>but they do seem suited for this very specific role, so let’s hire them.” That
>may work fine at a big company, but it’s deadly thinking at a startup.
It's deadly thinking at a big company too. It's just that a big company is, well, bigger, so the effects of the poison take longer to show. But, with time and with enough bad hires like those above, the effects will begin to tell on even the largest of companies. The best companies don't think in terms of, "Is this person well suited for this role?" They think in terms of "Is this person well suited for the company?" At places like Amazon and Google, there is no such thing as "would hire, but not for my team". And the converse, "would hire, but for my team only" doesn't exist either. Why? In both cases, that caveat means that the person is so specialized, they're going to have a hard time adapting months or years down the road when business priorities have changed and the team they were hired for has been reorganized out of existence.
This paragraph also struck me as very interesting. I wonder how the atmosphere and mindset regarding hiring at Yahoo! is today and whether Marissa brought Googles more healthy philosophy into the company.
I use to run a product team at a company that was acquired. Interestingly enough, all the candidates with Yahoo PM background that I interviewed I didn't like.
I think hiring PM can be rather easy and I used these steps to find a great candidates:
0 - make the candidate feel comfortable and at ease. Start with topics you might have in common. Hobbies, teams, movies, etc. some of the best candidates might be the most nervous people at interviews.
1 - eliminate slick talkers. Don't fall for articulate BSers and constantly be on guard if you maybe falling for one. Judge their substance not their delivery.
2 - undrestand how they think. Ask questions or carry the conversation in a way that they have to solve a simple problem.
3 - the most important one. Do a simple product meeting with them. If you follow item 0, you should kow what they are into. Ask them to build a simple app or site for their specific hobbies or interests. This gives you a chance to see how good of a fit they are and how they think and solve problem as a PM.
Lastly, experience was never as important to me as passion, raw intellect, curiosity and instinct.
We are in the process to build one of the best ressources on Product Management on the web right now and Ken's essay was a perfect fit. It's really hard to find good advice on hiring Product Managers so I've reached out to him re re-publishing his essay :)
Also make sure to check out what they are doing over at the Google Ventures Design Staff blog:
http://www.designstaff.org/
Most feed readers should be able to find the blog's rss feed. But you can also use our feedburner url if you want to be more explicit: http://feeds.feedburner.com/blossomio
We also have an email newsletter on the right-hand side of the blog (I'm still looking for feedreaders that also support email newsletters btw if anyone knows something in that direction).
A product manager shouldn't just be able to tell you good ideas - a product manager needs to be able to tell you how they come up with good ideas, and how they figure out if those ideas are right before they blow a bunch of resources on things.
If a product manager is trying to get hired on instincts, no matter how good their ideas sound, the appropriate thing to say is 'Okay, let's say my instincts 100% disagree with yours, and I happen to be your boss. Now what?' And if they can't articulate exactly how they're talking to customers, determining market needs, validating their assumptions, and doing whatever else they can to prove they're not just some charismatic lunatic advocating a fever dream, do not let them in the door - because they are a potential time-bomb.
Some people do have good product instincts. But if you've accidentally hired a product manager of the lucky variety, their performance is likely to revert to the mean - after they've persuaded your organization to sink a huge bunch of time and money into their dog of an idea that sounded good at the time.