
Mistakes startups make when talking to users - ckelly
http://survata.com/blog/10-mistakes-startups-make-when-talking-to-users/
======
TravisLS
I thought this was going to be a post about customer development, where the
issue is not generally the formatting of the questions, but asking the wrong
questions in the first place, or misinterpreting the answers.

Surveys can be a useful tool, but nothing replaces a real conversation. Even
asking more open ended questions in an email or survey form can give you so
much more information than radio button answers (and unless you're a huge
company, it's not too hard to read a few hundred open-ended responses).

Along those lines, I'll share a snippet from patio11's Microconf presentation
last year (posted on HN yesterday), regarding interpreting the responses you
get to a more open-ended conversation:

"If you’re solving a problem people actually have, they will say at this
point, 'Shut up and take my money.' If someone says, 'That’s kind of
interesting, tell me when that exists,' you have not successfully identified a
problem that people actually have."

You don't get that kind of insight from radio buttons.

~~~
wpietri
Agreed. One of the biggest mistakes I see startups making when talking to
users is using surveys at all.

The theory seems to be "Oh, we'll get more information that way." Or, from the
more honest ones, "This was less scary than trying to talk with people one on
one." But surveys are mainly useful for confirming or denying particular
generalizations from real conversations.

Surveys throw out all sorts of data. Every raised eyebrow. Every excited look.
Every long pause where they try to figure out what they hell you are talking
about. Every gesture, every tone of voice, every tangent. You really need that
data when you're first starting out. Early on, the problem isn't the things
you know you don't know. It's the things that you're pretty sure about that
are false.

~~~
usaar333
I'd say it is more of a "this is more tractable". Given that the barrier to
doing a survey is so much lower for your users, the guys who accept "real
conversations" will probably be significantly more optimistic about your
product than those who don't..

~~~
Evbn
People who are optimistic at the only people that matter. A company with only
1% of the world population as customers is one of the greatest companies in
history.

~~~
usaar333
If there are enough of them. Most B2Bs do not have anywhere close to 1% of the
world's population as customers.

------
WA
While the headline doesn't really match the article, because the article talks
about surveys and not customer development, I want to point out one error I
see quite frequently in startups or SaaS-businesses when they talk to users.

It can be summed up as: Asking the user for some sort of interaction and then
failing to respond to the user's action.

This can be blog posts where there's an open question at the end with the goal
to engage readers and failing to answer to interesting comments.

But I see it more frequently when I go for some sort of SaaS-trial, my account
expires after a week or two, they ask me for feedback and since I'm not
convinced of the product yet, I might give them feedback and state my
concerns. That is actually the last chance for them to make me change my mind.

And they miss it. They simply fail to reply or to address my concerns that I
stated in my feedback.

I don't have any numbers but my educated guess is that they lose quite a few
customers this way. Some lost me this way and I bet there are others.

~~~
mackwic
This comment worth gold. I respond it only so that I can see it later. Anyway,
thanks for pointing this out.

------
adrianhoward
If you want to get better at writing surveys - go read "Asking Questions"
[http://www.amazon.co.uk/Asking-Questions-Definitive-
Question...](http://www.amazon.co.uk/Asking-Questions-Definitive-
Questionnaire-Questionnaires/dp/0787970883)

If you actually want to look at how to get better at talking to users - rather
than writing surveys - I'd recommend getting a copy of Andrew Traver's pocket
guide "Interviewing for Research"
[http://www.fivesimplesteps.com/products/interviewing-for-
res...](http://www.fivesimplesteps.com/products/interviewing-for-research)

There's also an upcoming book on interviewing by Steve Portigal
<http://rosenfeldmedia.com/books/interviewing-users/> \- googling around his
name will find you some useful interviewing slide decks.

Indi Young's book Mental Models has some good advice on interviews in one of
the chapters, and is also a useful read
<http://rosenfeldmedia.com/books/mental-models/>

(Bias disclaimer: I know Andrew, and I do workshops on interviewing ;-)

------
ChrisNorstrom
They forgot the #1 mistake: Too Many Damn Questions.

There have been plenty of times when I have willfully taken surveys to help
out companies I like then immediately gave up after being shown a survey with
20-60 questions. No.

What pisses me off is that they're asking for my time and effort without
giving me anything in return. Like they feel entitled to taking 15 minutes of
my time.

The worst is when they only display 1 question per page and there's a little
progress bar at the top. And after 10 questions you only see the progress bar
7% full. At least it's not as bad as the time Starbucks/Chase Duetto Cards
hired someone with a studdering speech impediment to call me for a phone
survey... That lasted 30 minutes.

~~~
chiph
If they can't refine it down to 3 questions, they haven't thought enough yet
about what they really want to know.

Also - it doesn't have to be the same 3 questions to each visitor. Draw them
from a pool.

~~~
Domenic_S
A nice platitude, but when folks from UX, marketing, sales, engineering,
bizdev, and executives all want to know different things - that's how you end
up with the 60-question surveys.

~~~
ChrisNorstrom
Well if I'm being paid for it, then there's really no limit as to how many you
can ask.

------
replicatorblog
For anyone interested in the art and science of surveys/customer interviews, I
found this book to be quite helpful: [http://www.amazon.com/Asking-Questions-
Definitive-Questionna...](http://www.amazon.com/Asking-Questions-Definitive-
Questionnaire-
Questionnaires/dp/0787970883/ref=sr_1_1?ie=UTF8&qid=1366916840&sr=8-1&keywords=asking+questions)
It's written for sociology/anthropology students, but the principles are
equally applicable in product conversations. It's filled with lots of best
practices for designing questions, interview flows, surveys, and other
customer tools. There's actually a lot of mistakes that can be made without
realizing and this book helps prevent many of them. e.g. switching up types of
questions from yes/no, to scales, to free response to prevent answers based on
momentum.

------
lawnchair_larry
The problem with this data is that it comes from Survata, which is one of
those god awful spammy survey walls that makes users fill out a survey before
they can access the desired content. When your data comes from being hostile
and annoying to users, I don't think it can be trusted.

------
nickburlett
Sometimes unbalanced scales (#4 on this article's list) are appropriate. If
you have a five-option list where 95% of your users are answering 4 or 5, then
it's time to unbalance the scale so that you can get better information.
However, you need to have collected the balanced scale version for long enough
to know it's an actual trend.

------
onemorepassword
I still couldn't truthfully answer most of the "improved" questions with
anything other than "it depends" or "none of the above", because they still
make too many assumptions about people's behavior and choices that simply
don't match the diversity of reality.

Pay for a streaming video services? Depends, 99% of current services I
wouldn't pay for. How often do I get a new mobile phone? Depends, sometimes
two in a year, but the current one is already 3 years old. Fuck if I know the
"average". How often do I check my mail per hour. Depends on where I am and
what the fuck I'm doing, of course.

Etcetera, etcetera.

------
jroseattle
Mistake #11: assuming that an online survey equates to discussions with your
users. This is not a knock on survata, but rather talking about what it means
to engage with a company's userbase.

Survata is one tool, but nothing beats direct, 1-1 customer interaction. Can't
remember where I read it, but one CEO of a startup made it his job to handle a
decent amount of customer support email. Of course, that only scales so far,
but out of the gate -- that's priceless interaction.

------
danesparza
Also: Not having an actual conversation with your customers

