
Encouraging interviewees to say “I don’t know” improves performance - dncrane
http://morehappy.me/2013/10/08/encouraging-interviewees-to-say-i-dont-know-improves-performance/
======
lmkg
I think part of this conditioning from taking tests in school. For twelve or
more years of your life, in any situation where you are asked to demonstrate
your knowledge, a null response is treated in the same negative fashion as an
incorrect response, and getting outside help is disallowed. No wonder we try
and guess, we've literally been trained for years that it's the best approach!
Compounding that, an interview is the one situation in "the real world" that
most resembles a testing situation from grade school, which will bring back
all those bad habits.

~~~
tootie
Actually, the SAT (American college entrance exam) penalizes wrong answers
more than non-answers. A non-answer gets you no points, but a wrong answer
gets points subtracted.

~~~
iyulaev
Barely. IIRC it's 0 points if blank, -0.25 if wrong, so if you eliminate 1
choice it's worth it to guess. I'd like to see it be -5 points if wrong, since
this is more reflective of the real world. When I entered the industry as an
electrical engineer this was a difficult transition for me. In school, you're
incentivized to guess and BS. But in the real world, the penalty for being
wrong is HUGE.

~~~
wwweston
> in the real world, the penalty for being wrong is HUGE

In _some_ situations (committing an integrated circuit design to manufacture
probably being one), the cost of being wrong is pretty big.

Other situations where there's a tight feedback loop and you can
make/propagate changes quickly (say, most web app software development), it's
less so.

~~~
iyulaev
True. But generally the cost of guessing and doing something wrong is much
higher than saying "I don't know" and asking someone else or Googling the
result. School, through homeworks and especially exams, trains us to do our
best without outside help. I would argue that this behavior is maladaptive to
engineering in the real world.

------
patrickyeon
Some commenters, and possibly the author, want this advice to extend to job
interviews, but the reason an interviewer is pushing you to answer a question
isn't that they want you to be right or wrong. A good interviewer wants to get
a look into how you think and how you approach a problem.

An even better interviewer will make that clear, by telling you "there's no
trick here", "I don't know the answer myself, let's see what we can figure
out", or "there's no one right answer, I just want to know what you can see
here". A trick I've employed a fair bit lately to get a reserved interviewee
to start working with a question is "What is the worst solution we could
provide to attack this problem?" I'll even possibly go as far as offering my
own horrible solution, and asking them where we fo to improve on this.

And I do mean worst. I haven't met a candidate yet who can't at least throw
out ideas on how to improve my horrible solutions, and at that point the ball
is rolling.

~~~
jlgreco
> _" And I do mean worst."_

I am not convinced that throwing INTERCAL at a candidate is a good way to
roll. ;)

But seriously, I think that you are spot-on with this approach. Interviewing,
done effectively, is about getting a fleshed out idea of the capabilities of
the candidate, not just getting correct answers. Perhaps the best response to
give or expect, when the interview gets stuck, is _" I don't know, but..."_
Even on questions that aren't coding questions, these are often good
responses: _" I don't know, but [this is how I would find out]"_ or _" I don't
know, but [this is a topic or problem that I believe is related]"_ .

~~~
jaggederest
For what it's worth, my 'worst solutions' always involve a guy with a van
driving around hassling people in person. Somehow people never accept my
proposed solution, so I haven't been able to test it.

------
asgard1024
I have long suspected this to be true. That's my argument against mandatory
voting, because if people can choose not to vote, they can express they don't
know and hopefully leave the result on people who feel they do know.

However, it's a bit unclear how this ties together with Dunning-Kruger effect
(which is also used as an argument against democracy). The D-K effect would
suggest that if you allow people to express doubt, the experts will doubt
more, and overall performance will decrease.

It would be interesting if someone did a psychology experiment on that. (I am
actually in favor of doing these kinds of demonstrations in high school,
because it's in my opinion important for people to understand how democratic
voting works, for example the fact that voting won't get you a simple average
of the results - which is usually used as an argument against democracy.)

~~~
IanCal
> Dunning-Kruger effect (which is also used as an argument against democracy)

Democracy isn't about having an efficient system, it's about having a stable
system. Democracy is an awful way of getting things done, it's just a
reasonably good way of stopping very bad things from happening.

~~~
nether
Favorite quotation from Kelly Johnson, former head of the Lockheed Martin
Skunk Works:

> Voting on everything prevents anything very stupid from happening, but also
> anything very brilliant.

You can guess who called the shots in his aircraft design teams. He did,
because he _was_ truly fucking brilliant.

~~~
IanCal
Yes, exactly. It's inefficient, that's the point of it.

He didn't have the same control over things as a dictator would. You can do
these things in business more safely because it's less likely to kill large
numbers of people.

~~~
nether
Yeah, that's the value of a "benevolent dictator" in business. Skunk Works had
one, Apple, Python, Ubuntu, and SpaceX too, and all blazed incredible trails
in their fields.

------
croddin
When reading the title, I assumed this was talking about job interviews, a
much different type of interview. How do these ideas apply to job interviews?
Is it a good idea for an interviewer in a job interview to encourage
interviewees to say I don't know, and should you be more ok with saying I
don't know and then elaborating in a job interview?

~~~
bennyg
On the job interview point:

I'm always as truthful as possible in a job interview - especially technical
ones. My background is not in tech, however I'm an autodidact with programming
and whenever technical questions are asked that I may not know, I'll say it.
And then I'll say how I think it should/would be done, or the steps I'd take
to figure out the answer and solve the problem. Not sugarcoating some of my
technical inability has faired me well so far. And then after I hear the
question, think about the steps to do it, and/or get coached by the
interviewee - I know the answer, and am ready to use it in my
technical/creative arsenal. I feel like it's a win-win to not bullshit about
something I don't know.

------
samograd
I'll usually ask at least one question in an interview to see if the candidate
_will_ answer with "I don't know" or ramble on incoherently making something
up. Guess which I prefer.

------
beat
One of my best technical interviews was from a lead engineer who simply asked
more and more difficult questions until he found your limit (believe me, he
knew more than anyone he would interview). His interest wasn't in what you
knew so much as how you reacted when you hit the limits of your knowledge.

There's a lot of value in that approach. I don't use it myself when
interviewing people, but I do try to accomplish the same end.

~~~
mistercow
>believe me, he knew more than anyone he would interview

He wouldn't really need to. By asking questions about very domain-specific
knowledge, he could still achieve the same effect with anyone who didn't have
a freakishly similar set of expertise.

------
triplesec
Corollary: "There's no such thing as a stupid question", because you're not
afraid to be wrong. All creative and innovative people are like this, without
exception, IMO. As Richard Feynman said in the title of an autobiography:
"What do you care what other people think?"

Everybody wins when people leave their egos behind.

~~~
ape4
Sure. But its not so hard to come up with a question that sounds pretty
stupid. What is Monday? Can I eat your shoe for lunch? Why do you have to
interview me before hiring me?

------
Spoom
I've found that the ease with which someone answers "I don't know" has a
positive correlation with their wisdom in general. (Of course, if they answer
"I don't know" to everything, they probably just don't care, and that's a
separate problem.)

~~~
yeukhon
I don't necessarily disagree with you. I think it is true that having too many
I don't know means the person is probably not familiar with the area he's
facing. The I don't care part is probably a bit rare. You probably have to
look at body language (i.e. facial expression, for example).

------
gcb0
One thing i always ask on that chat before the trivia questions, is what tech
news sites the candidate read. And where he usually goes to were he wants to
know the correct argument order of a function he does not use too often.

That usually gives me a better grasp of how he will solve a real world problem
than the trivia questions.

But when hiring for teams that need to get specific work done fast, i do not
underestimate the value of the trivia questions as it will show me the ramp up
time to get that work done. Of course i apply that on top of the other means
to get an idea if the candidate is a good fit overall.

There is no silver bullet for hiring.

------
mathattack
The answers may get better, but don't you want to find out which candidates
are more likely to bluff when they're wrong? The point of the interview isn't
for candidates to give you answers that are as correct as possible. If this
were the case, you'd mail them a list of questions, and ask them to get back
to you in a week. One goal of interviews is to find out how people behave when
they don't know. This is why I try to take people to the end of their
knowledge, no matter where that end is.

------
adamconroy
In an interview i have never had a problem saying that I don't know something
or that I do know about something but can't recall the specific answer.

I normally try and turn that scenario into a quick discussion where I
hopefully let the interviewer display their vast knowledge and insight.
Assuming the question is relevant I am always geniuenly interested in finding
out about something I don't know. Seeing that I am not being paid to take the
interview I feel it is only fair that I get something out of it.

------
yeukhon
Nice study. I think it's quite intuitive. When you are forced to spell you are
force to say something. So by saying "I don't know" it is natural to continue
with "I don't know but I think" and then it continues with more words.

Whereas to tell someone in advance "don't you dare to tell me I don't know"
the person will just hold on to the thoughts until he or she has to spell that
thought out (because he or she has to say something to get out of the misery).

------
judk
This is obvious. See also the child abuse panic of the 80s, where kids were
pressured to make up stories of abuse. And the general fact that eyewitness
testimony is unreliable.

------
ufmace
Being comfortable with saying "I don't know" a lot is something I had to kind
of train myself to do. It's part of realizing that this is a truly enormous
field that we're in, and nobody could possibly know more than a fraction of
it. Anyone who doesn't say it much is either spending all of their time in a
subset of the field that they know very well, or is throwing around bullshit,
either from fear or bravado.

------
WhitneyLand
How about just ditching contrived interview questions because companies have
no good science or data to establish the correlation with hiring the best
people?

It seems past performance would be more reliable. What have you created and
what was your role in it? Can you articulate your passion for the problems we
are solving? Can you show that your colleagues respected your work and
benefited from collaborating with you?

------
m0th87
A good interview question should give space for exploration - it shouldn't
involve esoteric questions or things that are easily google-able. Good
interview questions will give you insight into the candidate's thought
process, not what they've memorized beyond fundamentals. So the results of
this study seem irrelevant.

------
pron
It's a nice idea, and it makes sense, but the sample size – 26 people per
group, multiplied by 2 for two experiments – is too small to reach any
definite conclusions.

------
agumonkey
lucidity as the fastest path to clarity ?

