
People are still bad at gauging their own interview performance - nns
http://blog.interviewing.io/people-are-still-bad-at-gauging-their-own-interview-performance-heres-the-data/
======
fecak
>some portion of interviewees are losing interest in joining your company just
because they didn’t think they did well, despite the fact that they actually
did.

This is the key takeaway for me. This is IMO and experience probably based on
a "sour grapes" type defense mechanism, where candidates will internally try
to talk themselves out of wanting a job that they felt they didn't get. They
walk out of the interview feeling they didn't perform, and instead of
regretting the lost opportunity they start to focus on even the most minute
"negatives" about the job/company.

The more time you give a candidate to stew about those reasons they don't want
the job, the more time they'll have to find reasons that may not even exist.

Positive feedback should be given almost instantly. Some companies and
candidates are reluctant to give positive feedback quickly because they feel
it may hurt their negotiation leverage.

I don't have data on this, but anecdotally (over almost 20 years in
recruiting) I've seen this countless times while collecting post-interview
feedback from candidates. The ones that feel they did poorly are likely to
mention a negative about the interviewer or company, which becomes somewhat
awkward when we come back with a job offer.

~~~
nsxwolf
>some portion of interviewees are losing interest in joining your company just
because they didn’t think they did well, despite the fact that they actually
did.

Wait. They think they didn't do well, got an offer, but turned it down because
they think they didn't do well on the interview?

~~~
fecak
Usually, in my experience anyway, it doesn't get that far.

The blog post just refers to interviewees saying "I wouldn't want to work with
the interviewer" when they mistakenly feel that they did poorly in the
interview.

Anecdotally from the real world, when I (headhunter) get feedback from a
candidate shortly/immediately after an interview, if they feel they did poorly
they tend to add that there were things they didn't like about the opportunity
(a person, the office, something). People that feel they did well don't
usually go negative as easily.

But I also go back to the employer and get their feedback ASAP, and when that
comes back positive I immediately pass that message to the candidate. Assuming
the employer's feedback is somewhat prompt, the candidate doesn't get too much
opportunity to go negative, and once they "feel wanted" by the employer they
usually trend back positive. Offers come shortly thereafter.

~~~
sksnxjis
It blows my mind that any applicant gives negative feedback to the company
before finding out of they got an offer. Even if they hated the interview and
would never want to work there, an offer is still fodder to get a better offer
somewhere else. Being positive is a strictly dominant strategy before an offer
is made.

~~~
fecak
My comment was as a 3rd party recruiter, and the blog post was based on
feedback from anonymous interviewees after anonymous interviews. I don't
expect that candidates would give negative feedback to a company directly
without having a sense of the employer's opinion.

------
RyanCavanaugh
I would draw the opposite conclusion from that data. The mode difference was
0, and 91% of people were within 1 star of the interviewer.

Without knowing what the interviewer's bar is for any particular star level, I
don't see how the interviewees could do any better. Rounding to 1-star
increments amplifies relatively small changes -- if the interviewer rounds up
3.6 to 4 and you round 3.4 down to 3.

~~~
throw_away_777
Yeah, the way the data was presented was not informative. The heat maps are
interesting to look at, but don't really convey the information well - the
colors are basically useless. You have to hover over everything, at which
point it should be presented as a table or a better picture.

It is pretty clear from one of the plots that interviewers and interviewees
grade on a different curve - the mean interviewee scores themselves lower than
the mean interviewer does. But once you correct for this, it seems that people
are actually pretty good judges of how well they did. I'm honestly surprised
by how low the R-squared was, I wonder if they included an intercept?

------
Sleaker
>some portion of interviewees are losing interest in joining your company just
because they didn’t think they did well, despite the fact that they actually
did.

Maybe this is because the interviewee feels partly abused, as if communication
isn't a two-way street. I think interviews would be less terrible if
interviewers actually told you at the end that you did above/aswell as/below
expectations, and/or why instead of just waiting 1-2 weeks before getting back
to you and then telling you that you are/aren't moving on to the next step. A
huge part of the interview process produces emotional ups/downs that no one
wants to deal with and which ostracize potential employees, but the
interviewers don't really do much to help with this.

I also think that having a 1-4 range is not a good guide for gathering
sweeping datasets. Adding any amount of numbers will help to show actually how
much. Or actually using human language to define them would be better. 1)
Terrible/Unhirable. 2) Unhirable/Not Passing 3) Passing 4) Exemplar would
actually be beneficial right now arbitrary numbering doesn't help to give
people much context as to what they should choose, and the numbers are so
close that you're going to get outliers where maybe some people think that a 2
is passable but not great, and interviewer thinks 3 is passable but not great,
or vice-versa.

~~~
fecak
>I think interviews would be less terrible if interviewers actually told you
at the end that you did above/as well as/below expectations

This happens quite often in phone interviews. My clients have regularly told
candidates "We'll be inviting you in" at the end of a phone interview.

The issue with this for on-site interviews is that they tend to be with
multiple individuals/groups, and getting those groups together to discuss the
candidate's performance while the candidate is still on-site is a mess
logistically. It would be ideal, but usually there will need to be some
discussions among the panels.

~~~
bcbrown
> usually there will need to be some discussions among the panels.

The way we've handled that (startup, ~30 devs) is that the VP of Engineering
goes last, and will go around getting quick reactions from everyone on the
loop. If everyone says "I'm all for hiring", the VP will go ahead and let the
candidate know we'll be making an offer. If anyone says "I'd like to discuss",
then we don't give feedback until after getting together to discuss the
candidate.

~~~
fecak
That's a solid idea, so long as the VP trusts the devs' judgement and ability
to make a decision on something this important quickly. The one issue I see
with this is that you'd need the "I'm all for hiring" to essentially mean "Not
a single red flag - perfect candidate", which I'd assume must be the case.

There are lots of positives in providing that immediate positive feedback -
showing the ability to make a quick decision, organization, and not least
making the candidate feel wanted right away.

------
sushid
> Therefore, every bar above 0 is impostor syndrome country, and every bar
> below zero belongs to its foulsome, overconfident cousin, the Dunning-Kruger
> effect.

Dunning-Kruger effect is attributed to "a metacognitive inability of the
unskilled to recognize their ineptitude." [1] Seems like a harsh accusation
for being off by a single star.

And the author makes a generalization that the imposter effect is "better"
than Dunning-Kruger effect. I don't clearly see why this would be the case for
the interviewee.

~~~
sushisource
I'm so mind-bendingly sick of these two terms (Dunning-Kruger and Imposter
Syndrome). They're very much in-vouge as of late and it seems like an
overwhelming number of /r/programming and HN posts have at least some
reference to it.

Pulling some piece of psychology jargon out of thin air doesn't make it
immediately applicable or accurate. Presumably, the terms get used so
frequently for the "look how smart I sound" effect, which serves only to piss
me off.

I would categorize these two groups as "overconfident" and "under-confident",
at most. They're not suffering from some genuine long term psychological
condition just because you think it sounds cool to use the terms.

/rant

~~~
Myrmornis
Yep. You can pretty much tell when people use those terms that they're more
driven by a desire to use those terms than out of having anything interesting
to say.

~~~
AstralStorm
Isn't a single instance not indicative of a syndrome? The syndrome presence or
not can only be inferred if the same candidate heard multiple times on various
interviews is consistently under- or overvaluing. And by a noticeable amount
too.

------
p4wnc6
Maybe I'm missing something, but why isn't the implication going the other
direction?

In my experience, the problem is that interviewers have no idea how to
correctly value a candidate's performance. Maybe the candidates are closer to
being well-calibrated, but their self-assessments don't match up with the
interviewers' because the interviewers don't know how to gauge what they are
looking for?

Making the assumption that an interviewer knows how to measure the response of
a candidate, even in cases of extremely quantitative questions with well-
defined answers, is highly suspect to me. I think virtually no one knows how
to do that effectively.

~~~
GoToRO
:)) Just the other day a manager said "HR is overwhelmed. They have a student
that filters the CVs and she is doing all she can." The senior programmer
exploded "How is a student any good at filtering programmers, she does not
know anything about programming!".

~~~
hnal943
Based on your comments in this thread, it sounds like you may need a new job.

~~~
GoToRO
I already have it. Not a long time ago I was searching for a job and this is
how I learned all these things. Because I always learn :).

Strange experience.

------
Kephael
These interview training programs seems to be really reminiscent of coding
bootcamps and appear to have grown rapidly in a very short period of time. I
think I'd almost prefer interviewees being judged based on the schools they
attended and companies they worked at. I'm not sure that I find the ability to
cram for data structures and algorithms questions to be indicative of
intelligence and problem solving skills. Schools attended and companies worked
at could be a very strong potential signal of applicant performance and not
involve several rounds of "implement that algorithm from memory". Are
interviews becoming more difficult as a result of the proliferation of all the
interview test prep material?

~~~
kristopolous
Algorithms and elite schools doesn't tell me whether you'll use the right-
sized solution for a problem and be able to prioritize and focus on tasks
accordingly. It doesn't tell me if you can perceive and extract requirements
well and have a process of communication and a professionalism of management
in how you go about getting to solutions that work for the humans that will be
using it.

It also doesn't tell me about your personal value system and whether you are
passionate about creating good user experiences or if you are just interested
in a technical playground or if you are a social climber or what.

It also doesn't tell me if you are highly opinionated and want to do things
your own way or if you are flexible enough to deal with the imperfections of
the real world. I don't know whether you value the mantra that the perfect
shouldn't be the enemy of the good.

No, none of these brain teasers or abstract bootcamp style things tell me that
- and those are the actual things that will make or break a company and
determine whether your product will hit the market on time. Those are the
things that will get you a customer base and will create a company culture of
productivity and not something that doesn't lead to results.

Until interviews become more about how people approach work and their ethic
and process of conducting it (which is a major part of classical engineering),
these are ineffective tools for finding the right people.

\---

[1] although as a disclaimer, I strongly prefer companies of under 20
employees. If you have 200, then the employees can be more insulated from the
exposure to these types of risk and the necessity of this type of vertical
thinking. But it's an "insulation" \- everyone is working either directly or
tangentially on something that is intended to be used by something in the
world (usually humans).

------
GoToRO
I often wonder how would these people hire soldiers? "So I see you have no
previous experience in killing people... We are looking for people that have
at least 5 years of experience in killing people."

My point is you need smart enough people for your job and nothing more. You do
an IQ test, a personality one and that's all. The rest can be learned.

~~~
rifung
Not that I disagree with you, but sadly, most companies don't seem to want to
train people anymore. They rather push that responsibility to either
universities or the applicants themselves.

In fairness, I do think that someone who is very comfortable with a language
or framework will be able to do things much faster than someone who isn't.
Considering how quickly people change jobs nowadays, I can also see how
companies don't want to train people only so they will leave shortly after or
even before they're useful.

Obviously this is just based on my own experience in the US.

~~~
GoToRO
I agree with you about the training. However if the stack is too specific,
then the stack it's like a fingerprint and they are looking for somebody with
the same fingerprint...

~~~
GoToRO
@Jemmeh I "failed" a similar interview (VB, C#) Don't they know that the VB
and C# examples are shown on Microsoft's site side by side?!

The only conclusion I reached is that one should navigate in life in such a
way that your well being should not depend on people that are less smart, less
experienced, less curious than yourself.

~~~
sshumaker
I think this is probably because there is still a stigma associated with being
a VB programmer. If you knew F# instead you might have been hired anyway.

------
delphinius81
Once you have an engineering job, a typical project tasking involves: some
team design discussion, some engineering research, some work, some review,
more work. The requirements are either known upfront, or determined as a team.

I do not understand why interviews are not conducted in this manner. Pick some
arbitrary idea and have the candidate work with the interviewer to design
something, then let the candidate do a little implementation research/design,
review with the next interviewer, wipeboard code and review some part of the
project with next reviewer. Have a coffee break at some point to get to know
the person. Treat the interview process like a normal day at work.

~~~
dogma1138
That's a good idea but logistically is hard, both requiring the interviewer to
spend considerable time on site as well as costing your own employee's time.

It might work well for a small company especially with a small number of
candidates but it doesn't really scale, unless you are a really big company
(googleish size) that can dedicate so many resources to the interviewing
process.

Doing design and research takes time, you can't do it in 30 min, you can't
also ask someone to spend 8 hours interviewing, and if you have 10 candidates
you can't spend half a man month or more on interviewing them either.

I've seen some companies that "interview" by contracting candidates for 1-3
days of work and paying them for it, but it really only works on positions
where you hiring effectively interns/fresh graduates, but as most candidates
especially those who end up in the face to face phase are currently employed
you can't really do that.

~~~
delphinius81
I dunno, I've personally spent on average half a day (3-4 hours) with on-site
interviews as a candidate. When on the other side of the table, it's been
about the same (~3 hours). With a tractable enough problem, I think you can
fit it into a half-day. You could also begin the design/research process as
part of a phone screen, have the candidate further prepare on their own time,
and then present/discuss as part of the on-site, along with the more in-depth
technical stuff.

After you've run this process enough times, your interviewers will have a good
grasp on the various solutions and technical problems. Most people's training
on how to interview comes from how they themselves were interviewed - which
was probably poorly done. Improve the interview process and you can train
people to be better interviewers just by being interviewed. :)

------
Ologn
After having interviewed dozens of people, the idea occurred to me that my
ratings for interviewees were a Gaussian curve with a normal distribution.
Most people blended into each other - they knew the topics the same amount.
Enough to be employed in their current job. Then one in six was a standard
deviation below or worse, and one in six was a standard deviation above or
better.

That might be one reason for the impostor syndrome. Because if we interviewed
six people, one would be unqualified, four would be OK but not spectacular,
and one would know the subjects better than anyone. So in an interview of six
people the best qualified one of the six would have the job. The average four
were not truly impostors, but would not get the job unless they had an inside
recommendation or if we were really needed someone and there was a shortage of
candidates coming in.

------
Apocryphon
I understand companies are unwilling to give negative feedback to applicants
for fear of legal repercussion. But often not getting any form of feedback is
highly detrimental to an applicant who's invested days if not weeks of going
through the process, only to receive a boilerplate "thank you for applying,
please try again" response.

One wonders if it's possible to devise some sort of neutral, yet helpful,
assessment that could be given to applicants without fear of legal
retaliation. Or, if eventually an applicant will sue a company for _not_
providing feedback after they were rejected.

------
andrewstuart2
In related news, "People are still bad at knowing the unknown" and "Everybody
thinks they're average; perhaps because their sample size can only be 1."

~~~
EliRivers
"Everybody thinks they're average; perhaps because their sample size can only
be 1."

It's my understanding that this is not the case.

People typically think they're above average.

[https://en.wikipedia.org/wiki/Illusory_superiority](https://en.wikipedia.org/wiki/Illusory_superiority)
[http://www.livescience.com/26914-why-we-are-all-above-
averag...](http://www.livescience.com/26914-why-we-are-all-above-average.html)

------
Fricken
I've never interviewed for a job I didn't get. Possibly because I aim too low.
But if I were to present myself honestly as a potential employee, I would
enter the interview without greeting or making eye contact, then I would sit
as far away from the interviewer as possible and doodle in a notebook for 15
minutes before raising my head and explaining everything I think is wrong
about the company.

------
EdiX
Mmmhhh I would have drawn the opposite conclusion, most people estimate ±1
star away from the real result, it's not bad considering the actual
performance is not entirely objective.

Glad to see that once again Dunning-Krüger proved wrong.

~~~
tom_mellior
> not bad considering the actual performance is not entirely objective

It's not even clear (to me) what's being compared here. What's the "actual
performance" that the blog post very carefully does not define? The screenshot
highlighting the technical performance question seems to suggest that it might
be only that one point out of the three that interviewers fill in.

But it's completely possible to be technically competent and still feel that
one didn't do well _overall_ because one came across as nervous or boring or
whatever. In other words, the comparison here seems to be between apples and
undefined oranges.

------
collyw
Surely a lot depends timing and circumstances. One interviewer said they gave
me the job based on asking if the guys accent was Dutch (he was Norwegian and
found that hilarious). It was a maintenance programming job and had a high
turnover of staff. Other interviews you may have done well but had some
serious competition.

------
nfriedly
I used to do a lot of interviews for front-end engineers. The process I
settled on was to give them the following setup:

\- some mock-ups from when we first designed our product gallery

\- a json file that listed the products names, pictures, description, etc.

\- a blank css file

\- a html file that loaded the css and jQuery, and then made an ajax request
to load the json file. (This was back when jQuery was pretty much state of the
art for front-end.)

With all of that in place, I'd explain that I'd like them to work on building
a product gallery that looked similar to the mockups. I would further explain
that, since we only had an hour, I didn't expect a polished or even fully
functional product, but I'd just like them to dive in and see how far they
got. They could use their editor of choice and ask any questions they wanted.

I felt like this gave better insight to candidates skill and workflow. My
colleagues all gave more traditional interviews, so I also felt like there was
a good counterbalance if someone didn't do well with this kind of interview.

~~~
xxbondsxx
Doesn't that heavily bias towards candidates who have used jQuery before
and/or built a product gallery before?

Skewing interview performance based on some specific previous experience might
be what you want if you expect your candidates to hit the ground running, but
might not be the best to gauge long-term performance.

~~~
nfriedly
This was back in the day when everyone used jQuery for everything, and the
company was only looking for experienced people, so a candidate who hadn't
used jQuery before probably wouldn't have been a good match for the position
anyways. I don't think a single candidate had that issue though.

As far as the product gallery goes, I wasn't looking for a pixel-perfect
product. Just spit out a bit of html and matching CSS that look vaguely like
the mockup. The mockup images had things like the hex values of the colors
written on the image to make that part easy.

Basically, it was trying to focus on things that would be the main part of the
candidate's job there. The folks that we did hire all did well on both that
interview and long-term at the company.

------
merb
That there still needs to be discussions about that topic. On a workplace
people should be treated equally. No matter how many topics will be brought
up. A world with different genders, stereotypes, whatever can only work if
_both_ sides won't interpret too much into it and just enjoy their work with
their colleagues.

Actually we don't employ a woman (I'm working at a 3 person company). And we
actually hire, but the only woman will probably decline on their side, she
actually has experience in things we actually not need yet, so I guess we
can't provide her a new challenge. way probably the best application.

------
mark-r
Don't miss the footnote: "I’m always terrified of misspelling “Dunning-Kruger”
and not double-checking it because of overconfidence in my own spelling
abilities."

------
kafkaesq
What a hatchet job.

Unfortunately we don't have the raw data available; but all we can infer from
these heatmaps is that yes (and exactly as one would expect) there's a bit of
variance between between the two sets of ratings (in both directions). Maybe a
bit more than can be explained by the sample size, or from the simple fact
people (on both sides of the process) are forced to discretize their
assessments (so even if both parties were to objectively agree that the "real"
performance was in the 2.4-2.7 range, which it is a good chunk of the time....
inevitably you'll find them be off by +/\- 1 in their quantile assignments a
good chunk of the time -- which is exactly what the data seem to show).

In particular: independent of how we explain the variance (whether due to
sampling effects, or bona fide DK/IS) note that the vast majority of the
variance is in the +/\- 1 range. And on the alleged DK side (-2 variance) the
measure is apparently quite miniscule.

However, that's not the original D-K effect. What D-K specifically predicted
(and were able to show with much better data and methodology -- in their very
specific sample population, at least) is that, _among the general population_
there will be a striking degree of -2 variance, specifically in the bottom
quartile. Going by the oft-quoted abstract from their original paper:

 _Across 4 studies, the authors found that participants scoring in the bottom
quartile on tests of humor, grammar, and logic grossly overestimated their
test performance and ability. Although their test scores put them in the 12th
percentile, they estimated themselves to be in the 62nd._

In other words: "Performers in the bottom quartile estimated themselves to be
performing, on average, at mean performance in the third quartile."

So if were to look for something analogous to a D-K effect, we'd expect to
find a "hot spot" (or at least the strongest amount of red ink, for that
quartile) in the upper left of the first chart.

Instead we find most performers in the bottom quartile (correctly) identifying
themselves as in the bottom quartile; with a smaller portion bleeding up into
the second quarter; and an an almost negligible portion placing themselves in
the D-K zone -- namely, the third quarter.

If anything what the data show is that (even assuming there's nothing to
contend with in how the actual data were taken), while inevitably there's some
variance (both ways) in the two sets of rankings, and probably some of due to
distorted self-perception -- on the whole there's actually _much less_ of a
D-K effect among engineers than among the general population (or rather,
"Cornell undergraduates taking psychology classes" \-- which, lest we forget,
were the actual subject class used in the original D-K study).

So basically the opposite of what the author of the original article is
claiming to infer from this data.

------
matmann2001
Probably because interviewing companies have stopped giving performance
feedback.

