
Engineers can’t gauge their own interview performance - leeny
http://blog.interviewing.io/people-cant-gauge-their-own-interview-performance-and-that-makes-them-harder-to-hire/
======
shoyer
I actually interpret their data quite differently. There is a statistically
robust and positive correlation between perceived and actual ratings:

\- 44% of the time, the actual and perceived ranking was identical.

\- 91% of the time, the difference between the rankings was at most one star.

\- Only in 9% of cases was there was a strong disagreement about how well the
interview went (2 or more stars out of four).

\- There is also a bias towards engineers thinking they did slightly worse
than they actually did.

Even in an alternative universe where the same interview was repeated, you
would not expect rankings by the same individual to be perfect. A more
meaningful benchmark (though probably unattainable) would be
interviewer/interviewee consistency relative to interviewer self-consistency.

~~~
andimus
The problem with that interpretation is that it ignores how much something
deviates from the expected random. If everything was completely random:

\- 25% of the time, the actual and perceived ranking would be identical.

\- 62.5% of the time, the difference would be at most one.

\- Only in 37.5% of cases would there be a strong difference in scores (2 or
more stars out of four).

Would you say that 44% is a good score on a multiple choice test? Obviously
candidates did better than random guessing, but not by enough to call it a
strong correlation. Determining the strength of a correlation is what
R-squared is for, which makes it the right choice for this analysis.

~~~
mbizzle88
But the problem with R-squared in this case is that a linear model isn't
appropriate for ordinal data. These rankings are ordered categories, not
numbers.

~~~
andimus
The comment I was replying to was not about ordinal data and I stand by it in
its context.

But on your point, the Jury's still out on what to do with Likert-type data
like this. Purists do say that it's ordinal and therefore you can't do any
numeric analysis on it, but practically speaking you can learn a lot from it
by treating it as interval data.

In my opinion, statistical analysis like this is ALWAYS two parts speculation
and one part science (if you're lucky). It's designed to give insight without
declaring fact so you're allowed to bend the rules a little if it's in the
spirit of the math.

My "gut" feeling agrees with the original post. If we treat the data as
ordinal data, then we can only look at the accuracy, and 44% is, if anything,
more compelling.

Cards on the table: I was involved in analyzing the data in the first place
and have a vested interest in things-- but I am saying what I believe without
any intentional bias.

------
jMyles
...another reasonable conclusion from this data is that _interviewers_ have
difficulty judging interview performance.

~~~
JFlash
Beat me to it. When I interviewed at Google there were multiple interviewers
and then there were multiple committees and people who look at their feedback.

Comparing interviewers to other interviewers makes a "He did above average"
from a person who's hard to impress as good as a "She did really great!" from
someone else.

~~~
robrenaud
Well, the hiring committees see the entire prior rating distribution from each
interviewer as well as the hires/no hires for each previous rating.

So the committees has a decent way of correcting for interviewers who use
different scales.

~~~
JFlash
Right. You'd only know someone was "hard to impress" with that information.

------
stcredzero
So many of you 20-somethings interviewing really make me cringe! Basically,
many of you are a stark mirror in which I can see how terrible I was as an
interviewer in my 20's and 30's.

Be honest now, do you do this?

    
    
        1) See something or think of something
        2) Decide that it's the most important point
        3) Go on a "fishing" expedition to try and get the interviewee to see it
        4) If the interviewee can't figure out what you're hinting at, decide
           s/he's completely worthless! (Or if she gets it, decide she's the ultimate!)
    

A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty
arrogant view of the relative value of your own perceptions. What you really
need to be doing is _listening._

I had the experience of someone doing this to me with concurrency, then
concluding I'm an impostor, _while I 'm telling him about 3 concurrent systems
I've worked on._ The disconnect? He was "fishing" for the current undergrad
spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in
the same way when I was an undergrad, and the poorly thought out puzzle you
came up with on the spot wasn't such a great hint.

Given the number of times I was mis-identified as an impostor programmer
during interviews (a minority of instances, but still) there is clearly
something going on. You guys are just like the young Ivy league grads in Mad
Men. You just have different clothes and different language, but you are
looking out for the signs of your own tribe just the same.

~~~
ArkyBeagle
Ah, the password game. You don't wanna work for those guys anyway.

It's interesting isn't it? This was all much easier in decades past. The
questions don't matter. The answer doesn't matter.

What matters are the body language cues.

------
invalidfunction
Eh - humans generally have a hard time rating things on a 5 star rating
system. That's mainly because the discrete intervals between 1 and 5 stars
aren't well defined so everyone will have a different opinion of what 3 stars
"mean". I encountered this when I ran a rating system in Mechanical Turk. If
you ask turkers to do something arbitrary like "rate the quality of this
picture" it takes quite a number of ratings on the same image to get an
accurate consensus.

It would be interesting to see if you can replicate experiment but replace the
5 stars with 5 yes/no questions regarding their performance and see if those
match up

~~~
dogma1138
Whats worse is that any type of rating system has an utter bias towards the
very very upper end of the scale by it's consumers.

If you have a true rating system then 2.5/3 out of 5 means average or more or
less what you should be looking for in most cases 4 is out standing and 5 is
bloody icecream pooping unicorn.

You see this every where a rating system is implemented heck in entertainment
movies and games don't fight from 0 to 100 anymore but form the 90-100 if you
see any score below 90 or 4 stars people usually avoid that product.

------
edw519
I never cared about my own "interview performance".

But I always cared about my interviewer's assessment of me.

So much so that I've always reserved the last 10 minutes of every interview
just to find that out.

They almost always ask if I have any questions, but even if they don't I ask
them anyway:

Based on what you've just learned:

    
    
      - How well would you say I did in this interview?
      - How well do my skills fill your requirement?
      - Where do you see me in the first month?
      - Where do you see me in 3 years?
      - What's the first thing you'd have me work on?
      - What gaps do you think I have?
      - What are your 3 biggest concerns about me?
      - What can I do to address those concerns?
    

and of course:

    
    
      - What is the next step?
    

I have never had an interviewer afraid to honestly (or appear to be honest)
answer these questions. See how easy?

~~~
NickLarsen
The last 10 minutes where you ask questions is a bad time to waste asking
someone to judge your 3 year plan when they just met you. Most of these
questions are better asked when you fail an interview and request feedback as
to why.

I specifically state when I perform interviews that I will not answer the
question "How well would you say I did in this interview?" The interview isn't
over yet, and that's impossible to answer.

You should really spend this time to see how much you would want to work at
the company. Ask about their processes, the things that would absolutely turn
you off to a company or the things that would absolutely turn you on to a
company and make you forget about all their other bullshit. Ask about the
problems they are trying to solve. Ask about how smart the other people are
who you'll be working with.

Don't waste time in the interview checking if you passed. You'll know very
soon anyway and there is always the chance to get this feedback afterward.

~~~
scott_karana
> Don't waste time in the interview checking if you passed. You'll know very
> soon anyway and there is always the chance to get this feedback afterward.

I think that would be a mistake. If the interviewer isn't able to answer those
questions satisfactorily, it might be a sign that the _hiree_ isn't a good
fit. Nobody wants to change jobs to something that doesn't work out.

------
kelukelugames
Are they being nice to me because I killed it or is it because I did so poorly
that they are showing pity?

Are they being mean because they think I wasted their time or because I killed
it and they want to push me?

------
joeax
I was once in an interview where the interviewer was a hard ass. Asking all
kids of hard questions about languages I knew, then he got deeper and asked
about specifics into those languages. Then started asking about really
abstract stuff like different sorting algorithms, OOP concepts like
polymorphism, etc. I felt I was holding my own but as the interview went on
the interviewer seemed to grow more impatient and irritable. Needless to say I
left the interview thinking I blew it. Two days later I got a call from the
recruiter saying I was their top choice and got an offer soon after.

P.S. It turned out that guy was just grouchy because he had interviewed so
many underwhelming candidates. He admitted he was just trying to break me.

~~~
pgodzin
Trying to break a candidate seems to be very related issue to the one the
article is talking about. Most people will take that kind of interviewer
attitude as a negative experience that reflects poorly on the company and
would make them less interested in taking the job.

The goal of the interviewer should be to evaluate a candidate, not dig and dig
until you find a weakness. If it takes that much effort it's not worth doing
in the first place.

~~~
joeax
Agree. Luckily attitudes have shifted over the years and companies realize
there's too much risk in alienating a good candidate.

------
soham
Terrific analysis, thank you! I love reading your posts.

I also think that the dynamics differ based on the mode of interviewing viz.
Faceless (like [http://interviewing.io](http://interviewing.io) does) vs face-
ful (like normal onsite or video interviews, or the way
[http://interviewkickstart.com](http://interviewkickstart.com) does).

I have an observation to share for the latter kind of interviews:

When I was hiring at Box for one of my teams (or even now), I'd generally go
towards the end, to meet the candidate and talk to them about how the day
went. I'd ask them about specific interviews, the interviewers and how they
think they did on the given problems.

I found that it was surprisingly rare to get a candidate who was aware enough
of how things went (positive or otherwise).

Digging into it, I realized that the correlation of what the candidate
perceived, was simply to how expressive the interviewER was. If the
interviewer was friendly and chatty, the candidate would feel like they have
done well (or at least forgivably well). If the interviewer was more silent,
the candidate would feel they haven't done very well.

Now that I do interview training for a living, I try and hammer this point
home: that interviews are like a date. It depends a lot on the other person
(in this case, the interviewer). Understanding this well often works like
magic.

------
exelius
Nobody can judge their own interview performance - engineer or not. The hard
part is that you as an interviewee don't really know what the interviewer is
looking for. You may have nailed the back half of the interview, but you don't
have the one skill they're looking for. Or sometimes you don't blow it at all
and they love you, but there's one other person they interviewed for the
position that blew them away.

Trying to guess how well you did in an interview is folly. Hell, one interview
tactic I use with people is to push them until they break: I keep asking
technical questions until I reach a point where you say "I don't know". This
sometimes makes people think they "failed" the interview, when all I'm really
trying to do is see how much you know while also seeing how much you'll
bullshit me on topics you don't know as well. Nobody can be an expert on
everything, and interviews are as much about behavioral profiling as they are
subject matter expertise.

------
cryoshon
I've found this with myself as well. In one of my worst-ever interviews, I had
the flu really badly the entire time, and read the signals of all of my
interviewers to be both negative regarding my technical chops and negative
toward my weakened state. I got the job, and was very well liked. In another
trainwreck of an interview where I seriously considered standing up and
leaving after getting extreme negative signals, I was offered the job as
well-- but I should have paid attention to the negative signals, as they were
indicative of larger problems with that group.

In other interviews, I've massively overestimated my own performance, seeing
positive signals everywhere and being very impressed with myself-- of course,
these didn't pull through, and my confidence was dashed. In another incidence,
I did get the job offer.

The most confusing are the interviews in which I don't have a strong feeling
of doing well or poorly either way-- not the median case, but certainly it
happens frequently enough that I've gotten a job where I didn't think that I
stood out. If anything, these cases are a consolation that others may see
something positive that I don't see in myself.

This tells me that my ability to correctly predict my interview performance is
no more efficacious than random chance. I think that there are a few
confounding factors which make interviewee measuring interview performance as
perceived by the interviewer difficult. Interviews are a time of endless
posturing, flagrant lies, propaganda, and overt deception hopefully mingled in
with actual mutual interest, excitement, and good will. There are many social
signals to keep track of in addition to self-monitoring to ensure the correct
outcome. For people who are not the strongest socially, I think that these
social signals tend to get dropped intermittently as attention shifts inward
(don't say the wrong thing) or outward (get this technical problem correct).

~~~
vdnkh
It's a fools errand to try and predict interview performance. I've been
interviewing for a short while now and haven't gotten an offer yet despite
feeling really great (and for some, really crappy) about my interviews.
Feedback from the interviewers is equally inaccurate. On my most recent
interview, I was told that my solution to their coding challenge was one of
the best they'd ever seen, and that the feedback so far was extremely
positive. Two days later I got the "another direction" email. I don't really
take rejection to heart anymore but I hate not knowing what I need to do
better.

~~~
ju-st
I ask myself after a job interview if I really like my potential tasks, the
people and the work environment. If "yes", then I will not get the job. If I
don't like it then I will get an offer. This metric worked for me the last 5
out of 5 times.

~~~
vdnkh
I've yet to be met with success but I'm getting closer. I'm refactoring the
way I present myself to emphasize the value I bring. I believe I was too
focused on how the company could benefit me. Not salary or perks, but
mentorship and training since I'm still early in my career.

The thing is that I already bring a lot of value - I could definitely bring
more, but I failed to emphasize what I already have.

------
vonmoltke
Hopefully this gets more love than it did last time:
[https://news.ycombinator.com/item?id=10739343](https://news.ycombinator.com/item?id=10739343)

I found the analysis very interesting and consistent with my personal bias (I
almost always think I did worse than I actually did).

~~~
dang
> _Hopefully this gets more love than it did last time_

We invited the submitter to repost it for that reason. I'll try to merge the
comments from there into here.

Invited reposts are mostly deprecated now in favor of re-ups [1], but when it
looks like the submitter might also be the author (as e.g. with Show HNs), we
still send them. It's nice for an author to know that their post may still get
discussed, and it's good for HN when an author jumps into the thread.

[1] All the details at
[https://news.ycombinator.com/item?id=10705926](https://news.ycombinator.com/item?id=10705926)
and previous posts linked there.

~~~
leeny
Author here. Very much appreciated.

------
imh
>And by extension, it means that in every interview cycle, some portion of
interviewees are losing interest in joining your company just because they
didn’t think they did well, despite the fact that they actually did.

I hate the assertion of causation here. As always, this relationship means
that A causes B, B causes A, and/or A and B have a common cause (or even more
possibilities, given that we're essentially conditioning on this event taking
place). Aline argues that "some portion of interviewees are losing interest in
joining your company just because they didn’t think they did well," but I
think it's more plausible that there is something that happens in the
interview that simultaneously makes the interviewer think they did poorly and
makes them think poorly of the company. If you're rating me on my ability to
name country capitals in a statistics interview, I'm going to think you don't
want to hire me and I'll think you're an idiot I'd hate to work for. Or if
you're being kind of a jerk or being impatient and checking your cell phone
every 30 seconds, I'll think I'm doing poorly and that you're rude and I
wouldn't like to work with you. Common cause seems much more likely to me.

------
lewisl9029
> "resumes suck"

Do they?

Maybe resume's specifically do suck, but surely it's beneficial to have at
least some way to see an overview of what a candidate has accomplished in the
past. Especially when you're hiring for more senior roles where you need to
place more value on experience and a proven track record than raw technical
excellence. I can't really see these "anonymous" interviewing platforms
working well for those kinds of roles.

~~~
6d0debc071
> Do they?

I have seen hundreds of CVs that show no achievements of note - and tend to be
more along the lines of describing the duties of the job and little else
besides. I've also had hundreds of subsequent conversations with those people
that revealed significant accomplishments.

CVs are better than nothing. But there's no set way to write them that's
widely known, and people tend to be rather reticent about their own
accomplishments - perhaps not even thinking of them as such. There's a lot of
value being left on the table.

~~~
vonmoltke
Furthermore, not everyone's accomplishments boil down to nice bullet points.

------
lazyant
So Dunning–Kruger effect (poor candidates over valuing themselves) and
impostor syndrome (the opposite).

~~~
skue
Dunning-Kruger actually accounts for both.[1] But agreed, this isn't specific
to hiring.

[1]
[https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect](https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect)

------
cjrjdjcnd
Nobody on HN has taken the PE or FE exams.

~~~
vonmoltke
Neither have most of the engineers working for large corporations, unless it
was required by their degree program (mine did not).

~~~
cjrjdjcnd
Except that's wrong. If you want to get hired with a job title of engineer,
you at least have to take the FE exam.

That is unless you're a webdev, in which case you can call yourself head
fruitloop and it would still be acceptable, apparently

~~~
jnbiche
What is it with software developers who take such offense at the use of the
word "engineer"? Clearly, you're not a "real" engineer, or you'd know that
your statement about the use of the word "engineer" in titles was wrong in the
US (and Canada doesn't use the FE exam).

I've never met a engineer who gave a crap what my title was, software
developer or software engineer. Which isn't a surprise, since we pose no
threat to the jobs of "real" engineers. Not sure why the word is so
threatening to some software developers.

Do railroad engineers have to put up with this?

~~~
cjrjdjcnd
Actually I'm a mechanical engineer. Webdevs calling themselves engineers is an
insult to the amount of work and effort that I've put in to get where I am
now. It's like medical technician calling themselves a doctor because they
took a training prep course.

~~~
khedoros
A technician could call themselves a "doctor", but they couldn't call
themselves a "Medical Doctor". A developer can call themselves an "engineer",
but they cannot call themselves a "Professional Engineer". If I started
putting "PE" after my name, I'd expect the NCEES to come after me. No one but
you can decide what you take as an insult, of course, but choosing industry-
standard (and legally uncontrolled) terminology like "software engineer" to
take offense at seems like a losing battle.

