
A hiring test I'd like to run - edent
https://shkspr.mobi/blog/2019/12/a-hiring-test-id-like-to-run/
======
ditonal
I interviewed at Twilio. My resume and phone chats made it abundantly clear
that my professional experience was 95% backend in languages like Java and Go.

I looked at their Glassdoor where people wrote they were heavily biased to the
algorithm type questions, for better or worse.

In reality I was given only a single coding interview my whole onsite, and it
was to build a JavaScript SPA, including routing/linking, without the use of
any framework, just vanilla JS.

I was allowed to Google whatever but 50 minutes is still a crazy time crunch
to figure out a lot of the stuff that needed to be figured out.

It felt like a massively unfair way to evaluate me. I have honestly never felt
more screwed over and time wasted by an interview than at Twilio and I would
highly recommend avoiding interviewing there.

My overall point being, is there’s no silver bullet. It’s not necessarily
about this magical process will work well and this one will not. It’s about
the small details, especially things like making sure all of your interviewers
actually read the candidates resume and that everyone knows what type of role
they are interviewing for, and that the interview is designed to discover both
the candidates strength and weakness. I’ve been on both sides of the process
and I know too often shortcuts are taken and churning the pipeline forward
takes priority over doing a diligent job and respecting the time of every
candidate you engage with.

The very fact that the foundation of tech recruiting is inexperienced
recruiters, with minimal tech knowledge, who are incentivized by sales type
numbers, and churned through themselves at a comical rate, speaks to how the
industry views recruiting. College athletes get recruited, top law school
graduates get recruited, engineers aren’t recruited but relentlessly spammed
in hopes of finding more bodies to feed into a process straight from a Kafka
short story, the problem starts there and no a-ha process improvement will fix
that.

~~~
avip
I don't do much hiring, but when I do I make an active effort to avoid reading
candidates resumes.

I've found it's not possible for me to enter an interview with an unbiased
mind having read a resume.

~~~
josephorjoe
This feels weird to me.

If you have advertised a job and I've given you my resume, it is to give you
some information you can use to avoid wasting your time and my time.

Just like I would hope your advertisement is accurate and useful enough to let
me decide if it is a good use of time for me to send you a resume (e.g., saw
an ad for "react developer" \-- got to the interview and they were looking for
a "server side java dev" \-- I mean -- wtf?).

Not reading a resume before an interview feels... rude. It feels like the path
that leads to you asking a Java/Go programmer to do a Javascript coding
challenge in 50 minutes, which is a waste of everyone's time.

~~~
avip
It is rude. People put lots of effort into resumes. But it's ruder to reject a
candidate based on age, gender, martial status, education or lack thereof or
doing non sw jobs in the past.

I always talk on the phone before and that's where we both make sure the
interview is of relevance.

~~~
tristor
If you can't read someone's resume without rejecting them based on age,
gender, marital status, education or lack thereof, or for their non-sw work
history, then it sounds like you're not a very good person to be interviewing
candidates. The concept of "unconscious bias" might be true in an academic
sense, but in a very real way you should be treating people with respect
regardless of their characteristics, especially during an interview process.
Not reading someone's resume isn't just rude, it's supremely disrespectful.

I take a completely different approach, because I have been on the receiving
end of disrespectful interviews and I won't stand for it and I don't expect
the candidates I interview to accept it either. I carefully read the resume
and research the candidate far in advance to the interview. I only ask
questions during the interview which are open-ended, not trivia questions, and
are directly related to either the content of the candidate's resume or things
we're actually doing day-to-day on my team.

The dog and pony show interview style is intensely disrespectful and so is the
idea that you won't even read a candidate's resume. I've walked out of
interviews where both have happened, and I hope everyone on HN gets the
personal confidence to do the same. I'm a professional, I expect to be treated
like a professional, and I return that by treating those I interview like
professionals. End of story.

~~~
avip
>it sounds like you're not a very good person to be interviewing candidates

We agree on that one, but the alternatives are worse. Would you like to take
my place interviewing? It's a time consuming task I'd love to delegate to
someone more experienced.

~~~
GreenJelloShot
I would honestly love to take a job that was 100% focused on interviewing and
hiring technical people. I have been through so many bad interviews (on both
sides of the desk) that I think it really should be a dedicated position
staffed by people who actually understand the major defects in the industry.

~~~
avip
You can always apply to triplebytes and get exactly such position :)

<disclaimer: my affiliation with triplebytes is that they've rejected my
application twice - possibly cementing parent suggestion that I'm
underqualified to interview developers>

------
dcole2929
The single best interview experience I've seen was given by Oracle when I
worked there a few years ago.

The onsite was a panel interview where candidates were given a laptop with a
configured IDE and codebase with a specific set of bugs. Their task was to
debug the various issues and complete as many as they could with the
understanding that it was not expected for them to solve them all or even most
of them. There were also tests to help them. And a different section of the
interview asked them to right some tests and some code for again brain dead
easy coding algorithms like reverse a string.

I thought it was great because it closely resembled what being a dev is like
and actually tested for the skills we cared about. No you didn't get to use
your own custom vim configuration but I'm not sure how that would have really
worked even if they wanted.

------
notkaiho
This reminds me of the story of the young intern that went around a few years
ago. They sheepishly admitted to their boss at the end of their summer
internship that they had just googled all the Excel things that were required,
that they hadn't actually known any of it.

"But your generation KNOWS to look for help, and knows where to find it!
That's where you were doing it right," was the boss's response. And that's
heartening.

~~~
dillondoyle
This is such a valuable skill. For our work we've found this ability to google
or find answers without incredibly specific step by step instructions to be
lacking from many younger workers. I've wondered if others have found this for
many smart people under 30 as well?

~~~
hinkley
I really believe there are too many important things to know at this point,
and so I instead concentrate on memorizing the bounds of the Realm of the
Possible. I know what my tools can do, I know where the pitfalls are, but I
don't always recall the exact details of those boundaries. I often have to
look up why, and I'm perfectly okay with that.

Where that doesn't work well is in interviews, and in places where people
think it's perfectly reasonable to have technical discussion meetings where
laptops are banned. We're really going to make architectural decisions without
looking at any code? Who thought that was a good idea?

------
city41
At my previous company, the technical interview was 2 hours long. The
candidate was given a choice of about 10 very small, simple projects. Then we
sat at a pairing station and paired on the problem. The candidate can use
google, their favorite editor, write tests, anything they want. The goal is
not to finish the project, but just code with them for 2 hours. It's the best
technical assessment I've yet seen in an interview setting.

One caveat is this company really believed in pair programming, and virtually
all code was created in pairs. So that's part of why this interview approach
worked.

~~~
organsnyder
I did an interview once at a company that is religious about pair programming.
I found the exercise unsettling: would they evaluate technical ability or
culture fit more? If they were focusing on the former, I should take the wheel
and try to demonstrate my chops (perhaps making me appear arrogant); if the
latter, I should try to demonstrate how I can collaborate with the other
developer to make both of us better (perhaps at the expense of me not
demonstrating as many skills within that brief window of time).

It turns out that they were really focusing on how I interacted with the fake
"client" and whether I had the proper "consultant mindset". Didn't see that
coming (perhaps that's my fault). Though I feel I ended up dodging a bullet in
not getting an offer from them.

My suggestion is to try to give candidates specific times to demonstrate
various competencies (technical, colleague interaction, customer-facing), and
tell them which ones are the focus of each exercise. Yes, that's not how the
real world works (you need to utilize them all at the same time, of course),
but an interview setting is hardly the real world. And you'll still see
glimpses of their overall capabilities in each stage.

~~~
Iv
> would they evaluate technical ability or culture fit more?

Why not ask them that? "What are the roles in this roleplaying game? Am I a
driver and you are evaluating me? Are you a client? Are we colleagues?" Or
even more directly what you asked: "Are you evaluating technical ability or
culture fit more? "

I don't think there would be a point in keeping that a secret.

~~~
organsnyder
Yeah, from a candidate perspective that's a good idea. From an interviewer
perspective, though, it's important to remember that many very good candidates
will feel uncomfortable doing so.

------
ses1984
Some of the tests in my current company work like this. We don't expect you to
have everything memorized. We give you a problem and then say, "feel free to
use any reference you want, help, man pages, google, stack overflow,
whatever."

Some people still stubbornly refuse to look for help.

Some people act on the first google search result even though it's obviously
wrong.

Some people don't know how to read a man page.

I find it very illuminating, but I don't really have enough data points to say
how successful it is.

~~~
reallydontask
> Some people still stubbornly refuse to look for help.

The thing is that some people might feel that they will get assessed lower if
they look for help rather than come up or even luck into a solution.

Despite making it abundantly clear to candidates that they should take the
test(s) as a pair programming sessions, we had candidates struggle with some
basic stuff and not ask questions. They just kept trying things while getting
more nervous.

A lot of people would feel that it might be held against them. The thing is a
lot of people under pressure will forget the basics

~~~
codyb
One thing I’ve had help melt the pressure on both sides of the table is a
laugh. I’m fortunate enough to be able to sort of slice through tension and
get a rise out of most people.

Stressing exactly what you’re looking for, that you’re not perfect either,
that there’s no one solution, and letting them know things can be
collaborative has been helpful too.

Let’s ping pong off each other and see where we get, I mean that’s what we’d
be doing in the day to day right?

To this end I’ve sat down and added new questions to the interview guides at
my previous position because I was concerned that any person asking the same
question over and over again would really start to be biased towards some sort
of “ideal” solution.

But there’s really nothing like a good laugh to put people at ease.

After one interview I mentioned that I felt I did a good job of interviewing
because I could usually help relax the tensions which generally gets the best
out of people and someone said that they didn’t think they’d ever gotten a
laugh out of someone during an interview.

That’s a bit too serious for me.

~~~
flowcont
Thank you for doing this! I'm doing several processes now and it's so
unpleasant and stressful just before the interviews, that taking some weight
off of it is really helpful.

------
reallydontask
> I'd tell them that at the start, obviously. This isn't The Secret Rules For
> Getting Hired.

> "I know you've never used Blender _, " I'd say, "And this job doesn't
> require it. But we want to see how quickly and accurately you can learn to
> use something unfamiliar while under pressure."

>_ Or whatever.

The main problem is that the level of pressure that people feel under
interview conditions will vary dramatically. Personally, I would find this a
very bad test as I get horrendously stressed at interviews yet somehow manage
to portrait a facade of calmness (to be fair I've been told once that I looked
remarkably calm).

To the usual refrain of:

If you can't handle the pressure of an interview then maybe you are not a good
fit for our company

I normally reply:

If day to day working at your company generates interview like levels of
stress, then please do not offer me a job

~~~
perl4ever
If someone says there was a lack of culture fit, it stings and certainly could
be some type of illegal discrimination. But at the same time, it's necessarily
true! Unless your culture is discriminating unfairly...

------
bryanrasmussen
Well this is exactly like a test I went through at one time. It even used the
same reasoning pretty much.

The job was fullstack node and react, I went in and they said we want you to
look at this game written in Python which is really slow and tell us what is
wrong with it (I hadn't done python in years and back then wasn't very good at
it). We will sit with you. I said I was game, they said lots of people weren't
- one guy evidently got offended and said he didn't do that language (I can
definitely think of one friend of mine who is totally competent who would get
offended at being asked to work in a language that did not have anything to do
with the job he was applying for)

The game was using a big dictionary in an array and you had to look up words
from it and it was slow (about 7 minutes to do searches). So I thought to
myself, damn I cannot remember the name of that algorithm for this (binary
search), ok let's stall a bit maybe I will remember the algorithm name.

So what's the first thing you would do?

Uhm how about checking a profiler. So we checked and found some simple things
we could optimize. And then I looked through the code and I saw there were
parts I could cache etc.

So after all those things were done the time to do a search was a little bit
more than halved and I still couldn't remember "binary search", so I stared
dumbfounded at the screen for I think about a 3-4 seconds and I started to say
"uhm" and then the guy sitting with me I guess ran out of the time he wanted
to spend and said "ok well there is an algorithm for this blah blah blah"
showed how to do it in python. Thanks for your time.

I actually didn't want the job but still felt somewhat let down that I
couldn't remember.

~~~
marvinjames
Of course there was pressure, but why not just say that you know an algorithm
that can do this faster, but you can't remember the name of it, and then just
provide the intuition behind it?

~~~
hellisothers
In my experience on both sides none of that matters, also none of the “how do
they look stuff up” matters, what matters is did you just know what to do and
do the right dance.

~~~
peter_l_downs
My experience has been the exact opposite.

~~~
mixmastamyk
Most interviews and interviewers are poor, meaning jump thru ten hoops, miss
any and disqualified. Highest score wins.

------
scrooched_moose
I'm a mechanical engineer by training and still occasionally do some entry
level hiring in it.

My favorite question in recent years has been posing an advanced project - not
unreasonably difficult, but one I know they won't have covered in school. Make
it clear I don't expect them to have the answer, but have them talk through
their approach to this project with me. "How would you get started?" "What
would you need to know?" "What resources would you use?", etc.

For us it's been a great filter - both a gauge of their current knowledge and
ability to tackle an unknown.

~~~
cushychicken
I'm an electrical engineer, and I have a similar question that I ask my entry
level candidates - "How would you measure the voltage coming out of a wall
socket?"

It's familiar enough to be accessible, but technically just far enough out of
their reach that they don't know exactly how to do it. We talk about
approaches, tools they could use, some potential problems or issues they'd
need to overcome.

If they get flustered about where to start, I laugh, and say "Don't worry - I
don't know the right answer. I'm actually afraid of AC mains electricity."
Never fails to calm them down, because, well, it's kind of funny to hear an
electrical engineer admit that they're afraid of electricity!

It's a question that I've gotten a lot of mileage on.

~~~
_AzMoo
Couldn't you just jam a multimeter into it?

~~~
cushychicken
You could!

My immediate followons to that would be:

\- How would you jam a multimeter into the socket _safely_?

\- What information will the multimeter _not_ tell you about the AC voltage
signal?

------
motohagiography
This is very specifically a terrible idea.

I passed an analogous interview to this once when I was much younger, and then
subjected several applicants to it thinking it was a useful filter.

It's not, it's abusive psychological torture that rewards sadistic amateur
psychologists and interview administrators. The only way to succeed is to
submit to the interviewer by asking for help. It filters for people who know
to implicate others in failure while taking credit for small successes, and
bullies people into submission.

The interview I passed was a variation (I learned after) on the bridges of
konigsberg, where instead of asking for help, I asked if it was impossible.
The person administering it hired me on the spot, but even this was wrong
because the place was full of people who thought the reason they were there
was because they were intelligent - something they didn't have control over -
and they acted As pettily as psychotically as you would expect.

The problem on teams isn't individuals not asking for help, it's managers who
don't have the confidence of their teams to ask.

If you are reading this and I put you through that exercise 15+ years ago, I
apologise.

~~~
dlkf
From the article:

> I'd tell them that at the start, obviously. This isn't The Secret Rules For
> Getting Hired.

> "I know you've never used Blender*," I'd say, "And this job doesn't require
> it. But we want to see how quickly and accurately you can learn to use
> something unfamiliar while under pressure."

~~~
shantly
I don't think I've ever had to learn something very complex under the same
short time constraints and high do-or-die pressure as an interview. Tiny
things (think: new-to-me unixy commands), yeah, but nothing very complex. With
as many unknowns as that introduces, should it ever come up in a real job
situation with a very short time window available, my expert (I'm quite
serious using that term here) advice would be that we should count on having
it take quite a while, and immediately begin whatever mitigation is required
to allow for that, rather than waiting to begin planning for and mitigating
the harm that'll do and simply gambling that this complex thing won't surprise
us and end up taking longer than we'd hoped (likelihood that it will:
extremely high).

~~~
hinkley
I end up doing a lot of technology selection and so I have to have these
skills and the ability to ask hard questions.

I don't for a second believe this should be the skillset of every member of a
team. A team where everyone is adding new tools all the time is chaos.
Learning new tools isn't really a goal. Getting better at your job is the
goal, and learning new tools is either a means to that end or just moving the
goalposts over and over so nobody knows what's going on.

~~~
shantly
Yeah, sure, I do evaluative work just fine, but I don't think "build [non-
trivial thing] with [fairly complex software or tools you've never used] while
we watch you, you have 50 minutes" resembles that very much. The closest I've
seen is situations along the lines of "meeting's in 50 minutes, could you find
out about [thing similar enough to other things you've done or worked with
that you are qualified to evaluate it on short notice]? I just learned we'll
be talking about it in there and your take on [a couple specific aspects of
it] would be useful." Not "you've never used this, I want a demo of [more than
hello-world] in 50 minutes, or it's your ass", ever.

------
stedman
We do this at retriever.co for our virtual assistants, in two areas:

1\. we have applicants do a data transform with an esoteric excel function
that they've /almost surely/ never used, and prompt them to find videos /
tutorials on how to use it if they don't know (which, we assume, they wont)

2\. we have people answer fake customer support tickets that require them to
do some on-the-job learning about the Internet Service Provider industry -- to
a level of detail that requires that they read through and grok a page on
wikipedia explaining up-time calculation.

While neither of these pieces of knowledge are necessary for their day-to-day
at Retriever, they allow us to stratify candidates' ability to learn new
skills and grok new information.

100% would recommend!

~~~
aniketpanjwani
I've been thinking through how best to screen personal assistants, and these
are both very helpful ideas. I particularly like the first idea to screen
applicants on esoteric utilities - thanks for your comment!

------
learnstats2
This made me laugh - I first applied for temp jobs around this time and there
was a typing test.

"Wow," said the person who had left me alone with a computer that felt old in
the 1990s, "I've never seen anyone score 50WPM before."

"Uhh... I could type a lot faster, but the letter T is sticky and I had to
delete a lot of Ts"

She looked mortally offended by this.

I was given a job as a dishwasher which I was sacked from after one shift.

------
tracker1
I partially agree, and it depends on what you are interviewing for. In the
end, getting a job done, on time, without error/bugs is generally most
important to the people you are going to be working for.

As for algorithm questions at some FAANG or wannabe startup, it can be rough.
Personally I tend to blank on a lot of terminology, similarly on names a lot
of the time. Some trivia games kill me, I can see the answer's face, the
movies they've been in, but just blank on the name.

All the same, in my life and career, I've managed to learn, adapt and create
both simple and complex systems in new languages and platforms.

On the other side of the table, I tend to ask maybe 3-5 trivia like questions
only to gauge where a person is at, and let the conversational parts determine
if the person would be a good fit. I think you need some of both. I feel drive
accounts for more than even skill, knowledge and experience.

~~~
perl4ever
Maybe you just can't figure a person out in an interview. I got my current job
as a contractor/temp, and thinking back to a previous company, one of the
people that was the best and stayed the longest was a contractor that
"graduated" to permanent. The simplest way to prevent people from keeping up a
facade is just time.

------
commandlinefan
> Are they able to read a manual?

Uh, I can read a manual (in fact, I enjoy reading software documentation), but
it's going to take me longer than an hour...

~~~
edent
Let me amend that. Are you able to look up information in a manual - either
using the index or hitting CTRL+F?

~~~
GreenJelloShot
It completely depends on the manual. If this is the first time I have ever
seen it, and it's about something I have never used, it very well might take
me a while to figure out how to look up information quickly and effectively.

Does the manual having working code examples? Then is it probably a good
manual. If it does not have working examples that you can copy & paste (and
they actually work), then it probably is terrible.

------
arkanciscan
After a couple of years working as a web developer, I decided to go to
College. I took a "Web Development" class. I failed the final. The assignment
was to code a website by hand. I used BBEdit to pretty print my hand coded
HTML and it inserted a <meta name="editor"> tag that I didn't catch. I dropped
out and went back to getting paid. Been doing it for 20 years now, though I
use Prettier to do my formatting these days.

~~~
avgDev
OOf I had the same final. I had to create a website by hand in a basic text
editor like notepad++. I passed but I honestly don't see any value in it.

------
rsstack
This is very similar to what I do for roles that are supposed to be
independent and to work with customer software. I ask the interviewee to do
something with a technology or software they don't know, but give them no
other constraints. They can use whatever programming language, whatever OS,
whatever IDE, whatever existing libraries online and even IRC/Slack if time
permits - the goal is to see that they can solve a problem that is new to them
without panicking. In real life they'll have a lifeline or two in the form of
colleagues, but those colleagues might also not have the answers.

Obviously a terrible solution for a standard product engineering position, it
is only relevant for special roles.

------
gota
I don't mean to be derisivs but at that point - testing for reasoning,
understanding, personality - Why stick with a software tool?

You'd be better off arranging to play a strategy, preferrably collaborative,
boardgame.

One day I'll use Pandemic as an interviewing tool. I wonder of it'd work as a
pre-formal interview stage

~~~
edent
I've been to group interviews where they pretty much do that. Randomly divide
all the candidates into groups, give them a task, see how they work together.

I can't say I enjoyed the experience - and I've no idea how good it is as a
tool - but it was illuminating to see how people treated each other.

------
Msurrow
First test of any software dev: Do you know how to look for answers on
stackoverflow?

Second test of any software dev: Do you know how to ask a good question on SO?

~~~
war1025
I've never found anything of actual use on StackOverflow in my ~decade of
doing this professionally. The only time StackOverflow provided useful answers
to my questions was during my first year of programming courses in university.

~~~
Insanity
StackOverflow is fine when learning a new language. But indeed, hardly seems
to help in real-world problems.

~~~
Msurrow
Thats why I put in the "know how to ask questions" part. Sure, a lot of the
time you don't get an answer. And you end up figuring out stuff the hard way
yourself. But then I always force myself to go back and answer my own
question. It might help the next guy, but more importantly if I have to answer
the question myself I need to put what I learned in writing.. and in an
understandable way as well. Often the exercise of explaining it will make me
understand even better.

------
FiddlerClamp
I had that same automated Microsoft Office test at a temp agency in the 1990s.

Funny thing, it had a bug. As long as you didn't let go of the mouse button,
it would let you open and navigate all of the menus. So I simply browsed the
menu UI until I found the correct choice. 100% score.

------
LanceH
I've always wanted to run one where we bring them to a room to interview.
Maybe take a break. Move their stuff to a slightly different room. Pick back
up right where we left off. Maybe change out the furniture and people outside
during the interview. Introduce them to numerous employees, who will respond
to names swapped amongst each other. Another break - switch interviewers. Pick
up right where the first one left off - insist that this employee has
conducted the entire thing. If they drove there, it would be best to move
their car.

I'm not sure what position I would be hiring for, but this is the interview I
would like to conduct.

Other possible ideas would be a background check that is either insanely deep
(so you're 3rd grade teacher was unavailable for a recommendation) or raises
issues that they have no part of, "we see you've spent time in North Korea."

~~~
notkaiho
... But why?

~~~
jedieaston
To see how they respond under pressure.

~~~
LanceH
Today's changing markets demand employee's who can adapt quickly.

Or static old companies seem to want employees who don't notice change around
them.

------
bilekas
I love this.. Definitely going to set something like this up.

And anyone who does go for the help gets an immediate offer.

I hate people too stubborn to admit they don't know.

------
ngneer
Interviews should be personalized. That they are conducted en masse with
little consideration for applicant background is an artifact of the difficulty
of adequate personalization. Targeted ads are valued precisely because they
are targeted, presumably increasing the chance of a match. Has anyone heard of
any startup companies in the personalized interview space?

------
Davertron
I hear a lot (and believe myself I suppose) that I would rather hire someone
that knows how to learn than someone who currently knows thing x but can't
pick up and learn things on their own. But I feel like it's VERY hard to
figure this out about someone. It's pretty easy to test how well someone knows
a particular technology or tool (given some variance for operating in a high-
pressure scenario etc.) but it seems pretty hard to tell if someone can learn
new things quickly, and learn the right things. I feel like this is what
algorithm tests are trying to do, but I also feel like a lot of algorithms are
based on having a key insight into a problem that is, at least in my
experience, pretty hard to have when you're in a room with strangers on a
timer and being judged, so if you don't know the particular algorithm to
apply, good luck. And also it's pretty easy to go study algorithms and hope
you remember how to implement DFS if that's what they ask you in the
interview. And there is some non-zero value to just knowing about different
data structures and algorithms for certain types of work for sure. I don't
think many of us web developer types will be implementing DFS ourselves, but I
run into issues pretty often where having knowledge about different data
structures and algorithms comes in handy.

So I can see the appeal to doing a test like this that has nothing to do with
a particular skill, where you're trying to evaluate someone's judgement and
how quickly they can absorb new ideas or figure something out on the fly etc.
BUT, I just think there are too many factors that can really muddy this up,
and I still don't feel like there's an objective way to measure "smart and
learns quickly". Even "learns quickly" is somewhat vague; I personally
wouldn't expect someone to pick up, say, functional programming in an hour
just by saying "go read about it." Some things take longer to learn than
others.

Anyway, there's no silver bullet. I still feel like we end up making pretty
subjective decisions about other people when interviewing them. I don't think
we're even all that aware of them ourselves. I would say though that if you're
interviewing somewhere, you're getting useful information about that place by
how they interview you, and so I wouldn't feel too disheartened if you feel
like the interview is unfair or doesn't work for you. That's probably a good
sign that you wouldn't enjoy working there anyway.

~~~
GreenJelloShot
I totally agree. The best solution I have come up with for trying to assess if
someone can learn new things quickly is simply to ask a candidate to describe
the last new thing that they learned. It doesn't have to be job-related at
all. Just the last thing they learned. And then ask some follow up questions
about how they learned, why they learned it, and how have they used the
knowledge.

If nothing else, it will give you a general idea of how often they learn new
things.

------
lettergram
I think the primary issue with interviews is that the interviewer doesn’t
share expectations. I think it’s totally fine to evaluate how a person asks
for help, as long as it’s clear they aren’t expected to solve the problem.

Further, you’d need some standardized way to evaluate how well someone asks
for questions. Otherwise you’ll have interviewer bias.

Which is why designing interviews are very difficult.

------
myself248
I was pretty happy with the interviews I went through for my current position,
although that might be selection bias because I passed them. They all shared
some important attributes:

1: Fairly open-ended, there was no "bzzt-wrong" moment like the Office 97 case
in TFA, so I didn't feel a lot of pressure. It was definitely enough rope to
hang oneself, so the interviewers could form an actionable and objective
judgment, but as a candidate, I didn't feel that I was on trial, more just
chilling and talking tech with some other nerds.

2: The underlying skills (understanding one's audience, breaking a large
problem into pieces, making reasonable assumptions) were obviously directly
applicable to the position. I never felt that I was doing senseless work, even
when the questions were clearly contrived, because the connections were self-
evident.

3: Despite the above, the actual questions didn't require a lot of domain-
specific knowledge. Even the one that sorta did, could be answered by assuming
that it was similar to a more common system, and that's how I approached it.

4: In every case, there were "plan B" options and accommodations for having a
gap in one's knowledge. "I don't know this off the top of my head, but here's
where I'd go to find out and here's the guess I'm going to use in the
meantime" was a valid answer. Perhaps the best answer in some cases.

The result is that I work with a team of people who are super adaptable, can
think on their feet, but have a keen interest in pushing towards correctness
as soon as real information is available. Whenever I think about possibly
going elsewhere, I remember that most HR departments don't select for any of
those attributes, often quite the opposite. (And those places tend to be the
clients whose problems we get called in to help solve, as a result.)

------
Tade0
There must be a serious dearth of candidates in my area, because the last time
I was subjected to a time-constrained test was almost three years ago, and it
was a Codility challenge, so I had some time to prepare for it.

I quit my job recently, and in order to find another I went full-auto with the
applications - I've sent close to thirty of them I think, so a long series of
recruitment processes followed.

They were all mostly the same - I either received some homework to do or had a
1-2h technical interview during which I was asked a set of questions which
were easy for the most part. And that was it.

My point being: I've discovered that I can't really relate to the stories
other people posted in this thread. Does this mean I wasn't ambitious enough
regarding my applications, or are the others subjected to a difficult job
market? Maybe both? Or none? I really don't know.

------
techstrategist
I experienced an extremely simple (and IMO effective) version of this.

For a dba-adjacent job, I sat down with a dba and was shown how to connect to
a sql server instance and change some particular setting. At the end of our
hour of discussing other technical matters and my resume, he asked me to
demonstrate recall of this simple command.

------
Iv
I keep hearing and reading horror stories like this one on HN and other
website. I never did interviews at big companies, nor did I do any in the US,
in 15 years of freelancing, I only had once a technical interview, with a C++
guru, I failed most of the questions but still got hired because "you still
did good, kept calm and asked good questions".

In my last gig, it was for a work on a Java application. I was upfront that I
only did a bit of Java in school, but that I was more of a C++ guy. The Java
guru hirer was like "Yeah, if you know C++, you'll get Java quickly" they were
more interested in my experience with video and image processing.

I basically avoid to do HR-based interviews. Short-circuit them when you can.

------
sylvainkalache
That's how Holberton School (we train people to become software engineer)
select and train students. We are a college-alternative where students learn
by working on projects, in group.

The whole education is flipped compared to a regular one. We give students
problem sets and they have to find the answer. We never give lectures (we have
no teachers), just guidance so that they can get started on a topic, but never
enough that they can actually achieve the tasks we give them.

> Are they able to read a manual? Candidates (no prior experience in coding
> needed) need to answer questions on a specific command that can be found by
> reading the manpage. At first we provide the manpage, then we show them how
> to find a manpage, and finally, we don't provide any guidance, because at
> this point they should be able to do it on their own. In the curriculum,
> once students, we have a quizz to makre sure they can answer basic questions
> on the topic before jumping into the coding part.

>Can they formulate a search query? We ask trivial questions that can be
answered just by Googling. Might sound like a no-brainer to the developer
community, but not everybody is good at this.

>How do they assess whether the tutorial they found is suitable or reliable?
That's the tricky part, but basically, because the school is not providing
material that contains the answer, candidates and students have to navigate
the ocean of information/tutorial that can be wrong/incomplete of correct. By
pressing a button, students work (code at this point) is automatically and
instantly corrected, so they can easily figure that what they built is correct
or not.

> What steps do they take to make sure they're finding - and learning - the
> right information? On this one, we as a school, are the one providing this
> guidance. Basically when students are doing their projects, we define
> learning goals. Getting coding part done is great, but we also want to make
> sure that students have been able to grasp concepts that are key to becoming
> a great software engineers. So students look at these "learning points"
> before, and by the end of the project, they must be able to discuss all the
> points.

------
scarejunba
Haha, that's funny, and an awful idea. At a previous employer, as a hiring
manager, I offered candidates an option of bringing in their fully-configured
IDE in their own laptop with the language of their choice and I'd give them
the problem to solve. Essentially, act in the environment that you've already
mastered.

Only one guy did (I had maybe 10 a month) and for some inexplicable reason he
couldn't figure out how to create his `main` equivalent in his language in his
IDE. He must have been incredibly nervous because we tried "perhaps you could
sketch out your thoughts in broad strokes and then we can return to the IDE"
etc.

------
Iv
I just got interested in the way the school 42 works. Especially the month-
long "test" that they have to accept tuition free students.

The idea is to make you complete a lot of programming projects in languages
and tools you normally don't know at all. When you start. The evaluation does
not only take into account if you finished project but also if you helped
others, knew how to look for help (basically: google or other students). Its
goal is to select students who would work well in a peer-learning project-
based-learning paradigm. This, in my opinion, is much closer to real-life
conditions than any academic test.

------
haolez
Another take on what the author proposing is to simply not care about the
output of the task, but focus on how that person approaches the problem (even
if they don't manage to complete the task at all).

------
andrewfong
Honestly, I wish more companies would just tell you in advance of the
interview what the specific prompt is going to be (and perhaps the evaluation
criteria too).

Sure, you could "cheat" by Googling the answer in advance or asking someone
for help. But if the question's sufficiently open-ended, it's not that hard to
suss out in follow up questions whether you have a deep first principles
understanding of the subject area or you're just repeating something you read.

~~~
bagacrap
I'm pretty sure if you give me a subject I know nothing about and a couple
days to study I can teach myself about it, at least to the level of someone
who has real experience in the subject which may be a couple of years out of
use. So then how does a hiring manager know to filter me out?

~~~
andrewfong
Presumably, at some point, you'll be interviewed by someone with more up to
date experience. And if you study enough to fool that person, that seems fine?
It's a good sign you're capable of the job, prior experience be damned.

------
catern
I don't know about this. I don't think I should spend most of my time learning
completely new tools.

If people spend most of their time learning tools, that encourages tools which
are easy to learn but lack deep and powerful functionality. I think it's fine,
and even good, for tools to require focused learning to understand, and in
exchange have a worldview and tool use that is really powerful. Like Emacs or
vim or Excel, for example.

------
tantalor
See also _Kobayashi Maru_

"a test of one's character or a solution that involves redefining the problem
and managing an insurmountable scenario gracefully"

------
neo2006
I do hiring interview in my company and one of the test we like to do is to
give the interviewee a problem out of his area of expertise and open a browser
and an editor for him and tell him that he is allowed to search for things.
Most of people I interviewed do not use Google help they are afraid you judge
them negatively for it even if you say you will not.

------
lbarrett
This reminds me so much of Vernor Vinge's story "Fast Times at Fairmont High".
It's set in the near future, and one plot point is that high school tests
cover this skill--to read through the manuals quickly and get things done with
new tools. I recommend it, and the related book "Rainbows End".

------
Aeolun
I like how one of the comments is about someone getting a job by telling the
interviewer they knew what a computer was.

------
bulutzuku
This is an excellent idea!

------
itronitron
rather than give them the application, just show them the file (location,
name, and extension) and ask them to make some useful revision to it

------
myegorov
fwiw this describes my experience of interviewing at Databricks

------
throwawayhhakdl
Our team uses hacker rank or something similar (I forget) to weed out bad
profiles (most of them). We combed through their question base to remove
things that felt too specific. But I’m happy with the result, because there’s
nothing you can’t google, and googling is allowed.

We’re a python and R shop. You can choose either test. As a non R user, I can
get a decent score on the R test, because I can read docs. And that’s great.

We have also dabbled in game based assessments, and while I have qualms with
it, the same kinds of skill sets seem to be the primary differentiator of
success.

My 2c on two types of assessments people probably normally disregard

------
onion2k
If the test doesn't reflect the job you'll be doing then it's not a good test
of whether or not you'll be able to do the job. If you're applying for a role
that expects you to know an application then looking in the help _is_
cheating. If using the help is acceptable in the job then it's not.

It's also worth noting that people don't like "proxy tests" for the skills
they'll need to use on the job. If the test only tangentially assesses a skill
then it's going to be influenced by other skills or flaws and won't give an
interviewer a good idea of how good you are at something. This is why people
complain about whiteboard exercises in developer hiring. Unless remembering
and explaining an algorithm on a board is something you'll actually be doing
in the role then it's not a good test - it _sort of_ assesses how good you are
at explaining an algorithm, but it really assesses memory, presentation
skills, etc. Someone with brilliant whiteboard skills isn't necessarily a good
developer, so hiring them on because you were impressed by how well they
explained something in front of a board feels wrong, especially to people who
aren't good at the tangential stuff.

In the case of the suggested hiring test in the article, it doesn't test
someone's job skills. It tests how well they look for help, but it _also_
tests how well someone copes in unfamiliar applications, or with esoteric UIs
(in the case of Blender), or if they've actually used the app that's being
used as a proxy. None of those things are necessarily a good assessment of
whether the person can do the role they're applying for, so the test fails.

~~~
dangerface
> If you're applying for a role that expects you to know an application then
> looking in the help is cheating.

If you discourage your employees from learning you are going to have a lot of
dumb employees. Even the developers who made excel do not know how to do
everything that excel can do.

~~~
onion2k
Some businesses need to hire people to do a job using an application in a very
standardized way from day 1, there is no expectation that the person doing it
will learn anything in the role, and when the software changes training is
provided. The employer is in complete control of what is done and how the
software is used. Figuring something out yourself is considered a bad thing
because it's not the standardized way of doing the thing.

Please don't go through life imagining every job is like a developer job. We
get far more freedom, creativity, and responsibility than a lot of other
workers.

~~~
hvidgaard
That is all well and good for those types of jobs. But no one can remember how
to use all of Excel, and they should be able to search for help and understand
that help.

