
Harder programming questions do a worse job of predicting outcomes - Harj
https://triplebyte.com/blog/interview-questions-are-too-hard-and-too-short
======
drblast
Google recruiters call me a lot. I think I'd do a good if not stellar job
working there. I've passed multiple FAANG interviews and been very successful
as a senior developer.

In my email I have an "interview prep packet" from them that essentially tells
me to brush up on algorithms and read Cracking the Coding Interview to prepare
for their interview process.

I'm fairly happy in my job. If they offered more money or a really interesting
project I'd consider working for them. But I'm pretty lazy about redo-ing
college algorithms class during my free time at home to go work there, so I
probably won't.

There's an opportunity cost with interviews like this where an M.S. and long
career of getting shit done counts for very little and memorization of
undergrad level topics that you can look up in two minutes in Knuth if you
have a problem that requires it can make or break an interview.

I've made a career fixing a ton of horribly shitty, inefficient code that's
been produced exclusively by people who pass these interviews.

~~~
warp_factor
You wouldn't be a good fit for Google. With their algorithmic interviews that
require college-grad level of studying, they filter for people that are ready
to follow orders without complaining.

That's who they want to hire at the end of the day: some coders that don't get
too critical about their job and do what they are asked to do, even if it is
repetitive, stupid and doesn't really make sense (such as re-studying
algorithms implementation details for two weeks before an interview when it
can be looked up super-easily online).

~~~
sfw1k

      some coders that don't get too critical about their job
    

Couldn't be farther from the truth.

As someone who works there I've noticed this general trend of animosity
towards Google, it seems to be fueled by the fact that people feel a sort of
inferiority complex when they don't clear an interview or they feel they
wouldn't be able to crack it if they ever gave it a shot, this leads to an
overcompensation of attitude in the other direction, a sort of "sour grapes"
narrative where there is an effort to downplay the prospects of working at
Google or ridicule the people working there like you did just now.

~~~
RichardCA
I think it's possible for both things to be true at the same time. Yes it's a
selective meritocracy, and it's also true that the process unnecessarily
alienates people. It comes down to recruiters being given the very difficult
task of finding candidates for a process with such a high rejection rate, and
the added stress of being given very specific rules about the things you can't
be transparent about. The overwhelming majority of job seekers are mature
adults who can handle rejection and constructive negative feedback. It's the
sense of having wasted one's time that generates the "sour grapes" feeling.

~~~
warp_factor
There is also the general attitude of Googlers//Facebookers that brag a bit
too much about working at Google//Facebook as if it was the dream that
everyone was looking for.

There is nothing more annoying than discussing with a Googler that tries to
convince you that you should apply and work for Google.

------
bunderbunder
Aside from all the things mentioned in the article, this also seems like a
fairly predictable application of Goodhart's Law: "Any observed statistical
regularity will tend to collapse once pressure is placed upon it for control
purposes."

Once upon a time, skill at doing these sorts of problems might have correlated
(imperfectly) with general aptitude as a programmer or software engineer. But
the very act of trying to leverage that correlation for hiring purposes
probably also made it go away. Now you've got a whole lot of people practicing
hard on these sorts of problems, spending huge chunks of their free time
grinding away on Project Euler and Advent of Code and HackerRank. That muddies
the quality of this stuff as a proxy for what it was originally trying to
detect: natural aptitude. I'm guessing having time to level grind like that
also correlates inversely with other traits that are desirable in a
programmer.

~~~
ohyes
What’s frustrating to me is that we haven’t found a way to teach programming
that doesn’t rely on natural aptitude.

It’s really a travesty that we can’t teach it the same way that we teach maths
or natural languages.

The end result is we’re left trying to divine whether someone is the
programming equivalent of being illiterate. As with illiteracy people find
ways to fake it.

~~~
gizmo686
As someone who has studied both math and computer science (and is now a
professional programmer), I have no idea what you are talking about. We are
far better at teaching people to program than we are at teaching them to do
math. Its just that most people self select out of math [0], so, due to
selection bias, it appears that we are great at teaching those that remain.

As an aside, as someone who also studied linguistics [1] (and a couple of
foreign languages at a beginner level), I am very confident in saying that our
approach to teaching natural languages relies almost entirely on natural
aptitude. It is just that, absent a serious mental disorder, all humans have a
very large natural aptitude for natural languages

[0] Probably because we are absolutly terrible at teaching it.

[1] I assume not what you mean by teaching natural languages, but the topic of
language acuisition (including in adults) does come up; plus it gives some
perspective on how teaching language would look if we didn't rely on natural
aptitude.

~~~
Gibbon1
I took math as part of my engineering degree. And that was 30 years ago. The
older I get the more I think the way it was taught was terrible. Same reason I
think teachers are bitching about the US's fetish for academic testing.
Testing pressures teachers to teach students to mechanically solve problems.
But with shallow understanding.

~~~
crooked-v
I think of monads, which are real easy to explain in a programming context as
soon as someone understands lists and map/reduce, but are total gibberish in a
math context even if you've gotten through calculus.

------
zinclozenge
The less talked about absurdity is that successfully passing programming
interviews is a skill itself. It's especially absurd because the time I spend
developing that skill is less time spent developing skills and knowledge more
directly relevant to my job. Yet, programming interview skill is more relevant
to progressing my career.

edit: now if you'll excuse me, I need to do some dynamic programming problems.

~~~
rightbyte
This is kind off strange for a non US resident to grasp. I've been employed as
a programmer three times and noone tested my coding abilites what so ever on
the interviews. No samples, nothing. Never heard of any collegue doing that
either.

~~~
castlecrasher2
>This is kind off strange for a non US resident to grasp.

I've worked for two non-technical companies as a software developer and one
highly technical company, and interviewed at a few Silicon Valley companies.

The difference between the interview processes is staggering; my current job's
interview was two hours of conversation, no code tests, just a general
assessment of "do you know what you're doing" by the hiring manager and a
couple other members of the team. The highly technical company had a code
assessment then the in-person interviews had zero coding.

The SV companies must have a good reason for this, but golly the amount of
coding in those interviews is nuts. I'm a process over code speed kind of
coder, and I've failed every SV-level test because of it; my code comes from
talking to non-technical users like medical researchers and study operations
managers and tossing something together in Python or a cloud service that
makes their lives easier. Needless to say, I don't go over algorithm
fundamentals on a regular basis, and I generally fall out after the first or
second interview.

It's especially odd that interviews are so intensely focused on those couple
hours since I personally don't see any dev or any resource for that matter
contributing in any meaningful way in so fast a time, or even within 90 days.
I'm not sure how this problem could be solved with the limited time companies
can dedicate to interviews, though; maybe rely more on portfolios?

~~~
mlthoughts2018
> “The SV companies must have a good reason for this”

In fact, no, nobody has a good reason for it.

~~~
lovecg
Go start a company. Hire people without doing any sort of coding interviews.
Report back in a year.

The reality is that these are extremely desirable positions with a staggering
number of applicants who _do not know how to code_. Not as in “I can’t solve a
dynamic programming problem without studying up on it”, but as in literally
don’t know what a for loop is.

The process is far from perfect and the frustration is understandable, but it
works well enough as a filter from these companies’ point of view.

~~~
castlecrasher2
I imagine technical screening is necessary, but in-person coding assessments
are nonsensical. It's not like programming is a spectator sport, so why are we
tested on it live? I suppose the process is self-selecting, because I
personally have given up responding to SV recruiters or interviewing with
those companies.

------
esoterica
Why do developers complain so much about hard interview questions that can be
supposedly be gamed by studying to the test? Every high paying industry
heavily engages in gatekeeping, because the number of people who want to make
400k/year is far larger than the number of 400k/year jobs available. The
traditional forms of gatekeeping involve requiring that people have the right
personal/familial connections, or have an elite school pedigree (finance,
consulting, law), or that they spend several years and half a million dollars
in post-college schooling (medicine).

The tech industry's preferred form of gatekeeping is asking people to do
algorithmic puzzles, which is far tamer and less exclusionary than what other
industries do. If you believe that the only thing standing between you and
400k/year is a few dozen hours of practicing leetcode, why are you whining
about it instead of taking advantage of the situation to wildly enrich
yourself with a fairly modest amount of effort?

~~~
drblast
I already make plenty of money in my current job and I'm fairly happy.

If I wanted to switch companies I'd be interviewed as if I were a recent
college graduate.

Anyone experienced and good at this isn't going to have time for that because
they're not going to jump through hoops to increase a 275k salary to a 300k
one.

So what the interview selects for are people desperate enough to study hard
enough to fool the interviewer.

And this is one reason the industry leans heavily toward young privileged male
candidates and loves to reinvent the wheel every six months.

~~~
esoterica
If you're experienced and good you can do way better than 300k at some
companies. Also if you think spending a few hours practicing leetcode makes
you desperate, wait till you hear the hoops people have to jump through to
become a doctor or investment banker.

~~~
Chyzwar
There are more interesting things to do. Coding your side project,
contributing to the opensource project, learning a new programming language or
even reading random GitHub project code and issue trackers. All of these
activities help to improve your programming skills more than leetcode. If I am
making max for my local area and I am not willing to relocate, why would I
waste time for interview preparation?

~~~
ggggtez
I don't know what you are replying to the parent poster at all. If you don't
want to interview for a job in a new location, or for more money... then the
advice to study isn't meant for you. Like, sure some people find studying fun
for it's own sake, but the purpose of studying _is for the interview which you
just said you don 't even want to do_.

~~~
Chyzwar
What I am saying that there are more interesting activities that would
actually improve your skills. Leetscode and drilling interview questions are
waste of time even if you are looking for a job. There are places that hire
based on take-home interview, onsite pair programming or based on your
portfolio/recommendation.

------
PopeDotNinja
> However, whether or not a candidate answers a question correctly is not the
> only source of signal during an interview. You can also evaluate their
> process by, for example, observing how long it takes them to finish, how
> clean their code is, and how much they struggle while finding a solution.
> Our analysis shows that this second source of signal (process) is almost as
> predictive as the first (correctness).

I seem to do OK when an interviewer:

\- asks me a question that sounds like a real problem, not a contrived one
(although on occasion I'll have fun with a contrived puzzle if the interviewer
has a sense of humor & makes the process light hearted)

\- doesn't push me down a path that requires me to implement a "simpler"
solution I'd never consider (e.g. asking me a question that clearly wants an
O(N) solution & then pushes me to try the O(2^n solution first)

\- talks like a person with a problem, and not as someone who clearly knows
what they want & simply won't say it

\- doesn't try to "see how I think", because I code as much in my head as I do
on a screen, meaning most of the code I throw into a text editor is the latest
thought in a stream of random ideas until I get to one that works

\- doesn't constantly interrupt me

\- states their actual expectations, such as "I don't expect you to finish,
what I'm really looking for is X"

~~~
kyralis
As a frequent interviewer I will say that you are not alone in that: any
interviewer that isn't following the above guidelines is unlikely to get a
good signal from anyone.

To your final point about expectations: One question that we try to ask
ourselves about any particular interview question is "how may bits of
information are we getting out of this?" Trivia questions are usually less
than 1 - you'll find out if they know some trivia, but you don't honestly
really even care. The best questions are those that get you several bits over
the course of the interview period, rather than a single yes/no answer at the
end. Questions where there is no final end point or where we explicitly do not
expect candidates to finish, are often the most effective; we can explore the
pathways that are working, dive into side channels that seem interesting, and
get data along the way rather than just a checkmark on some individually-
useless problem.

~~~
PopeDotNinja
I like the idea of extracting as many useful bits of information as possible!

------
joker3
There's a problem here. The only thing that Triplebyte can claim based on
their data is that easier programming questions are more predictive of
performance _among candidates who received an offer_. Since candidates who get
offers are (in theory) different from candidates who don't get offers, we
can't necessarily generalize from one population to the other.

There's also a question about how to mix question difficulty. Should you ask
nothing but easy questions, or is it good to throw in a harder question or two
to see how the candidate reacts to something they can't answer? I can see a
good interviewer getting a lot of signal out of that, but in the hands of a
bad interviewer it would not work well.

~~~
avip
Yep. Lot's of "data analysis" without even defining what would be a "false
positive" (someone who did well in an interview, got an offer but turned out
to be a bad hire? how is TB exposed to that critical data exactly?)

~~~
walamaking
It wasn't clear at all in the article what it even meant when it referred to
"final outcome".

------
davidhyde
I think that asking a candidate to perform a code review can be an effective
method of evaluating quite a few desirable qualities. Can they understand
someone else's code? Can they engage in constructive critical discussion? Are
they able to effectively refactor something to make it better? Can they spot
mistakes and do they have an opinion about how to avoid such mistakes?

~~~
djhworld
Yeah that's great but what if they can't reverse a binary tree?

~~~
almost_usual
You spend a lot of work days reversing binary trees?

~~~
reddit_clone
I think he was being facetious. :-)

------
Zelphyr
I see one aspect of this trend of asking programming questions that require a
lot of memorization: We have had for the past ~10-15 years people in the
workforce (and thus acting as interviewers) who went through a public
education system where heavy emphasis was placed on passing tests that
required a lot of memorization.

I'd be interested to know how many of these interviewers actually think
they're able to identify a solid candidate this way? Not to mention, are they
even factoring in how many people don't test well but are otherwise superb
software engineers?

Ultimately it seems like there is a soft element to interviewing that is being
tossed out now, which is: do I think we can work with this guy/gal? Are they
someone that can become part of our team on a personal level? Can they get
good work done? Fizz Buzz can't tell you that. What can tell you that is
experience. It's a hard-to-put-your-finger-on-it X-factor that I think
companies think they can ignore.

~~~
nybblesio
So much this. I've been programming for 30+ years. My brain only has so much
cache space and it dumps frequently. Asking me questions that clearly are
testing my ability to hold large amounts of data in my meat computer isn't
testing my ability to design and write computer programs.

Sure, I'd love to have a "mind palace" like Sherlock. Alas, I do not. I often
admit this as early in the interview process as possible to avoid wasting
time.

~~~
bunderbunder
If anything, it's testing your ability to _mis_ design computer programs.

I hate working on software that was written to be read by an audience with
perfect recall, because, as a human, I just don't have that. Give me code that
assumes I have the memory of a goldfish, and can't keep track of anything that
isn't right in front of my face.

I'm pretty sure that's what half of Dijkstra's papers were trying to say,
weren't they?

~~~
nybblesio
Agreed and Knuth's Literate Programming. Our industry is caught in the event
horizon of the black hole named, "cargo cults".

~~~
bunderbunder
The black hole that gets me is a love of complicated things. It seems like,
given a choice between two different things that are equally capable of
solving the problem, 9 out of 10 hairless apes will pick the one that has more
switches and knobs.

I haven't tried literate programming, but one of the things that entices me
about it is that my instinct says that it's a vaccine against unnecessary
complexity. If you can't express it comprehensibly in both English and code,
there's likely an easier way to do it.

------
je42
I usually ask what's your strongest language; then ask questions about that
programming language. f.e. if it is python:

> how would you explain the with statement to a junior developer ? then
> increasingly difficult questions that go into the language runtime/concepts.

one other favourite question of mine is:

> Imagine, you got a standard website the serves data from a database. When a
> customer types in the url into the browser bar what needs to happen until
> the customer see the website.

> Go as deep as you can in the answering the question.

When you got an answer, you'll see frontend engineers explain more about the
browser, while backend engineers talk more about the backend.

There was one very senior engineer, that actually talked about the ethernet
layer, he talked for more than 15min. Most medior engineers are done in 5mins.
;)

~~~
matwood
There is so much that can happen between the browser making a request and
receiving a response, that someone could completely gloss over the OSI model
and still talk for a very long time.

I've toyed around with starting my answer with what happens when the physical
return key on the keyboard is pressed. Could spend quite a bit of time on what
happens before the browser even knows the return key was pressed :D

~~~
caspar
See [https://github.com/alex/what-happens-when](https://github.com/alex/what-
happens-when)

~~~
je42
hehe. however, if you look at this. do you think this is a server engineer or
a client engineer ? :D

------
acroback
What is important to employers?

1\. A candidate who can solve puzzles but is not willing to do the dirty work
with team, solving production issues, doing debugging, bug fixing with usual
stuff. 2\. Another Candidate, who is willing to learn, is ready to work with
team and do the dirty work.

I am on a hiring committee as a Tech Lead, and I always try to weed out 1.

Works great, we hire as interns and then assess them. Someone from Google who
we hired full time, _was_ detrimental to team's morale, grunting, complaining
about code, complaining about food and what not.

Another experienced smartass was self centered on his skills and didn't want
to teach junior engineers anything or willing to admit he needs to update his
skills. The moment he realized his skills have no values, started attending
pointless conferences. Now his LinkedIn profile has "aware of block chain
technology", "attended machine learning seminars".

I said not to interviewing at Google and FB because I don't have cycles to
spend months on leetcode. Did I erred? Perhaps. But I am sure neither can
provide me same work quality I execute in my current mid size company. I
regret nothing :).

~~~
pmiller2
I didn’t interview with Facebook or Google either, both because I didn’t want
to grind leetcode, and because those companies are pure evil.

~~~
acroback
For me evil is not what I consider. Its pure business, if other party don't
need for my skillset, I aint gonna put my time to change my skillset to suit
theirs.

------
SkyPuncher
The best interviews I've been a part of have been a short cultural style
interview followed by a (roughly) 2 hour paired-programming session on a
relatively simply to-do application.

1\. The problem is pretty well understood (but does offer room for
interpretation).

2\. Provides time to cover all key aspects (Frontend, Backend, Database,
Networking, Debugging, Testing, Caching, etc) in at least some capacity. In
particular, it shows you what areas the developers focus on.

3\. Provides a more relaxed/realist environment. It's also more accommodating
to developers switching stacks - familiar with good programming patterns but
not the specifics of stack (e.g. "Here's how I'd do [some specific task] in
[other stack]. How do I do it here").

4\. It's clearly a throw away task so there's no concern about "interview
labor". It can also be pre-prepped so you don't have to worry about jumping
too far in.

5\. You can cut short with bad candidates and expand the problem for more
complex candidates.

------
komali2
>how clean their code is,

I got bit in the ass by this one as triplebyte itself. They asked me to make a
tic tac toe game, and gave me iirc 30 minutes (less?) to do it. Except, it
wasn't "build a tic tac to" game, first it was "draw a board to the console,"
"take user input from the console," etc a bunch of instructions in a
convoluted path that perhaps another engineer would do when knowing from the
outset that the goal was to build a tic tac toe game in 30 minutes, but not
me.

So we'd get to a portion where I'd be writing a quick test on user input, or
extrapolating something to a function, and the interviewer would say "don't
worry about that, just worry about {getting the grid to print to console or
whatever}."

Later on I got my feedback and they said they were disappointed with my user
input tests and repeated, extractable code in the tic tac toe portion.

Triplebyte is trying to do good things in the interview space but I think
they're still learning. All in all my interview with them was about as
positive an experience as a harried and bad interview could be, from my
perspective.

~~~
sigfubar
You failed your Triplebyte interview because you neglected an extremely
important aspect of the job: communication. You made assumptions about the ask
which turned out to be grossly out of tune with those of the interviewer. In
the real world, engineers are often left holding the bag when other
participants of the process leave out important details. It’s our job to ask
questions and establish the boundaries of each problem before diving into a
solution.

~~~
komali2
As verbatim as I can remember our interaction:

Me: "I'm going to just put some user input checks here, no guarantees they'll
actually input X or O, yea?"

Interviewer: "Oh, don't worry about that, assume for now that you'll get X on
X's turn, O on O's turn."

Later:

Me: "Normally I'd extrapolate this to a function, so let me just - "

Interviewer: "For now, just focus on getting the program to response to the
next user input."

Me: "Ok... well the fastest way to do that right now is just copy paste this
code down here."

Interviewer: "That's fine."

But, it turns out, it was not fine.

I felt I was bamboozled.

~~~
halfjoking
Triplebyte sounds like a bad girlfriend.

When she says "don't worry about it" or "that's fine", you can't take her word
for it.

You need to be psychic and understand her needs without her saying anything.

~~~
dang
Please don't do this here.

------
wiremine
At my consultancy we recently streamlined our interview process:

1\. Phone screen which takes 15 or 20 minutes. 2\. The candidate fills out an
essay, including showing us some code they're proud of. 3\. If the essay ticks
the boxes we conduct a 1 hour on site interview. We use the same a set of
questions for every candidate, so the investment is easy to manage, and our
team has a shared set of expectations on what is good or bad. 4\. If the
interview goes well, we give them a take home assignment. Takes between 2 and
6 hours, depending on how experienced the candidate is. Problem is in C and/or
Python (or both) 5\. We wrap up with a 2 to 3 hour onsite interview. We walk
through the assignment and have a deeper conversation about culture and fit.

The results have been positive for us: we've made some great hires and weeded
out some candidates who weren't a good fit.

We've also been able to scale it down to the process we use for interns.

The 1 hour interview has some typical programming interview questions, but we
wrap them into a real-world example. The goal isn't to prove they know how to
program, but more about allowing them to show us how they think/work out a
problem.

~~~
Zimahl
If you want to do it for junior developers, fine, but as a senior dev, I'm not
doing any take home assignment. It's an immediate pass from me. There is very
little possibility that you can glean anything from my code other than picking
apart it for code review bullshit. I'm not going to waste 2 to 6 hours on this
- done it before, never again.

~~~
JamieF1
And this is why I built a site to allow people to search for jobs based on
interview type. Although I haven't done any proper marketing yet, still a work
in progress ([https://softwarejobs.xyz](https://softwarejobs.xyz))

Everybody is different.

------
dahart
My group likes starting with easy questions and ramping up the difficulty, not
to eliminate people with wrong answers, but for two other reasons: first, to
see whether people understand the questions and whether they try to make up
answers, or ask questions, or say “I don’t know”. Second, to see what the
limits & boundaries of their experience is. We know that people don’t know
everything, and we measure more for potential than for knowledge, but it’s
still useful to understand someone’s experience and exposure level.

More important than question difficulty to me is attitude, and I’d love to see
whether attitude is measurable and how it compares to later performance, but
curiosity and optimism and communication really do go further than right or
wrong on math and engineer questions for me. That point might even be tired
already, I know people say it all the time, but I’m going to keep saying it
because we still have blog posts on question difficulty, when easy vs hard
engineering questions are pretty low on my list of what matters when I’m
hiring.

~~~
ehsankia
At the end of the day, the question itself is a tool, not the goal. When
interviewing, I look at how the person approaches a problem and works through
it. I don't really care if they get to the end or not.

People who complain about memorization and difficulty are kinda missing the
point. Just like learning math at school isn't really about knowing how to do
trig, but being able to think logically and do problem solving.

------
antibland
A few jobs back I was tasked with hiring new developers to bolster a thin
front-end team. The job was very CSS/JavaScript heavy, so I asked questions
that were pertinent to what the candidate would be doing if hired. Of the five
candidates, only one answered all the questions perfectly, and he turned out
to be the biggest bust for us.

The other candidates, after answering some of the harder questions
incorrectly, seemed very upset with themselves. They knew they were cracking a
bit under pressure, but actually showed that they knew the answers when we
chatted further. I hired 3/4 of those people because of how well I felt they'd
do given the opportunity. All three became leads within a year and a half.

I think personality has a lot to do with outcomes. If you are someone who
shows they are hungry to learn and knows how to improve their skills, I will
never dismiss you for screwing up a few coding questions.

~~~
TheOtherHobbes
Personality has a huge amount to do with outcomes. Especially in teams, where
the best teams have a mix of complementary talents.

If you're hiring for Generic Developer Skills you're going to get generic
developers - and much less development than a more flexible approach would
give you.

------
paul7986
One interview of mine asked something i didn't know yet I said there's no
doubt I figured it out via Google. The interview pretty much ended there and
I'm glad it did! Any place or interviewer that says you shouldn't use Google
of OverFlow to get your work done is no place I want to work for.

~~~
darpa_escapee
What, you couldn't derive a solution in a vacuum from first principles? I bet
you think referring to API documentation isn't cheating, too!

~~~
paul7986
Hmmm hard decipher your comment.

But, that interviewer is the type of developer I don't want to work with. A
know it all who only acts like he knows it all and uses Google secretly. Who
puts others down to make themselves look good. AKA an insecure P word I want
nothing to do with.

~~~
darpa_escapee
> Hmmm hard decipher your comment.

I'm being facetious and agree with your sentiment.

------
angarg12
I got my yearly review recently and I got very good feedback. At the same time
I have been doing Leetcode at home for fun, starting with easy problems, and I
get my ass handed to me.

I find it hard to reconcile these two experiences. How can I thrive at a top
tech company while failing to solve an 'easy' coding challenge. It makes me
concerned about what would be of me if I had to look for a new job now.

~~~
jhasbro
Thats exactly what your top tech company wants. For you not to look for a
another job. For you not to try to get a higher salary.

------
minimaxir
This is an interesting post given that up until recently TripleByte's
facebook/Twitter ads were asking very nonrealworld programming questions:
[https://twitter.com/minimaxir/status/1054596563585052673](https://twitter.com/minimaxir/status/1054596563585052673)

Now, the ads ask simpler things like floating point precision and function
variable scoping
([https://www.facebook.com/triplebyte/ads/?ref=page_internal](https://www.facebook.com/triplebyte/ads/?ref=page_internal)
); legit problems, but not sure if they are an indicator of how good a
developer they'd be in the real world working on a CRUD app.

~~~
ummonk
How is that nonrealworld? Dealing with things like endianness is quite normal
for a low level programmer.

~~~
pmiller2
You nailed it. For someone who needs to determine the endianness of a machine
on the fly, the question is trivial to solve; for everyone else, it’s trivial
in the sense of “unimportant.”

~~~
ummonk
Their pre-screening quiz uses a bunch of questions on various technologies to
ensure that some subset of the questions tests you on technologies you're
familiar with. Which is good, since not everyone is a web engineer.

------
curiousDog
The bigger problem is the inconsistency amongst interviewers when judging
candidates. All these articles from TripleByte and Gayle (who've built
business on the flaws of interviewing) focus on the questions instead. Doesn't
matter how hard the question is if the interviewer knows what they're looking
for, are experienced enough, show no nepotism and have good communication
skills.

My worst interview ever was with Facebook when a non-native, new college grad
gave me a Leetcode hard problem in half-broken english and went back to his
work without even looking up or walking with me through the problem.

------
aantix
If they want something that mimics the common, everyday..

Give the candidate a project with 300,000 loc, tell them to make the most
local change possible that fixes the reported bug. Update the tests to reflect
the new logic.

Bonus: discuss architectural changes that would have resolved the bug and/or
improved performance.

~~~
auston
They do something like this with a couple thousand lines of code and failing
tests.

------
xiphias2
,,Hard questions do filter out bad engineers, but they also filter out good
engineers (that is, they have a high false-negative rate). Easy questions, in
contrast, produce fewer false-negatives but more false-positives''

The philosophy at Google is that it's better to filter out 3 good engineers
than to let in a bad one. The consequence of this is that it's really hard to
get kicked out of Google.

The other part (whether it's more important to work on long easier questions
to see how the candidate works on a large code base) is orthogonal reasoning,
and that part may be true, depending on what type of engineers somebody is
looking for.

~~~
ashelmire
> The philosophy at Google is that it's better to filter out 3 good engineers
> than to let in a bad one.

Just fire the bad ones. You're going to get bad ones anyway. At larger
companies you might never notice whether someone is good or bad.

From the way they structure their interviews, it seems like they'll still get
plenty of bad ones - it's just they'll get bad ones that are great at
algorithms, with unknown skill at everything else (like the actual work done).

~~~
marcinzm
>Just fire the bad ones

Who and what decides what a bad software engineer is (note this is very
different than deciding who is a good)? Is it a manager? Managers are single
people so shouldn't be the sole deciding point. So you need to gather
feedback, aggregate it and decide what constitutes bad. Then have that
feedback reviewed by independent people to ensure that no one is being unfair.
That takes time and bureaucracy.

------
taylodl
Another solution to this problem is contract to hire. I realize this is
kicking the can down the road to the contracting firm but hear me out: that's
the business the contracting firm is in. They can get really good at their
hiring practice since that's their core business. That's not our core
business. We've been doing this for the past two years and it's worked out
great. Now you can see how well people do the actual job and if you're not
satisfied, which happens from time to time, just get someone else.

~~~
pmiller2
Why would I leave my W2 job for your contract to hire position? All I can see
here is a company that doesn’t want to fire people who aren’t working out,
possibly to save on benefits and unemployment insurance premiums. And why do
you have so many people not working out that you have to do this? None of this
inspires any confidence.

~~~
extr
Yeah, in my experience asking the basic question of why I would even entertain
moving from my W2 job to a contract position where I assume 100% of the risk
is met with dead air on the other end, followed by a hasty assurance that we
don't really have to go the contract route.

It gives me the feeling that I will be working with people who either don't
know their own value and so are willing to agree to this kind of deal, or are
so poor performing they don't have the leverage to insist on a regular
employment contract.

------
Waterluvian
The thing that bothers me about these questions is that there isn't a shortage
of practical questions to ask that will test if the candidate can truely
contribute to the real problems you're trying to solve.

"Right now our roboticists use a hacked together QT based GUI to manage
customer robot fleet data. It takes 1-10 minutes to load on a slow network and
is hard to add more features to. I know you have far less information than
you'd want but walk through your thought process for how you'd replace this
system over the next 12 months. We can make assumptions."

And then the next 15 mins can be an organic conversation about the problem
space. You can direct the conversation into corners most relevant to their
potential role: "you mentioned using web tech because we discussed how all
usage is across the internet. Can you talk about the merits of Http vs.
websocket?" "How would you ensure that we don't accidentally take every single
customer offline if we centralised our data store?" "What kinds of UI
technology would lend itself to robot mapping? Can we just use Google Maps?"

If you really need to dig deeper into technical prowess, find something
relevant in your conversation and dig deep into it. "We talked about saving
changes to floor layout. Can you whiteboard/laptop how you might implement
undo/redo for floor elements?"

------
wglb
Instead of doing this, read, then follow
[https://sockpuppet.org/blog/2015/03/06/the-hiring-
post/](https://sockpuppet.org/blog/2015/03/06/the-hiring-post/)

------
dmolony
This whole rock star/ninja interview process is ridiculous. They need to paint
a house but they want to hire Michelangelo.

------
high_derivative
Having recently gone through some interviewing (for machine learning
research), a very cynical but overlooked aspect (in this thread) is the
following:

After receiving an offer from a big tech company, the interviewing process has
already completely turned me off from the idea of working there.

Now despite this being a dream position for many and me having no alternative
but to take it currently, the smug interviewers have already gotten me in the
corporate mind-set: No matter, the reputation and salary, treat it like any
other job, do not bother being loyal - they will not be.

So the terrible interview process has at least the advantage of reminding
future employees what they are signing up for.

I wonder what kind of psychological filtering is at play. Do employees feel
loyalty after an interview process that is best described as hazing? Are they
projecting the humiliation they experienced when interviewing future
employees? That's always been my impression.

~~~
seanmcdirmid
Many interview processes generally paint the company in a grimmer picture than
it actually is. So I would do additional research than turning down an offer
because of a bad interviewing experience. Of course, you are completely in
your right to do so, but it might not be the most optimal behavior.

Usually, interviewers are just bad at interviewing and aren't really aware of
what proper behavior is. It actually isn't easy, nor is it a desirable
activity nor do you really get points for it in your performance review.

~~~
high_derivative
Oh I won't turn down the offer, as said, I have no choice and it would be an
irrational career move to do so.

The ultra-negative interviewing experience just completely changed the mindset
with which I will go into that job. This then got me thinking about the hidden
cost of a negative interviewing process.

~~~
seanmcdirmid
Sure, but in general all of the big company interviewing processes will convey
the similar amounts of dread these days. They are so worried about false
positives and have so little concern about false negatives that they push many
candidates away simply by not focusing enough on the "why should I work here"
part of the interview.

------
cshg
Specific examples of what classifies as a “hard” or “easy” interview question
would be very helpful to have reference points and assess one’s own interview
process.

~~~
akhilcacharya
Easy - reverse a string, determine if a string is a palindrome, reverse the
digits of an integer, determine if one string is an anagram of another.

Hard - implement a subset of regex match in optimal time+space, find the
operations required to turn 1 word into another word given a list of
transitory words, find the median of 2 sorted arrays in optimal time, find the
next permuted value.

~~~
pps43
That's more like small/large than easy/hard.

~~~
akhilcacharya
I'd argue finding the next permutation is actually very difficult if you don't
know the standard algorithm.

------
rajeshmr
You realize that this approach is flawed the moment there are blogposts /
books ( e.g. Cracking the coding interview ) on how to crack it. The how to
crack 'x' becomes a field altogether ( coaching / youtube videos / blogs /
books etc ).

Also, platforms like hackerrank are adding fuel to fire. I read the CEO write
somewhere that he wished the below "were taught in schools :

1) Communicating complex ideas with clarity 2) Systems thinking 3) Grunt work
tasks 4) Boundaryless thinking 5) Self-awareness / EQ"

Please note almost all of these are not evaluated on their platform (they
profit from coding tests) or during interviews and almost all are soft /
intangible skills (skills which are not immediately obvious about the
candidate during a typical programming interview). [ Side note : some could
say that coding tests are the problem they have chosen to solve - in which
case, why are they worried about these skills ? Are the companies seeing
coders crack the tests on their platform, while not performing well on the
above skills post hiring ? We could only speculate. ]

All good work is done by teams, and to be effective in a team requires a lot
of intangibles which aren't even assessed in a typical interview.

A better approach could be from this article:
[https://leerob.io/blog/technical-recruiting-is-
broken/](https://leerob.io/blog/technical-recruiting-is-broken/)

Or : A couple of weeks of work with a task being assigned and the mentor or
interviewer looking at how the candidate is approaching the problem and
whether he is able to solve the problem within the time constraints (an easy
task shouldn't take long, and a hard problem shouldn't be short circuited to
give a sub-optimal solution.) and other such observable traits can be
evaluated.

My two cents!

EDIT : Poor wording above (i.e., couple of weeks). A task should be assigned
and evaluated post a time (which is ideal for task completion as per the
interviewer). No constant interaction with the candidate and spending loads of
time with that candidate - that isn't scalable when the demand-supply equation
is imbalanced already.

EDIT 2 : The idea above is not about spending weeks for recruitment. The idea
was about being practical about the kind of questions / tasks that are given
during interview (example : code a feature or fix an issue we have, as another
user has suggested well in the comments). Took me a while to realize we have
missed the point I tried to convey for the logistics of how it should be done.

~~~
selestify
Who has time to spend a few weeks off from their full time job just to
interview?

~~~
davidkellis
I agree with the sentiment, but I noticed recently that TaxJar (I considered
applying there at one point) mandates a trial period along these lines. From
[https://life.taxjar.com/distributed-team-hiring-
process/](https://life.taxjar.com/distributed-team-hiring-process/), "... We
hope a candidate is able to spend somewhere between 80 and 160 hours working
with TaxJar during their trial. ..."

Presumably there are some people who are willing to do a trial, either
concurrent with their existing employment, or on the chance that they will be
hired after the trial.

~~~
rajeshmr
This is good to know, typically it should be in concurrent with the existing
employment. If we plan to switch companies, we commit time apart from existing
office work for interview prep anyway. This interview prep time is what we
would end up committing to work on their tasks etc. I think working on a
project or task from the prospective employer would be much fun than preparing
/ memorizing interview questions etc. What do you think ?

~~~
pmiller2
Most companies won’t allow such a thing, especially if the other company is a
competitor. In any case, doing so would likely require me to take 1-4 weeks
off from my regular job. Again, why would I want to blow all my vacation time
working, just for the possibility of another job?

~~~
rajeshmr
Okay my experience was, i didn't have to take leaves from my regular work. I
had to simply dedicate a couple of hours post office time, to work on the task
(the couple of weeks was given owing to time constraints due to my regular
work). Post office hours, i would have to anyway commit to interview prep if i
had decided to switch anyway is what i meant to say.

In any case, this is not the only method or approach that is the better
alternative :) Maybe it will work for some and not for others. But evaluating
alternatives is a good exercise. Currently we are stuck with this method
because we do not have a better solution to the hiring problem.

------
gumby
I really agree. At companies I've worked at for the last 20 years (at least)
the approach has been to ask pretty simple questions; people who struggle on
those we don't want; people who don't struggle, even when they make silly
mistakes (due to feeling interview pressure) are good.

Questions about the standard library of the programming language in question
are good. Questions about the dusty corners of said library: bad.

And don't ask about floating point: most likely the candidate won't really
know more than the usual things; anybody who really _does_ understand them
will probably give answers over the interviewer's head :-).

------
zinckiwi
I like some coding in interviews. I try to make the tasks fairly real-world
though. I am a UI engineer so my tasks typically fall into one of two buckets:

\- A take home (2-3 hours) task for retrieving tabular data from an API and
displaying it. Here I'm looking for general framework chops, readability, some
design sense.

\- An in-person (~45m) not-quite-pair programming task, with a real computer,
tools, editor etc., for doing a typical UI operation, e.g. truncating text.
Starts simple and gets more complex as time allows: make a function to
truncate text to x chars; now add an ellipsis only if truncation occurred; now
make sure not to truncate in the middle of a word, etc.

------
systematical
I've never been giving given a practical programming test. Nearly all are
useless algorithms. Create a function that takes a integer greater than 0.
Build an array equal in size to the argument. Fill the array with numbers that
when summed together equal zero.

It was worded far worse than that. What exactly is that telling you about the
engineer?

On the flip side I'm asked to code full fledged applications but not to spend
too much time on them... okay...

Another time I was asked to code a luhn algorithm. Oh and do it while a room
of people watches you on giant screen cause that's what your day to day job
will look like... I failed miserably and still got the job. What?!?!?

~~~
kyralis
FWIW, I regularly pass interview candidates who "fail" at certain questions.
The point of those questions isn't to complete it perfectly (and sometimes we
get less information from those, honestly), it's to see how you go about
getting there, _how_ you fail, and how you deal with that failure.

For instance, one of the problems I frequently ask has a structure that really
encourages people to try inventing heuristics to solve the problem, even
though ultimately all of those heuristics fail. Seeing how people react to
"but what if your input looks like this?" questions is often very
enlightening- can they rethink their approach? Do they just keep glomming on
more special cases? Can they deal with someone pointing out that sort of flaw?

------
innocentoldguy
Most programming interviews are a waste of time and energy for everyone
involved and the results are, for the most part, a complete facade. Asking
puzzle questions gives interviewers the feeling that they're really testing
hard for top talent, when in reality they're just demonstrating their lack of
interviewing skills.

I recently interviewed managers at two different technology companies, both
nationally known, for a report I am working on. Most managers admitted their
technical interviews were flawed, but didn't know any other way to assess
skills. They also admitted that a significant number of people they recruit
refuse to even take the technical challenge and end up working elsewhere.

In interviewing a couple dozen engineers, I found most just don't want to
waste their evenings and weekends on a technical puzzle for a job, especially
when there are a lot of companies out there who don't bother with them, so
they end up searching for companies that don't waste their time with technical
challenges.

Another funny thing I discovered during my research is that just under half of
the employees at both companies I've interviewed so far were not able to
successfully complete their own technical challenges.

Another problem with technical challenges is that often times the interviewer
knows less about the topic than the interviewee. I recently went through the
interview process with a local technology company who uses Elixir and Go (both
of which I know). During the onsite interview, the interviewer kept saying
things like, "Don't forget to..." or "You forgot..." I kept explaining that I
didn't need to do as he was suggesting. In the end, my code worked, my tests
worked, and I passed the interview. In spite of this, I was rejected because
the interviewer, "Wasn't feeling it."

I still have a lot of research to do, but I haven't found anything, so far,
that suggests that technical interviews predictably result in top-talent
getting hired. It seems to be the same crapshoot interviewing people without
using technical challenges is, because in the end, most people decide within
the first couple of minutes if they like someone and hire based on that,
regardless of the rest of the interview process.

------
g9yuayon
I just finished reading Bill Kilday's Never Lost Again. It was amazing that
four engineers produced the ground-breaking Google Maps in 16 months.
Algorithm/Math questions used to be really effective at finding engineerings
like those four engineers. There was a reason that Microsoft resorted to
algorithm puzzles in its hay days as well.

It's just unfortunate that there's so much prepping materials online nowadays
that the programming puzzles have become ineffective. It gets worse as many
interviewers were not good enough to ask follow-up questions. For instance,
addition with big integers is a pretty easy interview question, right? But if
a candidate can go as deep as this article:
[https://bearssl.org/bigint.html](https://bearssl.org/bigint.html), I can be
pretty confident that the candidate is really really good.

That said, I personally don't find it necessary to join the rat race. Instead,
I'd suggest engineers just take time to thoroughly study just one book on
algorithm designs. In fact, an introductory book, such as Kleinberg's
Algorithm Design or Udi Manber's Introduction to Algorithms, will be good
enough. It may not get you into Google, but it will likely get you into
another damn good company. The best part of this approach is that passing
interview is really just the byproduct of you trying to become a better
engineer.

------
charliewrites
Another clarification: this data only applies to general programming exercises
and not domain-specific knowledge. For example, if your role requires bio-
informatics knowledge (or ML or AI etc.), then by all means ask about bio-
informatics—even if it means asking harder questions. (We do, however,
recommend keeping the content of general programming exercises as vanilla as
possible, so that you don't filter out someone because they lack mastery of a
subject that you don't actually intend to measure.)

------
weird
On the interview its build a tic tac toe but when you get to the real job they
give you a week to fix a line of code. This industry is fucked up.

------
sytelus
If you are asking interview questions that has one beautiful precise answer,
you are doing it wrong. Good interview question should start with something
very simple that even very beginner can think and answer, then gradually add
complexity and constraints little by little.

Example:

1\. Write function that multiplies two integers.

2\. What if these numbers were real numbers but computer can only operate on
integers? How do we use same number of bytes as ints to hold a real number?

3\. What if I wanted infinite precision? What would be run time of your
algorithm and storage complexity? (don't insist that candidate must hit the
known optimal).

4\. Can I have complex numbers as well?

5\. Imagine complex numbers not only has "i" but also "j" and "k". How do we
handle this?

It is astonishing how many candidates won't be able to move past #2.

The key is to look at how candidate approaches handling complexity, create
representations and use it to craft clean solutions. Whether they eventually
arrive at known optimal/great answers is unimportant.

~~~
Osiris
I couldn't get past #2. Does that mean I'm a bad programmer or just missing
the experience for the particular field?

My point is that questions should be tailored to the work the programmer is
expected to perform.

~~~
sytelus
Above example is not a test of your programming skills. May be you are great
NodeJS ninja who doesn't need to care about any of this. I think there is two
types of hiring that often happens (1) for a specific project, specific task
(2) long term member of the mission that company has.

For #2, I prefer to hire people who are generalist problem solvers. They need
very strong coding skills but more than that they need to be able to operate
in new domains easily and adapt. If I was running a startup, I would need them
to work on MySQL database one day and react-native stuff other day and perhaps
also pick up some of deep learning practitioner skills two months down the
line. Above example question doesn't expect candidate to be familiar with
floating point representation but it is interesting to see what they _might_
have thought if they were the ones doing it. Assuming everyone works with
numbers all the time, people are hopefully familiar with basic issues of
precision, rounding etc.

------
pps43
So they saw a correlation among people they hired and made a conclusion about
people who applied, with no reject inference?

------
ggggtez
>Harder questions ultimately filter out too many qualified candidates to be
optimal.

This was the key sentence if anyone missed it. "Optimal" for whom, exactly?
Not for FAANG certainly. They don't need to worry about filtering too many
candidates out, because they have a nearly infinite pool of applicants, and
infinite money to conduct a search.

They can ask as difficult questions as they want, because they can pass
hundreds and thousands of qualified candidates, and still have plenty more
where that came from.

Edit: If you consider "optimal" to be the expected cost of a hire compared to
the expected profit, it is fully plausible that if your margins are big
enough, asking hard questions is the most effective way to ensure low false-
positives. But as everyone knows, comments on articles about interviews are
never about the economics, it's only about human ego of feeling rejected.

------
StreamBright
Is it a surprise to anybody? My favourite example of interviews going bad is a
tech company in SF asking to find cross currency arbitrage loops with all
known currencies over the phone in the first round hiring data engineers for
the ETL team. I am glad I did not pass that round.

------
kylestlb
I've seen this borne out in practice after administering dozens of phone
screens and in-person interviews over the years. I started off with questions
that were a little too hard and couldn't get a good read of the candidate
unless they happened to have already done the problem on some interview prep
site.

Switching to more practical, simpler problems allowed me to really observe how
they work and solve a coding problem. As the article said, I was also able to
add requirements or features to the problem which let me see how the candidate
adapts to changing requirements, or refactors their own solution to handle a
new edge case. Simpler is generally better if you are timeboxed to 45 minutes.

------
duxup
I changed careers from another technical non programming career to web
development. I got a handful of interviews but I never felt like I was showing
them what I could do... learn.

Until one finally gave me a fairly straightforward homework style project. It
was probably more apt for a noob and I threw myself into it and submitted my
work and explained what it over the phone. I was in the office the next day
and we talked about it and I had an offer.

The homework style interviews are understandably controversial, but at least I
got to show my work, me doing my work, my thought process, outside of a few
moments at a keyboard.

------
alboy
I always thought that the primary quality measured by this popular
interviewing style is "being-like-your-interviewer" which leads naturally to
the avalanche effect due to which the practice becomes even more popular
within the company and simply common sense after a while. The interesting
part, to me, is how exactly similar the methodologies should be for companies
different by size and domain. Even if you assume that some flavour of
technical interview worked for Microfaceboogle, it might as well kill a
copycat company (or may be completely irrelevant to its success or failure).

------
echevil
I wonder if hard questions, especially hard algorithm questions, has high
false positive rate as well. Being good at solving those questions simply
requires you to do tons of practice on leetcode, and those who are willing to
spend tons of time are probably those who find it difficult to get jobs (or
new grads). I don't think the skills for solving hard algorithm questions
correlate well with the actual performance at work. Especially if someone is a
new grad, they could be good at thise questions but don't have a clue about
how real software systems are built.

------
austincheney
Hold on, who is qualifying the outcome?

At one of my jobs I nailed it as the top candidate, by far, of the 79 people
they interviewed in person. The interviewers were looking for competence,
freedom from frameworks, experience and so forth. I have in this line of work
more than 20 years and do it as a hobby so I nailed the interview.

At the job though I worked with a bunch of fresh juniors who only know how to
write code the one way they learned in school. According to them I am shitty
developer because I didn’t write code in the one way they understand.

Who qualifies the outcome?

------
dudul
This article makes a lot of good points. After going through the "implement a
red-black tree on this whiteboard" experience as a more junior dev, I always
promised myself I would never use this kind of stupid questions to hire.

Now, 13 years later, I mostly rely on "homework" type exercises. I think they
address most of the issues. They are more "real world", no time pressure, etc.
However, even those now are being heavily criticized. What's left to be used?

~~~
RomanPushkin
There is a problem with homework, candidate spends couple of hours and you
spend couple of minutes to assess candidates. It's not fair. Google doesn't do
that, Netflix doesn't do that. Why anyone would believe random startup and
spend their weekend unless you're desperate looking for any job?

~~~
dudul
It is both true and false. Yes, the candidate may spend 1 hour (I don't give
difficult homework exercises) while it takes me 15 to 20 minutes to review the
submission, but I have to do it for maybe 5 candidates.

I totally understand the main criticism for homework, it takes time, the
company may never call you back after you poured 2 hours into their stupid
exercise. But it is an attempt at fixing all other alternatives: * the onsite
whiteboarding is BS * using open source contribution is totally unfair to
candidates who don't participate * the "contract for 1 month and then maybe
we'll hire you" is also total BS in my book. Who would leave a FT job with
benefits for a contract that may end?

I'm not sure I can think of any alternative that has 0 drawbacks.

~~~
RomanPushkin
> the candidate may spend 1 hour

> after you poured 2 hours

and at the end it turns out to be 5-10 hours. Companies say "you should be
able to finish it in 1-2 hours", but it's almost never true.

I think alternative is to have 1 hour coding session on a good problem. Most
problems are complicated, but there could be something else. I was once asked
by Uber to implement timer in JavaScript that will update DOM, also create
APIs to stop/start/pause. This kind of a challenge doesn't involve any
algorithm, but it shows your ability to code.

It should be enough to bring you onsite.

But I open source should work. If you contributed to Linux Kernel and it was
accepted - onsite no questions asked. If you have github repo of 1000+ stars -
onsite no questions asked. If you have already passed Google tech screen and
was invited onsite - invite onsite no questions asked.

------
rdiddly
Given the 5th & 6th paragraphs about "correctness signal" and "process signal"
it seems like the obvious solution would be to do both, i.e. ask questions
that are easy alongside those that are hard. The easy ones are "process
questions" and the hard ones are "correctness questions." Or maybe you have a
range of difficulty from easy to hard, and each question has a number attached
to it from 0 to 1 that reflects its intended use.

------
jorblumesea
One issue I've noticed is that the elite companies are trend setters, and
their interview methods are being copied by non elite players.

Perhaps Facebook needs and wants engineers that can bust out A* on the spot,
but I doubt Nordstrom or Starbucks needs that level of talent.

This has changed the field where now to even get an average job at an average
company you need to study at the new normal, whose interview ideas were
designed to look for the top 1% of the field.

------
spullara
I totally agree with this. You shouldn't ask a question where some amount of
luck of figuring out the problem is necessary. Instead ask something practical
and relevant to your domain. Also, leave room for exploration if they sail
through it and recovery for someone that goes down a bad path.

[https://github.com/spullara/interviewcode](https://github.com/spullara/interviewcode)

------
dilyevsky
I’d agree but on the condition interviewer actually know ehat they are doing.

As I gained more experience (300 interviews and counting, baby) I realized
that I pick up more on candidates skills that are not directly related to
their performance on particular question. At this point the question itself is
just a conversation starter and so it’s better if it’s simpler and more broad
because it leaves lots more avenues for the conversation to go.

------
graphememes
I ask what they've built, and how it was done.

This shows you way more than what a technical question can answer.

There are edge cases to this. Expertise positions specifically.

------
insie5
I'm like 0/15 on interviews in the last 2 months. more than half of them were
with foreigners who i could barley understand.

------
kappi
Another use of these kind of interviews is age discrimination to filter out of
older candidates who don't have time or motivation to prepare for this BS.
This is just another fad.. In late 90s there was fad for MSFT style puzzle
interviews..then it went out of fashion.. this will also go out of trend..
just wait for couple of more years.

------
wolfgke
"Easier interview questions are also less stressful, which is an important
advantage. Stress causes candidates to under-perform. But, on the other hand,
when candidates are more comfortable, they perform their true best, which
actually makes interviews more predictive."

I cannot believe that _any_ interview situation is comfortable.

------
throwawayinter2
Interviews are like taking the SAT or ACT. For better or worse they’re highly
standardized and you just need to prepare. There are a bunch of different prep
service like InterviewCake.com, Pramp.com, and PracticeCodingInterview.com
that are good if you feel like you need more than just leetcode grinding.

------
harry8
Where I earn my money is thinking of a solution to a hard problem while not
actually at work. It probably only comes up a few times in a year.

Questions you can answer immediately simply don't test if you can do this. I
have no idea how you would test to see if you can do this.

------
ummonk
Isn't this just saying that easier questions are more correlated with overall
interview performance? That might be a good thing (a sign of consistency), but
it could just as easily be a bad thing (i.e. it doesn't provide unique
signal).

------
crispytx
I took a technical interview today over the phone, and you know what, they
were pretty fair with their questions. I don't know what kind of competition
I'm up against for the position, but very fair questions.

------
dfilppi
I've worked for several companies who made up tests that nobody on staff could
pass, including the authors. But the test did imply that the existing staff
was superhuman, since no applicant could pass.

------
cbau
Interesting post! I appreciate this being shared.

Would Triplebyte mind sharing the data? I'd love if the numbers could speak
for themselves, rather than having to rely on an interpretation of the
numbers.

------
skybrian
It seems like this article would be stronger with some examples. What are some
examples of good and overly-difficult questions, according to TripleByte?

------
forrestthewoods
I’d love for this article to provide examples of what they consider “hard” and
“easy”.

------
jamestimmins
Heads up I think there's a typo here: "questions that carrying".

Interesting piece!

------
crimsonalucard
An example and actual numbers would be a good addition to this article.

------
RomanPushkin
Let it stay here
[https://twitter.com/dhh/status/834146806594433025?lang=en](https://twitter.com/dhh/status/834146806594433025?lang=en)

"Hello, my name is David. I would fail to write bubble sort on a whiteboard"

------
otabdeveloper2
That's because professional outcomes for programmers aren't related to
aptitude. Nobody ever got promoted for writing good code. (But the reverse is
very common.)

------
avatarbl
What alternative would you suggest?

------
drugme
Because they're designed that way. It all comes down to:

* Rejecting far more candidates than you need to -- so you can feel like you're hiring "the top 1 percent"

* Giving yourself the feeling that you have an objective hiring process (when really you don't)

* Making your own team members feel like they're super brilliant and special when really they're not

That's what the modern hiring process is designed to do. And in fact it works
quite well, to serve this purpose.

~~~
CoolGuySteve
Don't forget that they introduce market friction to reduce developer turnover
and reduce salaries.

I suspect one of the reasons Google is so open about their process and the
need to study is so that everyone follows suit. Thereby forcing people to take
days off and do homework for even the most mediocre of positions, causing the
switching costs of interviewing anywhere to become higher.

~~~
LaserToy
Interesting idea. I got an email from Facebook with invitations for the
interview. Replied: “I can do it but i’m Letting you know, I will not spend a
minute preparing for it. So, we can do it today”.

I never heard from this person again, which brings a question - is probability
of passing it without putting personal time is so low it makes no point in
even trying?

~~~
bcassedy
More likely the phrasing of your response provided signals that they shouldn't
proceed.

------
dcplogic
A while ago we had a job applicant who had travelled very far and long to
reach us. (literally from the other side of the world.)

So, as a courtesy we figured, why not spend a few hours extra with this
applicant in the programming test. We set up a laptop with a clean Ubuntu
install, devised a programming test that was quite involved. Not algorithmic
hard, just more complex than what can normally be done within a 20-minute
whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack
overflow/etc access was allowed and encouraged. "Just act as like you would
normally do when solving a problem."

We spent like 2x4 hours devising this problem, based on our codebase (cutting
out something somewhat easily digestible and making it able to run
standalone).

It took like one hour to get productive. Explaining the problem, setting up
editors, compilers, etc.

We took turns, but most of the time someone in the interview team (of two) sat
next to the guy. We did give him some alone time.

This is probably nothing new in terms of interviewing techniques, but to us it
was such a revelation. We learned so much more about the applicant. Perhaps it
worked well with this guy because he happened to be a bit more outgoing than
our typical successful applicant. We'd never felt so confident about giving
someone an offer before.

I'm really looking forward towards testing out this approach with local
candidates to see if we can replicate this "data gathering success".

~~~
peteri
This can be problematic, you waste a lot of time setting up the PC /
explaining the problem.

Last time I was involved with this interview style it always seemed to take an
hour to get setup which meant a long interview of 3-4 hours particularly if a
candidate went down the wrong path.

In the end we optimised for SOLID principles with a blackbox dll that had a
function that slept for 2 seconds and a calling class that had mixed
responsibilities (logging and calling the dll). We started folks off with a
test or two and hoped they'd inject a mock to get rid of the delay and split
logging off into a separate class.

I'm not saying it was a great test but you could do something within an hour
or so then maybe spend half an hour talking through what techniques they'd use
for a more complicated scenarios.

If you can afford the time then more realistic testing is great and I do think
you should try.

------
gammateam
I don't give questions like this primarily because:

\- Workers are going to be around for 15 months or less and they have domain
expertise on 1 stack already and I don't need to screen for how they would
hypothetically function across all stacks

\- Worker's process and resource finding skills are more indicative of the
time they will spend on a task

\- Worker's process includes collaborate use of version control and code
reviews, if they pass the screening but can't really integrate on these things
then thats what will get them booted from the team

It isn't always more expensive to have a not great developer. Look in your
organization and see if what I experience is true for you, and you'll save
everyone a lot of time.

------
james_s_tayler
Why do we talk about this every week on HN?

I'm pretty sure it's either once a week or once every two weeks.

