
How to Interview Engineers - FabioFleitas
http://blog.triplebyte.com/how-to-interview-engineers
======
pklausler
The sad reality of programming interviews is that it's absolutely necessary to
ask several near-trivial questions in order to flush out the candidates with
awesome resumes and impressive degrees who simply have no idea how to analyze
a simple problem and solve it using a computer.

Lately, I've been asking "given the starting and ending times of two calendar
appointments, determine whether or not they conflict." No loops, no fancy
algorithms, no tricks... and asking it has not been a waste of time. I get
people writing doubly-nested loops over all of the seconds in the two
intervals, comparing for equality; I get multi-line predicates full of
redundancy that still yield false positives and negatives; it's depressing as
hell.

~~~
pinot
I'm not a programmer but wouldn't

if (endtime[1] > starttime[2]){status=conflict}

work? Assuming time is encoded in epoch format.

~~~
aldarn
This is close but you need to ensure the start time is before the other ends,
and test for the second one starting before the first also. There's four
cases:

[appointment 1 start] [1 end] <\-- some time --> [appointment 2 start] [2 end]
(case 1 - no overlap, appointment 1 first)

[appointment 1 start] [appointment 2 start] <\-- some time --> [1 end] [2 end]
(case 2 - overlap, appointment 1 first)

[appointment 2 start] [appointment 1 start] <\-- some time --> [2 end] [1 end]
(case 3 - overlap, appointment 2 first)

[appointment 2 start] [2 end] <\-- some time --> [appointment 1 start] [1 end]
(case 4 - no overlap, appointment 2 first)

So you need to do:

if (appointment1.end > appointment2.start AND appointment1.start <
appointment2.end) OR (appointment2.end > appointment1.start AND
appointment2.start < appointment1.end) // conflict

~~~
jon_richards
Your cases don't consider [1 start][2 start][2 end][1 end] or the opposite,
but your pseudocode still covers those cases.

~~~
tzs
> Your cases don't consider [1 start][2 start][2 end][1 end] or the opposite
> [...]

Your notation there, where instead of a pair of start/end pairs you have a
list of tagged times, reminds me of a good approach if one is doing a
generalized version of the problem: given a list of N appointments, find
conflicts.

Make a list of tagged times, where a tagged time is a triplet (time, 1, name)
if appointment named "name" starts at time "time", and is (time, -1, name) if
appointment named "name" ends at "time".

Sort the tagged time list with time ascending as the primary sort key, and the
start/stop tag ascending as the secondary key.

Now to find conflicts you simply scan through the tagged times list, keeping a
running total of the start/end tag values. If the running total is greater
than 0 when you begin to process a given entry, that entry has a conflict with
an earlier appointment, and the running total is how many earlier appointments
it conflicts with.

As described above, this lets you print a list of what appointments have
conflicts with earlier appointments, but it doesn't give an easy way to say
which earlier appointments conflict. If you want to do that, it is
straightforward. Just add a set data structure, and during the scan of tagged
times add "name" to the set when you encounter an appointment's start, and
remove "name" when you encounter an appointment's end. When you find a
conflict, the set contains the names of all of the earlier appointments the
present appointment conflicts with.

The above assumed that two appointments do not conflict if the ending time of
the first is the same as the starting time of the second. If that should be
counted as a conflict, just change the sort so that the secondary key is
sorted descending instead of ascending.

------
padobson
_Spending a week working beside an engineer (or seeing how they complete a
substantial project) almost certainly provides a better measure of their
abilities than watching them solve interview problems for 1 hour_

 _Trial employment is expensive for the company.

[...]

Trial employment (and large take-home projects) are expensive for the
candidate._

I can't help but think there's way to mitigate that expense among several
(dozen) companies.

Triplebyte is already interviewing for other companies, why not set a low bar
and hire everyone who passes it for a week? Charge your clients accordingly.
If you have 25 clients hiring for overlapping skills, have them all pay to
hire a person for a week.

40 hours of of labor / 25 clients = 1.6 hrs of labor per client, or about what
it costs to do a single technical interview.

Meanwhile, the interviewee gets to do a single, focused project for a week,
but effectively interviews with 25 companies!

I can get on Dice right now and find over 100 openings for
python/java/golang/javascript/c++ engineers, certainly all of those companies
would benefit from something like this.

More power to Triplebyte if they can figure out how to take those projects and
turn it into a client deliverable on top of defraying interview costs.

~~~
angryasian
If I'm already employed, it be weird to take a week off just to work for
another company. Also ramp up time takes at least a day or two. This whole
process seems troublesome for both the employee and employer

~~~
padobson
When did you last switch jobs?

I recently went through a round of employment where I quit my job at the
beginning.

Between updating my resume/social networks, brushing up on academic CS,
finding leads, scheduling interviews and follow-up interviews and
managing/negotiating offers, it was absolutely a full-time job.

I can't imagine finding a new programming job while still working at the old
job.

~~~
mattnewton
I think this is a crazy expectation.

I last switched jobs while working at Apple last October. Having the BATNA of
your current employment is crucial to the negotiation process and I can't
imagine forgoing that- if you have no current income that's an insane
bargaining handicap.

Anyone with experience absolutely is not going to go for "quit your job, and
maybe we'll give you a new one after a week, maybe it won't work out and you
are boned"

~~~
stephenbez
I had a YC company that wanted me to do a trial week. They wanted me to quit
my job when I said I couldn't take off a week of vacation.

When I said no, they wanted me to come out Friday through Monday and since the
whole company works on the weekend, they would get 4 days to work with me.

At that point I said I wasn't interested.

~~~
bogomipz
Wow this is next-level outrageous!

Could you share the name of the company so others might avoid them and avoid
wasting their time with such nonsense?

~~~
stephenbez
Since it was in 2014 and I don't know if they still do this, I won't name
them.

I guess the lesson learned is to ask what the interview process looks like
before getting involved.

However for this company, I applied online, they sent me a
InterviewStreet/HackerRank coding test, and only after I passed that did a
recruiter call me, and tell me they wanted me onsite for a week.

~~~
bogomipz
>"However for this company, I applied online, they sent me a
InterviewStreet/HackerRank coding test, and only after I passed that did a
recruiter call me"

This is a huge red flag in my opinion and I would recommend people not agree
to this. Why? Because it shows complete disregard for the candidate. "Pass a
test and you can speak to an actual human being." Its demeaning. It also shows
just how useless many recruiters have become. Part of the job of a recruiter's
job and good ones do this, is to get a candidate excited about interviewing
with the company and get the preliminary deal breaker questions out of the way
to see if it even makes sense to proceed.

~~~
yetihehe
And for me that was a perfect interview. First quick question by email to see
if I was interested, then some simple test to see if I can really program,
then quick interview to see if I have required domain knowledge (all questions
were about knowledge required to program effectively, not abstract "how to
move mount fuji"). For an introvert programmer like me it's a perfect
interview schedule. Also it helps when you are already working and want to
check conditions in other company without risk of being thrown out from your
current job.

~~~
crdoconnor
I gave up on those the last time I wasted 4 hours of my life on one of those
tests and never even got a rejection email. It wasn't a hard test and I didn't
do badly.

Nowadays I just point them at my github. If a body of open source isn't enough
to get an interview I'm not interested in working there.

~~~
Jach
Are you against coding tests in general (especially when you have open source
work to point to) or just as a prerequisite to speaking with someone?

~~~
crdoconnor
Just as a prerequisite.

------
Steeeve
I find it odd that so many comments here highlight the fact that people are
giving interview questions that have little to nothing to do with day-to-day
employment tasks.

It seems to me like an enormous amount of time wasted for both interviewers
and interviewees.

When considering someone for a development/engineering role there are a few
basic questions to figure out:

1\. Does this person have competence within our technology stack?

2\. Can this person communicate clearly?

3\. Is this person productive/can this person be productive in our
environment?

4\. Does this person have characteristics that will allow them to succeed in
our environment?

Making people jump through hoops like some sort of dancing monkey while they
are on a job hunt is needlessly cruel and is a waste of everybody's time.

If you regularly see people who can't succeed with what you consider basic
engineering questions, you either have a recruiting problem or you yourself
have a communication problem.

If you expect an interview candidate to expend 8 hours or more working on
interview tasks, you should be presenting a unique enough opportunity to
justify it.

If you expect an interview candidate to whiteboard a solution to a coding
problem that they received in the room, that should be part of your everyday
work experience and you should go through the process yourself in front of
them so that they have an understanding of what your expectations are.

If you want to review their code and talk with them about it to determine
competency, provide an example of your own code and a set of questions and
answers that would be successful.

Don't put engineers through a gauntlet of challenges that you would have
trouble successfully navigating under the pressure of feeding your own family.
They have more than one company they are interviewing with and each company
has their own focus. Put them in a position to succeed and give them the
chance to do so. That's what you want out of your own people - the ability to
succeed given a reasonable amount of guidance and clear parameters for
success. If you can't provide the guidance or the parameters for success for
an interview, how can anyone expect to succeed in your actual working
environment?

~~~
danielbarla
The question of relevance reminds me of an anecdote. (Before that though; I
tend to agree with your points.)

A distant acquaintance interviewed for a secretary position. It didn't go
well, because she was asked a general knowledge question (something along the
lines of: "name some of the planets in the solar system"). She was furious
with how irrelevant and unfair this was. It makes me wonder though, would I
have hired her? I probably wouldn't have asked that question, but still. When
someone is missing some really basic knowledge - even if it's irrelevant -
it's not a good sign, per se.

~~~
gwbas1c
One of the things I like to do is ask a factual question about a technology
listed on the candidate's resume that isn't required to know in order to hack
something together, but is something that "you should know" if you've been
working in a particular technology for a few years.

These are part of the warm-up, I'll ask one or two right at the beginning of
the interview and then move to the real questions.

Examples are: "How does the autorelease pool work?" (Objective C before
everyone switched to ARC.) or "How do you ensure that an object is garbage
collected," (C# and Java.)

These are more about judging a candidate's real knowledge versus stated
knowledge. Someone who's spent 8 years in Objective C better know how to use
the autorelease pool; and someone with at least a year of experience in a
garbage collected language better know how to ensure that an object is
collected.

~~~
sobani
As someone who has worked in C# for about 8 years: you can't.

You can call `GC.Collect()` and _hope_. But far as I know the GC will do a
best effort and gives no guarantees.

Of course that's besides that fact that I would need strong arguments to
accept a `GC.Collect()` anywhere in the code. But to call it just to ensure an
object is collected? Oh boy..

Depending on the case you could add some code to the `Dispose(bool)` method to
know _when_ your object is collected, but also in that case you risk flipping
a Bozo bit.

------
notadoc
My own 2 cents for interviewing, for engineers and most other vaguely
technical positions but perhaps broadly applicable:

\- Don't ask them to whiteboard, especially don't ask them to whiteboard some
absurdity

\- Don't put a panel of 5-10 people against a single candidate, but if you
must absolutely (why?) do not do it on the first interview

\- Don't ask time-wasting stupid questions about manhole covers, boiling
bagels on mars, overlapping clock hands, or how many eggs fit into a phone
booth

\- Don't put someone into a socially awkward situation

\- If you ask a knowledge question and the response is something like "I don't
know, but I would google it" that should be acceptable if they are
demonstrably capable of using what they find

\- Do ask them to walk through their own projects, their code, their past
work/contributions, etc

\- Do try to effectively convey the corporate culture so the candidate can
know if they are a good fit or not ahead of time

\- Do try to accurately convey what their actual job duties and day-to-day
work will be

~~~
cyrux004
Can you expand on the boiling bagels on mars question ? Havent heard of that
before

~~~
notadoc
It's just a variation on the typical logic/creative thinking questions that a
lot of people and companies obsess over.

"Well they're a great candidate otherwise but they totally failed the boiling
water on alternate planet question, so let's keep searching" \- believe it or
not, that's a real mentality out there. I don't get it and never have.

~~~
collyw
Maybe its just me but I find a lot of algorithmic interview questions are just
as hit and miss as these obscure questions if you need to do it in an
interview.

I had a phone interview with Google and was asked to come up with a variation
of a soduko solver. Couldn't get it there and then, but half an hour later
walking down the road, a really elegant solution came to me.

------
mychael
> background-blind interviews, looking at coding skills, not credentials or
> resumes.

Subtext: "Did you create a famous open source tool, write a book, have patents
in your name or architect some amazing system at a big company? Doesn't
matter! Our hiring process prefers code monkeys who can solve our puzzles
instead of system thinkers."

> An interview can result in a bad engineer being hired and later fired (a
> false positive).

Subtext: "A hire can either be good or bad. It's never management's fault if
it doesn't work out."

~~~
fnord123
>Subtext: "A hire can either be good or bad. It's never management's fault if
it doesn't work out."

I agree that the article gets this wrong. It reminds me of Diego Forlan's time
at Manchester United.

Diego Forlan is/was an Uruguayan striker at Independiente (Arg) where he
scored 36 goals in 77 games (a very good rate). He then moved to Manchester
United (Eng) in January of 2002 and was anticipated to take off. He made 18
appearances in the 2001-2002 season but failed to score. It took him until
October to net his first Premier League goal.

He was loved by the fans as he has/had a great attitude and had some
delightful near misses. He also famously scored twice against Liverpool[1].
But he continued to struggle at United, scoring only 17 goals in 63
appearances until he was sold to Villareal (Esp) in August of 2004. In his
first season at Villareal he won the Pichichi Trophy (golden boot for La Liga)
and the European Golden Boot. What a turnaround!

Forlan went on to have a terrific career and is considered one of the best
Uruguayans to play modern football (soccer). His time at United was certainly
a misstep but it wasn't because he was bad. And it wasn't because United were
bad. It just didn't work. Sometimes it just doesn't work.

[1]
[https://www.youtube.com/watch?v=49Zym0gaZqA](https://www.youtube.com/watch?v=49Zym0gaZqA)

~~~
sqeaky
Did they really get it wrong? Very few candidates are Diego Forlan, very few
wrote books, verfy few created a foundational open source project, etc...

I suspect candidates that wrote the Linux kernel, Rails, Modern Effective C++
etc... don't need a recruiters services. If whatever candidate wrote is not so
big as to land them jobs almost automatically then was it really that big as
to demand consideration separate from college degrees and other ignored
history?

I think we dislike the idea of our history being ignored because we all feel
that we are special, but in reality very few of us are.

~~~
fnord123
>Very few candidates are Diego Forlan

How would one know? Usually we know a friend who works somewhere and they say
the company is shit. Or you work somewhere and someone doesn't work so you end
up thinking they're shit. It's not often that you have both sides of the
story.

>very few wrote books,

Books might indicate a good communicator and independent ability to complete a
task but it's not an indication that someone is a good team member that
fulfills a particular role.

>verfy few created a foundational open source project, etc...

I contend that this actually risks being a Balotelli. Good in the right
atmosphere, but very independent and often stroppy when they don't get their
way. They can be terrific in their element but often they're not good enough
to build a team around.

~~~
sqeaky
I would argue I know because I have done many interviews, not nearly as many
as the author of the article. Most interviewer candidate are about as
qualified as a stuffed animal, except I would allow a stuffed animal in the
office because they tend not to break things.

I agree with you that on your points about books and project founding, but I
would hear a candidate out and not make hard unchangeable presumptions.

As for how do I "know" I can't know, I am only in the interview for an hour or
two. The best I can do is get a sense. If the candidate nails fizzbuzz,
explain the nuances of git v mercurial for 20 minutes and provides a plausible
explanation for his previous place of employments toxicity then that affects
my decision making one way. If it feel as though their reasons are thinly
veiled excuses, they don't know foundational tools or fail at fizzbuzz... well
I don't let interviews proceed to where people fail all three when they start
with some minor points against them. Too many red flags and I call it done.

------
ammon
An interesting point my co-worker made is that, almost by definition, if an
interviewee does not know how to answer a question, they don't know why that
question is important or where that type of problem is applied. So from the
point of view of an interviewee who's can't do well on all the questions
(that's most of us in most interviews) some of the question will always seem
like minutiae / lacking application. (Of course, there's also plenty of just
bad interview questions)

~~~
sidlls
It's entirely possible to not be able to answer a question because the
question is objectively not meaningful, and therefore the candidate has
correctly never considered it important enough to learn about.

~~~
ammon
Totally. Both things happen.

~~~
filipecalasans
Sometimes questions are useful only if you are in the context of the specified
problem. I mean, if you are not struggling or have never struggled in the
problem, you will probably not know the specificities of it. This is specially
true for unexperienced engineers or those who are working on a different field

------
jobu
_" 15% dislike academic CS (and think that talking about CS is a sign that a
candidate will not be productive)"_

That seems really weird. Academic CS isn't really necessary for most
programming jobs, but I can't see how it would ever be a detriment.

~~~
rsyring
I had a guy work for me that was very academic minded regarding programming.
He had a hard time letting go of the "pure" way of doing things and taking a
good/practical approach to just getting stuff done. It can definitely be, but
won't always be, a detriment in my experience.

~~~
bg4
I've worked at places where well-meaning 'just get stuff done' people created
mountains of technical debt that hobbled the project in the long term. Made
them look great, though.

Great programmers should hopefully recognize and implement a proper balance
between practicality and rigor, as necessary. They should also exhibit self-
awareness and self-restraint, which sounds like the real problem for the guy
you mention.

~~~
FLUX-YOU
>I've worked at places where well-meaning 'just get stuff done' people created
mountains of technical debt that hobbled the project in the long term. Made
them look great, though.

Those people were just getting a bit better at building things. They were
building experience. It just so happens you found their bad code that they'll
realize was terrible a few years from now.

We can't simultaneously hate our own previous code and expect everyone else to
delivery perfectly when they first start (or even at mid-level).

------
mpetrovich
We've had success with take-home coding challenges.

After 1-2 technical phone screens, we send applicants a coding challenge for
which they have 48 hours to complete. The coding challenge is representative
of some of the work we do (e.g. Given our API, create a D3 visualization of X)
and should take applicants several hours to complete.

In submissions we look at everything from overall program structure and
approach to applicants' attention to detail and usage of good industry
practices.

This coding challenge quickly weeds out applicants that interview well but
code poorly and is an efficient way for us to see the quality of work we can
expect from them (and conversely, the type of work they can expect to do with
us).

After a successful challenge, we bring applicants in to meet the team. At this
point, it's largely just about culture fit.

~~~
aldarn
Do you specifically hire D3 developers? This sounds like a pretty difficult
"coding challenge" for a non-D3 guy, let alone someone who may not even
primarily use javascript. Having used D3 sparingly before i've found it
something that requires a fair amount of upfront domain knowledge to get much
done beyond something extremely simple (unless of course you copy an existing
example, which is what I always did). This seems unreasonably difficult for
someone with no prior knowledge to accomplish in a 48 hour window whilst
working etc -- you may find you are inadvertently optimizing your interview
process for the unemployed with this kind of time limit / coding investment.

~~~
mpetrovich
No, we're not specifically looking for experienced D3 devs. That particular
challenge can be completed with the simplest D3 viz if desired (of which there
are hundreds of examples online) and doesn't even require prior D3 experience
(I didn't have any when I applied). We do look for a solid working knowledge
of JS, however, and an ability to learn new tools quickly.

The 48 hour time limit is a relatively arbitrary timebox and isn't a hard
limit; we value quality over speed.

------
cl0wnshoes
I've performed interviews and phone screens for years and out of all the
techniques I've tried all of them typically end up with me thinking I'm doing
it wrong. Basic domain questions, problem solving, white boarding, white
boarding along side the person helping them out, coding in an IDE, take home
projects. As of today, in Kansas City, I'm looking at around 4% pass rate, and
close to 90% utter face palming failures.

I check my ego at the door, I try to make the person as comfortable as
possible, small talk, help them out as much as I can, provide them with
answers or explanations when they get something wrong. I've never asked an
algorithm or data structure question, I've rarely asked a patterns question.
Most people can't answer basic every day questions.

Overtime my goal in interviews turned into making sure the interviewee left
with more domain knowledge than when they came in hoping it will help them do
better somewhere else.

Ended up creating a pre-screening service that asks the same basic questions
and in the end provides the user with a study guide based on how they
answered. Currently piloting with two local recruiting companies and I'm
finding the same similar statistics of failure/success. Maybe that makes me
the issue :)

~~~
ubernostrum
You know the quote: if you encounter an asshole first thing in the morning,
you encountered an asshole. If you encounter assholes constantly all day long,
it's probable you're the one who's the asshole.

If you have a 90% rate of "utter face palming failures", the problem isn't
with the people you're interviewing, it's with the interviewer and/or the
interviewing process.

~~~
cl0wnshoes
That's what I keep questioning and why I've tried various methods. After
screening resumes along with coworkers, at various companies I've worked at,
we'll pre-screen or not, bring them in and ask them questions from their
resume.

Resume says 'I'm an expert in SQL'. Great, lets start some every day
foundational questions. What is the difference between an inner join and an
outer join. Why might we use a varchar instead of a char data type? Why do we
use indexes? What is the purpose of a foreign key? A good majority of the
candidates can only answer the join question.

'10 years experience, senior dev in <main language>' can you write a function
that returns the largest number in an unsorted array? Most struggle to even
pseudo code it. Tell me something, anything about interfaces. Basic security
question on SQL injection/xss/csrf/hashing/encryption. Without a doubt most
have only heard of SQL injection and then they get that wrong. Most say
hashing and encryption are the same thing.

I do blame myself, the failure rates are incredibly high. But when resumes
looks great and onsite can't pseudo code a simple loop, or answer basic
questions about something they claim expertise in, what can you do? Require
and call references to make sure the the lead dev with 12 years experience and
decent companies isn't lying? Because the result of most of my interviews
appears that people inflate their resumes and flat out lie.

~~~
ubernostrum
So let me ask you an "easy" question: for your interview questions, are you
consistent in what you ask, and do you have a consistent approach to
evaluating a candidate's response?

I ask this because some of your "foundational" questions are not as simple as
you might think (I could talk for literally _hours_ about XSS and the only
conclusion I could give you would be "there's no sure-fire way to prevent it",
for example), and others border on pop-quiz material, which is not really a
great way to evaluate someone.

Another question that's important to ask: you say you're working with
recruiting companies. Have you tried not doing that? Recruiting companies
don't exist to find you qualified people, they exist to spam you with résumés.
Recruiting companies have been known to flat-out _alter_ résumés to make them
better match the set of keywords you said you wanted. And my own experience is
that recruiting companies are the source of a huge percentage of headaches in
hiring. Cutting out the middlemen can and likely will drastically increase
your success rate.

~~~
cl0wnshoes
I try to be consistent with my questions, but it depends on how well the
candidate is answering them as to the depth I go into a subject. Generally I
ask the candidates about previous projects and work my questions into the
conversations. It usually isn't a rapid fire pop-quiz type interview.

Tell me about a project you had the most fun working on. You tell me you
started working on a SPA in Angular 2 and I ask if you've heard of some common
attacks like XSS or CSRF. You say yes to both and I ask if you had any
preventive measures in place or how you would go about trying to prevent them
(I'd stop you if you kept going on and on). We'll work our way through the
projects stack best we can in the same manner.

We generally used recruiters at all the companies I worked at due to lack of
applications. I agree they're part of the issue. I also phone screen for
recruiters so at least some of them are doing their due diligence (no idea how
they act on my reviews though).

------
0xffff2
As someone with an actual accredited engineering degree, I find it bizarre
that Silicon Valley has decided that the term "engineer" doesn't even need to
be qualified anymore. Even without getting into the flame war over whether
"software engineers" are real engineers or not, there's a very large pool of
engineers to which this post doesn't apply at all. I can't be the only one in
the latter pool that reads HN, can I?

~~~
null0pointer
I used to be reluctant to call myself an engineer. I don't actually get to do
too much engineering in my work. But I've seen too many less qualified people
calling themselves engineers so I might as well... Besides, my degree is from
the Engineering faculty at my university rather than Science where the CS/SE
degrees are. At least that's how I justify it to myself.

------
timebomb
Where I work, we'd be totally happy to ask interviewees how to reverse a
binary search tree. As soon as an actual developer creates a pull request that
does as such. Which hasn't happened yet. So we don't ask about things like
that. And everyone seems happier.

~~~
jkaptur
I don't get it. Programmers are surrounded by trees. The filesystem, your
code's AST, dependency trees, the HTML DOM, any JSON ... why do people
consider it so unreasonable to ask a question about simple manipulation of a
well-understood tree data structure?

~~~
walrus1066
Unless you're writing a database or browser engine, you will never have to
implement a binary tree. So it is effectively trivia knowledge. It's like
asking how the x86 CPU architecture works, because it's the platform on which
most code runs.

------
FilterSweep
Here's a concept I had a question about that I don't think gets discussed that
often...

If a company were to offer a base salary at a mere _third_ of the Google base
salary for the same job, why should an applicant have to copy Google's _depth_
of computer science abstraction in the coding interview?

Is the onus on the developer for _even entertaining_ such a demanding question
at a low pay, or does the current economy favor employers in such a way that
developers have no other choice? Or rather, is the knowledge of available jobs
asymmetric enough to where employers can pay pennies, piss, and peanuts for
A-level employees?

~~~
JoeAltmaier
I don't see a connection between pay and the interview process. Should the
company also make their chairs one-third the height? Have one third the
whiteboard marker colors? Feed you one-third of a lunch?

~~~
vorpalhex
My willingness to jump through hoops is proportionate to how much I want to
work with you. If you pay me well and treat me well, I want to work with you.
If you're trying to hire me for 50% of market, my tolerance is much lower.

~~~
JoeAltmaier
Then the filter would be "The skills of a Google Engineer, but can't get in
there so might work here". All others need not apply?

------
paulie_a
Honestly drop the question/answer aspect for technical checkouts. Offer a
problem then work with them to solve it. The most important aspects are two
fold, What questions do they ask, and now that you are being cooperative and a
team mate, how do they interact with you.

EDIT: But during the entire process keep in mind they are hopefully
interviewing your company. Far too many companies forget this aspect and the
entire process has become lopsided in favor of companies making candidates
jump through hoops.

------
expertentipp
Looking for a job has become a full time job. In depth technical discussions
about the technicalities, then about technical quirks, onsite visits and time
spend on researching what on earth the company is doing and what their project
is about, then the curse of the industry - take home assignments. Once I
signed my last employment contract I said to myself "I'm exhausted, there is
no way I'll move a finger during the first months of employment in this
company".

Also I learned to be more picky. "Want to see my code and apps? Show me your
code, show me how you work!" Eclipse required? Bad workflows with weird
limitations? "This is how we do it here, it works for us" attitude? High
discipline environment with daily standups? Cheapskate workplace/outsourcing
center? Sorry, I'm passing.

------
bogomipz
>"After an engineer passes our process, they go straight to the final
interview at companies we work with (including Apple, Facebook, Dropbox and
Stripe)."

Can anyone from any of these companies confirm this? This strikes me as really
odd. So only one employee from FB, Apple, Dropbox, and Stripe needs to
interview a candidate that Tripplebyte say is a "Go"? I'm having a hard time
believing that.

~~~
zippergz
I don't think "the final interview" means a single person. I took it to mean
the onsite round, with however many people that entails, skipping the
recruiter screen, phone screen, etc.

~~~
bogomipz
So why does it matter whether the first round of interviews is done by this
company or the actual company doing the hiring?

I would argue that as a candidate I would rather interview with as many of my
potential coworkers as possible. Interviewing is a two way street. The more
exposure to the actual people I would be working with allows me to make a
better-informed decision. How is reducing my exposure better for me the
candidate? It seems its really better for Tripplebyte though and maybe the
company looking to hire if they are short-staffed but not the candidate.

~~~
tzs
> So why does it matter whether the first round of interviews is done by this
> company or the actual company doing the hiring?

I think the idea is that if you do the first round with Triplebyte instead of
with Apple, Facebook, Dropbox, or Stripe you are effectively doing the first
round with Apple, Facebook, Dropbox, _AND_ Stripe.

~~~
bogomipz
>"... you are effectively doing the first round with Apple, Facebook, Dropbox,
AND Stripe."

Except that I am not. By doing the first rounds with Tripplebyte I have
squandered an opportunity to learn about and form an opinion about the people
and team I might be working with in the future. The more answers I can get to
my questions the better informed I am.

I guess I don't understand this whole "we will fast track you" as a value
proposition. The only efficiency gain("fast tracking") is gotten by the
company looking to hire. I myself am still subject to the same length of the
interview sequence. Only one is with Tripplebyte and one is with the company
doing the hiring instead of two or more with the company doing the hiring. And
in the process I have given up the opportunity to have more exposure to meet
and talk to the actual team I would be working with.

Should I be so grateful for an opportunity to work for "Apple, Facebook,
Dropbox, AND Stripe"(Tripplebyte seems to beat this drum loudly) that it
doesn't matter whether I think it would be a fit for me on a personal,
cultural and professional level?

~~~
itsdrewmiller
Assuming you don’t pass 100% of on site interviews or you don’t take the first
offer immediately, triplebyte will save you some amount of time. Beyond the
force multiplier for pre-onsite interviews, they also can predict which
companies you will have a higher success rate with for your on site. It’s not
a guaranteed slam dunk for every single person (and you still have to pass
their process) but I think the value proposition is pretty obvious to most
people.

~~~
bogomipz
>"Assuming you don’t pass 100% of on site interviews or you don’t take the
first offer immediately, triplebyte will save you some amount of time."

How exactly do they save me time? Can you elaborate?

>"Beyond the force multiplier for pre-onsite interviews, they also can predict
which companies you will have a higher success rate with for your on site."

What is the "force multiplier" here?

Again this is a job where you will spend many hours of your life, should the
only concern be the one whose interview I will have the highest success rate
with?

>"... but I think the value proposition is pretty obvious to most people."

Again what is that? It's not obvious to me. I am asking sincerely.

~~~
ryandrake
Let me try to answer: Say you plan to interview at Apple, Facebook, Dropbox,
and Stripe. Lets say each company requires 2 rounds of interviews. Without
these guys, you're doing two rounds at each company = 8 interviews. With these
guys, you do one round with them, and one round with each of the companies = 5
interviews. If that's how Triplebyte works, it would be a huge time saver.

~~~
bogomipz
Does every engineer unquestionably want to work for any and all 4 of those
companies? Aren't these very different companies? Is there any significant
commonality here other than Tripplebyte has some connection at all of these?

~~~
ryandrake
Pick N other companies, then. As long as TripleByte has this kind of
relationship with at least 2 of them, it's a time saver. I could easily see
myself interviewing for 20 companies in a single job search. Needing only one
"pre-screen" for all of them would be nothing short of miraculous.

~~~
bogomipz
Let's say you avoided interviewing with 4 future team mates from company
Company A by using Tripplebyte.

And come to find out most of those 4 people you didn't have a chance to
interact with during the interview process are completely toxic and
unprofessional. You only find this out after you have accepted the job.

If you find yourself looking for a new job again in month then Tripplebyte
hasn't really saved you any time have they? I am not sure where this idea
comes form that just because the company is "Apple, Facebook, Dropbox and
Stripe" does not automatically mean that everyone on every team is a great
personally or one you would want to work with.

When I interview with a company and I don't think I am alone in this I am also
interviewing the company. I am thinking "are these people I could work with?"
The novelty of working on Big S.V. Company eventually wears off and if it's a
bad fit for you on personality or culture then you are either going to be
unhappy, looking for a new job or both. I and I imagine many others are OK
with a slower process if it means being able to make a better informed
decision.

------
DenisM
Sometimes I ask candidates to come up with a question for the interview,
something that would allow them to put their best foot forward.

Any thoughts on that?

~~~
roguecoder
The question is really under-constrained and it's not clear what you are
trying to figure out.

I ask things like, "what's the most interesting bug you've encountered?" or
"what have you been excited about learning recently?" to try to find a topic
we can have a conversation about in a domain they are familiar with and
interested in.

~~~
DenisM
Oh, I forgot to add - I actually ask for a coding task they think will put
them in the best light. Like, "hey, I can balance a binary tree, let me show
you". In my mind if someone learned something like this very well they show
their learning potential, and complexity of the task will tell me their self-
assessed level of development.

------
LordHumungous
>I'm arguing for asking easy questions, but then judging fairly harshly.

I agree with this. Better yet ask a multi-part question that has an easy first
part and a harder follow on.

------
danthemanvsqz
Interesting read from the POV of the staffing agency and hiring company. But
why should I jump through those hoops when I don't even know if I'm interested
in the job? I really think the hiring companies need to sell me on them before
they ask me to spend an hour of my time.

~~~
Jemaclus
I agree. I had an interview yesterday where they passed, and the reasoning was
"You didn't seem interested in the job." And I sort of think that's hilarious,
because nobody tried to sell me on the job. Nobody told me exactly what the
job would entail. I never met the hiring manager, even though I was on-site
for 5 hours. And clearly they weren't interested in me... It just seems weird
that I have to be jizzing my pants to work for them, but they don't even have
to try to sell me on the role.

And this goes doubly true for any company that reaches out to me first. Like,
if I apply to your company, then yeah, you have an expectation of me being
interested in the company. But if you reach out to me, then maybe you're the
one who needs to sell yourself to me.

I don't really get that attitude by employers. Whenever I'm interviewing
candidates, I really try to sell them on the role. I try to tell them what's
awesome about it, why they should want to work here over literally anywhere
else in the world, and why you WANT to sit next to us for the next six months
and work with me and my team on projects. Part of my job as an interviewer is
to not only gauge your ability and interest, but to convince you that we're
the best place for you to be.

Or... do interviewers really think I selected their company is the only
company I even considered talking to? We're engineers. We have options. You
(the company) have to meet me at least halfway.

Incidentally, I've often wanted to ask my interviewers to solve a tech
challenge. I don't want to join a company where the team is incompetent and
I'm constantly cleaning up after them. Oddly, that's never a consideration
that companies allow candidates... ;)

------
andrewstuart
One thing I find interesting is the absolute conviction that every interviewer
has that they got it right when the reject a candidate.

No "reject" decision can ever be reconsidered or questioned by anyone, ever.

~~~
roguecoder
If I were in charge of a giant company I'd hire every 50,000th applicant or
whatever, just so we had a control to see if our interview process was worth
the time.

------
altharaz
At my company our engineers defined the following process.

First, we defined the skills we're looking for i.e. Programming / SysAdmin /
Cybersecurity.

Our process goes as follows:

1\. We ask candidates to answer a quizz by phone, with questions in the 3
chosen fields. Duration = 1h.

2\. We ask candidates to solve remotely with Google Docs 5 real-world problems
asking for skills in Algorithmics, Data Modeling, Object-Oriented Programming,
Software Design, and Parsing. We also add a bonus question like "how many
people can get into a train". Duration = 1h.

3\. We have a HR meeting at our office with a non-technical member of the
team. Duration = 1h.

The quizz helps us to remove of our list the candidates that do not have the
technical culture we're looking for, and to see what they already have worked
on.

The 5 problems helps us to see how the candidates can code, the bonus question
helps us to see how they approach new issues and manage their stress.

The final HR meeting is very typical: we try to see if we would be happy to
take a 6 hours long flight with the candidate.

Most candidates do not go through first step, and roughly half on them do not
go through second step.

This simple process definitely helped us to reduce the time spent on hiring
and to make the really good candidates shine.

~~~
Dzugaru
> how many people can get into a train

Can't speak for everyone, but I'd turn away and walked out the instant I hear
this or similar BS (which is totally unrelated to "approaching new problems"
or "stress management" in programming at least)

~~~
supran
You're too forgiving. I turn down interviews when they list "linked lists,
hashing, breadth/depth first search" on their study guide. I've never had to
write a linked list EVER in my career, don't fucking bother me with that shit.
Also, if the position is in a language where linked lists would be stupid
(i.e. python), then I definitely reject that company.

I can explain what everyone of them is, and why you'd want to use them. But
I've never written any one of them and I'm not going to spend the time to
memorize something I can look up in my CLR book or stackoverflow.

It's embarrassing that tech interviewing is still stuck in a 1990s mindset.

~~~
emerongi
I've written at least 5 linked lists in the past year. They're quite useful.

~~~
Apocryphon
What were you using them for?

~~~
mickg10
Not sure about GP, but very recently, an open-addressing hashtable that could
be traversed in both reverse modification and reverse insertion time order.
There are actually some interesting subtleties when doing this for open-
addressing hashtables (i.e., entries (and pointers to them) move around when
the table rehashes).

------
bambax
An alternative to an "interview" would be something more like an exam: have
candidates try to solve a (relatively simple) problem in limited time, with or
without external resources.

Candidates would be watched, would know they're being watched, but would not
see themselves being watched, which would probably take a lot of stress out.

You could even test many candidates at the same time this way. And you could
setup a system where each candidate could ask questions if needed, without
being heard by other candidates (like in language learning classrooms).

This must have been tried? but I wonder why it's not even discussed in the
post, as an alternative?

~~~
cr0sh
My current employer does this.

You're sat in the conference room, given a machine that has a screen monitor
program running, and a set of tasks to perform on the code. You are told where
to find the code in the introduction on the sheet of instructions. You are
allowed to use any resource via the computer (so nothing you've brought in
like a phone or laptop is allowed; basically, all work has to be "shown").
Then...go.

Outside the room, the dev team watches the process. We want to see how you go
about each problem, what resources you use, etc.

It's always surprising the number of people who go for an hour or more and do
nothing. Just sit there (or seemingly sit there - there's no keyboard or mouse
activity).

When I interviewed, they actually gave me an online test; the job was supposed
to be for javascript, but the online was in PHP (which my prior position used,
and I had been using for years). Passed that, and came in for the on-site
part. I thought it was odd that they first tested me for PHP. Did a bit of
face-to-face interview, then they wanted me to do the watched javascript
portion (single page app).

I first sat and read the problems, formulated some possibilities in my head,
then jumped in. 30 minutes later I was done, and popped open a new file to
write something to that effect (figuring if they were watching, they'd come
back in). I sat for several more minutes, then they came back.

"How's it going?"

"Alright...umm, hey, I'm done."

"No way! Are you serious? Are you sure?"

Odd...

"Yes."

Apparently they had never had anyone finish the test that quickly, and get it
completely right - especially someone who also had PHP skills, and had already
demo'd them in the first test, which was also done correctly. I found this
rather odd, as neither test was that particularly difficult.

Apparently, though, it's not uncommon in our industry. I'm not sure what to
make of that...

------
rf15
> 9\. Make decisions based on max skill, not average or min skill

This is a strict specialist search - here, too you should make a decision
depending on your requirements and future outlook: Do you need a generalist
that can branch out and specialise, or strictly a specialist that only does
the thing you need right now?

(Also let's not argue which is better, specialist or generalist - deep down,
each of us already knows and made the choice ;) )

------
partycoder
You can ask someone to solve a rubik cube in under a minute.

Or you can ask someone to beat a Go bot that is ranked over 5 kyu.

Or you can ask someone to find adjacent zeroes in a matrix of numbers in under
15 minutes.

All those exercises, however tricky, do not correlate much with how smart you
are or what your programming abilities are.

------
pavlakoos
Some engineers I interviewed got "offended" by ElonMusk-like questions and
refused to answer :)

~~~
andrewstuart
If you really ask questions like that then you shouldn't be allowed to be
involved in recruiting.

~~~
pavlakoos
Why?

------
Jemaclus
IMO, hiring is like a funnel. At the top of the funnel is the entire universe
of people. At the bottom of the funnel is your hire. The goal is to filter out
as many people as possible as quickly as possible. One key point is whether or
not they can write code.

Keep in mind __writing code __is the _BARE MINIMUM_ required for a job. If
they can't do that, they shouldn't be on-site in the first place. If you have
your candidate taking a day off work and taking a day worth of engineering
talent off engineering-related tasks, and you're spending that time asking
them to write code, you've failed to apply the filter higher up in the funnel.

Ideally, you verify their coding ability BEFORE you bring them into the
office, whether via a tech phone call, live-coding session, or take-home
assignment. Before the haters start saying "but they can fake it!", hold your
horses. Once you've established that the candidate can code, when you bring
them on-site, you _quickly VERIFY_ your previous conclusions. You quickly
_VERIFY_ that the coding assignment you gave them was completed by them. This
should take no more than 15-20 minutes. Maybe you just have them walk through
a chunk of code. Maybe you ask them to recreate their answer. Something.
Anything.

But if you send in three different engineers, and they all spend the entire
hour asking the candidate to solve programming tasks, you're literally
evaluating the BARE MINIMUM of the hiring requirements. And there's so, so, so
much more to being a good engineer than writing code.

Outside of a QUICK verification, your on-site questions, imo, should NOT be
writing code (or even pseudo-code). They should be about previous projects,
architecture, personality, professionalism, punctuality, communication,
teamwork, autonomy, and collaboration skills -- all the non-tangibles that
answer the basic question of "Do I want to sit next to this person for the
next 6-12 months?"

If your idea of a good interview is to ask someone to generate a calendar
between two dates or sort a list or find the minimum integer in a list, you're
completely, totally missing the point. You're evaluating the bare minimum to
get them in the door -- and if you're doing that in-person, you're wasting
your time. You should have filtered that out before they walked in the door.

Remember, these candidates are often lying to their bosses and calling in sick
or using up valuable PTO to take an entire day off work to come sit in your
office. Don't waste their time with redundant tests and things that you could
have figured out before they walked in the door.

(And dear god, some of the questions in these comments are... awful... at
evaluating anything except, I dunno, getting the right answer. Yikes.)

------
pards
"There is not a single set of skills that define a good _programer_. ... No
engineer can be strong in all _off_ these areas."

The ability to spell is one of them.

------
rezashirazian
I wouldn't use TripleByte to hire or get hired, but that was a well written
article.

------
gaylemcd
I agree with much of this. Some additional comments:

> Ask questions as close as possible to real work: This is achieved by making
> interview question as similar as possible to the job you want the candidate
> to do (or to the skill you're trying to measure).

This "or" thing in parentheses is really important. Interviews should not be
"as similar as possible to the job." They should be all about the skill you
want to measure. And there's quite a difference between these.

People can be trained, assuming you have time and resources.

The point of standard algorithm interviews _is_ to measure a particular skill:
intelligence. The belief behind this is smart people will tend to solve
problems well in the real world. It's okay if they don't know all the skills
they need; they can learn them. (And if there's a particularly challenging
skill, then maybe you need to add that into the process.)

> Avoid hard questions: If a candidate solves a really hard question well,
> that tells you a lot about their skill. However, because the question is
> hard, most candidates will fail to solve it well. The expected amount of
> information gained from a question, then, is heavily impacted by the
> difficulty of the question. We find that the optimal difficulty level is
> significantly easier than most interviewers guess. This effect is amplified
> by the fact that there are two sources of signal when interviewing a
> candidate: whether they give the “correct” answer to a question, and their
> process / how easily they arrive at that answer.

You don't give examples of what qualifies as hard vs. easy, so it's a little
difficult to judge this. But generally, I'd advocate for people asking harder
questions.

When a question is easy, a little thing -- small point of confusion, etc --
can make a big difference in whether the candidate seems to have gotten to
that answer well or not.

When a question is hard, this is when you can actually see a great
differential between great vs. good vs. okay. You can offer help and see how
effectively they pick up on that advice.

If your interviewers are just throwing out a question and then sitting back
and seeing what happens, then yeah, candidates won't be able to tackle a hard
question. But that's not what they should be doing.

To be clear here: hard does not mean "ask about some obscure data structure
[skip lists, etc]." It means a challenging question involving common knowledge
(hash tables, strings, arrays, maybe binary search trees).

> 5\. Ask every candidate the same questions: [...] there is no justification
> for asking different questions to different candidates. If you evaluate
> different candidates for the same job in different ways, you are introducing
> noise.

I think this is overplayed a little.

If you're doing algorithm-style interviews, any question should be basically
evaluating the same skill (problem solving skills). If I ask my favorite
question and you ask your favorite question, but they're evaluating the same
skill, then it doesn't really matter. There's nothing to make one question
better than another. Why not let each interviewer ask the question that they
like?

Now, if every interviewer asks a different question, you won't be able to give
a well-defined metric on what good performance looks like for that problem.
But that's okay. You can't really do that anyway.

When companies give well-defined metrics, they get too rigid. The candidate
takes slightly longer to complete the code because the candidate thought
things through more thoroughly, and now they're pushed into a lower performing
bucket. And that's wrong.

These standardized metrics sound good in theory, but they don't work well in
reality (for algorithm interviews, anyway).

~~~
ammon
So, we've gathered data on exactly these points over the last 2 years. And
what we've found that the decision that gets the best signal is often not what
feels most accurate to the interviewer.

For example, my guess before running these experiments would have been that
simply looking at progress (how far a candidate gets through a problem) would
be a bad measure of interview performance. I'd expect that things like style
and how communication how careful a candidate was being would render pure
progress a mad metric. However, when we compare pure progress to a subjective
score the interviewer gives the candidate (ignoring specific reasons, does the
interviewer think the candidate is a good engineer after the interview), we
found that pure progress is more predictive of success at companies! (To be
clear, we don't only look at progress at Tryplebyte. We've also found other
things to be predictive.)

The same is true (among our candidates at least) for question difficulty.
Easy, straightforward question (write a command line interface to store and
retrieve key-value pairs) are more predictive of success at companies than
question that try to target intelligence (calculate how much water would
collect in a histogram)

~~~
famil
Have you tried IQ testing instead of algorithm questions to measure
intelligence?

~~~
roguecoder
There is no such thing as a single "intelligence". "IQ" is measuring at least
three different forms of cognitive ability, is an inherently flawed and
culturally biased tool and may be illegal to administer unless you can prove
it's relevant to the work.

Additionally, what matters is not natural talent but the set of skills and
techniques an individual has built up using those talents, compensating for
their weaknesses and taking advantage of their strengths. I've met plenty of
very, very smart people who flounder the first time they encounter a code base
they can't hold entirely in their head at once: someone who was less "smart",
with a smaller working memory, but who has been developing skill with
abstraction and system metaphors since CS 101 is often a better actual
developer.

The myth that developers need to be "smart" is pernicious, and the cause of
most of the really horrific code bases in this industry.

~~~
dragonwriter
> [IQ tests] may be illegal to administer unless you can prove it's relevant
> to the work.

Not any more than any other means of assessment on which outcomes differ on a
protected axis of discrimination like race; yes, the rule was first
articulated in a case involving a fairly blatant use of IQ tests to effect
racial discrimination, but it is by no means restricted to IQ tests.

------
known
quiz != interview

