
Three hundred programming interviews in thirty days - braythwayt
https://data.triplebyte.com/three-hundred-programming-interviews-in-thirty-days-12c23c26b5ba
======
dikaiosune
I could have sworn I already read this, turns out the date on the article
isn't correct, I think this was written in June '15:

[https://news.ycombinator.com/item?id=9766816](https://news.ycombinator.com/item?id=9766816)

Which makes sense given that I tried out their online quiz this summer after
reading this exact post.

EDIT: Here's a link to when it was posted to LinkedIn, June 23rd, 2015:

[https://www.linkedin.com/pulse/three-hundred-programming-
int...](https://www.linkedin.com/pulse/three-hundred-programming-interviews-
thirty-days-harjeet-taggar)

~~~
braythwayt
My bad! I did an HN search for the link before submitting it, I find that gets
around cases where people change the title. But I could have done a title
search as well.

~~~
dikaiosune
Well it's clearly not that much of a problem, HN has such an appetite for what
they're doing/writing about that it's back on the front page. Also for what
it's worth even though I'd previously read the article I made it about halfway
through before it clicked for me that I'd already seen it.

------
Mc_Big_G
_" we’ve come up with a replacement for resume screens, and shown that it
works well"_

No you haven't. You might have, but you'll have no way of knowing that until
they've been on the job at least 3 months.

Personally, I think it's bullshit that I can show employers all the things
I've built and they still expect me to write a reverse sort btree or whatever
the fuck thing that has nothing to do with web development.

College grads are perfect at passing these tests and they have no fucking idea
what they are doing, but will get more offers and better pay every time.

Experience counts and if you disregard it then I'm firmly in the "Fuck you, I
don't want to work for you" camp.

Unfortunately, I've resigned myself to memorizing the typical interview
questions or going into management. I'd love to stay a dev, but being
ruthlessly quizzed about shit that you never use in real life makes changing
jobs a pain in the ass.

~~~
odonnellryan
> Personally, I think it's bullshit that I can show employers all the things
> I've built and they still expect me to write a reverse sort btree or
> whatever the fuck thing that has nothing to do with web development.

Aye. It's a hard problem: as an employer, you want to be _pretty sure_ that
you're getting someone on team that's going to bring you value.

How do you do that? I guess ask them as much as you can.

You definitely present a good problem, it is the same they're trying to solve
over at triplebyte.

> College grads are perfect at passing these tests and they have no fucking
> idea what they are doing, but will get more offers and better pay every
> time.

Yeah, you have to study for interviews. It sucks, but almost every software
company asks the same questions. Just prepare, I guess, and it'll pay off with
that higher salary.

> makes changing jobs a pain in the ass.

You might have nailed it here. Employee retention is good for everyone!

I think the solution is you need connections. When someone works at the
company and they know you're a good dev they aren't going to ask you questions
they ask other developers.

~~~
hamitron
>> makes changing jobs a pain in the ass.

>You might have nailed it here. Employee retention is good >for everyone!

I'd say having a good work environment is good for employee retention, not
arbitrary limitations to prevent people from switching jobs.

~~~
odonnellryan
Yeah, it's the XY problem. Sorta.

------
adambom
I would applaud the author on trying to take a data-driven approach to
discover the best hiring signals, but I must offer one major critique of the
approach: These data are ultimately not too meaningful unless you correlate
actual job performance with interview performance. By the author's own
admission, interviews are often faulty. By correlating final-round interview
performance with early-round interview performance, you are really only
predicting how candidates will perform in the final-round interview, which is
not necessarily a good predictor of long-term job performance.

~~~
lqdc13
Sure, but this is still really useful for many companies in the interviewing
space such as hackerrank, hackerearth etc.

They're optimizing for hiring, not job performance.

~~~
vonmoltke
Which seems totally at odds with what their clients want and furthers the
perception that tech interviewing has become a hazing ritual totally detached
from evaluating the ability of the interviewee to actually do the job.

~~~
sjg007
Beware premature optimization.

------
btilly
I certainly hope that they don't ever assume that experienced candidates don't
need a programming test.

There is a particular kind of dysfunctional candidate who once was a
programmer, graduated to being a lead programmer, and then became an
architect. As an architect they stop programming, and their programming
becomes rusty. Once that happens, the feasibility of their proposed
"architectures" becomes steadily less practical.

You can detect them by their first getting offended that they would be asked
to program during an interview, and then failing to actually write working
code.

By contrast people who really do program have no objection to being asked to
program in an interview. To the contrary being asked to actually program is
seen as a positive sign that they won't have co-workers to deal with who can't
actually program. (This is an unfortunate experience that most programmers try
not to repeat.)

~~~
jkot
I write lot of code, but I refuse to do coding at interview. For me its sign
of dysfunctional company. But I do not mind to do pair coding, and fix an
actual bug.

There are more reasons, in short it takes me couple of hours/days to warm up
and grasp the problem.

And secondly I refuse to waste my time, on half arsed throw-away code, with
zero results.

~~~
krat0sprakhar
> And secondly I refuse to waste my time, on half arsed throw-away code, with
> zero results.

That's a very wierd thing to say. Does that mean that each and every side-
project that you do ends up being substantive? If writing few lines of code
gets you a job at a good company where you can spend your time working on
exciting problems how is that wasteful?

I do program a lot too, and I typically don't mind writing code in an
interview. The only thing that sets me off are stupid level-order binary-tree
traversal questions which test nothing but rote-memorization skills.

~~~
sjg007
I mean you should know how the basic data structures work. Not necessarily for
a web dev job but for a software engineering job. Data structure interview
questions are a good small concrete problem to reason about.

But they completely ignore the other 80% of the job which is design, soft
skills, estimates, timelines, code reviews etc...

~~~
krat0sprakhar
> I mean you should know how the basic data structures work

Oh I'm not discounting that. I'm just saying that off the bat binary-tree
traversal don't really test my understanding of the data structure. Recently
in an interview I was asked a problem which eventually required using a
binary-search-tree to improve the running time. This, I believe, is the right
way to test but it requires the employer to spend more time designing
interview questions rather than relying on a problem bank[0]. (which is a
strong filter for good companies in itself)

> But they completely ignore the other 80% of the job which is design, soft
> skills, estimates, timelines, code reviews etc...

This is true but this is really really hard to test. In the interviews that
I've been giving, I've indeed been asked design questions. I assume soft
skills are implicitly tested while speaking to the interviewer (to some
extent, at least). But other factors, AFAIK are very hard to test. Also, I
would argue that a candidate doesn't need to know everything coming in.
Estimates, timelines etc can be learnt on the job.

IMO, if somehow employers could find a way to reasonably trust that the code I
have on my Github is indeed written by me then I think that should be strong
indicator of my competence. Evaluating candidates who don't have any code in
the open is a different matter altogether to which I have no solution.

[0] - [http://leetcode.com](http://leetcode.com)

------
vdnkh
>As soon as we started doing them however, I saw a problem. Almost everyone
was passing. Our filter was not filtering. We tried extending the duration of
the interviews to probe deeper and looking at code over Google hangouts.
Still, the pass rate remained too high.

To me, it seems like your team isn't that good at judging ability. To
compensate, you rely on coding questions. This what the industry has been
doing for years. What exactly is innovative here? I see the need for coding
questions (and think that your method sounds pretty good) but too often
culture fit & 'soft skills' are overlooked in favor of cramming as many
technical questions as possible.

One of the best experiences I had on an interview involved no coding on-site
at all. I completed a project for them, they liked it, I got an interview. 3
hours of culture/team/work philosophy. There were no surprises - if I wasn't
"technical enough", I wouldn't have been let in the door. It took time up
front, but at least the interview itself wasn't waste of time, PTO, and money.

~~~
ZanyProgrammer
3 hours of soft questions seems much more preferable to 3 hours of
whiteboarding data structures. Hopefully your project wasn't a bunch of
questions from an undergraduate data structures/algorithms class.

~~~
vdnkh
They were tough, insightful soft questions. And I asked a lot of them too.
Gave me a good feel for how they worked and what they valued (which is what I
really want to know). The challenge was pretty tough but it wasn't just graded
on performance. The team lead came in with feedback from each member which was
really good. Stuff like "you could have done this with less objects" or
questions for me to resolve for them. I've used the feedback to improve my
code since.

------
minimaxir
What the heck is going on with the statistical explanation?

The chart's y-axis is labeled "correlation coefficient" but the text blurb
analyzing it describes the definition of R^2 instead (% of variation of Y
explained by X). The clarification at the end does not disambiguate this.
Also, how do you numerically quantify "interview performance"/Y?

Additionally 3 out of 6 of the variables have error bars crossing 0, which
means that the correlation _could be positive or negative_ but we don't know
(it would handily fail a statistical test of significance at p = 0.05
significance). Why were they included at all?

Many startups (including statistical startups!) have been playing hard and
lose with statistical analysis lately and it's getting annoying.

~~~
gragas
I completely agree.

I also would not consider an r-squared value of 0.47 indicative of strong
correlation between two variables. I guess the point is that the composite
quiz score is _more_ correlated with performance than traditional methods, but
I still get the impression that someone is overselling something.

------
pklausler
I've conducted a lot of programming interviews over the past three decades. I
think that it's easier to do at places like Google where the objective is to
avoid a Type I (false positive) error and it's expected that a lot of time and
money will be spent to identify the best hires. It's harder in other contexts
where Type II errors matter more.

I have a big portfolio of technical questions, but I usually stick to a small
set that I consider to be well calibrated by posing them to smart coworkers
and seeing what they do with them.

I've found, counterintuitively, that the best indicators are the questions
that deal with "programming in the small", such as coming up with a predicate
expression to describe a test for a simple relationship between a few values,
because either you can do it easily or you can't do it at all. And programming
in the large depends on a lot of cases of programming in the small.

------
lewisl9029
I gave the TripleByte application a try very recently, but this step wasn't a
part of it:

> 1\. A fizzbuzz-like programming assignment. Applicants completed two simple
> problems. We tracked the time to complete each, and manually graded each on
> correctness and code quality.

Instead, I was presented with a single, rather convoluted, 1-hour long
programming problem that was _most definitely not_ anywhere close to the
difficulty level of a FizzBuzz test (the solution they're looking for probably
involved dynamic programming, which I honestly haven't had much practice
with). This was before I even spoke to any humans on the phone screen or
interview steps of the process.

I couldn't finish it in 1 hour (I kept going at it on my own time for another
2-3 hours before I finally arrived at a sub-optimal solution, at which point I
had enough). I'm sure there are plenty of people who are much better at this
than I am who can finish it in an hour. But I do want to question the wisdom
of using something of this level of difficulty as a screening test.

Your clients seem to value the "Product Programmer" very highly. Using a
question like that as a screener will weed out a significant portion of these
"Product Programmers" before they ever get a chance to showcase their product
skills. Most "Product Programmers" I know (myself included) don't do enough
convoluted dynamic programming problems in their spare time to be able to
solve a dynamic programming problem like that within an hour with any level of
certainty (and yes, solving problems like that _quickly_ involves either
extraordinary talent or lots and lots of practice). Instead, in their spare
time, these "Product Programmers" like to build things, and learn new
technologies and techniques to help them build things in a simpler, more
maintainable, and more productive way.

------
lordnacho
The best programmers I've worked with had this quality:

They'd always heard of everything, and they had a good idea of what everything
was, even though it was quite far from the current task to hand. For instance,
plenty of people have not had the opportunity to work with functional
languages, yet still have a good grasp of why they're useful, what languages
there are, who uses them, and so on. They tend to see coding as a vast field
where you can make many valid decisions, each with their own pros and cons.

The kind of people I'd root out are the ones who merely see programming as a
tool. Something along the lines of "I've got this data in a spreadsheet, I
need to calculate this statistic, so here we go..." They almost always end up
creating an unmaintainable mess, and they never mention any well known
pattern. Quite hard to identify in an interview, though. Perhaps the lack of
key words that suggest a deeper understanding?

(Hey, that's an interesting thought. You could machine learn what words good
devs use vs less good ones.)

As for coding in the interview, I think this can only root out the absolute
novices. It's quite hard to think of a short test that will demonstrate that
someone can't write elegant code. Perhaps a take-home, open-ended test would
work, but you're gonna lose people who don't want to spend a whole weekend
doing it.

------
ap22213
15 years ago you could get a software job by showing some passion,
intelligence, a CS degree from a average university, and some knowledge of
memory / space complexity of typical data structures and algorithms. And, if
the hiring manager (usually a programmer) liked you? Job offer on the spot.
Sure, you got paid 50-60k / year, but it was what it was. And, you actually
got to work with an entire team of people under reasonable deadlines, 40 hours
a week, with a real office, with clear objectives and scope, and an actual
schedule. You got to actually use your brain and some creativity (not just
google the 'best' answer or look on stack overflow or steal some code from
github). Projects sometimes failed, almost always because of requirement
failures, and sometimes the code was beautiful; most of the time it was
average; sometimes the code was really bad.

Today, you have to go through hours and hours of phone screens, technical
interviews, team meet and greets, etc. You have to know how to implement a
distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the
code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's
such a time consuming process. And, the smart (or motivated) people know how
to game the interview anyway (there are lots of resources out there on how to
beat the interview process). And, even when you ace the interview it's still
often difficult to get a straight answer on whether your hired or not
(strangely more so if you're over 40).

They say that software developers are in high demand... I say that people just
hate the idea that software developers make a respectable wage.

But, the funny thing is, after working in this field for over 15 years, after
working with 100s of people and seeing millions of lines of code, the truth is
that the projects are still failing, and the code is just as bad as it ever
was. And, projects still fail most often because of bad requirements, poor
vision, and bad management. In fact, I drink the koolaid: I hire people who
ace the interview, and you know what I've found? There's a 50/50 chance that
they're just as good as the person who I hired who failed parts of the
interview (sometimes I'm desperate). And, those people don't bring down the
project - it's the guy in the corner office.

Luckily, I was smart enough to save a lot of money over the years. I'm just
too old for this shit.

------
droithomme
This article is extremely interesting, but it seems to be equating success in
interview decision with competent performance at the job, which has not
actually been established yet. It is interesting though to see which
components of pre- and early-interview evaluation ended up being related to
the decision to hire.

------
nordsieck
Sadly, this is almost certainly illegal in the US due to the 4/5ths rule
([http://adverseimpact.org/CalculatingAdverseImpact/Four-
Fifth...](http://adverseimpact.org/CalculatingAdverseImpact/Four-
FifthsRule.htm)).

~~~
adambom
The language in that description refers to "protected groups". As long as the
author is not calculating selection rates for groups of candidates based on
some special criteria, such as race, gender, age, etc. he should be ok, right?

~~~
nordsieck
That's not nearly enough. The 4/5ths rule is designed to identify "structural
discrimination", or cases where no discriminatory act can be identified, but
the outcomes are nonetheless considered discriminatory. As long as a lawyer
can show that some protected group is disadvantaged by the quiz, that's enough
for it to violate the rule.

~~~
danieltillett
I am not a lawyer, but I thought the defence was if you can show that the
screening criteria correlated with on the job performance it is OK. No matter
what you think of protected groups using a criteria that does not correlate
with performance is not very smart anyway.

~~~
nordsieck
There may be some certification process you can go through to get a test
approved, but mere job applicability isn't enough. For example, this
firefighting test was found to violate the 4/5ths rule.

[http://www.scribd.com/doc/213380090/1999-NYC-Firefighter-
Wri...](http://www.scribd.com/doc/213380090/1999-NYC-Firefighter-Written-Test)

You'd have a hard time convincing me that doing well on it didn't correlate
with being a better firefighter.

------
johnrob
Why don't companies manage open source projects, and simply require a certain
amount of contribution as a gateway to a final interview? This would allow
candidates to prove their ability to write production ready code.

~~~
branchless
Because they'd filter out all the good candidates who don't work for free (for
for-profit companies)

~~~
johnrob
But they'll interview for free? They don't need to be large contributions (and
if small changes require too much time, that becomes a natural filter).

------
Ovid
I have to take exception to this because part of what I do for a living is
teach tech companies how to hire.

The claims are:

1\. Performance on our online programming quiz is a strong predictor of
programming interview success

2\. Fizz buzz style coding problems are less predictive of ability to do well
in a programming interview

3\. Interviews where candidates talk about a past programing project are also
not very predictive

Those claims may be true, but it misses a key point: no matter how well
someone performs in an interview, it doesn't mean they'll perform well on the
job. In particular, the post talks about hard skills (and does well there),
but doesn't talk about soft skills.

It's absolutely true that someone can talk a big game about hard problems they
faced that they solved, but failed to mention their use of Google or
StackOverflow. So unique coding tests can definitely tell us whether or not a
candidate can really code (though the tests need to be tailored to the sort of
real work the candidate will face). However, programming is a _hard skill_
(one that is easy to teach), and not a _soft skill_ (one that you tend to
learn over time or have a natural inclination for).

Case in point: early on we made the mistake of hiring a very talented
programmer on the basis of his reputation and a brief interview. He was a
disaster because he was completely unable to work without a very hard set of
requirements. In the very rapidly changing, agile, environment we put him in,
he flailed badly because he was good at implementing requirements, but he was
very bad at projecting himself into the customer's shoes. Later, we put him on
technically demanding tasks which required considerable talent, but had no
ambiguity, and he was fine. In fact, he was able to implement tasks that would
have been very difficult for other developers who were fine in ambiguous
environments.

Soft skills are tricky and most employers have no idea how to interview for
them. They require an understanding of the actual role someone will be hired
for and the set of skills necessary for them to succeed. Are there tight
deadlines? If so, can the developer prioritize tasks and make a case for
delaying less critical portions beyond the deadline? Is it a rapidly evolving
environment which makes it hard to nail down exact behavior? Are you on a team
with strong personalities who often disagree and can the dev handle that? Is
the developer required to report to non-technical people who have no
understanding of technology and can't appreciate what refactoring is?

Or here's a fun one for you aspiring managers: we've had one project with a
superstar developer who was so monumentally productive and so brilliant at his
work that some very talented developers who worked with him nonetheless had
_negative_ productivity at times because they spent so much time playing
"catch up" with the superstar and understanding how the project was evolving
that they spent more time keeping abreast of the code than developing it. The
code was "bus sensitive" because one brilliant developer was so competent. How
do you manage that?

These and other soft skills are part of what you really need to evaluate
because depending on your work environment, you'll have these and other soft
skills as critical skills that a dev will need, but you almost NEVER see
interviewers check for these because very few people are trained in
interviews.

Pro tip: start reading up about structured interviews. They've been proven via
multiple studies to have a strong correlation with employees being successful.
Unfortunately, they're harder to conduct and require training to understand
them. As a result, companies don't know about them or worse, ignore them,
because it costs more money. But what costs you more money in the long run?
Taking longer to interview or hiring the wrong person?

------
raitom
Why should I spend so much time with all these interviews when I can simply
reply to one of the countless job offer I receive every week?

~~~
to3m
Are these offers where you start next week if you sign the contact, or are
they expecting to meet you first?

If it's the former you should have no difficulty answering your question ;)

