
Bayesian Inference for Hiring Engineers - Harj
https://triplebyte.com/blog/bayesian-inference-for-hiring-engineers
======
KirinDave
Am I the only person getting progressively more creeped out by the series of
bizarre, unsourced, pseudo-scientific ads TripleByte has been running?

On Reddit right now they're running this weird faux-linear-algebra thing where
they imply that they can build a vector of your skills vs. a vector of job
requirements and get a meaningful answer via the dot product.

Which is a bit like saying you can predict the weather by taking the dot
product of a vector of ocean and air. What even are the units? What does any
of this even mean?

Hiring is a challenging, multi-dimensional thing. It involves a high-risk and
ideally informed decision by multiple parties. Doing it effectively is hard.
Doing it effectively and respectfully is harder still. And yet TripleByte
comes in and says, "We sound vaguely like machine learning. We got this."

Honestly, they make HackerRank, which was another extremely sketchy
organization making a lot of very questionable decisions, look reasonable by
comparison.

~~~
kofejnik
Vector thing actually does make some sense, you might want to read about
embedding. Or just watch fast.ai lesson 5.

Ironically, this was the topic of my final (and failed) conversation with them

~~~
KirinDave
No. Sorry, you're not forming a coherent vector space between both skills and
demands without published research on the subject.

You do not simply say, "Gosh it seems like I have said words related to ML and
therefore its application is plausible."

What's more, the notion that recruiting is in fact a skills demand model is
itself fundmentally misleading. Many of the skills you want are domain
specific and in fact cannot be expected to be acquired anywhere but on the
job. Given how many shops subtly permute "react" or "Golang" to mean a lot of
skills in a utility cluster, any space you form is going to be incredibly
specific to the employer and difficult to map anywhere else.

~~~
itronitron
>> subtly permute "react" or "Golang" to mean a lot of skills in a utility
cluster

This, and the tendency for shops to evaluate candidates' skillset against the
shops' most mission critical parts and processes which for whatever reason
have not been operationally hardened and locked down.

------
yosito
Not a fan of Triplebyte. I'm a full stack engineer with 10 years of experience
building web apps, and while I'm not the best in the world, I'm still pretty
damn good. I took Triplebyte's interview a few months ago, and they rejected
me with a link to a tutorial on how to build your first webpage
([https://learn.shayhowe.com/html-css/](https://learn.shayhowe.com/html-
css/)). This company knows absolutely nothing about hiring good engineers.

~~~
greenhatman
That is hilariously insulting. Sick burn.

I'd probably get similar. Also full stack 10 years.

------
compumike
Companies seem to be adding more screening steps to try to reduce their false
positive rate -- the rate at which they interview people who aren't hireable.

But most don't seem to understand that mathematically, there's a tradeoff for
a higher false negative rate.

Screening false negatives are people who would have done well in an interview,
but don't make it to that stage. These are more hidden to the company, but
quite expensive for the hiring process, and painful for people who are wrongly
rejected. If we put this in probabilistic terms, I hope we can have a deeper
conversation about what's happening and how this issue impacts engineers.

~~~
n42
At least where I work, we are painfully aware of this tradeoff, but are
willing to make the sacrifice because of the severe costs of making a bad
hire. We are small, and cannot afford to lose 90 days.

I realize this article refutes this point of view, but finance and HR do not
see it this way.

I am more happy to devote several hours of each week interviewing than entire
days/weeks mentoring and reviewing code for a bad hire, or worse, dealing with
the widespread consequences of a culture misfit in a small team.

Hiring is hard.

~~~
compumike
(Original article author here.)

I'd distinguish between _screening_ false positives (which my article focuses
on) and _hiring_ false positives. Screening rejections (before the final
interview) are usually done with very limited information, so there's more
room for bias and noise.

Hiring is hard! :)

~~~
n42
You're right, the difference is important.

I've been the tail end of the interview funnel when experimenting with our
screening strictness. The emotional toll that less stringent screening had on
my day-to-day work (faster, rapid, and high frequency on sites) was extremely
high. It led to a month or two of burnout as the lead engineer on the team,
despite probably leading to finding a hire sooner. The cost is, in my opinion,
immeasurable.

------
crispyambulance
I kind of resent these attempts at optimally cherry-picking "the right"
candidates using data.

Hiring is _intrinsically_ a subjective act.

People are very much a moving target. They change over time. Work experiences,
even bad ones, shape one's skills and ability to cope in organizations. Almost
everybody has a bad-fit job at one time or another. The experience of a bad-
fit is actually important to the growth of the individual and, I think, their
coworkers and employers.

~~~
AlexCoventry
When an organization scales to the point where they need something like
TripleByte, they're going to be using _some_ kind of impersonal screening
process, though. This is better than some disinterested HR drone scanning
resumes for keywords.

~~~
crispyambulance
I agree that an HR drone scanning for keywords is not good (though some are
better than others).

The most savvy candidates, however, tend to skip the step where you throw your
resume into a giant vat with the others and hope for the best. Isn't it still
the case that most jobs are filled using references and professional networks?

~~~
thomasmeeks
No, not at truly large companies -- Amazon could never fill its halls with
just references and such.

Even at smaller companies, however, there's some pushback to network hiring
because it isn't particularly great for getting a diverse pool of talent.

To be honest, though there are a lot of dangers with this sort of thing, I am
a fan. End of the day, engineers should be great engineers, not great at
hacking the job finding process.

~~~
mywittyname
> there's some pushback to network hiring because it isn't particularly great
> for getting a diverse pool of talent.

I'd be surprised if random resumes are really that much more diverse than
network hiring. Consider principles like the Erdos number[1] -- the
collaborative distance between groups of people in similar fields is
strikingly small. I bet, a company with a reasonably large development staff
(50 people) is probably no more than 2-degrees separated from 99% of the
talent pool in a given region.

Just about every position I've looked at was with a company where a former
colleague works.

[1][https://en.wikipedia.org/wiki/Erd%C5%91s_number](https://en.wikipedia.org/wiki/Erd%C5%91s_number)

------
acconrad
On a side note, I took the 20-30 min front-end exam that is referenced in the
article and a few things bug me (in case anyone from TripleByte is reading):

1\. I did "exceptionally well" but I don't know how many I got right or in
what percentile I fell in. Why? Firstly, I'm immediately suspect if I actually
did that well. For all I know, the "top percent" could just be anyone in the
top 50%. And the top 50% only gets 10 questions right. But more importantly, I
already stated in the beginning I was taking it for fun, so why can't I learn
what I got wrong so I can improve? I imagine there were areas I got wrong that
were repeated, which brings me to my next point...

2\. Why so React-centric? I happen to use it but plenty of people use Angular
or Vue. You can't expect them to know about the React events lifecycle.

3\. Okay, so I don't live in SF or NYC. But I know some of those bigger
companies have offices in Boston, where I do live. Why can't you make that
work if you already know what companies are in your pipeline and which offices
they have? Seems super expensive and wasteful to lose out on a great engineer
when you know your list of 200 companies totally has an office not in SF or
NYC (e.g. Facebook).

4\. Okay, so I want to work remote. Why can't I just take your final Google
hangouts exam so that way you have on file "okay cool this person is great, we
can fast track this person." If you green light companies who are remote-
friendly, you don't have to worry about this issue. Plus isn't TripleByte a YC
company, working with other YC companies? I know for a fact that GitLab is
also YC and is a remote-friendly company. Not to mention _you 've_ advertised
for remote engineers!
[https://news.ycombinator.com/item?id=15066073](https://news.ycombinator.com/item?id=15066073)

I dunno, given that the article is all about touting how effective the 30
question exam is at screening out candidates, you'd think you'd want to do
something useful with that quiz instead of locking people out, even if they
don't fit your current criteria.

~~~
compumike
(Original article author here.)

Feedback is important, so after the interview, everyone gets detailed personal
feedback on every section, regardless of the outcome. We get much higher
resolution data after the interview (part of the point of my article).

You may have hit a few React-specific questions, but getting those right is
certainly not required.

As far as Boston/remote: interviews are expensive for us, so it makes sense to
interview engineers who are excited about working in places where lots and
lots of our companies are hiring today. Currently that's SF or NYC. When we
expand, you'll be able to pick up from that step.

~~~
kofejnik
> after the interview, everyone gets detailed personal feedback on every
> section, regardless of the outcome

Wait, what? I never received any feedback _at all_

~~~
compumike
To clarify, my statement is true for every engineer who goes through the
Triplebyte technical interview in our regular process, i.e. to get matched to
jobs at the companies we work with.

(That was not your situation -- contact support@ to follow up in private.)

~~~
kofejnik
Well, I did email @support yesterday and also checked my spam folder, just in
case. Still nothing.

------
kofejnik
Just my $0.02 about triplebyte - I went through all interviews, seemingly
doing fine, but after the last informal talk (which also seemed ok to me),
there was about 10 days of total silence. Finally, I emailed to enquire and
immediately received an 'Unfortunately, ...' letter.

So, in the end, I've had 4 hangouts sessions over 3 weeks, plus time spent
preparing, and was rejected with no feedback at all. I'm still curious, was it
the bloom filter?

~~~
kofejnik
update: it has been fixed very quickly, and I'm a fan of triplebyte as a
result

------
asadlionpk
I will be harsh. This looks more like "hey, watch us force some math onto this
topic and look cool".

Sadly, this might impress some dumb CEO to use them though.

------
burnte
The problem isn't the computers, it's the people. You put HR-bots on the task
of listing the job, and they don't know Atom from Adam, so they list all sorts
of silly requirements like 15 years of SAP experience, 15 years of Ruby on
Rails, 15 years of COBOL, and all for a $20/hr entry position, or a list of
certifications that no human could ever accumulate. Then what happens is
applicants start keyword spamming their resumes just to get noticed, and now
as a technical person I get a stack of resumes that are absolute trash.

Two years ago I was hiring for a sysadmin. My HR department put my
requirements up on Indeed. I got 70 resumes that passed their screening. Of
those 70 I found 5 that I wanted to interview, 3 that showed up, and none were
hirable. I left a company several months ago, couldn't deal with the
management anymore, and the past few months of job searching have been
excruciating.

We need more technical people screening resumes and comparing to actual job
requirements.

------
closed
Edit: I think I've run afoul of an anti-triplebyte sentiment. I should clarify
that I think this post did a good job building a very simple example of the
statistical theory behind assessment,but I have no idea whether their product
/ approach is reasonable or not. Building a good assessment is much more than
just statistics, and it sounds from other comments that there are serious
concerns about the validity of their tool.

Really enjoyed the build up from simple cases, to more complex models!

If you're interested in the statistics behind estimating skill, and how well
questions tell apart novices from experts, check out item response theory :)

[https://en.m.wikipedia.org/wiki/Item_response_theory](https://en.m.wikipedia.org/wiki/Item_response_theory)

------
mlthoughts2018
My experience hiring machine learning talent over several years has been that
people over-hype the cost of a false positive. Both the article's false
positive (expending the cost of interviewing on someone who ultimately reveals
to be the wrong fit) and also a more fundamental false positive: actually
hiring someone who would hypothetically fail a lot of these interview
pipelines.

The discussions about making these pipelines more quantitative, with
assessments and quizzes, always couches it with a tacit assumption that the
_worst_ outcome would be to actually hire someone who fails at one of these
interviews. Rejecting a good person sucks, as they say, _but not as much as
hiring one of the multitude of sneaky, low-skilled fakers out there._

And of course, everybody's got their hot new take on how to spot the
supposedly huge population of fakers.

What I have learned is two-fold:

(1) That person who aced all your interviews and finally looked like the
perfect person to hire probably just spent 3-6 months utterly failing at a
bunch of other interviews, just to get into "interview shape," refresh on all
the nonsense hazing-style whiteboard trivia about data structures that they
had never needed in years at their job, etc. So it's totally asinine to
believe that someone passing through all your filters must be the sort of
person _who would rarely fail_ some filters. _That person almost surely did
fail filters, and the companies where they failed believe they dodged a costly
false-positive bullet, while you believe you just made an offer to the
greatest engineer._ Hopefully you can see the myopia here.

(2) The cost of passing up a good-but-failed-at-interview-trivia engineer is
often far greater than the cost of hiring them. For one thing, "suboptimal-at-
interviews" engineers are pretty damn good engineers, and they can do things
that differ from esoteric algorithm trivia, such as helping your business make
money. Another thing is that many engineers can generalize what they learn,
generalize from example code or templates, etc., very efficiently. So while
they might reveal a weakness by failing part of an interview (and _everybody_
has such weaknesses), why do you really care? They can probably become an
expert on that weakness topic in a matter of months if they work on it every
day, or if you have existing employees who can mentor them.

But the biggest thing is part of what Paul Graham wrote in "Great Hackers":
good engineers tend to cluster and want to work with other good engineers.

So if you're sitting there without already having a few good engineers on your
team, then most likely, the cost to mistakenly rejecting a great candidate who
happened to have a bad day, or a great candidate who happens to hate writing
tree algorithms on whiteboards, leaves you running a huge risk of losing out
on a good engineer who could help kickstart the phenomenon of getting the next
good engineer.

When your team is in this stage, you absolutely can manage with a few "dud"
hires who need a lot of help or who have skill gaps in key areas. The cost of
adding them to the team and managing their "suboptimality" is _far less_ than
the continued search costs brought on by rejecting good candidates with and
overly risk averse hiring threshold, and leaving your team in the state of
affairs where it still doesn't have a good engineer to help attract more.

In other words, the loss function penalizes false negatives more severely than
the combined penalty from effort spent on true negatives and suboptimality /
management costs of false positives.

But all these skeezy interview-as-aservice businesses what you to believe that
the opposite is true, that if you accidentally hire a "faker" because your
hiring process was too easy, then Cthulhu is going to rise out of the sea and
lay waste to your company.

Of course they want you to believe that. That's how they make money. Preying
on your fears over what would happen if you just unclench and treat candidates
like human beings with strengths and flaws and don't hold them up to ludicrous
standards that lead to self-selecting macho 22-year-olds getting hired because
they just spent 10 months on leetcode.

When you start to realize this, it becomes obvious that onerous code tests,
brainless data structure esoterica, hazing-style coding interviews, and
especially businesses that offer to outsource that nonsense, like TripleByte,
is all just snake oil junk.

------
greatamerican
I hope this company runs out of money. Writing an article like this shows how
deeply they misunderstand the problem they are seeking to solve.

~~~
paulie_a
Agreed, this approach is idiotic. I would avoid even applying for a company
that took this approach.

