
Technical Interview Performance by Editor, OS, and Language - compumike
https://triplebyte.com/blog/technical-interview-performance-by-editor-os-language
======
weeksie
If you're an emacs user there's nothing worse than doing an on-site interview
where someone wants you to pair program using their editor. Nothing works
right and you spend half the time flipper-handing it over the keys trying to
remember how normal editors work.

Then again, it's rare that I interview anymore. I've done maybe two of those
kinds of (pair programming, someone else's rig) interviews in the last ten
years and neither was for a job that I really wanted after spending time with
the devs conducting the interviews.

You want me to consult with you? Let me talk to your biz people to see if I'm
the right person to solve your problems, and let the technical folk go over
code I've written and we can deep dive on that if there is any question about
my chops. Otherwise, fuck off. I'm too old to play "dance monkey".

~~~
eagsalazar2
I sort of hear you. Having said that as someone who has done a lot of hiring
in the past we'd pass on someone like you immediately unless you came _very_
highly recommended. We need a consistent process to have any basis for
comparison over time. Also your "fuck you, I'm not a monkey" attitude almost
always shines through during initial screening call and is the last thing we
want to bring in.

I do feel for you. Maybe we're about the same age and level of experience? It
sucks to be put through some one size fits all hiring process and it sucks to
constantly feel like more jr people are reinventing process and best practices
every few years. But what are you going to do if you are a consultant? you
have to be the boss or you have to play ball.

While teams do need leaders with experience they also need collaborative (vs
combative) team players who can ride out the details and help improve hiring
and work process in a positive and constructive way.

I think more experienced devs get jaded. It is a trap IMO and probably
contributes to a lot of the ageism that is perceived and that really exists in
the industry.

~~~
zaptheimpaler
I mean you are justifying why your process is the way it is from the
perspective of the company. Great, no one doubts you have your reasons. But
does the candidates perspective not matter?

He rightfully doesn't want to spend 3-4 hours of his time on an artificial
interview process. You can call that combative or jaded, I call it knowing
what you are worth. Or just protecting your limited time on earth.

Seems like you ultimately do want monkeys who consistently put their own
interest behind the company's. No surprise that older people are less willing
to do this - its not because they are "jaded", its because they recognize that
no one else will put their interests ahead of their own.

Its a 2-way interview though, so overall you both steer clear of each other -
great.

~~~
mlinsey
I agree 100% with the direction of your conclusion - different companies can
have different cultures / different processes and candidates (especially
experienced ones with high market value) can just go with whatever style of
company you feel is best.

That said, your characterization of the parent as wanting "want monkeys who
consistently put their own interest behind the company's" is missing a key
third stakeholder besides the company and the candidate - and that's the other
employees! In my experience, a company built around a strong collaborative
working culture will have employees that a) want to see what it's like working
through a problem with you, in some form, b) feel that an environment where
every incoming candidate has to pass the same bar, regardless of pedigree, is
fairer c) prefer working with others that exhibit the humility you
characterize as being a 'monkey'.

Again, ymmv, choose the culture you want, etc...

~~~
zaptheimpaler
Hmm honestly there are better ways to let your employees get to know a
candidate than a 1-hour tech screen - how about a 15-minute chat instead? And
if you need a tech competency bar that fresh college grads generally do better
on than experienced devs to manufacture a social consensus on fairness, I feel
your team does not trust the interviewers enough.

My idea of a collaborative working culture is where

1\. we trust each employee to do their job well enough that we do not have to
look over their shoulder

2\. we respect each employee enough that we value their time

3\. we can work together with low friction

I would not characterize these positions as a lack of humility. The use of
'monkey' was in the sense of OPs use of "dance monkey" where candidates go
through a ritualized process that tests little but mostly serves to preserve a
social illusion of fairness. Pedigree is one thing, actual experience and
references to back it up absolutely is a reason to "bend the bar" unless
preserving the appearance of fairness is more important than actually
evaluating skill.

But overall, thank god for free choice. These are very different value systems
I think - manifesting as differences in interview style, but they are more
fundamental political differences.

~~~
PascLeRasc
I agree that the technical screening can go away and be replaced by a chat.
I've gotten to know new employees at my work by going for a beer with them and
asking "Oh so your iPhone's jailbroken? How'd you do that?" or "So I saw
you're using Gnome3, why do you prefer that?" and most of the time they're
pretty eager to give a well-thought-out explanation behind those sort of
personal computing choices. I really enjoy working with the people who are
well-versed in some niche topic like that and can communicate its 'good-enough
practices' well.

------
dtran
(Longtime emacs user but have now used sublime text for several years)

Super interesting post, Mike! Not surprising that emacs/vim users tested
better since use of those is likely correlated with having programmed for
longer or having programmed from earlier in life before some of the other
editors even existed. Would love to see it normalized by years programming.
Also, insert joke about how using emacs or vim probably led to natural
selection weeding out all but the most committed programmers or how the bad
vim programmers didn't start their Triplebye interview because they're still
trying to figure out how to toggle back to command mode to exit.
:stuck_out_tongue_winking_eye:

I was surprised to see such a big difference between PyCharms and Sublime.
Would love to see a language breakdown for generic text editors like Sublime
and Atom and compare python interviewees who used Sublime to python
interviewees who used PyCharms. I also wonder if there's some confounding
variable since AFAIK there's no free version of PyCharms whereas there are
free versions of Sublime and Atom, so an interviewee who has a PyCharms
license likely uses it in a school or professional setting.

~~~
msla
Using Emacs and/or Vim is also a test of how much you care, primarily because
they're not the default anymore, so anyone who uses them cares enough about
their programming environment to use a non-default editor, and one which has a
rather higher barrier to entry than most at that.

(Although, to be honest, using the GUI version of GNU Emacs as a Notepad
replacement is possible: Just ignore the keyboard commands and interact solely
via the menu up top. You even get access to at least some neat major modes
this way.

Still, Emacs isn't the default, so anyone using Emacs probably knows why, and
won't be the kind of person to use it without getting more out of it.)

~~~
komali2
I dunno man, I care enough to spend a big chunk of my free time improving my
code ability. Thing is, I find it easier to use hotkeys in vscode than in
emacs. I'm trying both right now and vscode just seems to allow me to do the
things I want to do without like learning lisp or something.

In terms of "caring," I care enough to learn literally all of the hotkeys
available by default in vscode, including some I added and changed. I'm a f'n
beast in that IDE :P

~~~
hashkb
You don't know what you're missing. Want to be a real beast? Learn vim and
emacs, both. You want an editor where "literally all the hotkeys" is literally
meaningless.

~~~
ShabbosGoy
Or you could just use Spacemacs.

~~~
erikbye
Not sure why this got downvoted. While I prefer my own customized Emacs or
Vim, Spacemacs is pretty decent from the jump. You just need to activate the
layer for your language.

------
Rantenki
This whole article reads like an celebration of confirmation bias, which could
be summarized as: "The people that pass our interviews tend to be the ones
that use the tools that we prefer."

I don't think there is any value in looking at this blog post and making any
conclusions about which editor and/or scripting language to learn next. You
might go interview at a .net shop and get the opposite result from the one
that you wanted.

Disclaimer: I personally match their preferred technologies very closely, but
work with some really smart people that write C#/F# in VSCode.

~~~
danielvinson
I agree on confirmation bias, but the actual correlation appears to be that
people from more wealthy backgrounds tend to do better in interviews.

Anyone who uses MacOS (ie. can afford a MacBook Pro) to work on has to have
enough money to buy one despite being in a job interview.

Most people who use vim and emacs are going to be older, more experienced
engineers, and as such, have the resources the acquire the tools they need to
succeed (time to learn, books, computer, paid IDE).

I see statistics like this and just think about all of the kids who didn't get
a free ride through college and a MacBook given to them who aren't succeeding.

~~~
kbp
> I agree on confirmation bias, but the actual correlation appears to be that
> people from more wealthy backgrounds tend to do better in interviews.

> Anyone who uses MacOS (ie. can afford a MacBook Pro) to work on has to have
> enough money to buy one despite being in a job interview.

> Most people who use vim and emacs are going to be older, more experienced
> engineers, and as such, have the resources the acquire the tools they need
> to succeed (time to learn, books, computer, paid IDE).

> I see statistics like this and just think about all of the kids who didn't
> get a free ride through college and a MacBook given to them who aren't
> succeeding.

I think having money and resources is definitely a deciding factor, but I
don't think it has much to do with wealthy backgrounds or free rides through
college. Like your third paragraph says, the bias seems to favour older, more
experienced programmers. It makes sense to me that more experienced
programmers would do better in interviews than kids fresh out of school, and
that experience just ends up being reflected in the data as being able to
afford expensive tools. Like the article says when it discusses the MacOS
bias: "if you walk into any tech company in San Francisco you’re inevitably
going to count far more than 60% MacBooks. While our interviews are
background-blind, having a few years of professional programming experience
usually helps!"

------
mizzao
Aside from the correlation !=> causation issues, there's a ton of endogeneity
and selection bias in this data.

The initial programming quiz has code snippets only in Javascript, Ruby, and
C.

The interviewers are usually more familiar with more modern technologies.
Indeed "enterprise" (C# or Java) developers tend to do poorly at TripleByte,
and Java is highly correlated with Eclipse usage.

TripleByte also recruits primarily for startups which tend to use certain
technologies. I'd say the final stage offer data is so biased by the selection
process along the way that it's almost impossible to draw any conclusions from
it.

------
compumike
We’ve always allowed engineers to interview using their own preferred
environment (which can be fun or challenging for us interviewers!), but now we
do enough interviews every week to see which tools tend to be more successful.

Also, there are some rising stars we couldn't include in this analysis, like
Go: candidates writing Go during their interview with us did _extraordinarily_
well on average, but we don't yet see enough of them to break them out of the
"Other" language category.

~~~
meggar
What's the most popular editor for Go?

~~~
ramenmeal
I don't have any backing data, but I think vscode with the go extension is
pretty popular.

~~~
drspacemonkey
Can confirm. VSCode's Go extension is really nice. I started using Atom for
Go, but quickly switched over. The only thing I miss from Atom is the
extension that shows inline test coverage.

~~~
dnasty
I'm not familiar with how inline Go test coverage worked in Atom but the
VSCode Go extension does include it. There is an action called "Go: Toggle
Test Coverage In Current Package" which I think will do the job for you.

------
hellofunk
I started coding professional on Xcode. Then I tried a couple of other IDEs
until my work in lisp required I give emacs a try. I eventually went back to
other languages, mostly C++ and Python, but still use emacs. It really spoils
you for text/code editing forever. Once you get passed the learning curve,
every other effort to edit code or text in other programs seems ironically so
archaic. What? I can't split the buffer quickly and get another one? What, I
can't quickly bounce back to my previous editing window configuration, then
bounce forward again? What, there is no visual undo tree? The list goes on and
on. I now use the Vim emulation inside emacs which has been the killer feature
in emacs to further boost productivity.

And, last year I passed several rounds of technical interviews with one of the
top 5 companies in SV, so perhaps I support the statistics presented here
(though I did not get to use my chosen editor for the interviews, so they did
not know I am an emacs user).

~~~
agumonkey
I colleged on Eclipse. But I always had a thing for emacs (it has some parsing
abilities oob, it's not just coloring and lexing).

Anyway the day I finally clicked on the extensibility of emacs. Which was as
trivial as defining a function, then defining a menu (which is duh.. a list).
And have the GUI [1] updated I lost the remaining bit of love I had for
Eclipse.

I got 2 or 3 years of Eclipse JDT/PDT, all the talks about the power of
plugins. It was nothing but pain.

Then that old emacs gets me the same thing like it's breakfast. It hit me too
hard.

Lastly, I revived a win95 box, with Turbo Pascal. Another old and tiny blast
from the past, and even though the 600KB IDE/compiler was amazing in most and
so many regards. The fact that I was stuck with the default text editing
features (no paragraph selection etc etc) made me feel what Emacs' freedom
meant to me even on the tiniest instances.

[1] note that I'm a keyboard fetishist, so I don't use 99% of Emacs GUI and
knew most keybindings, I was just trying for the sake of learning elisp and
emacs architecture.

------
sevensor
Does disseminating information like this cause a net improvement in the
workforce? Will people study high-productivity editors in the hope of getting
hired, only to become better programmers despite themselves? Or will this
result in people learning just enough of vim or emacs to use it badly in an
interview?

~~~
jordache
That's what you got out of the article?

Obviously it's just highlighting a single interesting correlation.

Wow someone is going to learn an editor thinking it will help he/she interview
better?

~~~
mulmen
Technical interviews are just about knowing the right dance moves. If coding
fizzbuzz in ruby using vim is more likely to get a good job than doing the
same in perl using sublime I'm going to learn just enough ruby and vim to
complete the interview. This is especially true when the company you are
interviewing with posts a blog with their selection bias clearly spelled out.

~~~
jordache
I don't think anyone beliefs companies out there are interviewing for
competency in legacy text editors.

~~~
mulmen
VIM and Emacs both have commits to their codebases in the last 48 hours.
Sublime released a new major version earlier this month. None of these can be
considered legacy. This is an article about the correlation between these text
editors and interviews.

------
jawns
Sample size is never directly mentioned, but:

> At Triplebyte we evaluate hundreds of engineers every week.

> I pulled the editor / language data for every Triplebyte interview over the
> last year.

Based on that, if we suppose "hundreds" means "low hundreds," this suggests a
sample size of 5K-15K.

That's not a bad sample size for the total population.

But if 35% use Python and only 5% use Ruby, then you're comparing a sample of
1,750 (on the low end) to a sample of 250. It wouldn't surprise me if the Ruby
data is more shaky than the Python data, and thus it's not a fair comparison.

Same goes for a lot of the other metrics where the sample sizes vary.

~ Disclaimer: I'm no statistician (but I play one on Correlated.org).

~~~
tedsanders
Be careful equating 'evaluate' with 'interview', when TripleByte has incentive
to obfuscate. I suspect hundreds of engineers may apply through Triplebyte
each week, but they may interview only a small fraction of those. If true,
this makes your point about small sample size even more important. I was
disappointed their charts didn't have confidence intervals or count numbers.
Makes it harder to guess at how the numbers generalize.

------
jorgemf
Flame war ahead... I want you to bear in mind that this depends highly in the
number of open positions per language/technology and the number of candidates
per position. Does it mean I suck if I do Java in Eclipse compare to someone
who does Ruby in Vim? Maybe they have 1 java position for 1000 java developers
and 100 Ruby positions for 500 Ruby developers.

So basically this article says nothing. It needs more information to be
relevant.

~~~
compumike
(Article author here -- I'm one of the interviewers here as well.) Our
technical interview is really designed to look for your strengths, such as
being productive while writing clean, correct code in any language/environment
of your choice. If you do well at that (or excel in other sections of the
interview such as system design or knowledge questions), we're going to work
with you as a candidate, regardless of the language you choose to interview
in.

~~~
jorgemf
Still is irrelevant. If you have 1 java position and 10000 Ruby positions, you
are going to interview more Ruby developers, your data is going to match more
closely the reality. For the 1 java position you are going to be more picky
and discard more candidates. But even with the same number of candidates and
positions your test are going to have bias, how do you know the java test is
as difficult as the ruby test, it could be more or less difficult.

You can do a proper analysis fixing some of the variables. For example you can
fix the language and compare how the different editors affect the selection
process. But still you have to show other variables as the years of experience
in the language.

And finally you have outlayers like who did an interview with you but was
discarded because of living in a different country. But I guess I might have
enter into the statistics.

------
tapatio
This is most likely because engineers that use vi/emacs have to memorize
language syntax and APIs as vi/emacs don't have code completion like most IDEs
(at least not by default). And having that stuff memorized means you'll do
better on a coding interview.

~~~
michaelmrose
This is complete nonsense. It's completely trivial to get code completion.

Its wholely unrealistic to suppose that someone would spend hours to days
learning emacs but not spend 1 minute adding a completion plugin.

~~~
sidlls
I don't use any completion plugin with my Emacs. Not coincidentally I'm
generally a better reference for our programs' APIs and organization than the
documentation or the other developers who use things like PyCharm and Sublime.

For me not using completion works to keep me disciplined about the structure
of code I work on. It also can make me vicious about arity and composability
in code reviews. Fortunately most of those details are taken care of by design
prior to implementation, so it's rarely an issue.

~~~
michaelmrose
I think not using code completion might have more to do with you having an
excellent memory and it might not be fantastic advice for everyone but I would
be interested if there were stats on usage of features like code completion in
emacs.

------
WhitneyLand
_Programmer interviews are broken. Whiteboard coding and algorithm questions
aren 't good predictors of how effective someone will be at writing real code.
Technical hiring processes harm both excellent candidates who don't interview
well and companies trying to hire good programmers._

Agreed. Great start.

 _[make an] investment by going through our interview process, afterwards we
can fast track you through to final interviews...O(n) process into an O(1)
process_

Makes sense. Use of complexity notation is transparently gratuitous, but I
liked it, because it shows the intention to understand and connect. Traveling
the world most people appreciate an honest attempt to speak to them in their
language even if you're not fluent.

 _You 'll have to take a coding quiz with questions in several different
languages, as well as some design questions. The coding questions involve
reading blocks of code, and answering questions about them._

Intuitively I hate the idea because I'm skeptical of how effective it is. Do
you have non-anecdotal data to show how well your quiz correlates to specific
goals?

 _After the quiz, you may be asked to complete a coding challenge..._

What triggers that requirement? Also same question as previous question about
correlation/validity.

 _Then, you will move on to our technical interview..._

What do the tech interviews consist of? They can vary wildly across companies,
interviewers, and formats.

 _...work history [is] meaningful but relying solely on [it] results in
missing good programmers_

This is a show stopper for me. If a company doesn't acknowledge that what
people have accomplished with previous contributions, projects, development
work is not just meaningful, but extremely important and a top criterion to
find great people, then I don't believe they have the correct perspective. But
hey, I can be wrong about things, I'll try to be open minded. Point me to
solid data showing tests and quizzes are a better predictor than
accomplishments and I'll have a look.

~~~
compumike
Accomplishments and work history are meaningful. The problem is that they're
also very noisy. There are false negatives from rejecting great engineers who
either can't communicate their experience well, or don't have much experience
yet. There are false positives from wasting time interviewing people who are
"too good" at puffing up their resume relative to their underlying skills.

Reducing the rate of both false positives and false negatives reduces the time
and cost of hiring an engineer for companies.

The data we look at for this is our "onsite to offer rate", which is how often
Triplebyte candidates get a job offer after each final interview we send them
to. Ours is roughly double what the companies see on their own for people
applying directly or referred by a friend. This means we're doing a good job
at pre-screening for technical talent and pre-matching for what each company
is looking for. If this "onsite to offer rate" wasn't substantially better
than normal, companies wouldn't work with us -- and especially wouldn't let
our candidates skip their resume and recruiter screens.

~~~
WhitneyLand
I agree using this approach on someone just out of school wouldn't work very
well, but I don't think that's the bulk of the problem space.

If someone has 3 years experience, there should be plenty of results to allow
an in-depth look at what they've been working on. It wouldn't matter if
someone 10 years senior had "more stuff", because it's not about quantity. I
think it's important to take in the context of where that person is at in
their career and what's realistic.

>>There are false negatives from rejecting great engineers who...can't
communicate their experience well

Well, it's true this is a problem isn't it? Whatever you call it, it's part of
a lot of us, part of who we are, and does sometimes result in unfair
judgement. The problem is, I'm not sure TripleByte's process gets around this
either. Part of their process is a dynamic interview that surely requires some
communication skills. Pass that and no doubt even more communication skills
will be needed at the employer interviews.

------
pfarnsworth
I find it kind of funny and rather llazy that they don't dig into deeper as to
why there's such disparity between programming languages.

To me, it's probably that the questions are geared towards Python/Ruby, rather
than C++. For example, parsing a CSV is one line of code in Python/Ruby by
default, but if you're supposed to do this in C++, that alone can take 10 mins
to write and test, etc.

If they really want to make their test/evaluation fair, and if they truly want
to be inclusive in the languages, they should figure out why there is such
disparity, it's probably with the questions they ask rather than the
candidates. To me, looking at these numbers, it just means that were I ever to
interview with them (I never will because I think they offer no benefits
whatsoever, since I don't want to interview with them, and then with the
external companies again when I can just go with the external companies the
first time), I would just choose Python or Ruby to interview in, since the
tests appear more geared to those languages.

------
mulmen
This screams of lying with statistics. There are no interesting conclusions
drawn here, just some correlation that is at risk of being based on selection
bias.

This is just fodder for the endless tech variants of ford-vs-chevy debates.

The elephant in the room is that we don't even know if technical interview
performance is a good selector. I'd be a lot more interested if this data
included long-term results of actual job performance like employee and manager
satisfaction, time in role, time to promotion, etc.

------
shmerl
I'm surprised macOS is more used than Linux among developers.

~~~
komali2
I'm not. I am not a fan of apple at all (for various arbitrary reasons, and a
couple reasons I think I could make a valiant argument about), but I have to
admit a macbook pro out of the box does "just work." I throw on chrome and my
editor of choice, maybe xcode or whatever if I need it, set up my ssh keys,
install git, swap ctrl/cmd, done. I can plug it into an external monitor and
unplug it without is losing workspace/desktop context, I know I'm getting the
best battery life, the resolution is working straight out of the box in all
applications, no random slowdowns, etc.

Meanwhile _every time_ I try to put Linux of any flavor onto a macbook or a
lenovo carbon or whatever, something gets fucky. Video drivers don't work, no
webcam driver exists, the mouse glitches, the tilde key doesn't work, plugging
into/out of an external monitor crashes it or causes all my windows to
collapse into a single workspace, same issue for if the computer goes to sleep
while connected to an external monitor, all sorts of weird things causes it to
not sleep on laptop lid close and nukes my battery, battery life will drain
excessively, arbitrarily, chrome will randomly cause the computer to slow WAY
down, etc.

Ok, that was a rant, but TLDR - I hate macs but coding on one just removes all
the distractions from programming and let me "just do my job."

(I am experimenting with windows as well right now because I love the Surface
platform as a whiteboard and prototyper, so far WSL is great and I actually
prefer it over linux because all the OS stuff "just works." Lots of weirdness
around WSL still though)

~~~
shmerl
I wouldn't touch Apple hardware. Using Linux there is a pain, so I'm not
surprised about your experience. Lenovo laptops though usually work pretty
well (as do Dell for example), so I'm not sure why you had hard time with it.
In my exprience things work pretty much out of the box. You might need to
install some firmware like for Intel WiFi, but that's about it. Stick to Intel
iGPU (avoid Nvidia/Optimus), and you'll be OK. With new AMD APUs things should
also improve for AMD based laptops.

I'm using Firefox, so can't comment on Chrome experience.

~~~
s4vi0r
Windows laptops are unequivocally trash. Regardless of spec advantage, I have
never used a windows laptop (e.g. dell, lenovo, hp, samsung) that hasnt had
issues with something. Its absolutely ridiculous and unacceptable that a
$1000-2000 laptop has a touchpad on par with its $200 base model cousin.

The day someone comes out with an actual mbp competitor is the day i switch,
but thus far its not even close :/

~~~
shmerl
Today Apple don't have better hardware quality wise. Your assessment sounds
like from some distant past.

~~~
komali2
Agreed, I recently found out the Macbook pro 11,4 I have that I don't actually
have 2 usb ports, just one and a splitter off it. Not sure why that was
legally allowed to be marketed as "2 usb ports."

------
minikites
As so many Hacker News posters are fond of saying, correlation isn't
sufficient to imply causation. Or perhaps their interview questions are
designed in a way that biases a particular development environment (positively
or negatively).

~~~
danjoc
Or perhaps there's some racial bias and a certain ethnicity tends to be
associated with Java and Eclipse.

------
look_lookatme
I would not extrapolate too much from our broken system of technical
interviews.

------
javajosh
Seems like a good time to mention this oldie-but-goodie from Oliver Steele on
how IDEs affect our thinking, from 2004. This blog post was a big reason I
started coding with basic text editors, and I'm glad I did:

[http://blog.osteele.com/2004/11/ides](http://blog.osteele.com/2004/11/ides)

------
luord
> if you walk into any tech company in San Francisco you’re inevitably going
> to count far more than 60% MacBooks

This surprised me. It might be because I'm not from the USA but in every
company I've worked for on-site (including the current one), the overwhelming
majority of developers used and preferred linux.

That aside, I'm glad python does well.

------
dsacco
With respect to languages: is it the case that Ruby programmers do better, or
are companies hiring Ruby programmers more lenient?

(Ruby being an example; Python also applies here, whereas the converse would
be a hypothesis that C++ programmers are not less skilled but rather are
judged more harshly in interviews).

------
c3534l
The low score of C++ and high score of Python in pass rate made me think for a
moment. How are they determining pass or fail rate? Because if they're failing
people because they forgot a semicolon or something trivial like that, then
this ceases to be a test of interview performance and more a test of how many
inconsequential mistakes does the average person make.

So for instance, Vim doesn't provide much help when you mess up. If you type a
variable name that doesn't exist, that's your fault (I'm sure with plugins you
can get more). So do Vim users do better because they're nerdier, don't mind a
steep learning curve, and like working in a terminal, or is it because Vim
doesn't tell you that you accidentally called "reduc" instead of "reduce"?

~~~
thomastjeffery
> If you type a variable name that doesn't exist [in vim]

Start typing a work and press C-x C-[nolk]

* n for a word in open buffers

* o for "omnicomplete" which is quite similar to "intellisense"

* l for the entire line in open buffers

* k for the word from a dictionary

There are several others, but those are the main ones. As soon as you start a
completion, you can use C-p and C-n to go up and down the list respectively.

> Vim doesn't provide much help when you mess up

Vim is _designed_ for _editing_ text. I don't know if a better editor (yet)
suited to editing, rather than writing.

> Vim doesn't tell you that you accidentally called "reduc" instead of
> "reduce"?

It can, so long as you have it set up to. Vim even has a spell checker.

~~~
c3534l
I didn't want to start a preferred editor flamewar.

~~~
thomastjeffery
I just wanted to let you know that Vim isn't the neutered editor you appeared
to think it was.

------
fcbrooklyn
I'd be interested to know how their prescreening works. Sometimes those things
are timed, and I can imagine a really short timeframe, say 30 minutes or so,
might make the specific editor more relevant. For example, I absolutely hate
Eclipse. Back when I was writing Java, I reached into my own pocket to buy
IntelliJ because I was at least 50% more productive with it. I know there are
people out there who dig Eclipse, but even if it's your editor of choice, it's
more appropriate for dealing with a big, long-lived project. Similarly, if
they had some existing project you are required to work with, full on IDEs
like Eclipse might require a lot more fiddling to work properly, compared to
straight up editors like vim/emacs/etc.

------
didibus
This is missing data about the job role. Are python devs hired as web dev, and
c++ as 3D engine programmers?

Does the triplebyte hiring process not take into account experience and role?
Don't they expect more for certain roles and a given experience?

~~~
dllu
It seems Triplebyte just tests everyone on web technologies regardless of role
or experience. It makes sense that C++ engineers are less familiar with large
scale web apps, but are valuable hires in the industry for robotics, finance,
game engines, etc.

~~~
compumike
(Article author here.) I'm one of the interviewers. If I had to guess, maybe a
third of our generalist-track interview is web-related. I've personally
interviewed many engineers who did not know the first thing about any web
stuff, but were particularly strong on other sections, so we accepted them.

You really don't have to be great at every section to pass the Triplebyte
interview; you just have to show your personal areas of strength. And we give
you lots of different opportunities to do that.

Of course, we will then match you with appropriate roles and avoid the ones
that require web-specific skills. FYI, we already have robotics, financial,
and game companies on our platform!

------
caconym_
Interviewing in Python for my semi-dream job is one of the best decisions I
ever made (on par with switching from Emacs to Vim). I don't even know the
language _that_ well, I'm an intermediate user at best, but the the reduction
in cognitive load from how straightforward and ergonomic it is is crazy.

It was also nice that they had an editor available with serviceable Vim
bindings. I think it was a big improvement over coding on a whiteboard.

------
coldcode
I have been coding for 36 years and currently do Swift/XCode. Used plain text
editors back in 83/84 time frame but have using some kind of IDE since Turbo
Pascal 1.0 and have zero desire to use either vim or emacs. I wouldn't hold a
lot to the idea that old programmers use them more than young. People should
use whatever they feel makes them more productive.

~~~
jhgb
But (Gnu) Emacs is _not_ a plain text editor. It can edit formatted texts,
embed pictures and widgets etc. While XCode's editor, even if I haven't seen
it in practice, most probably actually is. So it can't be the "plain text
editor" aspect that is the differentiator in favor of XCode here.

------
arjie
Interesting. Triplebyte is successful if the candidates they recommend are
candidates the companies hire. That means that by the time a company sees a
Triplebyte candidate, the candidate's language/editor choice should have no
signal left in it if Triplebyte has perfect performance.

Quite cool that they're willing to share fairly transparent performance
metrics with us.

------
KirinDave
And here I thought we had hit peak senseless-pattern-matching with the tabs-
vs-spaces dataset.

I was hoping for a quick section introspecting on correlation vs causation and
what this says about their process as opposed to a semi-serious presentation
of predictive power.

I can only assume a total lack of rigor and process if this is what they
choose to talk about publicly.

------
jedberg
I'd be interested to know how many PyCharm users were using standard editing
mode vs Vim mode.

~~~
otakucode
It's especially odd that the numbers support a strong Vim/PyCharm dominance
with emacs a distant third, but Vim/emacs are the headline and area of focus.
I suppose I can understand from the viewpoint that Vim/emacs are the
"old/weird" editors and PyCharm is seen as a modern 'modern IDE', but I didn't
expect to look at the charts and see such results from PyCharm. I happen to
think PyCharm is pretty good, and use it for almost all my Python work, but
didn't really think it was terribly popular overall.

------
wazoox
In my experience with many interns in the past few years, most of them came
using Sublime Text, in the unpaid nagware version; however all of the most
promising and intelligent ones moved forward to other editors, most of them
going to Vim.

------
walshemj
the OS break down was interesting as I would bet that most of the jobs are
using Linux's to run the code on the massive over representation of OSX is
odd.

Buying a more expensive Mac to use just as a terminal just seams an odd
choice.

------
todd8
Fascinating, but what does it mean. I can dream up explainations, now after
the fact, for this and the other interesting results in the article, but I
wonder what the real reasons are?

For example, Eclipse is the standard University Java course editor. Users that
have switched to IntelliJ have probably had more time away from University and
hence more experience. Is this the reason, or is it a "just so story"?

~~~
stupidcar
Perhaps, but a lot of teaching is done with Python nowadays, and Python has a
positive pass-rate, and actually tops the on-site success rate. So it doesn't
just seem to be a case of beginners getting rejected a lot.

Based on my anecdotal evidence of interviewing Java devs, I'd suggest the
problem is the opposite. While there are some excellent Java coders out there,
there are also an awful lot of "experienced" devs, with fifteen years of
experience and salary expectations to match, whose technical ability proves to
be pretty miserable.

------
btilly
I would love to see error bars on all of the graphs. Since we're estimating a
pass/fail ratio, any of the confidence intervals in
[https://en.wikipedia.org/wiki/Binomial_proportion_confidence...](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)
could be used. (Jeffreys interval may be a good default.)

------
tarboreus
The post is an enjoyable one, but makes a couple of assumptions, one being
that experience and coding ability are what get you hired. Sure, they help,
and they're a prerequisite, but showing up wearing the right clothes and
having the right attitude might actually matter more. People who use OSX are
more likely to be the kind of dev with the tidy beard and the square glasses
and the decent shirt. Devs using Emacs in Linux are more likely to be the
aneruotypical who doesn't look their interviewer in the eye. Both are more
likely to be good coders, though, since you don't get taught Vim and Emacs in
school, you have to pick it up through a non-school network or through self
study and research. Eclipse is the kind of thing you use because your
professor taught you Java and you never did any coding that wasn't for class.
Sublime is the kind of thing you use if you just don't really care about your
editor one way or the other.

I'd be curious to know what kind of shop they are at TripleByte. Something
tells me they might be Ruby coders using OSX all the time. Like attracts like,
especially in interviews.

------
CoreXtreme
I've not read it! But experienced folks will be hired more often than
inexperienced ones. And lots of old programmer use vim/emacs and other CLI
editors. So, it's not surprisingly. Do they've data adjusted for programming
experience?

------
seizethecheese
This just makes me wonder what factors are correlated with Eclipse editor
usage. Perhaps there's a sub-optimal channel to the developer career path
where people are taught Java on Eclipse. Does anyone know?

------
Karrot_Kream
I'd love to know whether these percentages were normalized by the number of
people in each bucket.

i.e. we should calculate P(pass | <emacs>) or P(pass | <c#>) and see how that
compares to P(pass) as a whole.

~~~
compumike
Yes, that's already what we're showing. On the "Triplebyte Technical Interview
Pass Rate by Editor" chart, Vim is shown as +38%. This means that:

P(pass | vim) / P(pass) = 1.38

------
smg
Is this more evidence that interviews are broken in the world of GitHub
profiles? There does not seem to be any correlation between the choice of
editor and developer productivity when it comes to open source development. It
is possible that developers who code with vim and emacs are forced to focus on
low-level details like method names while developers using IDE's can focus on
getting stuff done.

While I do understand that this is just content marketing to drive traffic, it
would be great if the following questions were handled with rigor.

Is editor usage related with age of the interviewer? If there is a strong co-
relation between Eclipse and older interviewees this could very well be
explained by ageism.

Is PyCharm ranking high in the rankings because of confounding with Python
also being high?

~~~
wutbrodo
There's no need to have such a defensive reaction. There's plenty of reasons a
result like this could appear, and most of them don't rely on "high-quality
developers prefer vim/emacs".

As a tangential example, back when OSX market share started climbing a ~decade
ago, it was a safe bet to say that the average OS X user was savvier than the
average Windows user. This was true at a baseline regardless of what you think
the "savvy" choice for an OS is: the simple fact that the sample of Windows
users includes the large chunk of the population that didn't give any thought
to their choice of computer (a sub-population that naturally is less savvy).
This dynamic would hold true whether you think that savvy users would prefer
Windows or OS X: by dint of being the default, the Windows-using population
had its average savviness dragged down, a fact that has no bearing on the
suitability of the OS for computer-literate users.

There could easily be a similar dynamic at play here: beginner developers are
going to be more likely to be on a big GUI IDE, which would drag down the
measured average quality of Eclipse users. This is entirely consistent with
high-quality developers preferring Eclipse to Vim, or being split 50/50
between the two, etc etc. As you point out, confounders like age and
experience are also likely contributors.

> It is possible that developers who code with vim and emacs are forced to
> focus on low-level details like method names while developers using IDE's
> can focus on getting stuff done.

FWIW, using Vim doesn't mean using raw Vim with no plugins. I used vim my
entire time at Google and I was the most productive dev on most teams I was
on. I'm pretty sure this would've been impossible without plugins at least
including auto-complete, jumping to definitions, etc.

~~~
nomel
> beginner developers are going to be more likely to be on a big GUI IDE,
> which would drag down the measured average quality of Eclipse users.

The data gives PyCharm and Visual Studio an "advantage", suggesting this isn't
the case. Or are you suggesting that only Eclipse is used by beginners?

> I was the most productive dev on most teams I was on. I'm pretty sure this
> would've been impossible without plugins at least including auto-complete,
> jumping to definitions, etc.

Pretty sure this was unrelated to the editor you were using. All sane editors
have these features you've listed, and I'm sure you would do equally as well
with them, especially since being a productive developer doesn't mean you
write the most code (usually).

~~~
wutbrodo
> The data gives PyCharm and Visual Studio an "advantage", suggesting this
> isn't the case. Or are you suggesting that only Eclipse is used by
> beginners?

I was just using a hypothetical example of how data can be confounded by
factors that have nothing to do with the central implication that "better
developers use vim/emacs". You're right that my particular example is weakened
by the fact you describe, but there could be _further_ confounders: in my
anecdotal experience, Eclipse is a lot more common as a beginner's IDE than
any of the Jetbrains IDEs (everyone I know in college used Eclipse, while my
first exposure to a Jetbrains IDE was a few years into my career, while at
Google. FWIW, I prefer JB IDEs to Eclipse (and prefer vim to both!)).

> Pretty sure this was unrelated to the editor you were using. All sane
> editors have these features you've listed, and I'm sure you would do equally
> as well with them, especially since being a productive developer doesn't
> mean you write the most code (usually).

I think you've completely misunderstood my comment. I didn't say these things
were related to the editor I was using. I said that the parent comment's
implication that you can only be productive with a big IDE (as opposed to vim)
is based on a misunderstanding of what vim is capable of. Obviously every sane
editor has these features: they're pretty much table stakes for development
IMO (which is the whole reason why I augmented my vim with them).

------
tritium
And yet, it's still all just plain text editing. Clearly, one kind of answer
is more fashionable than the other.

------
linguistbreaker
This "data" is not very meaningful.

It says much more about Triplebyte than about developers, technologies or
interviews.

------
brisance
Seems like machine learning could be used to provide greater insight.

------
otakucode
I think the headline kinda buries what I take to be the lesson from the
data... companies don't like Java devs. Which is understandable, I imagine
they find dealing with their
ResumeAdapterFactoryInterfaceFactoryAdapterInterface needlessly gross.

~~~
optimusclimb
That's such a dead and boring meme. Honestly the attacks I always see on Java
(such as this one) always come off as completely naive.

The amount of open source big data projects, and large profits and companies
relying on Java technology should clue you in that this is unfounded. Hell,
half of the modern smartphone market is all about Java.

I don't think companies dislike Java devs. I'd also take an article from a
company trying their hardest to commoditize engineers and turn them in to test
scores with a grain of salt.

------
AzzieElbab
Someone ought to put together performance charts by Zodiac signs

------
TomK32
vim ~/docs/cv.md

/Skills

}

I

* vim

:wq

~~~
castis
you just appended your skills list with "* vim

:wq"

and then left the editor running.

------
Naac
Article title is a bit misleading. Should be "Technical Interview Performance
by Editor/OS/Language"

~~~
sctb
Thanks, we updated the title from “Vim and Emacs users 2x as likely to pass
technical interview vs. Eclipse” a little while ago.

------
rawnlq
It is possible that eclipse users are more used to relying on their IDE for
auto-completion so they can't whiteboard as effectively...

~~~
wutbrodo
I hear stuff like this a lot, which seems to rely on whiteboard interviews
expecting people to know method names. Do anyone but the worst interviewers
actually _do_ this? That's a serious question, since my experience in this
regard is pretty skewed. I've been doing whiteboard interviews for years now,
including for a couple of years at Google, and the notion of ever expecting a
candidate to remember a method name perfectly is beyond insane to me. I'm not
OCRing their code and dumping it directly into a compiler: why the fuck would
I care if they used size() or length() when checking how long a string is[1]?

[1] Yes yes, I'm aware that this example isn't perfect because it could mean
different things depending on the language

~~~
Filligree
I'd like to hope they don't!

Interviewees, however, tend to get flustered if they can't remember the method
names. It's another rock on a pile that eventually leads to nervousness
collapse, something that happens far too often, so after a few dozen
interviews I've started stressing up-front that _no, it doesn 't matter, I'm
not scoring you on that_.

That, on its own, produced a measurable increase in average score.

~~~
jetsnoc
What do you score on?

~~~
Filligree
It all amounts to "Would I like to work with this person?", but a couple of
things. Persistence, memory, ability to learn -- not trying the same thing
over again if it doesn't work -- intelligence when possible, knowledge if I
expect them to have it.

That's pretty vague. I use the same strategy as the sibling post, ratcheting
up difficulty until the candidate fails, so I usually have a lot to go on, but
the exact criteria end up changing from one interview to the next depending on
what the candidate did.

Occasionally I end up with not enough data. I try to give my feedback in the
form of a confidence interval, but if it's _mediocre_ candidate combined with
not-stellar signal quality... figuring out how to score them can take a while.

------
kybernetikos
When it comes to a choice of programming language, ultimately a right tool for
the right job mentality is good, and for the kinds of tasks that commonly come
up in interview situations, C++ (or even Java) is not usually the right tool.

This of course doesn't mean that for real life situations, they're never the
right tool, but if someone chooses C++ to solve a simple tree inversion or
sliding window problem in an interview situation it probably means that they
don't have sufficient familiarity with any language more appropriate to the
situation.

~~~
dabei
C++ in the right hand is very succinct and expressive for solving algorithm
problems. If someone claim c++ expertise but switch to their secondary
language for some questions I'd think twice about hiring them.

~~~
kybernetikos
This is true - there are some people who can make C++ sing even for these
kinds of problems, however _most_ people who attempt to use C++ in an
interview situation to solve toy interview problems do poorly. For most
people, it takes them longer to solve the problem, to find and correct bugs
and to provide tests that prove that they've solved the problem correctly than
someone solving the same problem in a different language. I've noticed this in
my own interviews, and this article mentions it too. It seems to me that the
most natural explanations for this are either that the pool of C++ programmers
has more bad programmers in it than other languages (which I don't believe) or
the language and tools themselves are not as well suited to solving interview
questions in interview situations.

As to penalising people who switch languages for problems, if you give
guidance that you want to see a solution in a particular language, that's fair
enough, in any other case it seems unfair and misguided. Perhaps you don't
really agree with 'right tool for the right job' when it comes to languages?

