
Good at programming competitions does not equal good on the job [video] - jpn
http://www.catonmat.net/blog/programming-competitions-work-performance/
======
pitt1980
I'm not able to watch the video at my current computer

but its actually really typical that when something becomes an advantage for
being selected to a certain pool

the success of the those in the pool after the selection will negatively
correlate with that thing

\---------

the really obvious example of this is the hockey birthday thing from Malcolm
Gladwell's Outliers

people with the earlier birthdays were more likely to make it past each
selection stage in becoming an NHL player

but those with the later birthdays who were able to be selected in spite of
their later birthdays, were typically more successful after the selection

\---------

The authors contend that the strategy might actually work against a team's
success because they found that players born later in the year and drafted
later actually had more productive hockey careers.

Deaner said the study showed that men drafted in the second half of the year
were about twice as likely to have successful careers in the NHL ??? reaching
benchmarks like 400 games played or 200 points scored ??? than those born
earlier in the year.

"If the team wasn't making this mistake, they probably would have been more
successful," he said. "The guys born in the first part of the year are much
more likely to be busts."

[https://www.nhl.com/news/study-suggests-nhl-has-bias-in-
favo...](https://www.nhl.com/news/study-suggests-nhl-has-bias-in-favour-of-
players-born-earlier-in-the-year/c-657724)

~~~
MarkMc
That's an excellent point! I can think of another example: In the movie
"Hidden Figures" the black, female engineers at NASA are much better than
their white, male counterparts simply because it was harder for them to get
in. Perhaps the opposite is true today of engineering students at colleges
with affirmative action?

~~~
cma
Alumni preference is at least as big or bigger a factor in admissions than
affirmative action. Many of the schools were segregated not so long ago, so
you can guess how that affects the alumni pool.

------
fatjokes
I'm also a former ICPC world finalist and I'm willing to believe this is the
case in a BigCo because: 1) Top ICPC competitors tend to be extremely socially
awkward, and may not work well in groups. 2) Vast majority of engineering work
is very algorithmically simple, thus these folks don't get a chance to
"shine".

However, I've noticed that they are popular in prop trading firms, where work
tends to be in very small teams or individual. I don't know how their
performance correlates to fund performance.

If I were hiring, I'd still prefer to hire at least some top ICPC performers.
The hard algorithms are rare---but can make or break your product.

I also think the knowledge learned from programming contests is invaluable.
I'd like to be able to discuss bipartite matching or min-cut with my
colleagues without eliciting a blank look.

~~~
_0ffh
> Vast majority of engineering work is very algorithmically simple

anecdotal_evidence+=1 : Am engineer, only algorithmically interesting task I
had the last three months was to quickly find all minimal solutions of a given
instance of a certain class of constraint graph problems. That's about
average, at my job. Four algorithmically interesting things a year. And I
guess I might even be one of the luckier engineers.

~~~
0xfeba
I just match APIs up 60% of my time. 20% in 75% useless meetings. And another
20% yak shaving.

------
rkunnamp
Just like 'There is no such person as Good CEO', but only 'A Good CEO for a
particular company, at a particular point of time', there is no such a person
as 'Good Programmer', there is only 'A Good Programmer for a particular job,
at a particular point of time'.

Context matters a lot. One does not use an AK-47 to kill a mosquito. It is
terrible for that job. But that does not make AK-47 a terrible weapon.

Good at programming competitions does not equal good at any programming job,
is a more appropriate sentence.

~~~
kpil
I think that there are personal traits that make people perform good in most
situations, but perhaps not all extremes. As continuity is normally good, you
try to find those, and change only if you really have to.

A programming competition evaluates almost none of the traits thats doing a
marathon run as a highly preforming team.

------
lr4444lr
In the chapter of his interview in _Coders at Work_ [0], Norvig found that the
strongest correlate in the interview process with success at Google was
paradoxically to have been given the _lowest_ possible score by one of the
interviewers. He surmised that this was because in order for such a person to
have even gotten hired at the end of the process, someone trustworthy must
have seen so much potential in the prospective hire that he strongly advocated
for the person to be offered a position which worked _in spite_ of that other
low rating.

Kind of ironic for a company whose product values are so tightly tied to
quantitive data.

[0][http://www.apress.com/us/book/9781430219484](http://www.apress.com/us/book/9781430219484)

~~~
aoeusnth1
This isn't surprising - in fact, it would be surprising if it wasn't the case.
This fact says nothing about the ability of Google's hiring bar to distinguish
between good and bad hires, except that it is not a perfect signal.

~~~
lr4444lr
Totally agree. It just serves as a useful reminder though not to fall prey to
the fallacy of deliberately chasing the correlational measure as a matter of
policy.

------
mcv
I once had an online discussion about the difference between competitive
programmers and professional programmers, and which were better. Someone
argued that competitive programmers were better, because they had to perform
in more extreme circumstances. As he put it: they were sent into the forest
with a knife to kill a lion.

Everybody else ran with that metaphor. Someone asked what a lion was doing in
a forest; don't they live on the savannah? I asked whether he was sure there
was a lion; plenty of times I've been sent to kill a lion and ended up having
to kill a goat or an elephant instead. Are we even sure it needs killing?

I think that's the difference between a competitive programmer and a
professional programmer. The competitive programmer will be much faster with a
solution to the given problem, but the professional programmer will solve a
better problem.

~~~
wlesieutre
Mountain lions like forests just fine! Unfortunately for them, we chopped most
of the forests down.

Assuming you're in the US and somebody tells you there's a lion around, pretty
good odds it's not one of the big orange ones.

EDIT: Bay area published yesterday
[http://www.nbcbayarea.com/news/local/Several-Mountain-
Lion-S...](http://www.nbcbayarea.com/news/local/Several-Mountain-Lion-
Sightings-Shut-Down-Trail-at-Alum-Rock-Park-432968923.html)

~~~
mcv
A mountain lion is a very different animal from a lion. Different subfamily
and genus, even. I don't live in the US, but when someone tells me "lion", I
tend to assume they mean a lion, and not a puma. Though if there's any doubt,
that's absolutely something to ask questions about.

------
NTDF9
I have had a bad experience with an algorthmic topcoder.

The guy has a brilliant mind. But he doesn't understand the big picture or why
his code would fail when integrated with big codebases.

He thinks his work is "done" when he has written his tiny little code with
some queues and trees. Then he leaves it for someone else to integrate and
thus really solve the problem.

The best engineers I found were the ones who took ownership of their projects
and had the ethic to dig deep. Not the ones who could solve toy algo problems.

------
bit_logic
Here's a list of companies that don't do this
[https://github.com/poteto/hiring-without-
whiteboards](https://github.com/poteto/hiring-without-whiteboards) I hope it's
the beginning of an industry wide trend away from these types of interviews.

------
FTA
Look at programming competitions from an assessment perspective: what are they
measuring and is what's being measured good or bad for a job?

Ability to work under short time constraints (probably good), to hack out some
solution that will work temporarily (good) but probably not solid (bad), to
forego much time to consider the implications of design and implementation
choices (bad), to develop without communication if solo competition (bad) or
without communication outside of the small group of core developers (bad), to
build a solution without getting feedback and refinements from stakeholders
(bad), and so on.

------
kazinator
People who enter programming competitions are looking for some sort of glory:
to be stars. When they don't get it from the job, they get bored.

I suspect that the people good at programming competitions could easily
perform well on the job, if the motivation were there. I don't think it
necessarily has anything to do with short-term versus long-term problem-
solving focus.

There are plenty of short-term problems that you have to solve on the job to
be effective. You're not doing them in competition against anyone such that if
you procrastinate, you will lose, so the motivation vaporizes.

Also, since job interviews are like programming competitions, people who are
good at programming competitions figure they can easily get a job anywhere,
and to do that repeatedly. They are not motivated into working hard by job
security concerns.

------
js8
I would tend to agree, and while I was never very good at programming
competitions, when I saw some of the winning code on Topcoder, I somehow lost
interest. If that's the code I would write if I learned to be better at
competing, then I should rather spend time learning something else.

On the other hand, something like Project Euler or even Advent of Code is very
nice, if you do it at your own pace, or for learning a new language.

~~~
shortleon
Although, I do think that competitive programmers can become great programmers
more often than not.

They practice writing correct for-loops, branches, recursive functions,
efficient graph searches, input-output, data structure uses etc. in a very
fast pace. To be able to do that requires you to chunk a significant amount of
information.

I found that doing stuff very fast correctly makes you learn and internalize
concepts quickly and more deeply.

This is all unfortunately anecdotal, I'm not sure how to google for such
research (not just in cp but in "learning to do stuff fast improves learning
rate").

Yet every instrument I've played and tried to learn, the moment I tried to
play stuff fast but correctly (be it drums or piano), I had to improve my
technique, had to internalize the rhythm patterns (more complex rhythms are
insanely difficult to do fast), memory etc. With it came a significant amount
of progress.

Same thing happened to me with language. I was speaking English for 20 years
but still had trouble with fluent pronunciation (despite my writing feeling
natural), I knew what I had to say but somehow my tongue got all tangled. Yet,
when I tried learning some rap songs that use insanely fast diction, my
speaking improved up to a point where it felt normal.

------
0x4d464d48
I haven't done a code competition before but my understanding is that code
competitions reward cowboy coding over good engineering practice, i.e.
implementing features while paying no heed to maintainability.

When you have that mind set and start dealing with a codebase as monstrous as
Google's it sounds like a recipe for some serious technical debt.

------
potatolicious
It continues to surprise and frustrate me that as an industry we continue to
highly prize proxy signals for engineering skill, when engineering skill is so
directly measurable.

Why even bother with measuring things that are N-degrees removed from actual
engineering, when you can just get people to engineer things?

I know others on HN have been hammering this point home for years, but until
something changes it deserves to be repeated ad nauseum: work samples work
samples work samples work samples.

Stop with the trivia questions. Stop with the contrived algorithms questions.
Ask people to design systems, ask people to defend their designs, ask people
to write real runnable code that directly relates to the work they will be
doing at your job, ask people to review a real piece of code written at your
company, ask people to critique real design produced at your company. Anything
but what we're doing right now.

~~~
teen
I actually like the current interviewing approach... and I feel like we are a
silent majority. I think whiteboarding / coding algorithm and design problems
is pretty fair and reflective of engineering skill.

~~~
stupidcar
How is whiteboard coding reflective of engineer skill? At what time, during
any job you've had, have you needed to instantly write code on a whiteboard,
in front of peers, in a matter of minutes. Almost by definition, it's not
reflective of engineering as actually practiced in a job.

I'm a far bigger fan of at-home coding exercises. Yes, I know, some people get
annoyed at these because they think you're asking them to do free work in
their spare time. But what other way is there to test people's coding under
circumstances that most adequately mirror those of an actual job. E.g. they
have a (relatively) unlimited amount of time, they have access to Google,
their IDE, etc.

I'd much rather see what someone can come up with, in response to some novel
problem, based on a couple of days programming, than what they can hack out on
a whiteboard based on thirty seconds of thinking. The former seems closer to
what coders are actually required to do, day on day.

~~~
DavidWoof
> At what time, during any job you've had, have you needed to instantly write
> code on a whiteboard, in front of peers, in a matter of minutes.

Well, constantly, actually. Although I'm thinking of designs and snippets
rather than actual functions. I guess it depends on what you're asking people
to whiteboard during the interview. I agree that asking people to whiteboard
qsort is silly, but walking through design alternatives with occasional code
snippets to illustrate implementation options is a pretty basic skill.

> based on a couple of days programming

Either your company is very well-known and very attractive to candidates, or
this is going to incredibly restrict your candidate pool.

I think smaller work samples are a great idea, I think code reviews are a
great idea, but asking for two days sounds like a bit much, especially early
in the process.

~~~
kafkaesq
_Although I 'm thinking of designs and snippets rather than actual functions._

But that's the thing -- at whiteboard interviews, they don't ask you to
produce "designs and snippets". They make you write actual working classes and
functions.

------
pklausler
As a long-time interviewer, I've learned that a candidate being good at
programming competitions means that they're probably good at programming
competitions.

It's a weak signal either way for success or failure at interviewing and being
able to do the job. Part of the problem, as we've discussed on HN so often
recently, is that a programming interview has to waste time with FizzBuzz-
style questions just to flag the candidates with great resumes, transcripts,
and phone screenings who still actually can't program a computer to solve even
a trivial problem.

~~~
kafkaesq
_who still actually can 't program a computer to solve even a trivial
problem._

Or who get nervous, and freeze up.

------
bjacokes
Here are some of the positive qualities I would expect from a competitive
programmer compared to the general CS population _early in their career_ :

\- knowledgeable at algorithms and data structures

\- good at analyzing correctness and edge cases, even on simple non-
algorithmic problems (e.g. FizzBuzz)

\- accustomed to working hard and learning new concepts. this attribute is not
specific to competitive programmer – for example, I'd expect similarly from an
open source contributor – but it's higher than for a typical college student.

Some negatives I'd expect, which are fixable over the course of the course of
their career:

\- over-confidence in code, under-testing

\- less skilled in OOP, coding style, version control systems, as well as web
development or systems code (unless they have specific previous work in these
areas)

\- sometimes looking down on gruntwork/rote as beneath them (like the view
that pure mathematicians have towards applied math or statistics)

I think that list of positive attributes often outweighs the potential
negatives, especially during an internship or in the first year or two of
someone's career. After that, I would expect many non-competitive programmers
to have picked up on some of those advantages (code correctness, learning new
concepts).

I've tried to steer my own interviews away from algorithms (especially DP) and
focusing more on giving problems that are relatively straightforward, while
still being complicated enough that someone has to write precise code and
identify/fix a few edge cases.

------
kinkrtyavimoodh
Adding to the good points here, I think coding competitions superstars are
also potentially more likely to have a diva-complex, while others are more
likely to have an impostor complex when they get into a company like Google.
The right amount of impostor complex (where it enables you but does not
cripple you) is actually very helpful as it helps you learn and better
yourself.

------
sghiassy
Question?

Does being good at programming competitions make you good at interviewing for
programming jobs?

Obviously there's some irony in that question. But I also think there's some
truth to it as well.

~~~
msteffen
As a former ICPC world finalist who later worked at Google, I would say that
practicing for programming contests is almost like cheating as far as Google
interviews are concerned. The Google interview format is almost identical to
the ICPC practice sessions I did in college.

I've seen this video before, and IMO the extreme similarity between
programming contest questions and Google interview questions could explain the
negative correlation. Specifically, borderline engineers with programming
contest experience are more likely to get hired than borderline engineers
without programming contest experience, and therefore the set of Google
engineers with programming contest experience includes more borderline people
than the set of Google engineers without programming contest experience. Thus
the slight negative correlation.

~~~
cbhl
When I was in high school, being good at programming contests meant picking up
a bunch of habits that you'd never use at work -- memorizing the same twenty
#include lines that include every conceivable STL data struct you'd need;
using single letter variable names; no comments whatsoever. If you and the
next person come up with the solution at the same time, you might lose simply
because the other person could type faster. You have to un-learn these habits
for industry.

Just about anyone can get exposure to a set of representative coding questions
(see: the USACO training robot, or Cracking the Coding Interview), but
training for these contests means spending XX hours a year under a time limit
trying to write code from memory (because you don't have time to look things
up in the manual).

------
paulcole
Also important to remember that good at programming does not necessarily equal
good at job.

Excelling as a communicator, being an empathetic person, and having great
interpersonal skills are just as (perhaps more) important than how well you
can code.

------
agounaris
Winning the dunk competition does not make your team be an NBA champion...

~~~
BoiledCabbage
But winning the 3pt shooting contest is a pretty good proxy for shooting
ability.

~~~
NTDF9
And I'm so glad that Golden State Warriors didn't just Hire Kevin Durant for
his 3pt shooting abilities. He'd lose to a lot of people who are better 3pt
shooters.

~~~
sjg007
So what happened to teamwork? We see dominant teams last 4-5 years and then
fizzle. Wooden had 10 championships with 7 in a row. Celtics had 8 in a row of
10 championships.

~~~
NTDF9
Exactly! I'd hire a Durant (a high performing team player) over a sharp-
shooting 3 pointer genius (fast algo coder)

------
aqp
[http://ruberik.blogspot.com/2017/07/no-programming-
competiti...](http://ruberik.blogspot.com/2017/07/no-programming-competitions-
dont.html)

It looks like it wasn't "being good at programming competitions" that was
negatively correlated with job performance.

It was "participated in programming competitions".

And there are some more "how to interpret machine learning models" caveats in
that blog post.

It seems to me the biggest factor in explaining this is that the people who
are just below the hiring line but participate in competitions get a bump over
the line. Since there are more people just below the line than above it, the
"participates" group is bottom-heavy, producing the correlation.

I do a lot of interviews, and it seems to me that lots of people with
experience perform below how they "should", because they're not practiced at
solving problems from scratch, they work all day on modifying larger systems.
Programming competitions would fix that for them, as would most open-source
hobby projects.

------
AndrewStephens
I don't find this surprising. I played around on Top Coder enough to reach the
first division but the problems presented were in no way related to my day to
day work. While the people that did very well (I never got anywhere once I hit
the top grade) might have had an excellent command of very specific data
structures and techniques, non of the code would have been very useful in the
real world.

If I was interviewing someone, a career as a competitive programmer would not
be a detriment but it would not count for very much overall. We are looking
for creative thinkers that can work as a team.

------
CalChris
The market is pretty good at sorting this out and I'm sure programming
competition winners are fairly valued. I've seen more Berkeley, Stanford, MIT,
CMU and IIT degrees rise to the top than other schools. I haven't seen any
_top coders_. Maybe they're there and I haven't noticed. But I haven't seen
them on resumes that I've culled through.

It's probably something you might list on a first job resume but further on
down the road, you'd just list your work and your education.

------
WalterBright
All job interviews rely on proxies for whether someone will be good on the job
or not, because the only way to know is to actually hire them.

All proxies are inaccurate.

~~~
blacksmythe
Also, everyone is convinced that _their_ proxy is the best achievable,
although of course they test it only against false positives, not false
negatives.

~~~
WalterBright
> everyone is convinced that their proxy is the best achievable

In my experience, people merely hoped their process was good enough. And it
usually was. The "we only hire the best" is marketing propaganda and they know
it.

------
qdev
My viewpoint is taken from the context as someone who is a reasonably seasoned
developer returning to the job market. I never did programming competitions in
university, though I surely was someone who fit the profile (computer science
guy with a discrete math bent). As part of my recent prep for a job search, I
joined one of the programming competition websites and did a couple of
contests.

I posit that one of the dangers of spending a lot of time doing programming
competitions and becoming very proficient at them is that, perhaps, you can
come to believe that "true" programming, some sort of Platonic ideal of
programming, is about coming up with the clever insight that solves an
algorithmic puzzle.

But, in fact, a fair bit of _commercial_ programming is down and dirty, with
databases, and user interfaces, and a lot of the time is really just shuffling
data from one place to another, maybe filtering it or combining it with
another set of data.

And that's just at the beginning of your career. Later on in your career,
success means being able to work at larger scales in a team. That means
organizing the code in a way that supports the efficient development of the
codebase by individuals like yourself, by your team, by the development group
as a whole... And at the architect level, you perhaps are looking at designing
the system to support the efficient operation of the entire organization.

So I can easily believe that success at a programming competition does not
correlate with long-term success as a software engineer in commercial software
development. The two are really very different.

(Btw, I actually found the competitions that I did to be fun, but mentally
exhausting. I'd say go ahead and do them, especially if you have an
inclination for those types of problems. Just be prepared to use a different
mindset for commercial software development.)

------
bshanks
I see a lot of comments speculating that there may be something wrong with
programming competition winners, but this result might be a statistical
curiousity unique to Google. It's possible that for most companies, good at
programming competitions could still be positively correlated with job
performance. For example, what if most people whom Google hires could have won
some programming competitions if they had wanted to (a situation that most
other companies are probably not in)? In this case, 'won programming
competitions' could be data that is a lousy filter for Google, yet a good
filter for other companies.

~~~
closed
Good point, in essence it sounds like one theory people have is that
competition winners have over-specialized, but your theory is that there could
be floor effects in the range of ability at Google (i.e. the floor is high,
many people at Google could win competitions).

------
purpleidea
Completely agree with Peter. I'm shit at those competitions (maybe not the
worst ever) but I think I'm pretty good at my job. But I think there are also
some who are good at both.

------
x1798DE
In general, I think there are a lot of things like whiteboard interviews and
coding competitions where employers would _prefer_ "good at X means good on
the job", but are reasonably happy to settle for "bad at X means bad on the
job."

------
samlittlewood
When I am interviewing:

Candidate having competed in programming competitions - big green flag.

But, if success in competitions is, in their view, their biggest asset - amber
flag.

I would extend this to most competitive endeavours.

------
jpn
This:

[https://news.ycombinator.com/item?id=13739329](https://news.ycombinator.com/item?id=13739329)

------
hajderr
well if you're not good at programming contests would that entail a
contemplating/slow programmer?

I'd rather pick an algorithmist and teach him/her to reflect than a so called
"reflectionist" / "slow coder" and teach him/her how to solve algorithmic
problems.

I'd be interested in knowing the guy's (catonmat) own reflections and
experiences too.

------
twii
Depends also on what job I guess?

------
bluetwo
Overfitting?

------
williamsmj
The title of the post understates the claim in the link, which is that, "being
good at programming competitions correlates negatively with being good on the
job".

~~~
rquant
no the statement is ""being good at programming competitions correlates
negatively with being good on the job, conditional on passing Google
interviews" which is completely different statement.

"being good at programming competitions" hugely positively correlated with
"begin good on the programming job" unconditionally

~~~
sbierwagen
Do you have... any kind of evidence to support that claim?

~~~
Klockan
The overwhelming majority of people who can't code would fail horribly both at
jobs and competitions, this effect will drown out everything else no matter
how the distribution looks for the few percent who can code.

------
adavidoaiei
They are two types of business, first type business to business, software made
to run internal in a company, the business run on this software, generally in
this kind of software you use enterprise frameworks, the second type are
business to consumer where anyone can made an account and use application,
some software as a service are like that, they are too other types of web
applications business to consumer.

When you allow that anyone to make an account and use your application
sometimes you couldn't rely on enterprise technology and it's better to write
everything from scratch or to customize open source solutions.

What they are testing in algorithms contests: 1.) the algorithm is correct
with various test sets 2.) the performance of the algorithm, running time in
milliseconds, there you should to know how to measure time complexity of
algorithm with O(N) 3.) memory consumed, there you should know how much memory
you are allocating for variables, for example a variable of type byte consume
8 bits

I think that programmers which has prizes in algorithms contests are suitable
for business to consumer applications because there they will find a
challenge, this kind of programmers will be bored to dig in enterprise
frameworks.

