
We looked at how a thousand college students performed in technical interviews - leeny
http://blog.interviewing.io/we-looked-at-how-a-thousand-college-students-performed-in-technical-interviews-to-see-if-where-they-went-to-school-mattered-it-didnt/
======
Tade0
I'm leaning towards blaming this on the interviewing process.

Last year I spent some time interviewing job candidates and what I learned
from this is that the recruitment process in IT, in general, is broken.

A typical candidate is first faced with some kind of task to weed out the
lower 20% of applicants. After that comes the proper interview during which
the candidate is asked various questions like what was their major or how is
the abbreviation "SOLID" expanded.

In my experience, none of this correlates highly with future performance and
this may be the reason why regardless of the school they attended those
students performed roughly the same.

Time and time again the one thing that seperates the best from the rest is
their ability to perform code reviews. I have yet to find somebody who's a
poor developer, but a great reviewer.

~~~
dbcurtis
Interesting. Do you have candidates review code?

~~~
owyn
I have done this.

Print out a page of code and review it with a candidate. Have them explain
what it does, or how they might go about figuring that out, and what kinds of
improvements they might make. Include some glaring bugs or code style problems
if you want to ask those sorts of questions.

I feel like small fragments of code work well for refactoring questions, whole
programs tend to have a bunch of boiler plate code, but going over a bigger
program to find the real core features quickly is also a valuable skill.

I've done this with SQL too. Printed out a real statement from an application
and ask about performance, optimization, what the application might be doing,
etc.

Or load a 100,000k line application in their favorite editor and ask "okay,
try to find the frobzing subsystem, I want to know how it frobz bazzes". (I
haven't done this yet...)

------
capocannoniere
> What this means is that top-tier students are achieving the same results as
> those in no-name schools

This is an misleading conclusion that ignores a HUGE selection bias. I doubt
top MIT CS students, for example, would feel the need to practice coding
interviews on interviewing.io

~~~
leeny
We have a bunch of MIT students. The sad truth is that everyone is scared of
technical interviews.

~~~
rcheu
I went to MIT and knew a good number of people who were confident in technical
interviews. The problems in interviews were much easier than what I got in
class, or even CTY CS classes in middle school.

While there's definitely good people at non-elite universities, selection bias
is at play here too.

For example, in the senior year case, you're just looking at people who were
concerned enough about interviews to go to interviewing.io during their senior
year. Most people I knew at MIT had a full time offer they were happy with
from their Junior year internship by their senior year.

You can actually see the impact of this in your graphs too, notice that the %
of people with a 1 or a 2 in elite schools increases for senior year vs junior
year. People who have been doing well in technical interviews and have good
return offers won't spend as much time preparing.

~~~
akhilcacharya
>CTY CS classes in middle school.

The hell? What sort of problems are you getting?

Were you doing edit distance and word break and shortest palindrome in middle
school?

~~~
rcheu
Was a long time ago so I don't remember them all, but some examples I do
remember:

* Print pascal's triangle up to the n-th level, with proper spacing (so that it looks like a triangle).

* Make the game Go for 2 players. I got a sub problem of this during my Google interview (determine whether a piece is captured).

~~~
wiz21c
When interviewing in tech jobs I had : \- build an asynchronous communication
queue from scratch (which implies building semaphores) \- optimize the drawing
of a sphere on a screen

the rest was more regular (I didn't apply to super elite jobs, and both the
companies with these two hard tests failed, so it's no indication neither
about them, neither about me)

~~~
mrastro
> \- build an asynchronous communication queue from scratch (which implies
> building semaphores)

How long were these interviews? Seems like something I could do in a few days
not a 2-hour interview!

~~~
wiz21c
It lasted two hours. Now the way it worked was that I was doing on my own but
I had to constantly explain to a "peer" my line of reasoning. So I was not
alone in front of a white paper :-) This was an extremely interesting wy to
interview. I felt I had the opportunity to demonstrate my skills on a
difficult problem (which I didn't solve). I didn't remember the details of
Dijkstra's algorithm but I had the opportunity to show that I can approach a
complex problem in a logical and efficient way (well, that day, that kind of
problem; don't ask me to work on helicoidal tomography :-))

------
metaphor
> _What this means is that top-tier students are achieving the same results as
> those in no-name schools._

When NYU and Arizona State are "Top 50" while Michigan State and Vanderbilt
are classified as "no-name schools", I question the meaningfulness of this
blog's baseline and what it means for an interview to be technical.

~~~
kenhwang
I think the bucketing of schools by overall rank is about as good as
arbitrary. Why not rank by strength of CS program? If we use PayScale's
"Computer Science Majors by Salary Potential"[1] as an indicator for interview
success, it looks very much like the CS rankings.

[1] [https://www.payscale.com/college-salary-report/best-
schools-...](https://www.payscale.com/college-salary-report/best-schools-by-
majors/computer-science)

~~~
metaphor
> _Why not rank by strength of CS program? If we use PayScale 's..._

CS alone is far from representative of tech industry in the aggregate.

Furthermore, PayScale's list clearly doesn't control for locality bias--e.g.
consider Purdue #191, UW Madison #138, UM Ann Arbor #88, UT Austin #79 to name
a few; or what Duke #10 while UNC Chapel Hill #67 even means?? How do top
programs which feed industry around tech hubs like Austin and Research
Triangle meaningfully compare to the Bay Area and NYC if unadjusted gross
salary serves as the sole basis of value?

~~~
Meegul
Also, those numbers are absurdly low in my experience. It claims that Purdue's
starting is $64,000, but I don't know a single person who's graduated and
started at less than $75k, and I know plenty more that have started well into
six figures.

------
acconrad
After running over a hundred interviews within the last few years, this
doesn't surprise me at all. Interviewing is a skill separate from school, and
if you train for it, you can become good at it, regardless of your pedigree.

I actually think studying for the SATs is a better comparison - it's a test
that doesn't exactly translate to real-world performance, but has huge bearing
on prospects, and often the best advice for acing both (tech interviews and
the SATs) is to simply do as many practice tests/problems as possible in the
areas you're weakest in.

~~~
montyf
I was always under the impression that SAT scores correlate strongly with IQ,
not with practice. I had almost a perfect score[0] on my SATs, and I only used
about half the time available, and I did zero with a capital Z practice
problems beforehand -- many other kids took practice tests every weekend and
still performed abominably.

[0]: I only got two questions wrong on the entire test, both math problems.
Since I was practically unstoppable at math back then, I always suspected that
there were errors in grading.

~~~
wfo
A large portion of the SAT is knowledge of English vocabulary which obviously
has nothing to do with IQ.

The mathematics questions nearly all use the same basic rules and are
presented in the same format, and can easily be learned and trained.

SAT prep courses are ubiquitous and expensive because they work. Do people
without training do well? Sure. Do people with training still do poorly? Sure.
But a good SAT question would be "does this mean that training doesn't help or
that there isn't a correlation between training and success?" because the
obvious answer is no.

~~~
Consultant32452
Vocabulary is a component of IQ.

[https://en.wikipedia.org/wiki/Intelligence_quotient](https://en.wikipedia.org/wiki/Intelligence_quotient)

>The many different kinds of IQ tests include a wide variety of item content.
Some test items are visual, while many are verbal. Test items vary from being
based on abstract-reasoning problems to concentrating on arithmetic,
vocabulary, or general knowledge.

~~~
wfo
That's odd, when I learned about it a long time ago I was told it was intended
to measure "inherent" intelligence, not the level of exposure/memorization of
a massive set of (likely trivial) data points.

Vocabulary very clearly belongs on an achievement test, not an aptitude test
(as I imagined IQ to be).

------
BigChiefSmokem
People have forgotten how to train their core social and psychological skills
like resilience and confidence. Unfortunately, these are not things we are
born with or can code our way out of. These are things we have to train very
hard at similar to how martial artists or athletes approach their training.
The thing most programmers and knowledge workers in general are missing is
true belief in their own skills (you have been doing this how long? come on
snap out of it) and the resilience that comes from rejection after rejection,
and failure after failure. After a certain point in your experience you should
come to understand that raw skills are not really the end all, be all. If you
are not the guy then you are not the guy no matter how many degrees you have.
Sometimes the guy with the PhD from MIT is not the right guy because he
doesn't connect with the rest of the team, or the vision, or whatever, simple
as that.

Also, if you're testing only for specific skill sets rather than aptitude and
interest then you already failed as a hiring manager.

~~~
chrisweekly
Amen. Also "hire for strengths, not to avoid weaknesses".

------
_pastel
> Indeed, statistical significance testing revealed no difference between
> students of any tier when it came to interview performance.

Ah, the classic confusion between not finding enough evidence to prove there
is a difference, and finding enough evidence to prove there is no difference.

The juniors from elite schools in particular have fewer 1s and 2s and more 3s
and 4s than the other juniors. Really, you found statistical evidence that
they aren't from different distributions? I'd love to see it.

------
akhilcacharya
I think this is a bit specious, at least from anecdotal evidence. I know
bunches of people at my school (Top 50) that have interviewed at companies
that ask leetcode-style questions yet very few get offers or even make the 2nd
round or onsite. If you look at LinkedIn you'll see the vast majority work at
Cisco, IBM, SAS, Amazon etc and not Google, Jane Street, or Facebook.

The quality of the cohort and, to a lesser extent, the quality of their DS&A
classes matters quite a bit.

~~~
Ancalagon
Is the Amazon interview process no longer considered a top-tier difficult
interview?

~~~
akhilcacharya
As someone that has gone through it, no. It is possible to get intern and new
grad full time offers by passing one online debugging test and one coding test
(in rare cases a single one!).

Also the comp is a league below the others I mentioned.

~~~
akhilcacharya
*with that said, Amazon does work on a lot of really revolutionary stuff that I definitely wouldn’t be able to work on in 90% of other companies. Really enjoyed my internship there.

------
readams
Maybe what they've proven is that technical interviews are not adding much
information to the hiring process. This pretty much matches with my experience
that quality of interview is at best only very loosely correlated with actual
ability as a software engineer. Much more relevant is direct experience
working with them. School quality is a strong signal.

~~~
Spooky23
In my experience, school grades at good schools are filters for non-bozos.
Beyond that, nothing.

The best programmer I ever met was a musician with no college.

------
sologoub
Charts don’t have the same scale. As the result, it’s hard to visually compare
these - for example, for juniors, the shapes are similar for score 4, but top
tier school students are above 20 mark, while the rest are below.

Interviewing is expensive, especially if you hire the wrong person, and if a
given population tends to cluster around higher scores, that what you’d pick
first.

~~~
starpilot
Yeah, why didn't she normalize results and combine charts for easy comparison?

------
didibus
I started wondering why CS interviews aren't more based around reputation and
experience, like other jobs.

I've seen a lot of bad devs get hired into places with supposedly high bars,
or devs being let go and then ending up at Google next. While some places
often let go of good devs, I'm talking of cases where I believe it was
justified.

------
GreaterFool
We all know that interviews are terrible. Wouldn't _that_ be the problem? Then
students from top-tier universities achieving the same interview score as
student from lower-tier universities is meaningless. The interviews fail to
capture what matters.

The best jobs I had and also jobs I performed _the best_ I got through
credentials and experience, not scribbling on a whiteboard.

If you went to top 10 school in the world, you worked _hard_ (at least it used
to be this way). I don't need to look at specifics of what you did but I know
you needed grit, good work-ethic, personal time sacrifice, etc to get through.
Unless you're super smart. In either case it adds to your personal brand.

I think that's what perpetuates the system. Given unlimited time I'd be happy
to give everyone a shot. But time is precious so stick to what you know? I
know how things are in the Uni I went to. If a CV landed on my desk form
someone who did the same course, I'd put them on top of my list.

~~~
stmw
This sounds reasonable enough, but on the other side, so is the observation
that often people who get into and get through top 10 schools are actually
ones that did not work hard.

The point of using interviewing.io online technical interview as the first
step of the process is that it makes your "I'd be happy to give everyone a
shot" take far less time than alternatives. Still can't get unlimited time or
perfect fairness, but it's better.

~~~
opportune
I went to a "top 10 school." I'd say that only maybe 20% of the students (in
CS) were people that I would want to hire if I were running a company. A lot
of top school students get accepted because a) mom and dad are rich, sent kid
to feeder high school, pushed kid to do ECs/do tutoring/get help with essays
b) they fit into some admissions "bucket" like {has inspirational story} or
{huge volunteering effort} or {also amazing artist}.

The worst students were always the ones that went to feeder schools and had
wealthy parents, which are incidentally the majority of students at top
schools

~~~
Consultant32452
I wonder if your experience taints your opinion here. I don't know because I
barely went to a state college for a minute before I got the job I wanted and
dropped out. It's possible if you spent the same time at an average university
first you might find 50 or 80% of the "top 10" students to be worth hiring.

------
smt88
This tells us nothing about the talent or intelligence of the candidates
because these tests are bullshit.

I wish companies like this didn't exist to enable these types of interviews.

~~~
samueldavid
i wish i can upvote this more, bless this comment to the top

------
ericmcer
This really makes no sense. Even if every school had identical curriculum and
teachers, students who manage to get into an MIT or Stanford have to have work
ethic and prior experience that places them well above the average CS student.
The fact that these students have already been vetted by a respected school is
one reason why I think companies prioritize them.

I am all for big data analysis disproving common conceptions, but this feels
off.

~~~
rxhernandez
> get into an MIT or Stanford have to have work ethic and prior experience
> that places them well above the average CS student.

You're comparing apples to oranges. The worst at MIT is not necessarily any
better than the average at some state school. I've seen more than a few people
with bachelor's degrees from UCSD and UCLA be below the class average at Cal
Poly for their second bachelors or masters degree. If there's any metric you
can rely on it's that their average is usually better than the average of
another school.

------
jayess
Just goes to confirm my own experience with grads that the difference between
schools is largely cohort, not quality of students.

------
auganov
The graphs are super confusing. Without reading the story it's hard to figure
out if the point is their sameness or if you should keep trying to figure out
the deltas.

Given that sameness was the point - just make a single graph with 4 colored
lines.

------
ordu
There is one problem with conclusions, they assume that interview score is the
same as real world success of a candidate. It is not so, interview result
correlates with future success but there is no strict determinism. So
companies prefer students that successful in interview AND come from good
school, rather than just rely on interview results. It allows them to maximize
probablilty of the finding a good employee.

------
kriro
I suppose a bit of methods nitpicking is in order. Wouldn't it make a lot more
sense to use equivalence testing if one wants to write a statement like
"Indeed, statistical significance testing revealed no difference between
students of any tier when it came to interview performance." AFAIK statistical
significance testing cannot reveal this at all.

Of course there is no real explanation of the method that was used besides the
fact that it was some sort of "statistical significance testing". Equivalence
testing makes more sense to me if one wants to essentially say that MIT is
Aspirin but Ohio State is a generic drug that is similar enough to work just
as well (for a reasonable definition of similar enough).

------
ggggtez
On the other hand, they discount self selection bias, and the bias of
"qualifying".

Plus the charts don't even show up, probably due to load.

edit: And the bias of the interviewers who use their platform, who may not be
able to attract top tier talent, and are happy to get even barely competent
people... It's impossible to say since the charts don't show up and we know
nothing about the methodology and motivations of the different actors.

~~~
leeny
Graphs fixed. Just added static image fallback for when Plotly runs into
issues. Sorry about that, and thanks for calling it out!

Re qualification bias, we call it out explicitly in the article. Without the
coding assessment before the real interview, the result may very well have
been different.

Lastly, re the bar, our interviewers are coming from Google, Facebook,
Dropbox, AWS, and so on.

------
supersas
There is huge selection bias. The talent that already perform well in
technical interviews most likely spend less time practicing (and using
interviewing.io)

~~~
leeny
Would it be helpful to update the post with school tier distribution?

~~~
darpa_escapee
I'd be interested in seeing this.

------
InclinedPlane
The charts were moderately cool when one of them loaded the first time I
opened the page, but they're not really superior to static images, especially
when they don't load at all. And when they are elements that your whole
article hinges on, well...

~~~
leeny
Fixed! Thanks for calling it out.

------
erdo
Oh come off it, for an ounce of credibility can they maybe keep the scale of
their graphs the same? And the link to the "statistical significance testing"
is nothing of the sort. I'm calling bullshit.

------
blackflame7000
I failed my first 2 tech interviews and Google and Snapchat leaving me feeling
cheated by the dumbass whiteboard. Whiteboard tests are tailor-made for the
inside the book thinkers. The ones that stop learning when the chapter ends
and soon forget when the next subject begins. Computers are the thing that
gets you to the thing. You want people who can see that the software they are
working has a broader impact beyond 1s and 0s.

PS. 5 Years later, I'm now the CT0 of my company managing 50 people. So srs,
not srs Googs you missed out =P

~~~
falcor84
Not that I think that whiteboard interviews are the best approach, but I'd say
you're being quite too harsh. In particular, the type of student that you
mentioned who soon forget every topic they ever learned would surely not do so
well on a whiteboard interview. Or if they were able to "relearn" every aspect
of algorithmics in the week before the interview then they can't be that bad.

In any case, a good(!) whiteboard interview absolutely can focus on all of
that broader impact you mentioned and get the interviewee to discuss the
UX/efficiency/maintainability/etc of what they're creating.

~~~
blackflame7000
While I can see your point, I am of the mindset that Computer Science is too
vast to expect someone to memorize it all. Furthermore, it's not even a memory
test its a form of hazing, to be honest. It has little indication of how well
someone will do in a role as dynamic as software engineering. Im certain I
could make you look like a fool in front of a whiteboard and im sure you could
do the same to me. But have we actually proven anything about who is smarter
or more capable? No. I look for people who know where to find the answer even
if they don't know it off the top of their heads. At least that way I know I
won't spend half my day answering their questions if I hirer them. The kids
who spend their time in college depending on the book and the professor to
hold their hands aren't the ones who succeed in the long term. It takes
ingenuity and creativity for that. That's who I want on my team.

------
solomatov
As far as I understand, the interviews they conduct are preliminary
interviews, and questions on such interviews are much easier than questions on
on-site interviews. So, basically they test for simple CS/programming tasks
which any person who studies CS should be able to perform.

------
lifeisstillgood
A serious question- is there a syllabus of interview questions ? That is
something that says the most important subject areas are X, and this puzzle
covers X

(I am not sure how one decides if finding shortest substrings is important or
not ... perhaps most common usage?)

------
johnwalker
I would love to see more recruiting at state schools and community colleges
etc from tech companies. There are a lot of hard working people who could be
great to work with but didn't have the family background they needed to be
noticed.

------
lifeisgood99
The graphs/images do not show for me. Chrome 64.0.3282.140. Win 10

------
blueplastic
Looks like the server is overloaded... images in middle of the article aren't
loading for me.

------
elgenie
Garbage in, garbage out.

What is used in divvying up a _self-selecting_ cohort of _undergrads_
performing on technical interviews in 2018 are the US News & World report
rankings for _graduate schools_ composed in 2014 "based on a survey of
academics at peer institutions" ([https://www.usnews.com/best-graduate-
schools/top-science-sch...](https://www.usnews.com/best-graduate-schools/top-
science-schools/computer-science-rankings) … and I don't think the lack of
that link in the article is an accident).

Having a selective grad school doesn't mean much if anything for the standards
and teaching in that school's undergrad program, especially for state schools
with giant undergrad populations and relatively small grad programs.

For example, Illinois' grad school is selective and highly thought of by
professors and thus it's treated as an "elite" school and students of UCSD's
undergrad CS program are classified as "top 15".

Regardless, my alternate hypothesis would be "students of a specific level of
confidence in their ability to pass a technical interview use interview.io for
a limited period of time in which it provides value to them … that population
of students receives a certain distribution of scores".

------
bitmadness
One data point: I was at Georgia Tech for undergrad, now at Caltech for a CS
PhD. Both are good schools, but Caltech is definitely considered more "elite".
As a Caltech student, I definitely attract attention that I just didn't get at
a Georgia Tech student.

------
jeff571
People who bother to practice for coding interviews (using the same website)
and successfully land an interview do similarly well on coding interviews...
That's a lot of selection bias.

