
Google admits brainteasers were useless for hiring (2013) - gk1
https://qz.com/96206/google-admits-those-infamous-brainteasers-were-completely-useless-for-hiring/
======
jeffreyrogers
Here are two thoughts on tech interviews. One controversial and one
(hopefully) not.

1\. It's good that companies like Google are able to reassess their hiring
policies and learn what works and what doesn't. It's easy to criticize
companies for doing things that don't make sense or that we know to be bad,
but often these things looked like good ideas at the outset and you need some
amount of trying new things (many of which won't work out and may have
negative consequences) in order to improve things.

2\. The way hiring works in tech isn't really that bad. It doesn't really
measure what matters for job performance but tech pays well and so many people
want to do it that you need barriers to entry. The real losers of the hiring
process are the companies that waste lots of money on inefficient hiring
practices. If you compare the effort required to get a high paying tech job
(really just doing practice problems for a few weeks/months) to any other high
paying, professional field we have it much better.

~~~
k1ns
I recently interviewed at a well-known software company, was turned down, and
have very mixed feelings about the entire situation. I was tasked with a total
of 7 coding problems, 5 of which took place in an onsite interview that lasted
around 5 hours. The problems were fun, the interviewers were great, and I'd
ultimately rate the experience as the best hiring process I've been a part of.
I personally think that these problems were _exactly_ the type that tech
companies need to be using as they were focused on creativity and your thought
process as opposed to some "genius detector" brain teaser. I passed each
problem with flying colors, except for the final one.

The point of this final exercise was for me to talk through my thought
process. It was a debugging problem, and I eventually succeeded, but not
without fumbling around for 15 minutes or so. The solution was a stereotypical
example of being "right in front of you", but the nature of this exercise
threw me off. When I look back, the analogy that makes the most sense to me is
the boxing glove wall from the TV show, Wipeout [1]. It's a simple problem of
getting from point A to point B with the twist of getting punched in the face,
or the stomach, or the leg, at seemingly arbitrary intervals. I can see this
as a good simulation for the reality of an engineering job (being faced with
unplanned interruptions and having to bounce back from them) but, at the same
time, I've never had to verbally talk someone through every keystroke and
mouse movement that I make while debugging. I was frustrated with the exercise
and with my inability to immediately diagnose the issue. The interviewer was
_amazing_ at getting me back into the right mindset after noticing I had
fallen off the wagon each time, which showed me just how good of a manager
they were. However, the more I realized how terrible I was doing, the worse I
performed. I left with the unmistakable feeling of dread having set-up camp in
my stomach. Despite arriving at the solution, I knew I had blown that
exercise.

In the end, I was turned down. The official reason was that another candidate
had accepted an offer for the position, but the hiring manager also told me
that my programming skills were perceived as sub-par due to the final
exercise. They said I'd be a great fit culturally, and that their team enjoyed
the conversations we had, but my Python skills just weren't good enough. Being
someone that really identifies with their abilities as a programmer, with
Python being my language of choice, this verdict has kept me awake at night.
I'll never know if the actual reason was due to the other candidate accepting
or due to me being my own worst enemy during the final exercise.

The tech hiring process is one of the only times I've gotten 100% of the
problems correct yet still failed. Half of me says that's bad, but another
part of me understands in situations that require context like mine above.
It's not the manager's job to babysit me when I psych myself out, or when I'm
having a bad day. Engineers need to be reliable and stable enough to work
efficiently on their own. Had I been alone, I would have found the solution
much quicker. But you're almost never alone in a real work environment. Some
of these exercises are measuring much more than just algorithmic competency.

[1]
[https://www.youtube.com/watch?v=0XGmcQPCSG4](https://www.youtube.com/watch?v=0XGmcQPCSG4)

~~~
sjg007
I wouldn't worry about it. It is their loss. I would imagine that if Google
did a randomized controlled trial among a pool of qualified resumes (say after
they pass a phone screen) or a small coding challenge.. that they would find
that job performance is not correlated with technical interview performance.
They already found that GPA and where you went to school doesn't matter. In
fact Bock thinks it's only behavioral traits that determine performance. So
here's a challenge to Google or Facebook: ask the behavioral questions, ask
some technical ones. Score only on the behavioral and hire. In one year see
what the "performance" levels are... I'm guessing they did the latter test
where they accepted based on technical performance and ignored the behavioral
pieces which is why Bock nows says that behavioral are the most important...

My hypothesis is that the bar is still too high and that if you have a basic
technical baseline you'd still perform well regardless of your white-boarding
abilities. Google could run this experiment and in fact they might already
have enough data to answer it with a causal analysis.

The key issue is how do you define "performance" both in the interview and in
your yearly review... nobody has a clue how to actually measure performance.

[http://www.businessinsider.com/how-google-hires-
people-2013-...](http://www.businessinsider.com/how-google-hires-
people-2013-6)

~~~
fjsolwmv
You are misreading the research, which showed that _after_ filtering for
interview performance, excess programming talent or GPA wasn't important.

------
guyzero
This is a five-year old article about something Google stopped doing in about
2010.

But on the off chance you're 8 years behind the times, it turns out that those
infamous brainteasers were completely useless for hiring at Google.

~~~
theDoug
Surfacing old news for Hacker News to build karma points is too-frequently a
successful play. All you have to do is slap a (2010) and all gets forgiven.

~~~
slededit
It's not like the information has gone bad in the meantime.

------
g9yuayon
People tend to forget the historical context of a policy. Google was small in
the early 2000s, but it was one of the hottest companies in the valley.
Therefore, it could afford being picky. It was solving really tough computer
science problems in massive scale, so it needed to find talented true geeks,
those who live and breath math, engineering and computer science. The so-
called brainteasers were usually thinly disguised and well-designed math or
algorithm questions. There were few interview-prep books back then, either.
Therefore, those who could solve the teasers were likely creative geeks.

Fast forward to it today, we have huge amount of prep materials and prep
schools, so it becomes hard to tell who's truly talent and who has great
memory. Google doubled its workforce in the past three years from 35,000
people or so to 70,000 people. You just can't expect that everyone is
passionate and talented about math or engineering. Besides, Google does not
need too many of this kind of people either.

However, if a small company is tackling a hard problem that does require tons
of math, computer science, or pure ingenuity. Hell ya, brainteasers can be of
great help. They don't even need brainteasers, they can just directly ask math
problems.

~~~
opo
>...However, if a small company is tackling a hard problem that does require
tons of math, computer science, or pure ingenuity. Hell ya, brainteasers can
be of great help. They don't even need brainteasers, they can just directly
ask math problems.

Yes, if the job requires tons of math, it makes sense to ask math questions,
it doesn't necessarily mean that people who are good at brain teasers will be
good at the job. According to the article, what google found in regards to
asking brain teasers:

>...“We found that brainteasers are a complete waste of time,” Laszlo Bock,
senior vice president of people operations at Google, told the New York Times.
“They don’t predict anything. They serve primarily to make the interviewer
feel smart.”

------
akshat_h
Aren't a lot of so called 'algorithmic problems' similar to brain teasers? If
you have seen it before, you'll know it. But otherwise, there is not much
chance of you cracking it without an 'Aha' moment, which may or may not come
during the one hour allocated for an interview.

Weird tricks in bit manipulation, linked lists, array, hash problems etc. are
all standard in interviewing and are still used, even at Google, at least 2
years ago, when I last interviewed there.

This works though as Google's, and the industry in general has a policy of
rejecting candidates, rather than accepting them, because hiring is very risk-
averse. Candidates switch jobs frequently at the beginning of their career and
so there is a rotating pool of good candidates the companies can pick from.

~~~
xtracto
Yes they are, and as you, I consider themselves exactly the same. An example
one that I used to ask during interviews were the "traverse a matrix in
inverse diagonals" or "traverse a matrix in circular way" or the typical
"first duplicated number" (that one is on CodeFights and CodeWars).

Those are problems that you either know the "trick" or you don't.

The main problem with the majority of them is not the problem itself. The main
problem is that originally they are meant to understand the process of problem
solving of a candidate. However, as they are passed among interviewers, they
become an "solved or not solved", because it is the path of least effort.

I once created a problem that I love to ask in my interviews, it deals with
binary search to get the total elements from a list that is only avilable
through a "broken" API. A lot of people I interviewed told me that they loved
the question, however, when some colleagues started adopting it, I realized
that they were basically expecting the specific answer they knew... when the
real value of the exercise is to work with interviewees to "solve a problem
together".

~~~
akshat_h
I think this is also because no one really teaches you how to interview.
Giving multiple interviews doesn't, at least IMHO make you knowledgeable to be
on the other side.

It is kinda at a certain level in your career you are expected to just know
how to do it. For other skills like management, you at least have your peers
who are doing the work day in day out, who you can learn from. Interviews are
closed room, usually, though I have had some where one person was shadowing
the interviewer.

Also, if you take a terrible interview, its not gonna really impact you. There
is no performance report that'll negatively impact you. There will be another
candidate, who you'll have a rapport with. Or someone else will interview
someone else.

As you said in your anecdote about the binary search problem, not everyone is
best at being an interviewer.

But aside from there being a dedicated person for interviewing for big
companies, or outsourcing interviews for startups, I don't really see a
solution. And outsourcing, whether internally or externally comes up with its
own set of problems.

~~~
xtracto
> I think this is also because no one really teaches you how to interview.
> Giving multiple interviews doesn't, at least IMHO make you knowledgeable to
> be on the other side.

This is something that I have ameliorated in my teams in the following way:
When an Engineer is going to start interviewing people he goes through the
following process:

0\. Before starting, we provide some guidance on what we look in interviews
and what we ask (1 hour meeting with the new wannabe interviewer and the
manager)

1\. First he shadows an interview done by a Tech Lead or Sr. Engineer (those
are the ones that generally do the interviews)

2\. Then he does two interviews, where he asks the questions but is shadowed
by a Tech Lead of Sr. Engineer. He gets feedback about his interviews and is
asked to explain his feedback and we tune it to the expectations from the
team.

3\. Finally he starts interviewing people on his own.

This has worked well enough for me, specially once we had a shared GDocs with
a) The list of questions to ask (along with notes on what to look for and how
to guide guys) and b) A "competency matrix" ( like
[http://sijinjoseph.com/programmer-competency-
matrix/](http://sijinjoseph.com/programmer-competency-matrix/) ) tailored to
what we were looking for.

Finally, one of the things I always emphazise the people that is interviewing
for my team, is to remember how they felt during interviews. And be aware that
as interviewer ALWAYS you have the upper hand. I hate being interviewed, I
hate the feeling there, and the fact that you have 60-90 mintues to
demonstrate that you know what the company is choosing to ask you, without
regards of all other stuff you know that will be useful for the position, but
they don't ask. And the nerves.

------
loftyal
With interviews, you are essentially trying to predict the job performance of
a candidate when they are at your company, so why not make it as close to that
as possible?

With face-to-face tell them to bring their laptop or you can provide one, then
explain a real actual hard problem you've solved or one of your colleagues
has, that took roughly 2 hours, leave the room and let them try and solve it
on their own accord without any pressure of people staring at them whilst
coding. Come back and ask them to explain their solution and quiz them on
different parts.

This has been by far the best test I've given that predicted real on job
performance.

~~~
ggggtez
To be a true scientist, you shouldn't necessarily do what seems obvious. For
all I know, maybe whiteboarding is a _better_ indicator of performance than 1
hour of coding on a laptop. How well does it compare? I bet that's a closely
guarded secret.

~~~
AnimalMuppet
I bet that it's _unmeasured_ , and maybe unmeasurable.

------
Blackstone4
> Google now relies ... “behavioral interviewing,” such as asking people to
> describe a time they solved a difficult problem.

Is it just me? I feel like I've always had good reviews but I struggle to
answer questions like these in interviews.

~~~
bluGill
If you are not prepared you will struggle.

If you ever interview with me, be aware that I'm only allowed to asked from a
pre-selected list of this type of question. PLEASE PLEASE PLEASE spend time
thinking about your situations the day before! I expect to come with about 5
scenarios in mind that you can then twist to be the answer to the questions I
ask. Come up with some situations in advance, you don't want to have to think
of them on the spot. (actually you will have to think them up on the spot,
once you get past the first question you will relax and be able to think on
your feet better - for some questions a new situation will then occur to you
on the spot)

I know you had to work with a difficult person in the past, remember something
that happened. Don't make that person look too bad - you don't want to come as
bitter so select a different situation if that might happen.

I know you had to make a decision without enough information. I know you had
to do something controversial.

When I'm interviewing I'm only allowed to write down and consider The
Situation/Task, your Action, and the Result. (This is called the STAR system,
there are other variations) If you can make sure I clearly understand each
that is to your benefit, though I will ask clarification questions to get
that.

~~~
Blackstone4
Completely agree with you. Preparation is key.

I'm not sure why I struggle to answer these questions. Maybe I could be better
at telling stories.

On the other hand, I feel like it could be more to do with my ability to talk
about myself. For example, when asked of my major achievements and challenges,
I think of scenarios where we over came problems as a team. I think more in
"we" than "I". I feel like many of these questions are looking for answers in
the form of "I did all these amazing things".

Further to that, maybe I just don't rate myself as much as I should do. In the
past, interviewers who spend the time to go deep have historically said
something along the lines of "wow, you've done all this. Why didn't you tell
me earlier". I had one person say "It's like getting blood out of a stone". So
maybe I need to be more self-promotional. Gotta back yourself right?

~~~
bluGill
We vs I is an interesting question. If the company really values teams, then
saying we and pointing out your place on the team is better than successfully
convincing them that you did it alone.

Beware of this though. Some companies will say they value teams, but the
reality is they do not. Others value it so highly they hide that, lest you
hide that you don't work well with teams but need the job enough.

~~~
Blackstone4
Agreed. I feel like the best thing to do is say we overcame these challenges
as a team and this is the role I played. And these were my personal
challenges.

------
ncmncm
It is really amazing, when you get into it, how many people with excellent-
looking CV and experience just can't code their way out of a wet paper bag.

They don't know what cache is or how to tell when you've run out of it; can't
use pointers, can't write a loop; don't understand object lifetime or that
objects have one; or think a null pointer is, or should be, the same as an
empty string. (This last is most amazing because it seems to be official
policy at Google!)

The elementary coding and debugging exercises that are so annoying are
necessary because most candidates, by far, can't begin to do them -- MS:CS,
real-time OS thesis project, and N years experience notwithstanding. What have
these people been doing?

I have seen brainteasers used well, but only as a jumping-off point for a
conversation that tends well away from the puzzle.

I used to use a design problem (no coding) that bright undergrads solved in
two minutes, BSes in five, MSes in 20, and PhDs either never, or as fast as an
undergrad. Everyone who solved it used identically the same hand gesture when
describing the solution.

ITA Software used to publish great programming puzzles that you were invited
to solve and send with a CV. (Google bought them out and eventually took down
the puzzles, back in 2010, but they might still be in wayback). To learn how
your solution compared to theirs, you had to send it! It might not work so
well anymore, since everything gets posted nowadays.

------
mixmastamyk
Riddles fell out of favor, but now we're faced with the next disastrous trend
which I just wasted an hour on yesterday---coderpad.io.

Imagine a binary pass/fail test that determines your future, next to a
suitcase full of cash, clock ticking, and another developer watching over your
shoulder, also feeling smarter because they already know the answer. An answer
they researched in solitude at a leisurely pace.

Conclusion: "We can't find any qualified people, there must be a shortage."

~~~
0xfeba
I much prefer coderpad or HackerRank, live with the interviewers, over
whiteboarding. I find talking through a first approach, improvements,
debugging, etc. is very useful -- both as an interviewer and candidate.

The timed HackerRank and such sucks though. Way too stressful. And easily
subverted. They enforce fullscreen but you can just use another computer to
look things up.

I wonder if, comparatively, other fields have as many "fakes" as I've seen try
to get jobs in mine. I'd say 1/20 candidates I've interviewed aren't
imbeciles.

~~~
mixmastamyk
Better than coding on a whiteboard is hardly a complement.

Maybe I didn't write clearly enough, but the reason you are attracting so many
"fakes" is that you are testing for "nerves of steel" rather than coding
ability and misattributing a lack of the former with the latter.

~~~
0xfeba
We don't ask anything hard. For JS candidates, given [1, 2, 3, 'a'] return an
array without the letter. Debug an AJAX call (the URI protocol has an obvious
typo). For react candidates, implement an event handler for a button that
increments and shows a value (all prewritten, just add this.setState({value:
++value});

We also guide them and give hints and encourage interaction, or if they need
to think we leave them alone.

If that requires too strong of nerves, uh, oh well I guess. I don't know how
else to judge their technical ability and soft skills.

~~~
sireat
For the first were you looking for [1, 2, 3, 'a'].filter(el => typeof(el) ===
'number') type of solution ?

~~~
0xfeba
Sure, there are many solutions.

------
logfromblammo
The businessinsider.com article linked by posted qz.com article has some
hilariously bad answers for the leaked Google interview questions. Most of
them distill down to "here's how we would make a clever attempt to avoid
answering that question, by answering an easier one" and some are followed by
"here's a smarter answer, submitted by a reader infuriated by our published
response".

------
jl2718
If google really wanted a fair and objective hiring process, it would be easy.
They don’t. The coding interview is one of the most subjective processes I’ve
seen in hiring. The interviewer reviews the resume, and then gives a vague
problem that has many answers, only one of which they are looking for. Then
they either let you flounder or straight up tell you what they want. At the
end, code is erased, and interviewer gives subjective feedback. If they wanted
a good process, it would be double-blind, consistent, and adjusted for
efficacy. The stats that they do share show the opposite: Google’s top
performers were massively over-represented among those that had the lowest
interview scores. Steve Yegge himself interviewed 6 times before getting an
offer. one question you might ask is whether Google even wants top performers.
I don’t think so, honestly, and they say as much with their focus on culture.
They want people that will go along to get along and not cause conflict by
disagreement or doing things their own way. This type of cohesion is much more
important to google than individual performance. The idea is that the
interviewer is the expert and you are the Padawan trying to learn by
exploration. Be humble and take hints. If you disagree with an interviewer,
you’re done. If you do something they don’t understand, don’t try to explain,
just undo it. The interviewers are randomly chosen among volunteers, and the
ones that volunteer the most were shown to have the worst performance in
assessment. So the lesson here is that it’s a social game. You don’t need to
learn every algorithm, just know the basics of any programming language, and
I’d say even prefer things like for loops to functional maps because most
likely, they won’t follow. The social game, however, is an unlimited
liability. You need to make the interviewer feel like the expert in their
problem, and feel like they taught you something profound. This is not your
performance; you are just the magician’s assistant, acting out their will.

~~~
dukoid
"Google’s top performers were massively over-represented among those that had
the lowest interview scores" \-- source?

------
gwbas1c
The biggest problem with hiring, IMO, is finding an effective filter.

My employer now works with contracting firms where about 80% of the candidates
they send us are excellent and we hire. The interview is almost a formality.
When we reject a candidate, it's usually because the candidate is a bad fit
for the job. (Edit) The rejects are always very talented and would succeed in
different assignments.

When we hire for employees through a recruiter, the process is much harder.
Most of the candidates recruiters bring us are, to put it mildly, horrible and
incompetent. (Edit) The recruiter doesn't really filter for competency.

The medical profession has an industry-wide certification process. I wish we
had the same, because it would be easier to filter out the morons.

~~~
rogerbinns
What is the difference between a "recruiter" and a "contracting firm"? It
sounds like both are sending you candidates you hope to hire.

~~~
gwbas1c
Look up what out-sourcing is.

Most of the time, when a company needs to develop software, it doesn't make
sense to hire all developers as on-site, full-time, permanent positions.

Some people are only hired for the project and go away when the project is
done. Those people are hired through a contracting firm; and the contracting
firm finds them employment when the project is complete.

For us, it's easier to "hire" through the contracting firm because they
usually worked with each other in past projects. A recruiter has very little
direct experience with a candidate.

Ironically, things would be easier if recruiters always sent us "formality"
interviews and the contractors needed tighter screening. This is because it's
easier to hire a contractor for 3 months and choose to renew the contract.

------
ma2rten
[2013]

~~~
ggggtez
I was gonna say, I haven't heard anyone talk about brain teaser interviews in
years. Now everyone just complains that whiteboarding is not a good
measurement instead.

------
Bahamut
Maybe in a decade they'll admit their current interview process is also busted
:) .

~~~
nomel
Any insight into the current process?

------
chris_va
Google interviewers were never supposed to ask brainteaser like questions,
even before the company "admitted" they don't work.

The title is disingenuous, and implies that Google switched tactics after
finding out that these questions were not useful, whereas in reality Google
never started doing it because they found that these questions were not
useful.

(Source: I did interviews at Google pre-2013)

~~~
marcodave
so where are these "interview brain teaser questions from Google" coming from?

~~~
chris_va
I don't think these questions are uncommon in the industry. I've seen brain
teaser questions from Microsoft, for example. I think when people write about
them they just attach the examples to something emblematic for marketing, and
then it becomes common lore.

------
jaypaulynice
Google and lots of companies continued to use brain teasers even when they
realized it made no sense. Sure they wish to say this after using it for a
long time. What I’ve realized sometimes there are barriers to prevent certain
people and give a sense of meritocracy while justifying discrimination. It’s
like when looking for an apartment, some landlords will only check some
people’s backgrounds and not others. The same is true for credit card
companies offering some people much lower apr than others for no reason other
than prejudice. We could go on and on with examples, but that’s just a few.

In the end every company becomes average because there is only so many people
they can hire. This has nothing to do with brain teasers per se.

I’ve worked in tech for over 10 years and I have to say the places I did the
best work are interviews I somewhat bombed. Not a surprise!

------
perl4ever
I'm unclear whether "brainteasers" are synonymous with "Fermi problems", or
whether they are a disjoint set, or a superset. From some of the descriptions
in this thread, it seems like _some_ "brainteasers" are clearly not Fermi
problems.

I would be interested in knowing if anyone _is_ hiring based on ability to do
Fermi problems. I see these as not being about knowing a "trick" but about
knowing how to utilize your own knowledge, which is a much more general
capability.

------
dekhn
Laszlo Block was never a useful source of information on how Google worked. He
just used his position to gain attention and publish books.

~~~
CobrastanJorji
Surely the SVP who was in charge of hiring at Google is a useful source for
information on how Google hired people?

~~~
dekhn
You'd think, but no, not really. He mostly just wanted to sell a book.

------
bitL
They were IMO the only funny/interesting part of their whole interview
process. Now it's like a sterile academic discussion.

------
mfrw
At last, they do agree that brainteasers are not a measure of how well you can
code. In my _opinion_ code & communication skills should be enough, immaterial
of whether you can figure out how to measure 4 gallons using a 5 & 3 gallon
jug.

------
duckkg5
One they asked me in 2010 was "How many people are in the air right now?"

I've conducted loads of interviews at Google since 2012 and none have included
a question like this from myself or any other hiring committee members. IMO
the questions about what they do for fun are way better at determining if
they'll fit and do well.

------
contextfree
IIRC they took this practice from Microsoft, which had already abandoned it by
then

~~~
mikestew
Yup, I left MSFT in 2005, and sometime before that they packed all the hiring
managers in a room and told us: "brainteasers: quit doing them, they are not
an indicator of anything useful." The practice didn't stop immediately, but it
certainly fell out of favor shortly after amongst my crowd.

~~~
itronitron
I got asked the water jug brainteaser in an MSFT interview, as luck would have
it I had recently watched Die Hard With a Vengeance so I knew the answer :)

------
duckkg5
One they asked me in 2010 was "How many people are in the air right now?"

I've conducted loads of interviews at Google since 2012 and none have included
a question like this from myself or any other hiring committee members.

------
lgleason
and their hiring standards have also slipped......

------
s-shellfish
Google changes the way people interact with the internet -> Google changes the
way people think about how other people think -> Google uses it's own metrics
of intelligence to determine intelligence WHILE Google is actively changing
what intelligence is (or at least what people think it is)

Hm...Sounds like a paradox right?

