
How to Interview Engineers When You're Not Technical – Part II - greghausheer
https://www.greghausheer.com/articles/how-to-interview-engineers-when-youre-not-technical-part-ii
======
mcfunk
Some good questions in here, with one exception: > Programmers are unique.
They're one of the few professions where outside the boundaries of their
working hours they choose to do the same exact thing they do at work:
programming.

Hiring managers should definitely not be assuming that all programmers do apps
in their spare time. It's one thing to expect them to have a code sample
(anyone should), but many strong programmers have interests and
responsibilities that take them away from computers in their spare time. This
doesn't make them less effective or dedicated programmers, so while it's
always interesting to hear about the projects of those who have them, it
shouldn't be assumed (and you shouldn't consider it 'points off' in an
interview if a dev doesn't do side projects)

~~~
PragmaticPulp
Please read the article further down. The author explicitly says:

> It's OK too if an engineer doesn't have any side projects.

I always ask about side projects to give candidates an opportunity to
highlight their side projects, not because we require candidates to have side
projects.

I'm sure some hiring managers out there have side projects listed under their
"must have" criteria, but in my experience the good hiring managers are just
collecting as many data points as possible. In fact, I've known many hiring
managers who take side projects as a potential risk factor, because they can
pose a distraction to the candidate if they become too demanding.

All other things equal, a candidate with extensive side projects is going to
have more experience than a candidate who has never worked on any side
projects. In the real world, you never find two candidates who are identical
in every way except for their side projects. The goal of an interview is to
collect as much information as possible about the candidates in the limited
interview time. Asking about side projects is just one more place to search
for those data points. It would be equally unfair to disqualify side project
experience from the consideration process.

In my experience, have side projects is largely a boost for junior candidates
who haven't had enough opportunity to build a long professional resume yet. I
can't remember the last time we interviewed a senior developer where side
projects were the tipping point in our decision making process.

~~~
kemiller2002
One thing that I've never disliked about hiring managers asking for side
projects, is that when I work on one, I do what interests me. I like trying
new things, experimenting on stuff that I don't know how it will turn out.
When I get bored or I hit a dead end, I stop. A lot of it is unfinished and
messy. It's for me, not anyone else. My time is valuable, especially outside
of work.

~~~
PopeDotNinja
Yeah. I'm building an http server for fun. Last year I wrote a JSON parser one
weekend. A few years ago I wrote a YAML parser. I make things like this fun &
learning, not because I'm trying to impress anyone. When someone asks why I'm
reinventing the wheel, or is generally unimpressed that I didn't do something
they view as more useful, it can be awkward.

------
SilasX
There's also a general technique for filtering experts when you don't know
their domain well: ask them do something known to be impossible. [1]

Very bad answer: "Sure, no problem." [2]

Bad but tolerable answer: "Not possible but it's too hard to explain why."

Good answer: best attempt to explain why it's not possible, and the nearest
solvable version or what you would need to change to make it solvable.

There's a great Better Call Saul episode (s4e5) where an expert is idenified
as a unsuitable (in part) because he insists a project could have a firm
maximum duration without caveats about what could go wrong, and he could do it
without something it obviously needs.

[1] That's arguably what the unnamed MP was doing when he asked Babbage if
he'd get the right answer if he put the wrong numbers in the machine:

[https://en.wikiquote.org/wiki/Charles_Babbage#Passages_from_...](https://en.wikiquote.org/wiki/Charles_Babbage#Passages_from_the_Life_of_a_Philosopher_\(1864\))

[2] At best they might silently replace it with the possible version, but
that's still hiding the tradeoffs from you and means they could be
misrepresenting other answers.

------
rhombocombus
As a professional programmer I spend almost all of my free time NOT
programming, but I do spend it taking care of my family and social obligations
instead. While I love what I do I would love it less if I did too much of it.

~~~
bdamm
Exactly. When I was a junior I had plenty of side projects that I rotated
through. Then that got old, I developed deep passions for other non-computer
things, have a family, enjoy knowing people, and what do you know? Not many
side projects that involve programming. I haven't updated my blog in 15 years.
I'd rather go fishing with my kid, rock climbing with my bud, or banging
together a garden with my wife.

~~~
Viliam1234
Seems to me it still makes sense to only hire people who program in their free
time... if your real goal is to hire people willing to do a lot of overtime
work. (I am not saying this is a good thing to do.) Things that prevent you
from programming in your free time are more or less the things that would
prevent you from working overtime.

Or it could simply be thinly disguised ageism. "Hey, I am not saying we
shouldn't hire people who have kids... I am just saying I don't trust people
who are not passionate enough to write code in their free time!"

~~~
mikekchar
As someone who specifically chose not to have kids because I want to spend
more time programming (crazy, I know)... I'm not that interested in working
overtime for the company. I'll do it in emergencies, but my time is my time. I
only have so much of it and there are things I want to do. No amount of money,
or recognition, or pride in helping someone else get rich will give me that
time back. Just because I want to spend my time programming, does _not_ mean I
want to give it to someone else.

------
lxe
How do you calibrate yourself on these open-ended questions? How can you
compare one good answer to another completely different good answer and figure
out which one is better overall?

You can't rely on a "The very best engineers have a checklist they apply
depending on the type of bug." type of framework here -- each "very best"
engineer might have a completely different way of doing things.

You have to have at least some questions that are difficult to answer but have
a countable range of responses, and watch for specific lapses that can show
how well a candidate demonstrates specific experience required in your job
description.

For example, if you require some experience in distributed systems or scaling
things to many users, have them talk about "Could you talk about an or program
you built to scale to n users, what performance or reliability issues did you
see (or foresee), and how did you solve them?" Have a loose set of industry
specific things to watch out for, like caches, load balancers,
monitoring/observability, etc.... You don't have to have the specific
knowledge yourself, but if the candidate can explain to you what they did, and
how/why, in the framework of the specific experience you're looking for,
that's good signal!

~~~
hinkley
> The very best engineers have a checklist they apply depending on the type of
> bug.

This is misleading at best.

Masters do less work than beginners. They have years of intuition built up
that let them apply heuristics extremely cheaply and save their energy for
solving the novel parts of the problem. There is no 'list'. There is only the
verbal part of the brain trying to introspect on what just happened. Sometimes
accurately, but quite often not. The ones who get it even close to right
become mentors, instructors.

I can spend as long as you like explaining my debugging process and the first
time we sit in a war room you'll call me out for not following my own 'list'.
Every situation is either unique or the result of a lack of self-reflection
(why do we keep solving the same problem over and over? Are we not
programmers?)

The biggest danger to people who can't explain at all is when advances in
their field arrive and they can't introspect enough to adjust their behavior.
There are limits to how much you need to address this in an interview process.
For many of these people, they'll already be in that space when you meet them
and easy enough to spot.

~~~
rubidium
Most problems aren’t special. They occur because some good general guidance
wasn’t formed or applied.

Having a checklist does help masters. It helps make sure they don’t miss the
obvious issue in front of their face.

Checklist manifesto is good reading on this.

~~~
hinkley
Hmmm. I think the checklist exists, for sure, but tends to be more Plan B. If
your initial strategy is exhausted with no progress, fall back on the
checklist.

As a programmer I have an opportunity not everyone else has. I can codify
parts of my checklist in a way where I don't really have to think about them
at all save for a few times a year with the big outlier problems, teaching
someone to do basic troubleshooting themselves, or starting a new project.

------
supernova87a
You know, one of the things I found most amazing about interviews and the
hiring process was that to a large degree, the things that a person _says_ in
an hour session determine whether they will get hired.

Yes, not their references or resume (at a certain stage) or what they're able
to prove that they coded, but the things that they are able to come up with to
_say_. Doesn't that strike you as sometimes amazing?

Now, that may be a crazy notion and flawed in many ways, but I came to realize
the following. The reason that that is often acceptable is that it frequently
is a good proxy for weeding out the unqualified. In this sense:

If someone doesn't even know to say that something is important, or that they
solved a problem a certain way (regardless of whether you have proof the
actually did it), they probably have never even thought about those things
being important.

If someone tells you (just offhandedly) that they solved something by jumping
right in to doing x,y,z at the detailed level and coming up with the
_technical_ solution (the algorithm), but not the project planning or scoping
part of it -- they probably have no idea or experience about what's necessary
to plan out a project. They may not be good at estimating or communicating.
They may be someone who gets lost in the details. If they have no pitfalls to
warn against, probably they haven't even seen how a project has gone wrong
before. They are probably not a good candidate for a senior developer role.

Whereas, if someone knows to say that they had to do some top level estimating
or quick pass of which solutions would be the best from a cost/resource
standpoint, consider the timelines to implement and risk of doing so, how to
decide who on the team would be put on the project and in what sequence,
etc... Then I know that they at least realize these things are important.

(Yes of course you should probe deeper. And I know that this applies to some
kinds of roles and not others, but since the article is about interviewing for
non-technical people and is often about hiring for a role requiring a higher-
level hiring decision)

------
FalconSensei
> There are countless ways engineers choose to spend their off hours. Some
> contribute to open source. Others take online classes. Some are tutors, and
> others work a second job as a freelancer.

It's called off hours for a reason. Being a tutor or a freelancer is a second
job.

------
ggambetta
> [From the preceding article] how were teams split, tasks delegated, and
> projects managed throughout Sprints?

> Did they use Agile, Kanban, or Scrum? How long was their Sprint?

How about asking whether they used sprints, or formal agile methodologies, at
all? Why assume that every engineering org has drunk the kool aid?

~~~
marcinzm
The context is companies with 30+ engineers. At that point you're going to
want some form of formal project methodology that allows everyone to co-exist
somewhat peacefully. Generally, the results otherwise are varying level of
dysfunctional.

~~~
ggambetta
I worked in a handful of teams at Google and none used Agile (I believe some
do, but they're a minority), and very little formality besides a weekly team
meeting. From what I hear, FB and other big companies are in a similar
situation. Sure, you can stretch the definition and say they're dysfunctional
("omg lol google kills projects") but you can't deny they're successful.

~~~
marcinzm
First of all, I used the word Agile in my response zero times. I used the
words formal project management for a reason. Agile tends to be assumed but if
someone asks then either you can infer what they actually want or you're going
to be a terrible hire in a startup anyway.

Second of all, as people love to say here, your small or medium sized company
isn't Google so you shouldn't make decisions as if it was. What works for
Google does not necessarily work for random startup or small company number
7521. Every small and medium company I've seen without a project management
process was a disaster. Partially because, unlike Google, they can't throw 10
million at a problem and shrug if they waste 9 of it or cancel the whole thing
in 2 months.

------
paiute
> What's the one thing you would not compromise on as you write code?

Code that works.

I'll take a gordian knot that works over an over-engineered system that fails
or never ships. Some of the worst code I've seen comes from engineers who
focus on "unit tests, documentation, and refactoring code." Those are all good
things, but they aren't #1.

The "done" question is a very good one.

~~~
jerzyt
I completely agree about the unit test point. I'm not arguing that it's
useless - quite the opposite. But it's damn hard to write useful unit tests.
I've seen a case where an internal IT team took ownership of code I delivered.
A few months later I was asked to make some enhancements, and I found that
they've added a lot of unit test code. It was absolutely useless. Essentially,
they were testing the compiler, not my code. Complete waste of time.

------
mirekrusin
Or get two candidates and ask them to ask each other questions. Ask them to
score themselves and "opponent". Infer score yourself. Promote winner to next
level to interview other winners, until final battle. Brutal, but what can you
do when you're not technical?

~~~
Loughla
Jesus Christ. Somewhere a hiring manager is explaining the benefits of this to
human resources.

Step 1 - regular interviews.

Step 2 - group interviews including memory check with coding task

Step 3 - this thunderdome you just created.

~~~
WrtCdEvrydy
We never did group interviews but did try panel interviews as we noticed the
order of interviews ended up skewing how we felt about a candidate.

------
j45
There are some good questions here.

Still, if you're non technical when interviewing, how would you know the right
answers to technical questions?

Eliciting underlying thoughts about problem solving is important, but is more
than just saying the right keywords.

If interviewing is not much more than a game of keyword DDR, it makes
successful outcomes even more challenging.

Maybe hiring is broken because it is designed for linear working paths, but
engineering is not linear.

~~~
pdonis
_> if you're non technical when interviewing, how would you know the right
answers to technical questions?_

My reaction to the article was similar. My answer to the title question would
be: "Find a technical cofounder you can trust."

~~~
j45
Absolutely. Don’t take advice from someone who doesn’t have relevant
experience.

I find a much better success rate in not hiring he traditional HR ways. Tech
folks are not linear workers with floors and ceilings, they are growing much
faster and on multiple layers at the same time.

It’s humorous when non technical ppl try to manage things they don’t
understand.

That table seems to be currently being turning in society because of covid, if
you ask me.

Technologists will have to move into each area of management, because maybe
it’s technologists turn to learn and manage business after such a long time of
business trying to manage tech.

~~~
lonelappde
The biggest tech companies you know are founded and run by techies.

~~~
j45
Yup. Lots of non tech companies (pretty much most) which are still doing tech
hiring

------
greghausheer
Thought I'd write a Part II to my first article on questions non technical
founders and managers can use if they need to interview engineers.

------
crimsonalucard
I consider myself good at programming in general. But let's say the
interviewer was hiring me for a technology stack that I'm pretty much clueless
about... let's say assembly language or cobol...

Even then I can bullshit my way past all of those questions, and I'm sure most
people who can't program can also do the same too.

~~~
ape4
"What's the one thing you would not compromise on as you write code?" Its not
hard to figure out the right answer to that one: "I never compromise. Must be
perfect." But of course that's not true. There are always pathological
conditions that can reasonably ignored especially in the first release.

~~~
jrumbut
That's funny, I came to the almost opposite conclusion that the obvious answer
was that everything is tradeoffs and compromises.

I suppose very easy things like consistent indentation width might be done
perfectly in a codebase, but any non-trivial thing always could be "more X" or
"less Y".

~~~
greghausheer
"I came to the almost opposite conclusion that the obvious answer was that
everything is tradeoffs and compromises."

Probably the smartest answer there is.

~~~
crimsonalucard
No this is not true. Definitely not smart either. Not everything in the
universe is a nice yin yang tradeoff, this is a philosophical and scientific
mistake. There exists in this world solutions to problems that are the most
optimal.

When humans think in terms of design tradeoffs they are thinking in a domain
where there exists no theoretical or mathematical technique to find the
optimum. This does not mean a technique can never exist, it just indicates a
lack of knowledge. The keyword is design. How do you design the best looking
website vs. how do you design the shortest distance between two points? I
would say for the later, we have enough knowledge and theory to say that the
shortest distance between two points does not need to designed... it can be
calculated.

Design implies alternative designs and tradeoffs due to the lack of techniques
to determine or find an optimum. Calculation implies greater knowledge.

~~~
kthejoker2
Most real world problems have constraints that render global optimums moot.
Usually opportunity cost, but also risk, experiential components, satisficing
...

In fact, any interview problem with a global optimum is worthless for
evaluating candidate's ability to handle ambiguity or demonstrate capacity for
creativity, second order thinking, communication ...

Expecting candidates to lead with absolutism like in your post os a good
recipe for finding Brilliant Jerks.

~~~
crimsonalucard
Dude, the entire field of Machine learning is an attempt to find a global
optimum. It is an attempt to turn what is otherwise a design problem into a
calculation.

To say that global optimums are moot or don't exist is categorically wrong.
Global optimums and trade offs exist in equal measure everywhere. I value good
judgment in both.

Catering to one extreme is unwise on every level.

>Expecting candidates to lead with absolutism like in your post os a good
recipe for finding Brilliant Jerks.

First off I never said or implied this. My post is just saying that not
_Everything_ is a design tradeoff. Second off are you implying I'm a Jerk? Did
you just call me a Jerk?

------
seancoleman
I've been hired to perform "technical-interviews-as-a-service" for a handful
of non-technical founders / leaders. Has anyone partook in this format, either
as leader, interviewee, or interviewer?

------
jerzyt
I really like problem solving and estimation questions during interviews, and
I favor them over coding skills, etc. A question that I frequently use is how
long would it take to evacuate a city like Phoenix, if there was a 24 hour
warning that some dictator du jour is going to drop a nuke. It boils down to
assessing a throughput of various modes of transportation: cars, buses,
trains, airplanes, and yes walking. One candidate responded "by helicopters"
at which point the interview was pretty much over.

~~~
plasticchris
Google used to do this kind of question but found it wasn't predictive of
future perfomance.

------
MisterBastahrd
I'm a good enough developer who has worked on enough projects that if someone
tries to tell me that I need to show them personal projects I'm working on,
then that ends the interview. I used to be a recruiter, I know how to conduct
an interview and sit on the other end of that desk, and I know what I won't
put up for in the workplace.

My personal time is my personal time. If there's a good work-related reason
for me to put in more hours, fine. But you get none of them that you don't pay
me for.

~~~
j45
Another angle of this question is often to elicit what kinds of problems you
have solved in your personal time, ideally technical, but not necessarily so.

I have had great conversations with someone who has built an automatic pet
feeder that lead to the work.

It shows a different facet of one's personality. This, in turn may come in to
play when creative problem solving is involved. Quite often even non-technical
problem solving around the home can transfer to work

It can delineate the difference between a developer who sees what they do as a
profession, vs those who started with it as a passion and continue to solve
problems.

~~~
invalidOrTaken
This is presumptuous of a level of trust that is not there.

There is no talent shortage, and if there is, it's of _management,_ not the
ability to write code. The difference between a good developer and a code
monkey is that the first is self-managing. They _have_ to be, to do their job.
But even they can't do it without clear communication about the project---and
cash, of course.

That the shortage is of _management,_ not developer talent (if you protest
that there _is_ a developer shortage, I have bad news for you about where the
shortage of management ability is), means that the interviewer is generally in
the power position.

Ergo:

How great can the conversation be on such unequal terms? I'm not saying I
haven't had good conversations with interviewers or interviewees, but it can't
be expected to be a regular thing. What I _really_ think is a smorgasbord of
hunches, folk wisdom, and Alan Kay quotes---but I would never be so rude as to
express such in an interview.

Buy the girl a drink first, sheesh.

~~~
marcinzm
Pre-pandemic I've found that in many markets there is a shortage of good
senior developers. Good defined by those who can interview well in whatever
the current in-vogue interview style is. Or more specifically those that could
get jobs at Google, Facebook, etc.

~~~
invalidOrTaken
I feel like the caveats prove my point.

~~~
marcinzm
It's more that you can't lump all developers into one category and make
blanked statement like you did. Some developers in some markets aren't in
demand. Some developers in some markets are absurdly in demand.

