
The Machine Learning Software Engineering Interview - mgrover
https://eng.lyft.com/how-lyft-designs-the-machine-learning-software-engineering-interview-bbbb9fc8bb28
======
yanilkr
This blog post was so painful for me to read.

This is a symptom of "bullshit" going on around in big tech companies.
"bullshit" here is an economic term defined in the book "bullshit jobs".
[https://www.amazon.com/Bullshit-Jobs-Theory-David-
Graeber/dp...](https://www.amazon.com/Bullshit-Jobs-Theory-David-
Graeber/dp/150114331X)

Reading through the post, I was noticing

So much corporate Jargon which really does not mean anything important.

Dehumanizing language when describing people interviewing and being
interviewed and its process.

Too much obfuscation of ideas that can be very simply explained.

glorification of simpler problems into heroic challenges.

Delusions of Grandeur.

Today's such jobs are tomorrows layoffs.

I think I will stop here. I have crossed my negativity threshold for the day.

~~~
sidlls
"Obfuscation" and "delusions of grandeur" are practically synonyms for ML and
Data "Science" in this industry. I've been around for a while and I've never
quite seen something as over-hyped and hyper-glamorized as these two
specializations.

~~~
codesushi42
Really?

Were you around during the dotcom era?

Although I'm not old enough, I've heard that OR in the 80s was the same crap.

~~~
SenHeng
What's OR?

~~~
goatinaboat
_What 's OR?_

[https://en.wikipedia.org/wiki/Operations_research](https://en.wikipedia.org/wiki/Operations_research)

Basically a mathematical approach to problems of logistics and scheduling
developed first in WW2. Very powerful in the domains for which it was
developed but less generally applicable than enthusiasts hoped, leading to the
usual “hype cycle”.

If you have a problem OR could solve or just want to fool around with it PuLP
is very easy to use
[https://pythonhosted.org/PuLP/](https://pythonhosted.org/PuLP/) Of course the
ease of use means that it is a commodity skill now.

~~~
codesushi42
There is also Google OR tools.

[https://developers.google.com/optimization](https://developers.google.com/optimization)

------
starpilot
Good source of data for training a de-bullshitter.

Input: A year and a half ago when we began scouting for this type of machine
learning-savvy engineer —something we now call the machine learning Software
Engineer (ML SWE) — it wasn’t something we knew much about. We looked at other
companies’ equivalent roles but they weren’t exactly contextualized to Lyft’s
business setting. This need motivated an entirely new role that we set up and
started hiring for.

Target: We invented a position called a machine learning software engineer.

Input: First, candidates on the ML SWE loop go through Lyft’s hiring review.
The review is a regularly scheduled session for a committee to study
candidates with an unbiased perspective and decide whether to hire them.
Working alongside the review committee is a separate panel of interviewers
that provides technical feedback. This feedback is designed to help the
committee decide if there’s a fit and, if so, the candidate’s technical level.
At first glance, this review process may seem cumbersome. Examining the checks
and balances more carefully, however, we notice that they are intentionally
introduced to put friction on the hiring process. Having a consistent review
committee unifies standards and eliminates bias.

Target: A committee and a group of interviewers evaluate a candidate for fit
and technical level.

Input: Despite the what-ifs, being transparent about how we design interviews
can improve our interviews. Call it enlightened self-interest: candidates
invest time to talk to us and we mutually benefit from learning if there is a
good fit. Even if there isn’t an immediate fit, positive experiences build
brand and improves candidate sourcing. Maybe the candidate can reapply when
the timing is better. Practically, hiring an engineer easily costs tens of
thousands of dollars. By showing how we iterate on our interviews, we reveal
what we truly care about and how we try to probe at them, hopefully adding to
the virtuous cycle for the hiring pipeline.

Target: We don't respect the time of our readers and are hopelessly
unenlightened on this fact.

~~~
hcknwscommenter
Left unsaid is the "everyone gets a veto" crap that destroys hiring 10x
contributors. I have personally been involved in numerous interviews (as the
interviewer ), where my fellow interviewers have deliberately shitcanned
candidates because they seemed extremely well qualified and highly motivated
(as compared to my colleague). Everyone gets a veto is almost uniform in my
field and it is a massive problem.

------
aloknnikhil
This just reeks of mostly inexperience with a touch of arrogance.

> We looked at other companies’ equivalent roles but they weren’t exactly
> contextualized to Lyft’s business setting.

Ok, so what does Lyft need then?

> What are Lyft’s challenges (and can a specific role help)? > What should the
> role be with respect to the organization’s goals? > What are the desired
> skills, knowledge, and talents given the expectations for the role?

Umm. Isn't that what every company hiring looks to address?

> What’s left is to define the necessary ingredients for what a successful
> hire looks like vis-a-vis (3) in Lyft’s context: > > Skills acquired through
> practice, > Knowledge learned through study and personal experience, and >
> Talents that make each candidate unique.

Ok, I'll stop reading now since "Lyft's context" is no different from any
other company let alone one looking to hire a Machine Learning
Engineer/Scientist.

------
FpUser
Summary: we do not really want you to work for us, we are too busy trying to
understand WTF do we really want. Amount of BS the article is staggering IMO

~~~
thundergolfer
I thought it was because I was reading it right as I woke up, but I've re-read
it now and it's almost dizzying how it's cloaking the details but written as
if it's open and clear.

~~~
FpUser
When I was trying to decipher it I thought it was written by somebody in a
state of delirium .

------
thundergolfer
> In the context of the modeling onsite, we ask open-ended problems with
> sufficient business and problem context such that the candidate can clearly
> identify an ML-based approach to solve it.

I'm disappointed the author wasn't more specific about where the line is drawn
between "ML SWE" and "Research Scientist"/"Data Scientist" when it comes to
the core ML competencies like model selection, evaluation, and design.

Having worked in teams with Data Scientists and "ML Engineers" it's been murky
whether me and others in my team that were not the former were Data Engineers,
Backend Software Engineers, "ML SWE", or "Software Engineer - Machine
Learning".

This area of software is rapidly developing and there's no semblance of the
bright line that tends to exist between Backend Engineer and Frontend Engineer
for ML-involved engineers.

I'm personally interested in moving into the "Software Engineer - ML" space
(or is it "ML SWE"?) and thus I have to find out what the right balance
between Software Eng skills and researcher skills is. I think I have a decent
sense of real-world ML basics but would quickly flounder when pressed for
details on mathematical technicalities of different modelling approaches.

Lyft's job listings have these items for "Software Engineer - Machine
Learning":

> 5+ years (or Ph.D. with 2+ years) of industry or research experience
> developing ML models

This really seems like the purview of a research scientist or a data
scientist, if I understand the meaning of "developing" correctly.

> Proven ability to quickly and effectively turn research ML papers into
> working code

This seems fair enough, but my experience has been that the data scientists
are doing this mostly, while the "Software Engineer - ML" people are preparing
data pipelines or batch training systems.

> Deep knowledge of ML libraries like scikit-learn, Tensorflow, PyTorch,
> Keras, MXNet, etc

I know it's a job ad and thus it is a 'wish list', but this seems
unreasonable.

Guess I'll just keeping studying 'all the things'. Brb in 5 years.

~~~
mlthoughts2018
I think you underestimate the level of software architecture and engineering
skill that people with formal training in graduate level statistics bring to
these jobs.

I manage a team of machine learning engineers in a mid-size ecommerce company
and I can tell you that the same person who is optimizing Dockerfiles for
better layer reuse & figuring out how our CI pipeline will safely get secrets
needed to retrieve model files for integration tests is the same person
researching new variations of triplet loss in a paper from arxiv they present
in our team’s journal club & developing Bayesian hierarchical regression
models and explaining why partial pooling using an industry-specialized prior
distribution actually produces coefficients in the model fit that have a
meaningful improvement over OLS for some business outcome.

The same people that are refactoring a collection of unit tests to be
parallelizable and save us 45 seconds on every test run are also running huge
hyperparameter tuning experiments to get a learning rate scheduler that allows
us to reduce training time for an in-house deep GRU neural network from 24
hours to 15 hours, and they are defining infrastructure as code tooling to
even create the very GPU environments where this training is taking place.

The skill set really is extremely different from general backend engineer with
a proclivity for hacking around ML models and also is really different from
“research developer” who rarely deals with end to end systems or necessary
concerns of production or quality code factoring, and also is super different
from data science which is effectively just ad hoc business analytics but with
more impressive pedigree on the resume.

I’d define a machine learning engineer as taking someone with several years of
graduate experience in statistics / machine learning inclusive of formal
probability theory, analysis, topology, Bayesian stats (or work + research
experience that is equivalent, though it’s super rare for this to be able to
replace formal graduate math training), and then adding senior level skill set
in high-performance computing, generalist backend engineering, system
architecture, and full management of the lifecycle of complex production
systems.

The only piece of common modern engineering that I would say ML engineers
typically don’t have a senior-ish level of command over is frontend
development (though some do out of just hobbyist interest).

~~~
laichzeit0
Your definition of ML engineer comes off as being a high-risk individual to
have in an organization. I would rather split those into two separate
orthogonal roles and have redundancy in my resource pool. It seems like a very
difficult and scare resource to hire. How can Average Corp even think of
hiring someone like that? Dead no from me.

~~~
mlthoughts2018
The problem is you can’t really split them into two. You need the same person
designing models to also be making tactical decisions about software design
and production lifecycle because those things are almost always highly
dependent on nuanced specifics of the model itself.

There is not a division of labor where one person makes the model and then
throws it over the fence to a team that manages its production usage and
lifecycle. That team over the fence would not be able to do it, and I’ve seen
this attempted org structure fail hard everywhere its been tried for ML
services.

What’s telling is that you bring up the personnel risk but you don’t consider
the value add. When Average Corp hires someone, it’s because that person
brings more value than they cost, period. It’s not because the person is “not
risky” in some vacuum of decision making like you’re painting it out to be.

> Dead no from me.

That’s fine and all, but it usually indicates tech death of a company, and
likely you have brain drain in more area than just machine learning. If you
aren’t willing to take the risk and do what’s needed to structure the work and
job to extract value from high performers, that’s a sign of corporate
mediocrity and I think anyone with the skill set to be an ML engineer like
this would already not even be applying to work in a place like that.

Honestly the risks are even worse than your comment says. There are also big
risks around keeping this person intellectually engaged, giving them valuable
job experience.

You pretty much have to pay them a lot, give them good work life balance, give
them budget for conference travel & continued learning, give them meaningful
upward career & compensation growth, and give them meaningful projects.

I look at this and think, yes, if the business can’t give all those things and
still be coming out ahead on the person’s productivity, then you don’t want an
ML engineer.

But more often the company _needs_ an ML engineer and absolutely would gain
more from their productivity than they lose on supporting all those job
quality aspects — yet managers just take superficial offense at these demands
and balk at the idea that you have to provide meaningful projects and career
growth instead of just barking orders and expecting them to put up with work
that does not help them grow.

------
sportanova
> Most companies are open about the expectations for the role being
> interviewed for, the interview process, and preparation tips

When did this happen? I haven't interviewed for a few years, but nothing was
open or clear then

~~~
shantly
“Most” is certainly wrong. I’d allow “some”.

------
starpilot
I think this airy, self-aggrandizing post full of BS will be very offputting
to data scientists and MLE's. Very low information density here.

~~~
PeCaN
that sounds perfect for data scientists and machine learning 'engineers'

~~~
applecrazy
I'm confused at the animosity towards ML SWEs and data scientists within this
thread. Yes, there's a lot of hype/BS surrounding the field, but there's also
some very highly technical people making very novel discoveries.

My question is: why the hate? Isn't a machine learning engineer just as valid
as any other engineer?

------
brendanw
It would be great if companies would allow you to bypass the code challenge if
you grant them read git access to a relevant project that you have ownership
of.

~~~
codingslave
Problem is its easy to cheat about whats yours

~~~
davnicwil
I think the trick here is have people talk through what it does, how and why
(as in what are the tradeoffs, what other approaches could have been taken,
etc).

You can't fake this. Or to put it better, even if the code isn't yours and you
can do this well, it doesn't even matter that the code isn't yours as in doing
this you by definition have the skills and knowledge to reimplement it anyway.

~~~
pmikesell
You can absolutely fake this. There’s a huge difference between coming up with
and implementing something vs understanding it enough to be convincing.

Have you actually tested this out?

~~~
davnicwil
Yes I've done this both as an interviewer and interviewee, it's a pretty
valuable exercise I've found. Not a silver bullet, not the only thing you can
do, not appropriate for every situation, but generally good.

Regarding your other point I agree with you, but if the main point is to
evaluate understanding then the difference doesn't really matter. If you're
testing for ability to invent concepts etc, then this might be more important
to you.

------
fishingisfun
what an absolute mess. this is the private corps equivalence inefficiency of
the same madness i see with union jobs in the public sector.

------
high_derivative
To save some people the read, it's the same leetcode bs performance as
anywhere else.

------
ptd
Enjoyed the post but I found this sentence interesting.

>As a candidate, it’s easy to get a foot in the door and be evaluated by an
interviewer.

Do people really feel that way about a ML SWE postion at Lyft? I would be
interested to see what percent of applications they receive make it to the
door.

~~~
dahart
The article does hint at the percent: “YMMV: healthy numbers for an interview
loop for the phone screen to onsite, to offer, and to offer acceptance are,
for example, 50%, 25%, and 70%.”

------
syspec
Wow this post is really up it’s own butt, talk about self important

