
Business questions engineers should ask when interviewing at ML/AI companies - danicgross
https://medium.com/@danielgross/seven-questions-to-ask-when-interviewing-for-an-ml-job-1963ccee3a19
======
erikb
Anybody recognizes how similar these questions are to what an investor would
ask? This is what I'm always telling developers. You are an investor. You
invest your life, health, best hours of 5+ days a week, ambition. Your
investment is not as important to the company as a few million bucks would be,
but to you it should be more valuable than a few million bucks. It's all you
have. And you want to give it to someone in exchange for money, you better
make sure that someone is worth your investment. So read what investors are
asking a prospect company, and ask the same questions. Only accept if you
would give them a million bucks.

~~~
kemiller2002
Randy Pausch said it best (paraphrase), "You can always make more money, you
can never get more time." It's the most valuable thing you have, and one way
or another you are investing in a place where you work.

~~~
delibes
Thanks for reminding me. The Last Lecture is good. Since it's about 9 years
ago, some younger HN readers may have missed it:
[https://www.youtube.com/watch?v=ji5_MqicxSo](https://www.youtube.com/watch?v=ji5_MqicxSo)

Cancer sucks.

~~~
erikb
> Carnegie Mellon Professor Randy Pausch (Oct. 23, 1960 - July 25, 2008) gave
> his last lecture at the university Sept. 18, 2007, before a packed McConomy
> Auditorium.

So, actually 10 years ago already. Time flies.

------
Hyperbolic
After working for a few companies who (tried to) make ML products, I think one
of the most important questions to ask is "who owns the data you are building
models on?". It is way harder for a company to build good models off data they
don't have complete access to and full knowledge about. The worst (and
unfortunately common) scenario for companies trying to do AI is that data
scientists don't have full access to all priors necessary to build good
models, and the owners of the data don't really know much about it either.
Spells death of company

~~~
alvis
Well said. It is a good tester of knowing how clear the people understand the
problem themselves. Many people (particularly non-science background ones, no
offence) don't know which kind of data they should access to build a product
and, even worse, whether they have access to the data.

Not knowing the answer of this question = they do not know they are travelling
on Titanic at best, if not a boat full of holes.

------
okabat
The fastest way to make progress on these business questions is often to build
hacky MVPs that look like they're doing something smart, but behind the scenes
are powered by humans or dead-simple algorithms, and get them in front of
customers ASAP.

I recently joined a seed-stage startup solving a business problem via audio
analysis in the manner I described above. I'm not spending much time doing ML
yet, but I'm banking on my belief that we're solving a valuable problem
(customers want to buy our hacky MVP) and that ML can and will be needed to
scale our solution. By deeply understanding the customer as a first step, I
think the ML systems we build will be business critical and enduring. Time
will tell

~~~
cirgue
> dead-simple algorithms

The most successful models I've ever built have been logistic regression
models. If you can rephrase your problem in a way that's amenable to run-of-
the-mill statistical techniques, you can frequently achieve much better
results than you can with 'deep learning'.

~~~
madcaptenor
If you're going to do logistic regression, at least call it a single-layer
neural network with a sigmoid activation function.

~~~
inputcoffee
So true. I have had to do this.

I had a pretty good regression model but it was not taken seriously. So I
wrote it using "a neural network in TensorFlow" and the next thing you know
the whole company is asking me how it works and what it does.

------
bluetwo
Or, you can't figure out how AI is helping solve the problem they claim to
solve, and you realize they are just throwing every buzzword out there in
hopes you'll be impressed.

[https://www.youtube.com/watch?v=kzT3yfe2o-I](https://www.youtube.com/watch?v=kzT3yfe2o-I)

~~~
minimaxir
Presumably someone with a background in ML/AI can cut through the buzzwords
used by ML/AI businesses.

~~~
bluetwo
Sounds like a good use of ML. Buzzword detector with auto-remove.

~~~
akoncius
then it would be empty content :|

------
Eridrus
> Google got great because of PageRank, but it stayed great due to network
> effects.

I think this overstates network effects and under-appreciates branding and
simply reinvesting significant capital into continued R&D.

~~~
danicgross
Agreed. There are many flavors of moats. I originally wrote something more
complex but it quickly turned into an amateur HBS study about Google. I hope
the point is interpreted more metaphorically.

~~~
deepnotderp
I would suggest that in many cases the "data network effect" advantage of ML
startups is overappreciated relative to the branding advantage they have. For
example, DDG and others can, for the most part, match Google in quality of
search, but almost no one will use them because of entrenched branding, this
is especially true IMO in smaller ML vertical industries ideal for startups.
e.g., if there is only room for one ML company that analyzes farm pipe leaks
through camera, then branding and market dominance helps considerably there.

I may be wrong, I'm (mostly) a DL hardware guy, this is just my two cents :)

~~~
cycrutchfield
>For example, DDG and others can, for the most part, match Google in quality
of search

Even for the long tail? I'm pretty skeptical of this.

~~~
mcny
Google will I'm sure vehemently deny that it is unbeatable but I think with
60k+ searches every second, Google will keep getting better. I think Google is
unbeatable in search as we know it.

~~~
sharemywin
I'm not counting out google, but there are only two searches that actually
count from a business perspective. product and services. All the rest are cost
of doing business.

If you do your research on google and then buy from amazon or somewhere else
and don't use the adword link google's business strategy fails.

------
lkrubner
"Why does anyone need this? Like all advice, this sounds deceptively simple.
But make sure you get a very compelling answer here."

Right now I'm at the Web Summit, in Lisbon. I saw a few ML/AI startups here,
but I was surprised there weren't a lot more. There was some imbalance. There
were many startups trying to create new online social networks for niche
groups -- business professionals, women, entrepreneurs, sales professionals,
etc. I do think there is some innovation that can happen in social networks,
but generally I regard it as an over-crowded space. For the most part, the
benefits offered by the Internet seemed to have been mostly absorbed by the
1994-2008 cycle, and what's left is fairly minor compared to what happened
previously.

One of the few areas where I still see the possibility of significant traction
(the creation of large companies) is with information gathering and a
combination of ML and NLP. We've already seen a wave of startups which were no
more than pretty GUIs over existing technology. At the risk of being unfair,
BigML.com is just a nice interface over existing ML tools, and API.ai is just
a nice interface over some NLP tools. AmenityAnalytics goes a bit further
combining their web scraping and NLP scripts with a nice interface that
customers can use to filter the incoming data. But there is still a wide space
for companies to go much further in this field.

All the same, I agree this is a good question: "Why does anyone need this?"
You should really ask it whenever you are joining any startup.

~~~
PeachPlum
"It's like Facebook, but for $group"

Groan

------
udba
Typical of SV to focus solely on the business viability of a firm to the
detriment of everything else. I’d like to work somewhere knowing that I have a
clean conscience. Why not add:

Where do you get your data from?

If your data is sourced from users of your product, do you tell them what
you’re collecting?

~~~
jjeaff
They essentially all say "tell" their users where they are getting the data.
It's in the privacy policy. But people don't read it.

~~~
DeRock
Exactly. Informing the user does nothing except maybe clear your conscience.
Expecting the user to extrapolate to how you will use that data, what will be
inferred from it, what risks come with centralizing and storing it, all of
that is not the users job. It’s the job of the collector to be as conservative
as possible. Any company that treats users privacy as a resource to be
extracted is not somewhere I want to work.

------
inetsee
Some other questions I would want the answers to:

1) Is a Non-Compete Agreement required and what are the terms (time and
geographic coverage; compensation during the non-compete time frame)?

2) Is the company going to assert ownership over IP created before employment
began?

3) Is the company going to assert ownership over IP created that is unrelated
to current responsibilities (or IP in a area unrelated to the company's
current business), especially IP created on the employee's own time, not using
any company resources?

~~~
ryandrake
These are great questions to ask _any_ company, not just AI/ML companies.
Also, if you're planning to do it:

4) What is the companies policy on employees moonlighting / doing side
contract work for other companies not related to it's business?

Inquire about this _before_ you get your offer. Ask me how I know...

EDIT: Another one, which is kind of risky but I personally think is important
and would ask any small company:

5) How transparent/open are you about the company's financial health? Are the
books open or semi-open?

Keeping in mind that some companies will not share this information and may
even find the question offensive. One time I asked the owner/founder this, and
he would not share--I found out later through a side channel that he was very
offended that I asked for this information (but hired me anyway because he
needed someone like me).

~~~
abraae
Red flag if asking this question courteously causes offence.

------
siliconc0w
The best use of these questions is to get an idea whether your equity might be
worth anything. Ask how many customers they have, how fast they're growing,
how long does it take to onboard customers, etc. What do they see as an exit
(is it a 10M, 100M, 500M a unicorn?). When do they see it happening?

If, let's say, you have 0.01% of a Series A that exits at a billion in five
years. You'd expect to be diluted by half and there is around a 20% chance of
exit that high so roughly ((0.01% _billion) /2)_.20 /5 or about 2k a year in
'expected value'(or why, numerically at least, it's likely not worth it to
take a paycut to work at a startup unless you're seeing substantial equity).

~~~
dfee
Please let me know where I can find today’s Series A ready companies that have
a 20% chance of being worth $1b in 5 years time.

That figure probably oughta be at least one order of magnitude less.

------
aecs99
Did any of you succeed in asking such questions to companies (ML/AL or not)
and ended up receiving reasonable responses to your questions?

My experiences have never been great when asking such loaded questions. To be
clear, I interviewed extensively at small startups and I'm only taking those
interviews into consideration for this comment.

The HR and technical interviewers dedicate a large amount of time for
questioning you, and reserve the last few minutes to answer your questions. I
lost count of how many times the interviewers would just blabber something
meaningless in a rush, rather than answer a questions with patience and
honesty. Of course, I made my decisions on whether or not to join the team
based on such experiences. However, I cannot think of any remarkable instances
where the interviewers answered such questions without getting impatient.

Any chats with directors/PMs/C-level executives _after_ receiving an offer
were also not very informative. I walked out several times going in and coming
back from such chats with no questions properly answered because: (a) they are
still figuring out, or (b) they cannot discuss certain details because you
haven't said yes to them.

~~~
lmeyerov
I always answered this for Graphistry candidates. We take a vertical data
product mindset to AI -- so just one solid piece of solving the human-in-the-
loop investigation problem for security and fraud teams trying to help their
responders get a handle on incidents/APIs/logs -- so perhaps we're less
concerned about image. For startups with real tech, competition shouldn't be
the thing that matters, but execution.

For us, a great interview is one where the candidate teaches us and thereby
demonstrates the potential to be a force multiplier for the team. Also,
someone asking these kinds of questions suggests we may be able to put them in
front of a customer!

------
ordinaryradical
This is really solid advice when looking at it backwards too--if you, as a
founder, cannot produce reasonable and compelling answers to these questions,
you don't know what you're doing, where you're going, or how to actually get
there.

Great read.

------
justonepost
Not sure why ML/AI is in the title. Seems applicable to any startup.

~~~
logicallee
it's 2017. half of YC startups have AI in the pitch.

~~~
OscarTheGrinch
AI is the new UI...

------
vadimberman
These questions should be asked about every startup, not only ML/AI.

------
danieldk
I am in academia, but some of my students have returned somewhat disgruntled
from their internships in companies claiming to do machine learning, while in
practice their systems were primarily rule-based and the work consisted of
writing rules [1].

I now recommend students to find out whether companies are just name dropping
ML or doing serious ML work. Typically, the best way is to ask former
employees/interns. But I question directed at this also would not hurt.

[1] There is nothing wrong with that, but it sets the wrong expectations.

~~~
nl
I work in industry (sort of) and this is very true.

One of the reasons for this is that the people designing the systems don't
know how to do anything else.

There's such a huge, huge gap between using ML to do the same thing everyone
else is doing (say... image classification, or maybe regression on volume vs
sales...) and solving unique business problems using ML.

------
shanwang
So, if I'm interviewing with a startup in AI, and asked all these questions,
how many good answers I should expect? Is it ok for a startup to have a
defensible business to solve a problem 10x better, knows how to make money in
a big market, but have no experience in marketing and haven't talked to many
potentially users?

As a wanna be startup founder, I found my ideas have bad answers to at least
2-3 of these questions.

~~~
purerandomness
> haven't talked to many potentially users?

Any bad answer to these questions is a showstopper, yes. It's an indicator
that this company is definitely going to fail.

In your example, not having talked to many customers is the classical Juicero
approach: Build a solution, then search for a problem that this solution is
solving.

Not talking to customers means not understanding if there is demand for
something, and if there are customers who would like to pay for a solution.
(Problems exist in markets where target groups are not used or not willing to
pay for digital solutions)

If you're starting out, I highly recommend reading the blog of Amy Hoy and
Alex Hillmann
([https://stackingthebricks.com/](https://stackingthebricks.com/)) - they
defend the idea to first go on a "sales safari" where you simply obvserve your
target group, and _then_ find a solution for their pains.

As a founder, you want to continuously talk to potenttial customers about your
idea or your prototype: make them use it and perform thinking-aloud tests.

~~~
shanwang
Good points. But how many customers you talk to is enough? If I'm building a
product for enterprise, is talking to one keen potential customer enough? For
a first time founder, I can't imagine someone without much connections can get
the chance to talk to multiple potential enterprise customers.

What about a product targeting the mass market, if I'm building a mobile app,
is talking to a dozen friends enough? or should I pay for a market research
which can reach hundreds of strangers.

Do you talk to customers with prototypes of the features or do you just have a
basic skeleton and say 'what if I have this and that'?

I actually subscribed to Amy Hoy's mailing list but haven't checked it out for
a while...

------
YeGoblynQueenne
>> 2\. How was this problem being solved before the AI came around? Was the
pre-AI “manual” solution good enough? Common answer: “we’re replacing humans.”
That isn’t enough. Often having a human is desirable (bedside manner,
dexterity, perfection a requirement). Often a human is affordable due to
margin structure. You’re looking to get a sense that the product provided is
something that was never possible before, 10X better, or just-as-good but 10X
cheaper. Not 20% cheaper. 10X.

That "10X" sounds like a bit of a heuristic, but is it a good one? Surely,
what you're looking for is some assurance that the rewards from the use of
machine learning justify its cost and that the profit the company can make out
of it is higher than the profit they would be (or were) making without it?

It doesn't even matter even the profit is really much higher. In business, as
in war (and, er, board games) any edge you can get is enough to push ahead. If
a company can make even 1% more of what it did before, thanks to some new
technology or whatever other trick, then that trick is worth pursuing in
earnest.

~~~
sharemywin
I was going to say something about that my self.

I'm thinking it's more like this TED talk about something being remarkable:
[https://www.ted.com/talks/seth_godin_on_sliced_bread](https://www.ted.com/talks/seth_godin_on_sliced_bread)

------
murukesh_s
Should the engineers ask the same question when a high profile VC, like YC
have already invested on the company? The questions look more like what a VC
should (have) asked before funding.

Many ML/AI companies especially if funded and therefore highly visible may get
acquired as the market is very hot. Isn't that good for the engineers?

~~~
pjmorris
> Many ML/AI companies especially if funded and therefore highly visible may
> get acquired as the market is very hot. Isn't that good for the engineers?

Only if the potential acquirers are happy with the answers to the same
questions. As a percentage of net worth, individual engineers have a lot more
at stake than VC's or potential acquirers.

~~~
murukesh_s
Most of the big companies(apple/google/microsoft/fb/amazon) at current stage
are acquiring for talent/IP, not necessarily market fit.. they do have enough
money to do the R&D without worrying about finding a market fit soon

------
tboyd47
You can't really ask the hard questions during the interview phase. Before you
get a job offer, they have all the leverage, and people don't like being asked
hard questions when they're the ones interviewing.

Best thing to do is try to get a contact inside the company, then when you get
the offer, call back and start asking those questions.

Even better still - don't agree to an interview unless you know in advance
that the job will move your career in a good direction, then don't even bother
asking questions. It's not like they have to tell you the truth anyway.

~~~
jerf
"Before you get a job offer, they have all the leverage"

Only if you are in a situation where you need _this_ job, right now. Try to
avoid being in that situation. It isn't always possible, no, but there are
reasonably steps you can take to try, such as prioritizing saving up a cushion
over creature comforts and applying for multiple jobs.

And if you are good at a hot tech, which ML is right now, you probably do have
some leverage just from rarity.

Yes, the perfectly free market of labor may be a theoretical construct, but
don't give in to the despair that you have _no_ control over your employment
and that they have _all_ the power. It'll become a self-fulfilling prophecy.

~~~
tboyd47
True; I mean to say until you have a job offer, you have no idea whether or
not they are willing to negotiate anything with you. It's easy to fool
yourself based on how friendly your interviewers are. So many times I've come
out of the interview feeling like I got the job and then been told that
they're going in a different direction.

------
agibsonccc
The gotcha here is that most desirable AI startups SV wants to fund now
(understandably) follow a pattern more closely described here:
[http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-
star...](http://www.bradfordcross.com/blog/2017/6/13/vertical-ai-startups-
solving-industry-specific-problems-by-combining-ai-and-subject-matter-
expertise)

Generally, their advantage is in the way they use data to solve a vertical
specific problem. They more "bolt on" AI. AI isn't really the focus of most of
these companies. This is just what investors want to hear about.

The article here isn't calling them out specifically, but AI infra startups
are what are being discussed here as the "hammer looking for a nail" startup.

YC funded a few of them including Skymind (my company) and deepgram (mainly
audio). There are a lot more of these startups such as vicarious that mainly
publish papers but raise tons of funding hoping for a deepmind style exit
(they didn't really have a business model).

With that context out of the way, I'll also maybe just state some learnings
from the other side of the table.

Skymind has typically made money trying to reduce dependencies on cloud
providers by decoupling the AI infra from google cloud and co from the cloud
provider itself.

A horizontal play like this is by its nature very hard. We maintain the whole
stack including our own framework. We also started in 2014 and have a decent
foothold in enterprise. I wouldn't recommend trying to do this in 2017.

That being said, one other semi similar horizontal AI startup that is being
started are the chip companies. Those are _significantly_ harder to run than
even what we're doing.

The most common "failed" type of startup that I think of when we think
"horizontal" plays are Machine Learning as a service, which is hugely a loss
leader for the various cloud providers (some companies are building on top of
these providers though).

This is where you see Metamind, Alchemy API, Scaled Inference (founded in
2015), even Nervana before they sold were trying to a "nervana asic cloud"
among others.

Maybe a lot of investors from SV view this as a waste of time dissecting, but
I would personally love to see a bit more content on acknowledging some of
these trends in the market.

The allure of "AI as a service" and the horizontal dev tools infra play is
that you can try to build the next AWS similar to what the container and
database companies are trying to do. Execution is definitely key for this to
work though. Research also can't be the primary focus.

I won't comment on what will or won't work there

------
notyourday
So. Much. B.S.

There's only one question they need to ask:

"How much are you going to pay me?"

Edit: The reason anyone would wax poetically about everything else is because
he or she realizes that they are not being paid enough. It is like day trading
- the moment it goes against a bad trader he or she calls it "investment"

Edit: Thanks for the down votes! It makes me feel warm and fuzzy inside to
know that it is still possible to swindle people!

~~~
wpietri
Nah. This is a good list of questions. Partly because startups will include
equity, and you need to know these things (and more) to make a guess at
valuing the equity. And partly because a company without strong answers is
likely to be out of business soon; changing jobs unexpectedly can be
expensive, especially if you get stiffed on wages.

But mostly because it really sucks to spend a couple of years building stuff
that never gets used. There are a few people who seem happy to get paid even
if nothing is achieved. But most engineers I know do the work because the like
building things that get used, things that are valuable.

~~~
0xbear
Unless you are employee #1-5, the correct approach to valuing equity is to not
consider it to be worth anything. For employees 1-5, divide the numbers the
CEO gives you by at least 10 to account for risk, and then demand a
competitive base pay.

~~~
wpietri
Yes and no. I agree should expect zero to come out of it, as it's a pretty low
probability you'll see anything. But you still have to do the work of valuing
it.

First off, the people you're negotiating with see it as valuable, so it's a
very bad negotiating move to say, "Keep your dumb equity and just give me
cash." You'd be slightly better off just pointing to their family photo and
saying, "Wow, your kids are ugly." Second, going through the work of valuing
equity is necessary to see how the investors (that is, they people actually
paying your salary) look at the company. Third, on the off chance you are
successful, you're going to really wish you took the equity part of the
negotiation seriously.

~~~
0xbear
People you’re negotiating with want you to buy their line that their equity is
worth something so they could pay you less. What would you get as eg employee
#10? Half a percent or less. Most startups exit under $100M, and your 0.5%
turns into 0.25% due to dilution. It’s also vested over 4 years. We’re talking
$250k max over 4 years, with a 5% chance you’ll get anything at all. You’d
have to be seriously arithmetically challenged to take that seriously.

~~~
notyourday
People are arithmetically challenged. People also like to boast about doing
good etc. Typically they realize how stupidly naive they were just around the
time they no longer have ability to say "That's nice to know. So how much are
you going to pay me?"

The down votes on calling spade a spade demonstrate it quite well

------
_Codemonkeyism
If startup: When do you need the next financing round?

------
Brushfire
While I appreciate the nature of these questions, many of them exclude
network-driven consumer businesses. You know, those like Tinder, Facebook,
Twitter, Medium, Yelp, FourSquare... If you're spending time with a consumer
company like that, I'd focus less on LTV and more on organic growth rate.

------
bald
> "Reverse engineering from the technology to the market almost never works."

Completely disagree.

If the technology is creating some value for someone, and

if you have only one smart business dude that knows how to approach the right
people and validate markets,

reverse engineering from technology to market _does_ work.

Source: Did that myself.

~~~
harshaw
_almost_ never works.

Countless examples of where it fails.

~~~
cjalmeida
Then again, almost all startups fail. Knowing what worked and the
circumstances that made it work matters.

------
thisisit
Question 6. How often do people talk about ARPUs with potential hires? I am
sure people will tell you that there are 150 million potential users but not
their revenue models. It is easy to under shoot or overshoot ARPU.

------
minimaxir
Shouldn't the interviewing engineer know the answers to these business
questions beforehand during their research of the company? (particularly if
the company is upfront about their ML/AI-usage)

~~~
justonepost
Usually you're only going to get good answers to these questions after they've
made you an offer.

------
known
Is it different from
[https://en.wikipedia.org/wiki/Wisdom_of_the_crowd](https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)

------
alexasmyths
Yes, quite good actually.

Unfortunately, sometimes the hype is so strong, the cart comes before the
horse and gets acquired for quite a lot - which is why these companies exist.

------
OliverJones
This advice applies to all kinds of fledgling businesses, not just ML/AI.

------
raverbashing
Good questions, and they apply to many more companies than just ML/AI

------
agjacobson
This is silly. You’re the one being evaluated here. Can you code, or are you
just a “sophisticated” smartass?

The trick is to find out all these answers WITHOUT asking the interviewer.

------
anoother
And here I was thinking these would be questions about the moral / ethical /
social implications of the ways in which many companies use these
technologies..

