
Lessons for Building AI-Driven Products - mwakanosya
https://blog.insightdatascience.com/moving-towards-managing-ai-products-5268c5e9ecf2
======
pmcgrathm
As someone who is a product manager in the commercialized AI SaaS space, the
most important pieces of feedback I would give a new PM here:

1)Don't let your -brilliant- colleagues try to force their -brilliantly
complex- solution of a problem - clearly define market problems, and don't let
the team try to go the route of trying to force fit a solution to a market
problem. Market problems come first.

2)Frame the market problems appropriately for your ML/AI teams, and practice
trying to frame the problem from a variety of angles. Framing from different
angles promotes the 'Ah-ha' moment in terms of the right way to solve the
problem from the ML side.

3)Don't commit serious time to a model before having a naive solution to
benchmark against. Always have a naive solution to compare against the AI
solution. 'Naive' here may be a simple linear regression, RMSE, or multi armed
bandit/Thompson sampling.

~~~
alexbeloi
>Always have a naive solution to compare against the AI solution. 'Naive' here
may be a simple linear regression, RMSE, or multi armed bandit/Thompson
sampling.

This cannot be stressed enough, optimism bias will always push the scientist
towards the 'more interesting/complete/new' method and model, but a seasoned
practitioner will have the discipline to always establish a baseline (<1 days
work).

------
throwaway84742
If I could add to this: realize that “AI” will be a small fraction of the
overall product. And important one, one without which the product would likely
not exist, but in terms of effort spent it will be small. So no, you don’t
need an army of ML researchers to ship it. You need maybe one or two, and the
rest of your money is better spent on hiring engineers, sales/bizdev people,
and a great product manager. People don’t give a shit about models. They do
give a shit about whether your product solves a problem. Use AI as a means to
an end, not an end in itself. Focus narrowly on a concrete problem, don’t try
to do everything at once.

~~~
fnbr
Yes, exactly. And most of the time, you can get by with a simple model like a
random forest, so you just need to have a few analysts/engineers doing feature
engineering, and feeding it into your model.

Almost everyone using more complicated architectures (e.g. neural networks)
are adding unnecessary complexity. [1]

[1]: Unless you're doing computer vision.

~~~
e_ameisen
Agreed, we've also written about why the best use of your time is to start
with a baseline (even in Computer Vision!)
[https://blog.insightdatascience.com/always-start-with-a-
stup...](https://blog.insightdatascience.com/always-start-with-a-stupid-model-
no-exceptions-3a22314b9aaa)

~~~
throwaway84742
And before a baseline, start with formulating a problem in a way that’s
applicable to a real world product. Not just “image classification” but “image
classification to improve QA in underwater basket weaving, a 10 billion dollar
industry, from which we could generate $100m/yr in revenue by selling a
product that does A, B, and C to an estimated 100k customers each of which
would have $1000 LTV”.

I’m in an AI startup right now where the founders are hell bent on doing
nothing in particular, and I’ll be bailing in a month or so.

~~~
didgeoridoo
I'd love to hear about your experiences — email in my profile.

I've been doing some writing and speaking on why AI seduces founders into
pursuing lots of cool ideas instead of marketable products. I'd love another
data point.

Could do it anonymously if you'd like, or hold off until after you bail.

------
joshuaeckroth
Their lesson, "Build Breadth-first (Data/Pipeline/Model) instead of Depth-
first (AI model)" is insightful. Once the pipeline/etc. are built perhaps the
AI part is not even needed any more. Finding this out early can save you a lot
of trouble.

Another perspective, in which we talk about on integrating AI into existing
workflows, among other lessons:

Smith, Reid G., and Joshua Eckroth. "Building AI Applications: Yesterday,
Today, and Tomorrow." AI Magazine 38.1 (2017): 6-22.

[https://www2.stetson.edu/~jeckroth/downloads/smith-
eckroth-2...](https://www2.stetson.edu/~jeckroth/downloads/smith-
eckroth-2017.pdf)

~~~
gfdr
Not only that, but breadth-first makes it easy to: a) Iterate much faster on
models; b) Keep better track of results and code and models, which is already
quite difficult in ML; c) Enable other teams to work on other parts of the
pipeline, in parallel.

------
joshatidealspot
Could be better written but here are my thoughts on the same topic from
earlier last year: [https://medium.com/towards-data-science/hard-earned-
advice-f...](https://medium.com/towards-data-science/hard-earned-advice-for-
ai-products-c320d900862e)

------
tpkj
Question: looking at the article's "AI Product Management" infographic, one
item includes the phrase, "Adopt a bread-first approach to building a
product". Is that supposed to be "breadth-first"?

~~~
ScottBurson
It means to go for revenue first :-)

------
latenightcoding
A lot of good comments in this thread. I am a data scientist and of course I
love playing with the latest deep learning frameworks, but If I'm putting an
ML model in production I want something that: will pass QA, is easy to
interpret, will run in real-time and for those requirements it's hard to beat
random forest, boosted trees, logistic regression and naive bayes

------
wackspurt
Can anyone here offer advice for doing anomaly detection in distributed
systems?

I'm not looking for advice on which models to use, per se. I'm more interested
in how to go about things as a single-person team (building data warehousing
infrastructure, etc.)

