
Adventures in Improving AI Economics - oliver101
https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/
======
eanzenberg
In my experience, there are just a lot of "bad" AI/ML engineers who don't
fundamentally understand what data can do, what ML algorithms can handle, and
how to piece it together to produce something of value to the end user. A
couple of these people on a team can torpedo a project. Worse are those who
sabotage projects or are general pain points of hindering progress. These may
be jaded people who don't believe that ML has any value yet have titles like
Data Scientist or ML engineer, and can bring team morale down. The economics
are similar to a grad-school research project, yet is infiltrated by all sorts
of people with 3 month certificates believing they are the star of the show.

The most important element of AI project success is the right people and the
right team. Projects are long-term and failure can be often. It's not easy to
succeed but cultivating the right people and their mindset is in my opinion a
needle mover for AI projects, more-so than what data is available, what
algorithms are tried, and what shiny framework people want to use.

~~~
Swizec
Correct me if I’m wrong but isn’t this fundamentally true for any team working
on any project?

~~~
logicslave12
No, there’s more ambiguity in machine learning projects. When you develop a
website, aside from the design, it works or it doesn’t. Whether some kind of
ml product can work at all is often team dependent

~~~
cnasc
> When you develop a website, aside from the design, it works or it doesn’t.

This really isn’t true. There are websites and web apps that “work” but are
really suboptimal from a performance and UX perspective. It’s possible to do
this right, but it’s much easier to do a poor job. You end up with something
that kind of works and may even be profitable but which is a boat anchor
around your company compared to a better approach.

------
alextheparrot
a16z has a podcast where they explored gross margins a month back. The panel
called out AI as an example of a software business that has a high likelihood
of not having standard SaaS margins (Most of the panel thought this could be a
limitation).

The podcast is nice because I think it holistically explores gross margins in
a way that you start to understand how it might impact AI as a viable primary
business model and valuations related to companies who that is the case for.
Quite complementary to the article.

Might be interesting to people who are interested in this article:
[https://open.spotify.com/episode/79lJCrHB3nBn1qXCxKA5s7?si=R...](https://open.spotify.com/episode/79lJCrHB3nBn1qXCxKA5s7?si=RFlHPM0_QwWHZsqwf3nbxA)

~~~
Jack000
This is a great feature of the AI space for startups - in the short term it
reduces competition, in the long term it's not really a problem. If your
business is break-even currently, it will be profitable in 2-3 years due to
declining cost of compute. In 10 years the compute costs will fall by an order
of magnitude and more efficient models will become available, making the
economics closer to traditional SaaS.

This does disadvantage smaller bootstrapped businesses though.

~~~
kippinitreal
Not true if there’s enough competition that you need more resources for a
bigger model in 2-3 years. Anecdotally, it seems like SOTA model training
costs are rising much faster than computer cost is falling.

~~~
Jack000
maybe, but model quality doesn't scale linearly with model size. The
performance/dollar metric is more important, and that definitely will decline
over time.

Personally, I have a boots trapped business that uses a transformer model. I'm
not worried about supermassive models like gpt3 because it would be way too
expensive to deploy for my use case, with marginal additional quality.

------
motohagiography
Good analysis and great of them to share their thinking. Does feel like this
could have been a tweet that said the necessary condition for successful ML
solution is applying it to a problem that has asymmetric upside.

Great for telling people they should get tested for diseases, terrible for
diagnosis. In the alerting first case, consequences of being wrong are no
better than base rate as they wouldn't have been tested otherwise, and the
upside saves a life. In the latter diagnosis case, the consequences of being
wrong are catastrophic, and it is substituting for the best available
judgment. Similarly, it's great for fraud detection, terrible for making
credit decisions, because the false negative rate is essentially externalized.
It's good for finding opportunities, bad for providing services. So funnels
and conversion pipelines it's great for.

So perhaps there's an ironic Turing test for ML solutions that is related to
the relationship between the size of a group of people and the effect of mean
reversion of their collective intelligence on their behaviour makes them
indifferent to the perceived intelligence of the model, whereas a given
individual will find the results of the model unsatisfying. From an
indifference perspective, AI can fool some of the people all the time, and all
the people some of the time, but no confusion matrix satisfies all the people
all the time. Economically, ML will be useful for creating simple and cheap
services that people who can't afford better will use, and substitute up from
them when they can afford better, known as "inferior goods." There may be a
hard limit on ML providing "normal goods," to individuals at scale for this
reason. Lots of money to be made, but lots to be wasted tweaking your ROC
curve to in the hope of creating a normal good.

I yell from the rooftops every chance I get that "the confusion matrix is the
product." That is, your FP/FN/TP/TN rate is your product, and you are
optimizing your system for the weights your customer assigns to those
variables.

There is another ML/DL use case I'm hacking on that is about enabling privacy,
but even this reduces to the asymmetry of the upside/downside of the confusion
matrix. Obviously the article is more nuanced than this, but I think this
heuristic is a key tool for reading articles like it.

~~~
jmatthews
I appreciate the thoughtful commentary. I couldn't disagree more with you more
of course.

There are 2 instances where AI breaks the mold you've cast.

Executing rote tasks that no humans need do, and relatedly, while there does
seem to be a tough hurdle when it comes to "better than human" execution there
is also an inverted survivors bias. Once a technology is production ready it
is no longer AI. Cars aren't robots, antilock brakes aren't AI, Once a system
outperforms a human it's technology, not intelligence.

~~~
motohagiography
Our disagreement might be subtle. An old saw of mine is that the Turing test
thought experiment is covered by prior art in economics, where the idea of an
indifference curve describes the points between amounts of things where people
are indifferent to substituting between them.

I agree these things you state aren't intelligent, but nor are computers, or
can they be - people just become indifferent to whether we are dealing with a
human or a computer.

My assertion is that we are highly sensitive to substitutes when the downside
risk is large, but largely indifferent to them and even like them when they
resemble a lottery with good upside at low cost or risk.

Self driving cars are a good example, where someone asked me whether, if I had
kids, would I send one to school in traffic in an autonomous vehicle. I told
them it would depend on how many kids I had.

But this pretty much describes the dynamic.

~~~
nl
_Self driving cars are a good example, where someone asked me whether, if I
had kids, would I send one to school in traffic in an autonomous vehicle. I
told them it would depend on how many kids I had._

Pretty sure that answer is much less convincing than you think.

Infact, I thought you were right until that, then realised that was an answer
no parent would ever give which made me realise there's a lot missing in your
hypothesis.

------
oliver101
> This is the crux of the AI business dilemma. If the economics are a function
> of the problem – not the technology per se – how can we improve them?

The article focusses on the costs of resources to build a model (annotated
data + compute) but the economics are also affected by the ongoing cost of
making a prediction error. False positives and false negatives usually have a
different cost and each user might have their own preferences:

e.g. "show me all the content that's a bit relevant" vs "show me just the
content that's really relevant".

If you can write out the loss function in $$$ terms not just accuracy, then
you're closer to either abandoning the problem or finding a profitable AI
model.

~~~
mlthoughts2018
Great way of putting it. The trouble there is that it takes an exceptional
kind of senior ML person to basically wear a product manager hat all the time
and press to translate project success criteria into revenue impact or cost
reduction terms.

Having these “glue people” that connect ML engineering to product management
is probably the most important thing to running an ML organization.

------
tosh
this is more related to the previous a16z article on the topic but I found
"Data as a Service" by Auren Hoffman a great read for thinking about
businesses that sell access to machine learning models

[https://www.safegraph.com/blog/data-as-a-service-bible-
every...](https://www.safegraph.com/blog/data-as-a-service-bible-everything-
you-wanted-to-know-about-running-daas-companies)

------
zamadatix
"Andreessen Horowitz (known as "a16z") is a venture capital firm in Silicon
Valley, California"

In case anyone was as confused as I was about what a16z means - it's just the
company not a new abbreviated term related to AI.

~~~
gilgoomesh
Yeah, I find this kind of abbreviation annoying. But there's a few words that
are commonly abbreviated like this:

i18n -> internationalization

l10n -> localization

g11n -> globalization

l12y -> localizability

a11y -> accessibility

It bothers me because my brain does not jump from the abbreviation to the
underlying word. I really need to stop and think about each one. And I get the
numbers wrong when writing them.

~~~
LifeIsBio
There was a period of a few months when I was first learning about web apps
where I saw "i18n" multiple times. The first time I came across it, I tried to
sound it out:

i18n -> I-one-eight-n -> iwonation

I was already a couple of rabbit holes deep at the time and didn't have the
mental capacity to look it up and wrap my head around yet another new concept.

"Oh boy." I thought to myself, "One more word I've never heard of, probably
representing some complicated CS concept."

I was so annoyed when I finally found out what was going on.

~~~
rland
Wait, can you clarify what is actually going on? Is there any rhyme or reason
or is are these shortenings just random?

I'm having trouble parsing the grandparent comment...

~~~
websight
It's the number of letters between the start and end letters. Yeah, it's
annoying.

------
PaulHoule
Gr8 article.

I'd add that caveat that software dev processes can be well controlled or not
well controlled. AIML is not so much a new kind of project but it is a project
likely to be poorly controlled.

Another thing they don't mention is that AIML projects break the agile
assumption that you can manage with only punchclock, not calendar time.

Imagine you have a 2 week sprint and it takes 1 week to train a model. You
have to get the training started in the first week, and any tasks that need to
be done to start training have to start before that.

This of course means applying PERT chart thinking even if you don't make PERT
charts. It often isn't that hard but in an agile shop that mistakes the map
for the territory they will start the 1 week job consistently on the last day
of the sprint.

The 'containerization' process they describe is close to the methods used by
East coast defense contractors (in a band between research triangle park and
the applied physics dept at John Hopkins in baltimore) to get high accuracy.
Also they were what IBM Watson did as opposed to what people thought they did.

It's amazing those methods have remained so obscure, but the mind that is
impressed with BERT is going to be impervious to asymtopes. That article
should be telling people to run not walk away from those kind of models -- it
is how you always be a bridesmaid but never a bride.

------
mensetmanusman
Good analogy about discovery of Pharma molecules.

It’s really fun to think about the fact that Tesla has more than enough data
to unlock autonomous vehicles, but all that is missing is the correct AI
architecture to get it working...

Who will figure out how to code that? Will it be a breakthrough, or can sub-
optimal architectures eventually reach equilibrium with 10x or 100x the amount
of time/data processing.

~~~
mdorazio
> Tesla has more than enough data to unlock autonomous vehicles

Many people in the automotive industry, myself included, disagree with this
statement pretty strongly. Driving data quantity is not equivalent to quality
and they are severely lacking in advanced sensor data.

~~~
jointpdf
So is the claim by Elon Musk that current iterations of Tesla vehicles have
all of the sensors and compute power needed to be fully autonomous (Level 4+ I
guess?) in the future, via software updates only, a specious one?

~~~
nl
It's hard to be absolutist on the response to that: anything is possible, and
humans can drive without LIDAR.

But at the moment it seems a strange position to take: we know LIDAR data is
useful in many circumstances, and we know it can solve a number of the hard
parts of computer vision.

------
mijail
I'm happy a VC is providing these insights. If the economics of AI don't make
sense by helping increase profits or cutting costs... it's going to be a long
road to reaching the "promised land."

I'm biased but in a lot of industries synthetic data has the potential to
balance the costs from the perspective of data acquisition and preparation as
well as model testing.

This article doesn't focus too much on the edge side of things but one pattern
I'm seeing is that edge deployment can be notoriously resource intensive and
time consuming.

------
gk1
One way to deal with AI/ML shortcomings I've seen is to require end-user
intervention for edge cases; such as a support chatbot that transfers the
customer to a human rep if it can't understand the issue. Human intervention
isn't mentioned in the article but maybe they'd put that under "narrow the
problem," or they may not consider that a solution since human involvement
eats into margins.

I believe all software companies will be AI companies in <5 years. By then,
not having AI/ML would be like not having a database today. There will be no
choice but to deal with the long tail, and the competitive advantage will go
to the company that does it better. That makes this advice all the more timely
and important, and it also means opportunities for startups to innovate in
this space. Eg, better model optimization, low-cost operations without
regressing to colocated GPUs, etc.

"The critical design element is that each model addresses a global slice of
data... There is no substitute, it turns out, for deep domain expertise."
Totally true for marketing as well. Much more effective to define audience
segments and tailor the messaging and marketing for each.

------
known
[https://yts.mx/movies/robot-frank-2012](https://yts.mx/movies/robot-
frank-2012) show subtle issues related to AI in real life;

------
tigerbelt
Indubitably

