
AI Playbook - jbredeche
http://aiplaybook.a16z.com/
======
shadowmint
I found this really frustrating to read; on 'training your own models':

> Deep learning is a class of machine learning that has gotten a lot of well-
> deserved attention in recent years because it's working to solve a wide
> variety of AI problems in vision, natural language processing, and many
> others.

What others? Be specific.

The problem _is not_ that people don't know how to build neural networks.

The tensorflow tutorial is extremely comprehensive and there are lots of
resources for people who know what problem they're solving to try to build a
solution.

The _problem_ is that people don't understand what they can _do_ with this
technology, or even, in a meaningful sense, what it is.

Given a array of input numbers [i1, i2, i3...] it provides a black box that
can map that input to a set of output numbers [o0, o2, ...] where the
dimension of the input and output are different.

That's. All. It. Does.

It takes an input array and outputs an output array; a magic f(x) function.

The thing people don't understand well is _not_ how do I implement f(x); the
problem is how do I convert my domain problem (I have 2000000 shelf products
in my database and a tonnes of sales information) how to convert that into an
input array of numbers? What is the output array of numbers I want out of the
function?

The reason this works well for images is because you can say, ok, this output
number represents a tag (dog, cat, etc) for each discrete value, and the input
numbers are just an array of the raw pixel data.

That's why its good for images; because a traditional function implementation
of f(x) that takes a I of dimension say, 65535, and produces an output of
dimension 1 is a hard problem.

...but if your problem is, given an input of say [price, date, origin] and an
output of [value to business], you can solve that problem quite trivially
without needing any neural network.

tldr; The problem people are having is figuring out input and output for
domains other than image processing; _not_ how to build a NN, or how to choose
which NN to build.

Guide misses the point entirely.

(You might argue this is somewhat uncharitable to what is actually a well
written guide to ML; but you can read the about section for yourself and see
what you think; I feel like the guide needs to focus sharply on practical
business outcomes if it wants to do what it says its doing)

~~~
randcraw
> Given a array of input numbers [i1, i2, i3...] it provides a black box that
> can map that input to a set of output numbers [o0, o2, ...] where the
> dimension of the input and output are different.

> That's. All. It. Does.

No. That's. WHAT. It. Does.

Your emotional overreaction to this site has induced bias leading to a false
negative -- mistaking hype for signal. Fact is, using deep learning, newbie
grad students have routinely and resoundingly outperformed nearly all the
state-of-the-art techniques in signal processing in less than FIVE years of
trying. Image analysis and speech recognition have been revolutionized; 40+
years of R&D expertise has been largely supplanted in the blink of an eye.
Dismiss this tsunami at your own peril.

Does DL scale to harder problems? Why yes, it does. No computer implementation
of the game Go had outperformed the best humans until deep learning was used,
then it beat the best humans _almost immediately_ (unlike chess which took
decades to reach that level using traditional AI methods). I have little doubt
that significant achievements await DL's application to more complex tasks
like language analysis too. To believe otherwise would be severely short-
sighted.

Does deep learning also solve trivial pattern rec problems like regression or
curve fitting as efficiently as some linear ML models? No, but who cares? By
that reasoning, we should dismiss Einstein because many 12 year old children
can multiply numbers faster and for less money.

~~~
sillysaurus3
The parent didn't react emotionally and didn't even attack deep learning as a
subject. He said deep learning gurus are failing to make it understandable.

Here's someone else who cherry-picked a quote and ignored the central
argument:
[https://news.ycombinator.com/item?id=14347068](https://news.ycombinator.com/item?id=14347068)

Almost every one of your sentences are talking past the parent. If you want to
be argumentative instead of helping the parent as he asked, then it might be
best to find actual quotes to disagree with.

~~~
sjg007
Yeah I find that a lot of experts have a very difficult time trying to explain
things in simple terms. But maybe they haven't been asked to.

------
tontonius
Artificial Intelligence is like teenage sex: everyone talks about it, nobody
really knows how to do it, everyone thinks everyone else is doing it, so
everyone claims they are doing it.

Please don't write this off as a reddit-esque comment. I really think the
analogy holds -- just take a look at Product Hunt front page, or HN for that
matter.

~~~
rjtavares
Some people are really doing it, but they're not really doing it properly.
Some of them will get better at it, and some will realize they started doing
it way too early.

In a few years though, most people will be doing it, and it'll be great
(they'll exagerate how much they do it though).

Yes, the analogy holds.

------
sillysaurus3
It was pretty incredible to open up
[https://vimeo.com/215926017](https://vimeo.com/215926017) and click at a
random timestamp (29:30) and hear "we're trying to use AI to diagnose cancer
by examining the DNA that's free-floating in your bloodstream."

This is clearly the future, if it can work. That's probably one of the biggest
startup ideas that you could possibly have. I hope that whoever's behind it is
successful.

~~~
entee
Freenome is the company, they're doing interesting work but it's still early
days. This isn't actually a very new idea, it's been floating around for years
in academia (I've personally been hearing it about it for at least 5 years).
The hard part isn't really the AI per se, it's getting enough DNA from the
circulating copies (i.e. sample prep) and being lucky enough that the cancer
you're trying to diagnose actually puts out enough DNA for you to measure at
an early enough stage to be clinically relevant. In other words, if you can
only get DNA when it's at stage 4 and metastatic, your DNA test isn't actually
very useful. If you can get stage 1 cancers, now you're talking, but in those
cases tumors tend to shed a lot less DNA (among other reasons because they're
smaller).

A very good friend of mine is their CSO, I think they've got a really cool
idea and could indeed go far, but it's a fiendishly hard problem.

~~~
sillysaurus3
As someone who has zero experience but is extremely interested, could you ELI5
why it's difficult to detect whether some DNA is cancerous?

In other words,

 _The hard part isn 't really the AI per se, it's getting enough DNA from the
circulating copies (i.e. sample prep)_

It sounds like there's a way to detect "This DNA is definitely cancerous," but
the problem is that the test only works when you give it enough DNA. My
question is, where does this "enough" criteria come from? Why doesn't it work
with a single cell? (I know there are obvious reasons, but I don't know what
they are.)

~~~
entee
These are all good questions, and full disclosure I know only the basics in
this area. DNA can be identified as cancerous for a variety of reasons, though
typically what you're looking for is some mutation you know to exist in
cancer. This could be a single point mutation (insertion/deletion/change of a
given DNA base pair), a loss/gain/modification of several adjacent base pairs,
or whole scale genome rearrangements (deletion, insertion, inversion,
replication) that will result in of two pieces of DNA being adjacent although
you know they shouldn't be next to one another (see
[https://en.wikipedia.org/wiki/Philadelphia_chromosome](https://en.wikipedia.org/wiki/Philadelphia_chromosome)
for a quite well known example). You could also look for copy number variation
(gene X has way more or way fewer copies than expected). I'm probably missing
some as well, but those are the biggies.

To detect these, you're going to use either something like microarrays, QPCR,
or DNA sequencing. Today people talk almost exclusively about high-throughput
sequencing (HTS) because it has gotten quite cheap, gives you large amounts of
data, can be made massively parallel and isn't as hindered by needing to know
very precisely what you're looking for (microarrays and QPCR require somewhat
more targeting; HTS also needs you to know what you're looking for, but not in
quite the same ways).

In all of those cases you need a sample of DNA (duh). These can all be made to
work if you have a single cell in hand, it's tricky but sample prep does exist
for single cells at a time. Trouble is, you need that cell in hand. There are
about 5L of blood in the human body. A human cell will range from 1-10 pico-
liters. You will take a blood sample of 2 milli-liters. Chances are pretty
slim your one cell will be in that 2mL. Even if it is, there are billions of
other cells in that 2mL adding additional noise. Usually though, we're looking
at circulating DNA fragments with these technologies, and depending on the
stage of the cancer and the type of cancer, there may be enough to pick up on
in the blood. How to do that best is an area of active R&D.

Just dug up review if you want to learn more:

[http://journals.plos.org/plosmedicine/article?id=10.1371/jou...](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002205)

~~~
sillysaurus3
Thank you very much for the breakdown and the reference!

For some reason I assumed that this detector would be embedded into your
circulatory system, so that it eventually scans all 5L of blood in your body
rather than getting lucky with a 2mL draw. Now that I say it, it sounds pretty
unrealistic, but I wonder if it's even remotely possible.

~~~
entee
So if we want to get a little sci-fi but still within the realm of not-
physically impossible...

If you wanted to monitor for a very specific piece of DNA or a small number
thereof, you could have an inline piece of hardware that samples the blood
continuously, trying to enrich for those. You could then use a hybridization-
based (kinda like a microarray) test to check for the presence of that DNA.
Right now, the relevant technology just isn't there. You'd need new materials
(of the top of my head they'd need to be highly resistant to fouling, among
many other properties), probably a completely new filtering/enrichment system
(again fouling and clogging and specificity and sensitivity are huge problems
here), and a decent, cheap, high sensitivity and reliability hybridization
based detector. All of those things are quite hard, but I don't think they're
completely impossible. I do think it's highly unlikely we'll be seeing this in
the next 5-10 years though. But I could be wrong! :)

~~~
kanzure
A similar device was proposed a while back-- a nanopore DNA sequencing array
implanted into the heart that checks all circulating DNA. In context, the
authors had proposed embedding some enzymes in every neuron to record action
potentials, neurons would excrete the DNA into the bloodstream, and then you
would recover the entire connectome and all the action potential history by
DNA sequencing any circulating DNA that makes it to the heart.

------
joshatidealspot
I find a lot of discussion of AI skips over all the hard parts of
productizing. Recently wrote a quick article on that topic for anyone who is
interested: [https://medium.com/towards-data-science/hard-earned-
advice-f...](https://medium.com/towards-data-science/hard-earned-advice-for-
ai-products-c320d900862e)

~~~
jonobelotti
"AI products are hard. They require solutions to problems that are impossible
for the human mind to comprehend... "

The start of your second sentence seems to me pretty inaccurate.

~~~
derefr
How about "They require solutions to problems that the average human mind
shies away from attempting to comprehend..." (I assume the rest of the
argument is "thus making it hard for anyone with the problem your product
solves to realize they even _have_ that problem.")

~~~
eastWestMath
That's still remarkably grandiose.

~~~
Houshalter
How so? No one really has any idea how, say, NNs trained on machine vision are
working. Hand designed programs to do machine vision don't even get a tenth of
the accuracy of NNs. That's just one domain. And one that has been very well
studied by humans for decades with little progress.

AI solves problems that are vastly too hard for humans to figure out.

~~~
jacquesm
> No one really has any idea how, say, NNs trained on machine vision are
> working.

False. Plenty of work has been done to show what exactly is going on inside
NNs including some very impressive visualizations.

> Hand designed programs to do machine vision don't even get a tenth of the
> accuracy of NNs.

And that's false too. Hand designed programs to do machine vision can get
significant fractions of the accuracy of NNs and sometimes even perform better
and/or faster depending on the domain.

It is a _lot_ more work to write that code though and a NN might clue into
features that you weren't aware even existed.

~~~
Houshalter
Sure there are some pretty visualizations of what images turn on a neuron or
whatever. But no one could hand code an algorithm that does the same job
that's even half as good.

>And that's false too. Hand designed programs to do machine vision can get
significant fractions of the accuracy of NNs and sometimes even perform better
and/or faster depending on the domain.

Source? A few years ago people thought it was amazing an algorithm could
recognize cats 12% of the time. And even the best non-neural approaches use
_some_ form of machine learning.

>a NN might clue into features that you weren't aware even existed.

That's the whole point...

~~~
jacquesm
Your statements are nonsense. Yes, we do have a good idea of what goes on
inside a NN trained on images and Yes hand designed programs exist and work
and sometimes work very well.

It's just that NNs are _rapidly_ moving ahead and maybe some time in the
future both your statements will be true but today they are not.

Recognizing cats is not the sum total of machine vision problems and those
'pretty visualizations' go a lot further than what you give them credit for.

I've been doing nothing but reading up on machine learning and working through
piles of example code and models for the last couple of months and I think
based on what I've seen so far that we are definitely headed in the direction
that you suggest. At the same time we aren't there yet and such hype does not
help.

Keep in mind that the domain of machine vision is extremely broad and much
wider than Kaggle competitions and that saying 'we have no idea' when clearly
we do tries to sketch NNs as magic and they're not.

If anything the present generation is best characterized as an optimization of
what you could have done for many years, the only reason they are the way they
are is because we'd like to have our results in a reasonable time. That makes
applying NNs to large datasets possible and that in turn is what helps to
solve some of these problems.

~~~
Houshalter
>Yes, we do have a good idea of what goes on inside a NN trained on images

I'd love a source for this. The last time I checked I didn't find much. Sure
we know the first few layers do stuff like edge detection. But the higher
layers are much harder. And even if you know images are activating them,
writing code that can accomplish the same task is much harder.

>It's just that NNs are rapidly moving ahead and maybe some time in the future
both your statements will be true but today they are not.

The best NNs have already surpassed human accuracy at imagenet. That's insane!
No one expected that result a few years ago. Yes object recognition isn't the
only task in machine vision, but it's pretty representative.

~~~
jacquesm
> I'd love a source for this.

I will do some digging for you tomorrow, it's 3:30 am and I really should get
some sleep.

> Sure we know the first few layers do stuff like edge detection.

Yes, basic primitives.

> But the higher layers are much harder.

Well, those higher layers are built up out of the primitives the lower layers
contain. So you get things that are conceptually easier to understand rather
than harder.

> And even if you know images are activating them, writing code that can
> accomplish the same task is much harder.

That I completely agree with, I've basically gone that route in the last
couple of months.

> The best NNs have already surpassed human accuracy at imagenet.

Yep. It's extremely impressive.

It's a watershed moment in the history of computing and half of us are still
asleep or in denial. Which is fine by me. But please don't hype. It isn't
magic and it isn't perfect - yet.

edit: ok, found two of the things I've looked at:

[https://stats.stackexchange.com/questions/114385/what-is-
the...](https://stats.stackexchange.com/questions/114385/what-is-the-
difference-between-convolutional-neural-networks-restricted-boltzma)

[https://stats.stackexchange.com/questions/146413/why-
convolu...](https://stats.stackexchange.com/questions/146413/why-
convolutional-neural-networks-belong-to-deep-learning)

And this is interesting code:

[https://github.com/yosinski/deep-visualization-
toolbox](https://github.com/yosinski/deep-visualization-toolbox)

If anything I think that NNs as applied to vision problems are somewhat easier
to make sense of than doing the same for NLP or audio.

Though it definitely feels like cheating compared to actually programming a
solution.

An alternative name for 'deep learning' would be 'automatic multi-level
feature engineering', but that wouldn't be nearly as sexy.

------
eanzenberg
Is this a subtle ad for clarifai?

For the record, I've worked on building a classifier for nudity detection and
not much get's better than starting with a pre-trained model and fine-tuning
to your specific rules. I wouldn't be surprised if this is the future of deep
learning: Start with the closest pre-trained model that has trained for months
using custom clustered hardware and fine-tune further on commodity hardware.

~~~
strzzz
That's a very good business model as well to sell these pre-trained models.
Like a market where you choose what you need.

~~~
AndrewKemendo
That's what Algorithmia is trying to do.

[1][https://algorithmia.com/tags/machine-
learning](https://algorithmia.com/tags/machine-learning)

------
mifeng
While this site has some great info for non-technical folk who are serious
about learning AI, I think the site will fly over the heads of most "line-of-
business leaders" at whom it's aimed.

In my experience, those folks think of AI like...

My company is collecting tons of data in [insert data warehouses here]. How
can I bolt on some "AI" and find out what's "interesting"?

~~~
riphay
This is a great point, and I think I fall into that bucket you describe. I
have just enough information to be dangerous.

There does seem to be a lack of available tools even for someone with a well-
defined problem with available labelled data to validate implementing machine
learning to a problem. I think that there's a need out there for a software
with a GUI that can accept data and let you try some ML approaches with it and
take the coding out of it - and it doesn't seem like it would be insanely
difficult to build.

~~~
mifeng
Check out outlier.ai. Also I think there was a company in the most recent YC
batch that tackled this problem.

------
ahousley
Fantastic resource - thanks, a16z!

Do you accept pull requests? I'd like to make a suggestion for the software
and services page:
[http://aiplaybook.a16z.com/docs/reference/software](http://aiplaybook.a16z.com/docs/reference/software)

Also, I think there's an emerging category outside of MLaaS that focuses on
ML/DL deployment (ops).

Here's a quick intro:

Seldon's mission is to help people predict and shape the future with machine
learning. Our products increase efficiency and speed up the time to value for
data scientists:

Seldon Core - a machine learning platform that helps your data science team
deploy models into production. It provides an open-source data science stack
that runs within a Kubernetes Cluster. You can use Seldon to deploy machine
learning and deep learning models into production on-premise or in the cloud
(e.g. GCP, AWS, Azure). Seldon is framework agnostic, it supports models built
with TensorFlow, Keras, Vowpal Wabbit, XGBoost, Gensim and any other model-
building tool — it even supports models built with commercial tools and
services where the model is exportable. Fork Seldon on Github:
[https://github.com/SeldonIO/seldon-
server/](https://github.com/SeldonIO/seldon-server/)

Seldon Deploy - in development - a machine learning deployment platform for
data science teams. Features include workflows with approvals and audit
trails, advanced experiments, continuous integration and model explanations.

Seldon was part of the Barclays Techstars '16 and we’re working with customers
in banking, technology and media sectors.

Disclosure: I’m CEO at Seldon.

------
dchichkov
Does anyone have a list of VCs that are specializing in investing into AI
startups?

I'm looking for a seed and it'd be great to have a list like that (I'm
experienced in the field, first started with Deep Learning in 2010, it's on my
GitHub...)

~~~
lpolovets
There are tons: Data Collective, Array Ventures, Susa Ventures (my fund),
Switch Ventures, Amplify Partners, Zetta Venture Partners, AME Cloud Ventures,
Ten One Ten, Bloomberg Beta, etc.

That's not an exhaustive list by any means, just some of the first ones that
come to mind for me. To be honest, I think the majority of seed funds are
interested in AI/ML startups these days. Which one is the best fit for you
will depend on the specific flavor of company you're building, your location,
your level of traction, et cetera.

If you have any questions, you can email me at: leo at susaventures.com.

~~~
robbomacrae
Rob May states that he is an active investor in the A.I. space in his
excellent AI roundup newsletter: Inside AI (formerly Technically Sentient).

------
fiatjaf
I can't find an use to AI in any of my projects. Am I stupid?

~~~
saagarjha
No, of course not. You're just doing work that computers do well.

~~~
rs86
Crude but subtle in very subtle and insightful way!

~~~
bbcbasic
I like the twist

------
annnnd
Judging by contents, this should really be titled "ML playbook", but I guess
this doesn't draw that many clicks... :)

------
stared
Term 'Artificial Intelligence' is vague.

"Deep learning is a name for machine learning techniques using many-layered
artificial neural networks. Occasionally people use the term artificial
intelligence, but unless you want to sound sci-fi, it is reserved for problems
that are currently considered “too hard for machines” - a frontier that keeps
moving rapidly." from [http://p.migdal.pl/2017/04/30/teaching-deep-
learning.html](http://p.migdal.pl/2017/04/30/teaching-deep-learning.html)

In particular, two decades ago, letter recognition was considered a holy grail
of AI, see:

"Indeed, I once even proposed that the toughest challenge facing AI workers is
to answer the question: “What are the letters ‘A’ and ‘I’?" \- Douglas R.
Hofstadter (1995)

Not it as simple assignment to anyone learning ConvNets (BTW: I really
recommend notMNIST as a starting point).

~~~
ska
I've seen that Hofstadter quote used in this context before, and always
assumed it was being quoted out of context. Do you know where it comes from? I
suspect the idea is related to his "Letter Spirit" project - if so approaches
like notMNIST don't really speak to it.

Hofstadter often wrote about the difficulty in semantic understanding, and how
AI approaches often approached things in such a superficial way that they miss
the deeper meaning by design. Many of his projects were aimed at taking on
these much harder problems head on. I suspect this quote, while also being a
play on words, is aimed at this deeper understanding.

~~~
stared
I give some context in the blog post linked above.

This quotation comes from
[https://web.stanford.edu/group/SHR/4-2/text/hofstadter.html](https://web.stanford.edu/group/SHR/4-2/text/hofstadter.html).

And yes, he expected machine to have some 'deep understanding' of what is a
letter. My approach (partly in the lines of Dennett, partly my experience with
deep learning) is that we overestimate human cognition, which may be more 'a
bunch of heuristics' than something 'really deep'. Especially, as some
problems can be solved by... pigeons:
[http://www.sciencemag.org/news/2015/11/pigeons-spot-
cancer-w...](http://www.sciencemag.org/news/2015/11/pigeons-spot-cancer-well-
human-experts).

~~~
ska
In that context, it makes no sense to claim that convolutional NN have solved
his problem at all, let alone claim it is a simple assignment.

It is interesting to ask if he was asking the right question or not, but
misleading to trivialize it.

~~~
stared
Well, it is exactly what I meant this "artificial intelligence" is line that
changes. First we consider something some problem too hard for machines. Then,
once we solve it - it's "well, but it is just a singular case".

In any case, what do you mean by "the right question"?

------
monkeydust
For a business guy this is a very useful resource, nice and clear. Can
understand some of the criticism but perhaps people are expecting too much.

I specifically liked the Image Feature Analysis tool and the fact this is
exposed for others to use on github. Thinking about linking it to my home
security camera.

------
thr0waway1239
Collect underpants, er portfolio of AI companies

???? (release a self-anointed playbook)

Profit!

------
smefi
I found this quite frustrating. It gives the impression that AI is "plug and
play".. well it's not. :)

------
amelius
Deep Learning != AI

Deep Learning is more about pattern matching, not about "understanding"
patterns.

------
JabavuAdams
Ok, now it's a bubble.

~~~
rs86
100% agree

------
baxtr
Oh man. I wish back the old days where AI would be short for Advanced
Industries... ;-)

~~~
robotresearcher
For people who work with animals, it's Artificial Insemination. I worked with
robots and animals and had an intense code-switching problem.

------
beders
Site is incomplete. Misses out on whole categories of software classified as
AI (like Cognitive Computing with approaches like associative memory). Sad.

