
DeepMind AI reading list [pdf] - banjo_milkman
https://storage.googleapis.com/deepmind-media/research/New_AtHomeWithAI%20resources.pdf
======
godelski
This is an odd list. While it includes various topics is includes things I
would expect many to already be familiar with as well as
educational/entertainment sources. While I love 3B1B, I don't understand how
it belongs on a list like this. Similarly things like Lex's AI podcast.

The list also includes very beginner things. Maybe this could be ordered as in
a way to progress through subjects or at least something better than
alphabetical (and have a section for "entertainment" which would include
things like 3B1B, Lex, Robert Miles, etc, which are useful but not hard
literature).

Additionally: Title should be "DeepMind AI __Resource__ List," many items here
are not ones in which you can read.

~~~
stevofolife
I wouldn't easily dismiss 3B1B, Lex or whatever resources you've mentioned
here. Though you might find them to be entertainment but really people learn
in various ways and these resources provide excellent views on the subject.
And honestly with all the recent advancement in visual content and media, I
would reconsider the sole dependence on hard literature to learn or even
question the superiority of hard literature over newer formats.

Lastly, I agree that the title should be resource list.

------
stupidcar
I'd echo one of their recommendations: Robert Mile's YouTube channel focussing
on AI safety. He presents the material in a very interesting and accessible
way. For example, this video on whether or not corporations can be considered
a form of superintelligence:
[https://youtu.be/L5pUA3LsEaw](https://youtu.be/L5pUA3LsEaw)

~~~
longtom
Another great channel is Yannic Kilcher, especially the "papers explained"
series:
[https://www.youtube.com/c/YannicKilcher/playlists](https://www.youtube.com/c/YannicKilcher/playlists)

------
mlthoughts2018
This doesn’t make much sense. If you are not already trained in vector
calculus, functional analysis and basic classifier and regression algorithms,
then most of these reading list items are completely inapplicable (or even
dangerous, like when someone reads some blog posts about slapping together
neural nets in Keras and suddenly thinks they can build a model suitable for
production).

On the other hand if you are trained in ML, most of these are not detailed or
extended enough to give you anything useful. Doing “a hacker’s intro to X”
over and over really, really doesn’t give you any skills. This is particularly
true for deep domains like reinforcement learning, computer vision / image
processing, and natural language processing.

Meanwhile, basic design of experiments for A/B testing, explanatory modeling
and simple regressions is a fraught area. Not understanding extremely rigorous
details about hypothesis testing, model checking, limitations of statistical
significance, etc., can lead to wildly incorrect inferences from poor models
that non-experts will completely fail to detect.

At best this list seems like “special topics to seem trendy in AI without
getting deep / practical insight into any particular area.” There are one or
two minor exceptions in the list.

~~~
etaioinshrdlu
I think, like it or not, the end goal of the AI industry will result in people
with less and less rigorous education creating models used in production. It
may not be solid science, but it doesn't have to be. It will end up changing
entire fields anyways. Especially for less serious applications, no one cares
about rigor.

It's just what happens to a field as it becomes more universally accessible.

~~~
mlthoughts2018
I don’t think this is true. Security primitives are more widely available in
software libraries now than ever before, but you don’t see people believing
they can read a “security for hackers” blog post and then roll their own
encryption tool or secrets management tool and use it in production. More than
ever, the distinction between a security professional and a layperson who read
about RSA algorithms is hugely critical.

It’s precisely the same with machine learning and statistical inference.

The lower barrier to entry just means the danger of releasing extremely unsafe
projects is much higher, and the careful validation from experts is that much
more critical.

~~~
quonn
If the model produces good observable results in practice it works. For
security it‘s different: It may seem to work, yet be completely insecure
(security being the goal).

~~~
mlthoughts2018
Without statistical expertise, how would you know the model produces good
results?

(If you say something like, “just look at the business outcome of the model”
you are proving my point about the danger, because that would be a
catastrophically bad way to judge the performance of a model. What accuracy
metric did you use? Why? Did you understand training / serving skew? Did you
look at a confusion matrix or study class imbalance? What about missing data?
What statistical test did you use? Did you adjust for multiple-testing? Was
there peek-ahead bias? Did you test for discontinuities and non-linearities
that can render p-values inapplicable for tests of linear models? What was
your training convergence like? What simple baselines did you compare to?)

~~~
quonn
No, it‘s not. Most machine learning is used for practical applications where
it‘s easy to judge if it works.

For example, if I provide some automatic assignment of let‘s say a related
object, reducing the work required to find the correct one and the team using
it is much faster, then I can measure this and that‘s enough. Likewise for,
let‘s say a recommender system. The BI department will look at conversions and
that‘s in fact the only thing that matters. Nobody cares how good the model
is. What we care about is whether the recommendations have the desired effect
in that case.

And besides, what‘s hard about measuring how accurate a model is for
predictions? Compared to most things in a typical CS curriculum that‘s rather
easy indeed.

Finally, what‘s „dangerous“? ML is usually applied in a business context.
Better recommendations; automatic assignment of objects. Automatic
classification of pictures. Things like that.

Sure, in a medical context perhaps or financial. Then it may be dangerous. For
everything else, maybe you get a worse result. That‘s not unlike everything
else in software engineering where many people with different skill levels get
very different results (some better some much worse).

~~~
zwaps
Careful with this reasoning!

I have this anecdote of a industrial sorting/production machine that worked
for literally a year until someone opened the door on a windy day and
everything went flying thanks to the AI.

------
inetsee
Does anybody know where I might find a copy of these resources that are
readable without zooming to 300%? It's like trying to read a pdf on a phone.

------
raptortech
Lots of excellent introductory sources here, but I was hoping for papers!

------
doublesCs
Obvious submarine article / PR. The goal is to signal how concerned with
ethics DeepMind is.

~~~
galimaufry
This illustrates the problem with calling out virtue-signalling. I see two
references related to ethics. It seems the only way to avoid accusations of
virtue-signalling would be to have 0 references to AI fairness.

In practice, comments like this encourage self-censorship, even if that is not
their intention.

~~~
mlthoughts2018
AI fairness is really, really bogus as a research field.

Understanding and reducing negative impacts of bias is important, but the
current research field of AI fairness does not do anything like that and has
not yet reached any state of maturity where it can be considered a serious
subset of research at all.

I would really say even one mention of it on a list like this is purely to do
virtue signalling.

It’s the same for “explainability” of models too, another totally bogus field
that gets treated as being worthy of attention or societal prioritization
purely due to politics.

~~~
currymj
it is very strange to read someone claiming with extreme confidence that all
fairness and explainability research is completely bogus.

of course there are a lot of papers of questionable value, but that's a
problem with machine learning more broadly -- probably all academic science,
really.

lots of institutions (banks, medicine, etc.) really do need explainable
models. in practice this often means linear models with simple coefficients or
shallow decision trees. there is plenty of useful work on learning these while
maintaining performance, or "distilling" them from more complicated models. i
know for a fact some explainable models learned with these techniques do
actually get used in real life.

likewise with fairness -- end-users actually do care about fairer models in
all kinds of areas, especially lending and insurance. there's a ton of
frustrating debate about how to operationalize fairness in different settings
but it seems like there is actually progress on this front.

what do you find to be bogus about these research areas?

~~~
mlthoughts2018
Linear models with simple coefficients can often be some of the least
explainable models, particularly when the assumption of linearity breaks down.
[0] is a good classic paper on this, demonstrating a simple example where
coding error on the inputs leads to erroneous coefficient estimates that are
both _statistically significant_ and also _of the wrong sign_.

Meaning, you would believe the coefficients reflect a real relationship
between the covariate and the target, and even could claim it’s statistically
significant, and yet the actual relationship to the target is _of the opposite
sign!_ Any further feature importance scoring based off the coefficient
estimates would then become catastrophically misleading.

Meanwhile, a model like support vector regression on the same data is capable
of automatically handling the non-linearity, at the expense that there’s no
more such thing as a coefficient breakdown in the linear space of input
features.

Does this make it less “explainable”? That would make zero sense. How can it
be worse at explaining a data generating mechanism when it is better at
predicting that same generating mechanism. What could it possibly mean to
explain something you can’t predict or replicate?

The field of “explainable” models doesn’t even make the slightest attempt to
address this stuff - it just beats up on models that are arbitrarily labeled
as “black boxes” (what does that mean?)

If a given model X predicts or replicates a data generating process better
than Y, then X explains the process better than Y, period.

An analogy: Newtonian physics is not “more explainable than” quantum
mechanics. Newtonian physics is just more _wrong about how the world works_
than quantum mechanics.

[0]:
[http://www.saramitchell.org/achen04.pdf](http://www.saramitchell.org/achen04.pdf)

~~~
currymj
i think "explainable model" literally means "a model you can explain to
people".

You may have to explain your decision to a judge or customer after the fact,
or you might even be asking a layperson to actually compute predictions
manually (as is sometimes done in psychology and medicine for things like
triage).

the question of whether or not the model provides a good explanation for the
data-generating process is a distinct one. I think the Achen paper makes a
good point that people cannot safely turn linear regression coefficients into
stories about the world, although it doesn't seem like they've stopped trying.

but assuming you have a way to validate that it makes good predictions (not an
unreasonable assumption), an explainable model can be a useful thing to have.

~~~
mlthoughts2018
You’re totally missing the point.

“I can explain how this model works” is not something you can claim _about a
model._ You can only claim it about a tuple of (model, assumptions, data,
context).

In context A, some simple linear regression might be very “explainable.” In
context B, that same linear model might be totally not explainable (because
the mechanism of coefficients based on the regression’s fitting procedure
might be totally incompatible with other details of the situation.)

The analogy between Newtonian and quantum mechanics still holds.

The fact that you can more easily map Newtonian mechanics onto English words
or pictures absolutely doesn’t make it more “explainable.”

By that logic, saying “a magic wizard did it” would be the most “explainable”
model of all.

This is exactly my point. What is “complex” or “simple”? Arbitrary standards
of natural language words? The field of explainable models has done no work on
this. It starts from some totally arbitrary and confused idea about something
both being an accurate model and an explainable model as if they are separate.

The closest academic topic to making “explainability” a serious subject would
be the philosophy of language and connection to computability theory, like
Kolmogorov complexity, PAC learning, VC dimension, Occam’s razor.

But these are algorithmic aspects of model complexity in the face of a
specific data set, it’s absolutely not some hand wavy “oh but a person could
‘easily’ understand certain verbal acoustic vibrations about this” based on
nothing.

~~~
currymj
i understand your point perfectly well, but I think we've gotten to the crux
of the disagreement, at least.

I'm thinking about this pragmatically: a model is explainable if you can
explain the model to some people you need to explain it to. I mean this in a
fuzzy, "arbitrary", natural language sense -- the test of it is to try to give
the explanation. I don't think metaphysical questions about the nature of
explanation itself are really relevant.

Whether this is true definitely does depend on what data, which people, what
context, but it's possible to try to make techniques that will be generally
useful in many contexts.

Getting into the broader issues, I think Newtonian physics is absolutely more
explainable than quantum physics. For example, I've seen court cases around
car accidents where consultants had to come in and explain basic mechanics
(F=ma, friction, etc.) to the jury without reference to any math; and they did
an okay job of conveying the essential ideas. By contrast I've never seen a
math-free explanation of quantum mechanics that wasn't a complete mess.

Of course, "a wizard did it" is even easier to explain, and may be appealing
for that reason, but ultimately that model will turn out to be totally
useless.

~~~
mlthoughts2018
I still don’t agree that it’s possible to make the distinction you’re trying
to make.

For example, let’s say someone starts asking “why?” every time you make a
statement about the car accident in the court case. If it’s Newtonian physics,
eventually we bottom out at questions that require statistical mechanics to
answer and it falls apart. The “explainable” answer is unmasked as a fiction
that is incompatible with reality.

So then you might say from a pragmatic point of view all that matters is that
the jurors were happy with the explanation. But then who decides what that
metric is? What if Feynman is on the jury? What if a medieval religious leader
is on the jury?

The standard of “explaining it to people” is not a thing.

This just circles back to exactly what I said before, which is that
explainability becomes politics.

You’re trying to say it’s pragmatic, but who gets to control that standard?

Why do banks need “explainable” models? Because of some political fight about
who can regulate them. Now you’re trying to convert that arbitrary political
goal into some type of formalized standard of machine learning models, which
is totally intellectually dishonest and rigged.

If a surgeon uses a computer-vision-assisted robot to save lives, does the
robot need to be “explainable?” Who decides what that even means? Is it a
political standard because of insurance liability? What if a “smart” person
can understand the model but a “dumb” person can’t? What if the model deemed
more explainable by a political oversight committee also saves fewer lives,
thus condemning people to death for the sake of explainability.

I just don’t see any way your appeal to just some “reasonable” or “pragmatic”
idea of “explaining it to people” can be carried to a logical conclusion that
makes any sense.

