
Linear Algebra for AI - stablemap
https://github.com/fastai/fastai/blob/master/tutorials/linalg_pytorch.ipynb
======
aqsalose
"All the Linear Algebra You Need".

The title is misleading. In general, I like an applications-first approach,
but I have hard time liking a tutorial with the title like that and which
proceeds to say things like

>Neural networks consist of linear layers alternating with non-linear layers.

without providing a proper definition nor intuition what is linear and what is
not. Next step they are constructing NN layers of ReLU's and not telling what
is a rectified linear unit and only barely hinting it is supposed to do ("non-
linearity").

The article is not a worthless, it's a nice tutorial to building a NN
classifier for MNIST, but don't expect full mastery of the mathematics
relevant in understanding NNs after reading this tutorial.

Chapter 2 of the book by Goodfellow et al. is a serviceable concise summary of
the relevant concepts, but I don't think that chapter alone is a good primary
learning material if you are not already familiar with the subject. For that
I'd recommend reading a proper undergrad level textbook (and doing at least
some of the problem sets [1]), one example:
[http://math.mit.edu/~gs/linearalgebra/](http://math.mit.edu/~gs/linearalgebra/)
and then continue with e.g. the rest of the Goodfellow's book.

[1] There's no royal road into learning mathematics. Or for that matter,
learning anything.

~~~
jph00
It's the notes for a 40 min talk Rachel will give at the O'Reilly AI
conference. It was originally meant to be a longer tutorial, so the scope had
to be cut down significantly, whilst the title remained. There's lots of links
in the notebook to additional resources with more background info.

Having said that, there really isn't much more linear algebra you need to
implement neural networks from scratch. You'll need convolutions of course,
although that's not too different from what's shown here.

For those interested in much more detail, Rachel has a full computational
linear algebra course online [http://www.fast.ai/2017/07/17/num-lin-
alg/](http://www.fast.ai/2017/07/17/num-lin-alg/) . Most of that isn't needed
for most deep learning, however.

------
jwdunne
Hrm. I find a lot of stuff on linear algebra not so intuitive or motivating.
It's a bit harder to get through. A big motivator is the sheer scale of
application, along with calculus. Yeah it's a must for AI but really linear
algebra underpins much much more and gives you awesome problem solving tools
in general.

As for intuition, a better approach may be to get 3 good books and cycle
through them. Where one doesn't feel so intuitive at a given point, the other
2 might. They will explain things in a different way with different examples.
Connecting the ideas across 3 sources will help it sink in.

For a light intuitive introduction, give this a try:

[https://betterexplained.com/articles/linear-algebra-
guide/](https://betterexplained.com/articles/linear-algebra-guide/)

Maybe it's a little too light but the idea is important: voraciously grasp the
ideas intuitively. If none are forthcoming, try to make one.

Another Linear Algebra book, by Heffernan, was a 'third' book:

[http://joshua.smcvt.edu/linearalgebra/](http://joshua.smcvt.edu/linearalgebra/)

It's free too. I think someone else has mentioned No Bull Linear Algebra.
That's a cool book too.

I had to put my learnings aside a few months ago but I made more progress this
way than just following one book.

------
ivan_ah
More linear algebra stuff to read with you weekend morning coffee:
[https://minireference.com/static/tutorials/linear_algebra_in...](https://minireference.com/static/tutorials/linear_algebra_in_4_pages.pdf)

See also Section VI in this SymPy tutorial
[https://minireference.com/static/tutorials/sympy_tutorial.pd...](https://minireference.com/static/tutorials/sympy_tutorial.pdf)
or LA topics covered with numpy [http://ml-
cheatsheet.readthedocs.io/en/latest/linear_algebra...](http://ml-
cheatsheet.readthedocs.io/en/latest/linear_algebra.html)

------
hal9000xp
Is there any book which actually explains where matrix and its rules come
from? Instead of throwing on you matrix multiplication rules in dogmatic way
so you blindly follow them like mindless robot?

I know there are lectures in YouTube from _3Blue1Brown_ :

[https://www.youtube.com/watch?v=kjBOesZCoqc&list=PLZHQObOWTQ...](https://www.youtube.com/watch?v=kjBOesZCoqc&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)

So I want a book which covers linear algebra in the similar manner.

~~~
dmit
[http://linear.axler.net/](http://linear.axler.net/)

~~~
jasode
_" Linear Algebra Done Right"_ is a fine book but its enduring popularity
leads people to recommend it as a universal default answer.

The parent asked if there was a LA book that covered the material in the _same
style as 3Blue1Brown 's videos_. If that's the criteria, Sheldon Axler's book
isn't the best book. One can compare a sample chapter to the youtube videos
and realize they use different pedagogy:

[http://linear.axler.net/Eigenvalues.pdf](http://linear.axler.net/Eigenvalues.pdf)

------
backpropaganda
All the linear algebra you need for AI:
[http://cs229.stanford.edu/section/cs229-linalg.pdf](http://cs229.stanford.edu/section/cs229-linalg.pdf)

------
derekmcloughlin
I found that chapter 2 of the Deep Learning book by Ian Goodfellow, Yoshua
Bengio and Aaron Courville is quite good.

[http://www.deeplearningbook.org/contents/linear_algebra.html](http://www.deeplearningbook.org/contents/linear_algebra.html)

------
Yuioup
Thank you for this! I've been re-learning Linear Algebra since my question[1]
was answered on an earlier HN thread[2] on fast.ai. This will definitely help.

[1]
[https://news.ycombinator.com/item?id=14878199](https://news.ycombinator.com/item?id=14878199)

[2]
[https://news.ycombinator.com/item?id=14877920](https://news.ycombinator.com/item?id=14877920)

------
auston
This left me with a lot of questions, it could definitely use more
explanations between code examples. I had to do a lot of "reading between the
lines".

I've been reading [https://www.manning.com/books/grokking-deep-
learning](https://www.manning.com/books/grokking-deep-learning) and have been
liking it better than this tutorial.

------
dragandj
_All_ the linear algebra that is needed for AI is matrix multiplication and
broadcasting?

Am I the only one who is baffled by this?

~~~
jventura
If AI == Neural Networks only, maybe matrix knowledge is enough, but when you
start to include NLP, Expert systems, SEarch algorithms such as A*, Minimax,
etc. in the AI category, you'll have to know more mathematics..

AI is much more than neural networks!

~~~
dragandj
Even for NN alone, those few operations are not enough. Not only that, but to
even understand what the operations that I need directly do, I need far more
background knowledge of linear algebra.

------
bra-ket
"Coding the Matrix" : [https://www.amazon.com/Coding-Matrix-Algebra-
Applications-Co...](https://www.amazon.com/Coding-Matrix-Algebra-Applications-
Computer/dp/0615880991)

------
colmvp
I wonder why there's so much more emphasis on Linear Algebra over Calculus
(I've seen a number of courses teaching LA to complement DL courses), given
that without Calculus, it's hard to understand optimization and
backpropagation. Rote memorization might help for copying the same network
over and over but isn't enough when you have to customize it.

~~~
chestervonwinch
How do you know a stationary point is a local optimum? Eigenvalues of the
Hessian matrix! Even though vector calculus is taught without assuming linear
algebra, a lot of the material is coupled. Further, many of the "first steps"
machine learning ideas are essentially linear algebra problems (e.g., linear
regression, PCA, etc).

------
1024core
> The purpose of this notebook is to serve as an explanation of two crucial
> linear algebra operations used when coding neural networks: matrix
> multiplication and broadcasting.

Um... no. While I like Rachel, and she's really smart and all that, this is a
vaaast oversimplification.

------
BenGosub
I have yet to go through the tutorial. LA is one of the areas where I am
trying to improve... Coding the Matrix course/book is well written for those
that find a good/intuitive approach to learning.

------
tw1010
Right now, sure. But give it five years and you'll no doubt also need at least
a years worth of undergraduate (pure) math.

------
methodover
Is there a website that could interpret tbis... iPython notebook, I think it
is? I'd love to read it on mobile.

~~~
c8g
Mobile friendly:
[https://nbviewer.jupyter.org/github/fastai/fastai/blob/maste...](https://nbviewer.jupyter.org/github/fastai/fastai/blob/master/tutorials/linalg_pytorch.ipynb)

View from github URL:
[https://nbviewer.jupyter.org/](https://nbviewer.jupyter.org/)

------
RayVR
First line of fastai imports is a big "Fuck you" to usability. Don't import *.

------
alishan-l
Thanks for sharing!

------
nnfy
I may be biased because I have a background in mathematics, but after working
with tensorflow for about 6 months now, I dont think you really need to
understand linear algebra or multivariate calculus to work with neural nets,
unless you're trying to implement your own engine.

Edit: I dont mean to discourage anyone from learning, as an understanding of
the relatively simple mathematics behind NNs may afford the user an additional
intuition about the behavior of neural nets, but it appears that one can treat
neural nets almost like black boxes, given a suitable engine to work with like
TF.

As an aside, TF is a pretty magnificent library. It pretty much works out of
the box, and in addition to python and CPP bindings, there appears to be an
unnoficial port to c#, although I haven't tried it yet. I strongly recommend
the tutorials at tensorflow for anyone interested in experimenting.

~~~
trentmb
Yeah, you don't need to be an automotive engineer to drive a car.

Most 'work' in AI seems to be using python to shovel data into libraries
anyways.

