
Fundamentals of Linear Algebra and Optimization [pdf] - mindcrime
http://www.seas.upenn.edu/~cis515/linalg.pdf
======
SonOfLilit
My test for linear algebra books is how they first present matrices and matrix
multiplication.

If they define a matrix as an NxM table of numbers with a multiplication
operation defined as this complicated formula with a couple of nested sigmas,
and then much later a lemma is mentioned that says every linear transformation
can be represented as a matrix and then the composition of two transforms is
the matrix multiplication of their matrix forms, I throw the book away in
disgust. I throw most books away in disgust.

This is the second book I've seen that does it right, but unlike the other one
[1] this one wraps the very correct presentation of matrices in so much
technical language and such a boring cover story that I almost threw it away
in disgust anyway.

My greatest wish in STEM education is that we teach linear algebra better.
It's such low hanging fruit and could change so much!

[1] which I saw linked in HN a year ago and don't remember the name of, but it
was something like "linear algebra taught the correct way" and was apparently
well known in the States so ask your friends

~~~
jknz
I would teach linear algebra with a book that barely mentions linear
transformations.

In finite dimensions, linear transformations and matrices are exactly the same
object mathematical objects, with very different notations (matrix notation
(boxes with numbers inside) vs the linear space/linear transformation
notation). I would rather the students to learn deeper mathematics only in
matrix notation, rather than to master less substantial mathematics with both
notations.

Teaching both notations may reinforce the idea that matrices and linear
transformations are different mathematical objects in finite dimension.
Teaching the more abstract notation is mainly useful in infinite dimension
(Hilbert spaces).

~~~
btilly
This is exactly backwards.

Whether you're discussing the Jacobian of a function, or change of basis
matrices, learning the matrix formula is a lot less useful than seeing how it
falls out of the linear function definition.

The formula is hard to memorize and gives no intuition for why anything is
true. But from the linear function definition it is easy to reconstruct the
formula.

In fact this is so true that I would say that anyone who only knows the matrix
definition does not actually understand linear algebra.

------
hvs
I may be a troglodyte, but I was bored senseless in my linear algebra,
calculus, discrete mathematics, statistics, etc classes in college. Then they
tag all the interesting stuff like AI (we didn't call it Machine Learning back
in the 90s) at the end of your major where you actually use it. I had to go
back and relearn it all because I didn't pay attention the first time. Note to
mathematicians (or at least math professors): lead with the interesting stuff
before the theory. It'll make learning easier and more fun.

~~~
wolfgke
> lead with the interesting stuff before the theory. It'll make learning
> easier and more fun.

As a mathematician I say:

What is considered as interesting is different for each person. I, personally,
for example deeply love this really abstract stuff (but of course I am aware
that other people have different preferences). So I would say even finding
"interesting stuff" that many students in the lecture hall might be interested
in is really, really hard.

Another important argument against your idea is: To be even able to formulate
the ideas from AI, one first have to learn and understand the words of the
language in which one will formulate this. And these words are like "vector
space", "linear map", "tensor product" etc. and understanding their meaning
means knowing theorems about them. Starting with advanced topics, such as your
AI example, is like giving beginner language learners a really advanced text
in the foreign language that they just begin to learn. In other words: A
reallz dubious idea.

Considerung your post I can only ask you why you did not have a talk with the
course advisor of your faculty. He would immediately have told you why these
lectures are important for the things that you are actually interested in.

------
jacobolus
Folks looking for introductory books in this area may also enjoy this new in-
progress book:
[https://web.stanford.edu/~boyd/vmls/](https://web.stanford.edu/~boyd/vmls/)

As a follow-up I would recommend Trefethen & Bau, _Numerical Linear Algebra_.

~~~
JustFinishedBSG
Boyd and Vandenberghe will have a monopoly in anything optimization related if
they keep on...

------
alexcnwy
This is great!

There's so much hype about machine learning but so few people seem to
appreciate the fundamental importance of linear algebra (especially in ML).

This fast.ai course is also great: [http://www.fast.ai/2017/07/17/num-lin-
alg/](http://www.fast.ai/2017/07/17/num-lin-alg/)

~~~
zengid
Cool new course! I'm thinking about taking their Practical Deep Learing for
Coders course at the moment: [http://course.fast.ai/](http://course.fast.ai/)

~~~
edanm
If you're at all interested in actually doing deep learning, then do the
course! It's by far my favorite resource for actually learning to do deep
learning.

Of course it won't give you the math background, but you can abd should pick
it up after.

------
avalonalex
I took Prof. Gallier (the author of this document)'s class when I was a
graduate student at Penn.
[http://www.cis.upenn.edu/~jean/home.html](http://www.cis.upenn.edu/~jean/home.html)

It is one of the class I enjoyed the most when I was there. Some of his other
books are quite enjoyable as well.

Dr. Gallier's PhD work was focused on logic, but his current research has a
lot to do with computer graphics and computational geometry. He spent a long
time study graduate-level algebraic geometry and algebraic topology while
having the duty to be a professor at computer science department. His stories
is always an inspiration for me to learn more.

------
dragandj
If you're into Clojure, there is a fast linear algebra library Neanderthal
(I'm the author) that covers both GPU and CPU, is easy to use, and comes with
lots of tutorials that cover a linear algebra textbook chapters, as well as
more advanced numerical computing uses (solvers, factorizations, etc.):

The list of guides is here:
[http://neanderthal.uncomplicate.org/articles/guides.html](http://neanderthal.uncomplicate.org/articles/guides.html)

------
macawfish
I'd recommend watching these videos for some motivation:
[https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2x...](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab)

And remember, linear algebra has broader application than just geometry!

------
glbrew
Is this going to be a textbook? I can't find it in print, but it looks like a
great text.

~~~
mindcrime
Good question. I'm honestly not sure. I just stumbled across it while looking
for linalg resources. It appears to be legit, since the author is with UPenn
and it's on a upenn.edu site. But I don't see an explicit license notification
or anything, so not sure if you're even allowed to print it out yourself or
not.

~~~
mpowell94
Was in his class at Penn, he gives all this stuff away with no issue. Didnt
have to pay anything for course materials and its all open to the public.

------
catnaroek
Did fields other than R and C eat the authors' children or something?

~~~
jacobolus
Often linear algebra tools are applied to Hilbert spaces or Banach spaces,
where fields other than R or C don't make much sense.

Looking at page 25 of TFA:

> _In Definition 1.2, the field R may be replaced by the field of complex
> numbers C, in which case we have a complex vector space. It is even possible
> to replace R by the field of rational numbers Q or by any other field K (for
> example Z /pZ, where p is a prime number), in which case we have a K-vector
> space (in (V3), ∗ denotes multiplication in the field K). In most cases, the
> field K will be the field R of reals._

~~~
catnaroek
Yeah, lots of courses pay lip service to the existence of fields of
characteristic p during the initial few weeks, then proceed to ignore them
completely.

~~~
jacobolus
How relevant do you think finite fields are for solving applied optimization
problems? What proportion of a course do you think should be devoted to them?

~~~
catnaroek
Every ordered field has characteristic 0, so I don't think finite fields are
likely to be terribly useful for solving optimization problems. But:

(0) There is way more to linear algebra than linear programming.

(1) A linear algebra course is not the right place for an extensive study of
how to solve optimization problems anyway. That belongs in a real analysis
course.

~~~
jacobolus
This is a graduate-level course in the department of computer and information
science.

> _Prerequisite(s): Undergraduate course in linear algebra, calculus_

> _The goal of this course is to provide firm foundations in linear algebra
> and optimization techniques that will enable students to analyze and solve
> problems arising in various areas of computer science, especially computer
> vision, robotics, machine learning, computer graphics, embedded systems, and
> market engineering and systems. The students will acquire a firm theoretical
> knowledge of these concepts and tools. They will also learn how to use these
> tools in practice by tackling various judiciously chosen projects (from
> computer vision, etc.). This course will serve as a basis to more advanced
> courses in computer vision, convex optimization, machine learning, robotics,
> computer graphics, embedded systems, and market engineering and systems._

------
Nydhal
You had me at Bourbaki.

