
An Intuitive Guide to Linear Algebra - Zolomon
http://betterexplained.com/articles/linear-algebra-guide/
======
btown
As an electrical-engineer turned machine-learning-grad-student, linear
transformations have been involved in most everything I do since my first year
of undergrad. But all this time, I've done matrix multiplication the way I was
taught in high school: "The (i,j) element of AB is what you get by walking
right across the i'th row of A while you walk down the j'th column of B,
taking the sum of products as you go."

It works, but there's no connection between that process and the intuition of
a linear transformation; it's just a rote computation. And checking a long
string of matrix multiplications to see if they intuitively make sense
(shouldn't everything intuitively make sense?) is especially aggravating when
you constantly have to interrupt your intuition to switch to a rote
calculation.

I never thought to think of the columns of B as vectors that physically travel
through A; to think of a dataflow or pipeline from right to left on the page.
Sure, it's not a cure-all, but it'll be a useful mental tool to have.

Oh, and it's also an excellent introduction to the subject, although the
Linear Operations section gets a bit muddled... first something's not a linear
operation, and then it is, wat? Still, an excellent post.

~~~
wging
You might get some mileage out of thinking of a row in the matrix product as a
dot product, with the intuition that goes along with that. The dot product is
perhaps easier to tie into the geometric intuition you have for linear
transformations.

------
valgaze
Strang from MIT has an extraordinary set of lectures (his textbook is even
better): [http://ocw.mit.edu/courses/mathematics/18-06-linear-
algebra-...](http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-
spring-2010/video-lectures/)

~~~
jackfoxy
Can't say enough good things about Prof. Gilbert Strang <http://www-
math.mit.edu/~gs/> He is one of my heros, and yeah, I have copied his Linear
Algebra lectures to CD so I can speed them up, stop them, and in case they
ever disappear from the net. Not to take away from the OP, who I think has
some good ideas for making this stuff intuitive.

------
klodolph
> Linear algebra gives you mini-spreadsheets for your math equations.

Okay, that lets you visualize it (in the finite case) but it's a terrible way
to sell it. Spreadsheets are booooring. Did you know that functions are
vectors? Okay, better. Did you know that quantum mechanics is all about linear
algebra? Okay, sold!

1\. Almost any time you work in more than one dimension you will want linear
algebra in your toolbox. There are a zillion methods for solving (non-linear)
equations out there, and in more than one dimension, they use linear algebra.
Newton's method? Incredibly useful in practice due (quadratic convergence
rocks!), and with some linear algebra sauce BOOM you have Newton's method in
as many dimensions as you can sneeze at.

2\. Oh, by the way... did you know that the Fourier transform is linear?

3\. Back to quantum mechanics... there's a thing you can do with a linear
operator (a matrix is a kind of linear operator) where you get the "spectrum"
of the linear operator. It's useful for making sense of big matrices. But in
QM, the wavefunctions for electrons are described as eigenfunctions of a
linear operator, and taking the "spectrum" of the linear operator gives you
the actual spectrum of light that the chemical under study emits. Hence the
name, "spectral theorem". It may be linear algebra on paper, but it's laser
beams and semiconductors in the real world.

4\. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some
other time..

5\. It's hella useful for modeling. Any model is wrong, but Markov processes
are useful. Say you run an agency that rents out moving vans, and you have
facilities in 30 cities. Vans rented in city A have a 10% chance of being
dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how
long till you run out of vans in city F? It's a differential equations problem
with like 30 different equations! Or you could rewrite it as a single equation
with matrices. You'll end up with weird things like 'e^(A*t)' where A is a
matrix, and you thought "no way I can exponentiate to the power of a matrix"
but spectral decomposition is like "yes way!" and you can solve the equation
by diagonalization. Radical! (Basically, linear algebra rescues differential
equations from the pits of intractability. I'm using rental vans as an
example, but it could be a chemical reaction or a nuclear reaction or a
million subway riders or whatever you want.)

So the question is:

Do you find economics, quantum physics, chemistry, engineering, classical
mechanics, machine learning, statistics, etc. useful?

Then get some linear algebra in you!

~~~
marshallp
Hate to do this (but will have to anyway because you're misrepresenting that
authors article),

\- you've mixed in with nonlinear phenomena (most physical processes) with
linear, and there's less and less reason to pretend linearity with increasing
computer speeds (although in fairness you alluded to models being wrong)

\- poo pooed spreadsheets without explaining why (you alluded to QM etc. but
they don't care about useless proofs in linear algebra unless they're pencil
pushing time wasters)

\- implied infinite-dimensions is practically useful when it isn't (unless
you're a mathematician/theoretical-somethingist seeking to extract tax-payer
money)

There's little need to get linear algebra in you unless you want to waste
time.

~~~
klodolph
Hate to do this, but it seems you just don't understand jack about linear
algebra. I'm not even sure you read what I wrote.

1\. Yes, I mixed in linear and non-linear processes. My point is that you use
linear algebra even when working with non-linear processes.

2\. Yes, I poo-pooed spreadsheets as a bad way to market linear algebra (I
thought that was clear?)

3\. Yes, I implied infinite dimensions is useful. Linear algebra in QM tends
to be infinite-dimensional.

I'm reading a lot of hostility towards mathematics in general and linear
algebra specifically (you've put a bunch of toxic comments all over this
thread), and I'm not really sure why.

~~~
psykotic
> Yes, I implied infinite dimensions is useful. Linear algebra in QM tends to
> be infinite-dimensional.

You don't have to appeal to QM for their usefulness. Function spaces are
important in applied and computational mathematics because of their use in
understanding integral equations (the theory of integral equations before
Hilbert spaces was a giant mess), partial differential equations, calculus of
variations, approximation theory, etc. No doubt marshallp will respond that
this is all bullshit because it ultimately boils to finite processes running
on finite state machines, where there are no infinite sets in sight, let alone
infinite dimensions. The equivalent approach to physics would be an extreme
form of empiricism, banning the use of concepts like electrons (as some
logical positivists actually proposed to do in the early twentieth century)
and requiring all physical laws to be stated in terms of directly observable
phenomena, whatever that means.

~~~
klodolph
Or put another way, I think infinity is a good approximation to "very large",
and big things are everywhere.

~~~
eru
Not only very large, but also to: Made of many small components.

------
btilly
I hope that this matches how some people think enough to help them.

For me it is too computational. I prefer understanding the topic from first
principles as described at <http://news.ycombinator.com/item?id=4086325>.
(Then again I don't particularly like spreadsheets either.)

------
lomendil
I found "Linear Algebra Done Right" to be a much more intuitive introduction
to the subject. It doesn't get to determinants until the end.

<http://linear.axler.net/>

After going through that course I finally understood things like eigenvectors,
null spaces, and projections. Now I see them everywhere (unless you think
that's a curse)

------
hdivider
Funny, I'm actually working on a game that intends to teach some of this stuff
to a non-mathematical audience.

Linear algebra is so far-reaching, I find it surprising that other branches of
mathematics often seem to get preferential treatment (usually normal algebra
and geometry), in spite of the fact that linear algebra is both:

a) fairly advanced (i.e. not often taught in school, at least not the deeper
stuff)

b) not very difficult to learn (unlike lots of other 'introductory' topics in
mathematics).

Perhaps there is something about matrices (being mere tables of numbers for
most folks) that people find unattractive, almost statistics-like.

(On the other hand, it could be a simple extension of the symbol barrier [1],
given those long vertical brackets.)

[1] Prof Keith Devlin introduces this concept here:
[http://profkeithdevlin.org/2012/02/22/how-to-design-video-
ga...](http://profkeithdevlin.org/2012/02/22/how-to-design-video-games-that-
support-good-math-learning-level-3/)

------
elteto
"The eigenvectors are the axes of the transformation" = mind blown. After
several engineering courses, studying eigenvectors in an advanced math class
and still _no one_ could put it this simple. This guy is amazing.

~~~
textminer
His description of the determinant, too. I hadn't heard that explanation until
a second semester of real analysis, when learning the proof of the Inverse
Function Theorem (an amazing thing to study, by the way, connects many the
dots between linear algebra and calculus).

Even then, it was a question I had to ask my brilliant, constantly-pissed
looking young professor. "Hey, uh, the Jacobian... what does the determinant
mean, uh, geometrically?". He looked at me like a slug, before explaining it
was the measure of the newly mapped unit square. Fireworks went off in my
head. Two linear algebra classes before only ever explained it by its
algorithm or its usefulness (e.g., ∃ A^(-1) for A \in R^{n,n} iff det(A) != 0)

Side note: that professor had the most effective teaching style for pure math
I've ever seen. Besides lectures that expanded on the contents of Rudin and
interesting problem sets, he gave us a list of a hundred theorems,
propositions, and exercises. Told us the final exam would be six problems,
four of which would come from that list, another of which would be a clever
new one, the last something truly hard.

Never learned analysis better than when sitting down and working through (not
memorizing) each of those proofs and theorems for possible later
recapitulation.

~~~
kens
Determinant as volume of the transformation is cool, but I still don't "get"
the determinant intuitively despite many years of math. In particular, why
does the determinant work to solve linear equations (i.e. Cramer's rule)? And
what's the motivation behind the formula for the determinant? (I realize these
questions are a bit vague, but I'm hoping for a more intuitive answer than
"that's just the way the math works out".)

~~~
jules
One approach to this is geometric algebra. I couldn't find a good reference
that explains it intuitively, but there's this:
[http://en.wikipedia.org/wiki/Comparison_of_vector_algebra_an...](http://en.wikipedia.org/wiki/Comparison_of_vector_algebra_and_geometric_algebra#Matrix_Related)

------
eru
What the author uses as his strawman is a linear algebra course for engineers.

As mathematicians, we didn't do any of this matrix and vector stuff with
numbers when introducing linear algebra in university. There were a bunch of
axioms, and you proved things. That's how you know.

What the author sees as abstract "(2d vectors! 3d vectors!)" was way more
applied than the stuff we dealt in.

But, granted, the purpose wasn't learning about how to get mini-spreadsheets
for equations. It was about how to rigorously navigate a useful axiomatic
setting.

(Later on, we proved that you can find a base, and write down your linear
transformation as a bunch of numbers and call that a matrix; same for points
and vectors. But we always saw that as somewhat ugly, and anyway limited to
the finite dimensional case.)

~~~
marshallp
Are there practical applications for proving theorems? It seems like it's the
full employment act for pencil pushers.

edit: downvoters downvote instead of providing refutation. A lot in common
with fundamental religionists.

~~~
klodolph
If you can't prove a theorem, then it's not a theorem, it's a conjecture.
Conjectures are not as useful. For example, the Shannon-Hartley theorem tells
us under what circumstances it is _possible_ to decode a radio signal. That
theorem lets us design, e.g., cell phone networks.

Theorems are useful in all sorts of unexpected ways. At the turn of the 20th
century group theory was considered a worthless corner of mathematics, but
understanding group theory allows modern chemical instrumentation to work.

edit:

> It seems like it's the full employment act for pencil pushers

> downvoters downvote instead of providing refutation. A lot in common with
> fundamental religionists

You've proven the downvoters right by piling one ad hominem attack upon
another.

~~~
marshallp
The example you've described (radio signals) is just a method of winning an
argument. It doesn't actually help you, you practically do it by writing a
computer program that tries out different parameters to get the right one.

~~~
klodolph
I'll explain in more detail.

Let's say you need a communications channel with a 300 Mbit/s capacity.
Shannon's theorem lets you know what kind of parameters make that possible --
namely, bandwidth and signal-to-noise ratio. From there, I can make an
informed decision about the entire signal chain. Once I choose the bandwidth,
the theorem tells me the SNR so I can assign a noise budget to each of the
components. If we assume that the noise is Gaussian, we can calculate total
noise level as the root sum of squares of the noise levels of the individual
components (don't forget to multiply by the gain). I can look at an amplifier
IC's spec sheet and immediately say, "that's too noisy" or "that's
overdesigned, I think I can get something cheaper".

And thanks to theorems we got from the field of real analysis and statistics,
we know that the sum of Gaussians is itself Gaussian. So rather than sticking
two components together and measuring the noise, or running a computer
simulation, I can simply square, sum, and square root.

Each of these theorems reduces the amount of work necessary -- whether by pen
and paper or by computer -- by an enormous factor. But if you can describe in
similar detail how the computer program would work, I'll acknowledge your
greatness.

~~~
marshallp
That's just a few parameters that you avoided having to tune, a few
milliseconds of computer time. I'm not great, just questioning basic
assumptions that have been handed down from a time before computers were
around.

~~~
klodolph
You keep on asserting that the task -- without domain knowledge of Shannon's
theorem -- is solvable by computer. Can you describe how such a computer
program would work? I'm unconvinced that the computer program would get a
reasonable result before you run out of money paying for it.

~~~
marshallp
The Shannon theorem is basically voided by compressive sensing. But all of
this (shannon+compressive) was a waste of practical people's time anyway. It's
justification for pencil pushers who don't want to do real work.

~~~
karategeek6
[http://www.youtube.com/watch?v=1orMXD_Ijbs&feature=relat...](http://www.youtube.com/watch?v=1orMXD_Ijbs&feature=related)

------
ced
Terry Tao has a great set of lecture notes available online, too.
<http://www.math.ucla.edu/~tao/resource/general/115a.3.02f/>

~~~
achompas
This should be at the top of this HN thread. A shame that it's buried down
here--these look extremely valuable.

Thanks for the link!

------
gtani
These're good too

[http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-
xm...](http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xml-
latestli14.xml)

<http://joshua.smcvt.edu/linearalgebra/book.pdf>

Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and
Friedberg/insel/Spence) can be found easily for cheap, used. The old edition
of Strang I used to have was good too, but some people react really strongly
when you bring it up. There's lots of ways ot sequence LA and the needs of
EE's, econometrics/game theory, prob/stats and applied mathies are different
from physics/math majors. Look at ToC's and read the Holyoke prof's writeup:

[http://www.amazon.com/Introduction-Linear-Algebra-Fourth-
Gil...](http://www.amazon.com/Introduction-Linear-Algebra-Fourth-
Gilbert/product-reviews/0980232716/)

(i _think_ they're 1st year texts, my Dad's a physics prof and he started
talking about determinants around 7th grade)

------
mturmon
Nice work. It's a richly geometric area.

Other things that might be nice to include are dot products (projecting one
vector onto another as a measure of co-linearity) and rotation matrices (you
could keep it to 3x3).

------
verroq
I think it's much clearer to define everything in terms of vector spaces and
linear operators.

~~~
backprojection
I entirely agree. People tend to run away from abstract math, with the
impression that it's difficult. But the whole point is that a more abstract
view of these things makes them easier to understand, not harder.

~~~
gizmo686
The problem is that abstractions are designed to make hard problems harder.
Someone who has never seen the type of problem that would make a given
abstraction useful would not be able to understand it easily. If you work
you're way up, then it becomes obvious.

~~~
adeelk
> Someone who has never seen the type of problem that would make a given
> abstraction useful would not be able to understand it easily.

Whether or not someone is sufficiently motivated to study something should not
affect their ability to understand it. Vector spaces and linear maps are
actually much easier to understand, and I think that at some point in
mathematics (probably your first analysis course) your motivation has to come
from the beauty of the theory itself rather than some real-world application.

~~~
marshallp
So mathematicians seek beauty instead of utility.

Should they deserve to starve like artists then?

Should taxpayers fund disciples (students) for them?

If yes, than should taxpayers be forced to fund students for any cult anyone
wishes to start?

~~~
nerme
There is nothing quite like trolling a bunch of nerds with the opinion that
the entire study of pure mathematics has been a wasted endeavor for mankind...

You deserve a round of applause!

(Also, mods, where are you?)

~~~
marshallp
Newborns don't come out with all knowledge in their heads already. I'm simply
alerting younger people to facts, there's absolutely no reason to get mods
involved. Otherwise most of HN should be banned (since a lot of it is not
completely fresh news). Why alert the mods, are you a scared pure
mathematician?

------
textminer
Great post. A suggested follow-up: "Banach Spaces, The Spectral Theorem, and
Your Changing Body: A Personal Introduction."

------
sadga
I think an intuitive guide should have more diagrams of affine
transformations. Don't just visualize the matrix, visualize the
transformation.

------
dsrguru
Eero Simoncelli's guide to linear algebra is very good too and likely more
concise, although it doesn't cover eigenvectors and eigenvalues.

<http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf>

------
soapdog
For the mathematicians here. I am taking Linear Algebra classes as a CS
undergrad. Can someone recommend a very good book?

I am looking for the kind of book that will make you fell in love with Linear
Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus
which was miles away from what my classmates were using. That along with Maple
help to double check that my stuff was correct proved a good combo. My current
Linear Algebra book is an honest book but it is a boring book, it fails to
entertain or to amaze or to give you those moments of insight that puts a grin
in your face.

I think I just wish I had better books as an undergrad :-(

------
pfortuny
OK can you explain kernels and their role in transformations (f.e. the
dimension formula)? No you can't this way. And it truly has an INTUITIVE
explanation (so much more than the intuitive ess of a spreadsheet!).

~~~
marshallp
What's the practical use for understanding that intuitively (for the 1000s of
people who have to go through it every year)? Genuinely curious.

~~~
pfortuny
Why there are things that, for example, block other things when watching TV
(2d TV) and where they are. Why they are all "in the same direction"...

This is the first, simplest, off-hand example I can come up with.

Things are blocked "in parallel" (assuming the camera is far away from the
objects) because that projection is a linear map (I repeat, assuming "far away
focus") and has a one dimensional kernel: 2d + 1d = 3d, which means (flat TV +
orthogonal lines = 3d world), that is the dimension formula.

------
ktf
Obligatory plug for the _Manga Guide to Linear Algebra_ :

<http://nostarch.com/linearalgebra>

:D

------
reyan
Interestingly this is more or less like the way I was taught linear algebra in
high school.

------
Evbn
Every now and again I think my highfalutin college courses were an overpriced
waste, and then a conversation like this comes along and I see that the
fundamentals of my courses are apparently radical and unheard of most other
places.

Do schools really not teach the underlying concepts of math, or do people just
fail to understand them the first time through and then blame their teachers?

/took linear algebra in the math department, then TAed it.

~~~
shardling
>Do schools really not teach the underlying concepts of math, or do people
just fail to understand them the first time through and then blame their
teachers?

It is absolutely the case that a great many students will not remember seeing
material that they definitely were exposed to. As someone who's TA'd physics
classes for a few years, I'd often ask a recitation whether they'd covered
some particular subject yet in lecture, and they'd often say (as a class) no.

I'd ask the professor later, and of course they had -- but students rarely go
into lecture having done the reading or prep work, and so have a piss-poor
retention rate.

~~~
snogglethorpe
> _students rarely go into lecture having done the reading or prep work, and
> so have a piss-poor retention rate._

I've found that even if _do_ the reading, my brain is often, well, a bit
leaky... TT

------
marshallp
I might be missing something big, but it seems like linear algebra is then
just an overly complicated way of describing what should be a simple toolbox
of spreadsheet manipulation functions, kind of like an extra module for
python. Why is it given such special emphasis then, it would be like teaching
the datetime or sched modules.

~~~
binarysolo
The other way around -- Linear Algebra is a (computationally) simple way to
describe a (computationally) complicated toolbox of manipulation functions
across a huge data set/space.

~~~
marshallp
Learning a few functions that deal with spreadsheets is a lot more simpler and
faster (and hence useful to a lot more people) then doing linear algebra
courses complete with proofs and 18th century symbols.

What will have more societal impact? Should anyone wishing to use a
wordprocessor be required to understand turing machines and computer
architecture?

~~~
weichi
I hope the author is reading your comments, because they make it clear that at
least one of his readers is coming away with seriously incorrect ideas about
linear algebra!

The thing that you are missing is that linear algebra is about much more than
matrix multiplication. It is about the properties of linear transformations in
general. While matrix multiplication may have simple implementations in
spreadsheets, many other concepts in linear algebra _don't_ , for example
calculations of and properties of eigenvectors and the corresponding
eigenvalues, or the various matrix decompositions (themselves closely related
to eigenvalues and eigenvectors). The matrix decompositions in particular give
results that not only are mathematically interesting, but are enormously
important computationally.

In terms of societal impact, other than calculus, no field of mathematics has
had a greater impact than linear algebra. From quantum mechanics to the google
search engine to machine learning, linear algebra is fundamental. Spreadsheets
are awesome and a brilliant idea, but they are aimed at solving a different
set of problems. Important though they may be, they pale in comparison to
linear algebra.

~~~
marshallp
My point isn't that it lacks utility, it's that the proofs are not relevant
(maybe they are to the teeniest tiniest number of people but I'm not sure of
even that - most research is about speeding up calculations on computers).
It's that practically it amounts to a library of computer functions. I can
type the date function on the command line, how it works behind the scenes is
not relevant.

~~~
weichi
The details of the proofs are perhaps not relevant to a practitioner. Although
I trust you would agree that the _existence_ of the theorems are very useful?

You seem to have the idea that the only useful thing to come out of linear
algebra are a set of algorithms, and that as long as you have computer
routines implementing those algorithms, the proofs that those algorithms work
don't matter. There is a grain of truth in this, but the _concepts_ that you
learn in a linear algebra course - particularly around eigenvalues and
eigenvectors - are very important in providing you the knowledge to choose
those algorithms well. And the process of working hard to understand proofs is
a great way (perhaps the best way) to make sure that you really understand
those concepts. It's also a great way to make sure that you really understand
what a theorem means / why a particular algorithm works.

 _most research is about speeding up calculations on computers_

A a large amount of progress in "speeding up calculations" comes from
algorithmic advances. These advances aren't being made by people who view
linear algebra as nothing more than a set of library routines. They come from
people who deeply understand the concepts and proofs behind the important
theorems of linear algebra.

