
Surprises from numerical linear algebra (2010) - th0ma5
http://www.johndcook.com/blog/2010/01/20/ten-surprises-from-numerical-linear-algebra/
======
gamble
Linear algebra is the foundation of modern engineering. Before computers, the
only mathematically tractable problems were those that could be reduced to
linear problems. With computers, we can take on those formerly intractable
problems by reducing them to massive systems of linear equations. Literally
everything around you in daily life is the product of linear algebra.

------
gruseom
Ok HN, what's the best book for learning this stuff? My criteria are: rigorous
is good, but so is fun. Bonus points for anything a hacker would appreciate,
such as computational applications or exercises.

~~~
nokcha
I've read and would recommend _Linear Algebra Done Right_ by Sheldon Axler.
[http://www.amazon.com/Linear-Algebra-Right-Sheldon-
Axler/dp/...](http://www.amazon.com/Linear-Algebra-Right-Sheldon-
Axler/dp/0387982582/)

For numerical aspects of linear algebra (and other subjects), I've found
_Numerical Recipes_ to be quite helpful.

~~~
Silhouette
While _Numerical Recipes_ has certainly earned its place in the history of the
genre, it is only fair to point out that it is very out-dated by now, and that
the example code is neither particularly well-written (from a software
development point of view) nor freely usable in your own projects.

As others have suggested, something like Golub and Van Loan is a better
choice. The technical notes to go with the LAPACK library, and similar
documents from those developing related software, are also likely to be of
interest to anyone doing this stuff seriously.

------
mturmon
I thought the OP would reference the fact that errors in solving a linear
system don't combine and magnify to produce unusable results beyond say a
10x10 system.

At the outset of numerical linear algebra, say in the 1940s [until 1948], it
was not known whether errors would blow up or not. [best bound was 4^n where n
is matrix order]

Turned out they did not (if you do things remotely right) but even von Neumann
[in a 1947 paper, with others] was surprised that large (say 20x20, at that
time) linear systems could be solved accurately.

[edits in braces: This history, and more, is in
www.maths.manchester.ac.uk/~higham/talks/twge96.ps.gz ; Higham wrote probably
the best and most comprehensive up-to-date book on the accuracy of numerical
linear algebra.]

~~~
eru
Don't you still have to look out for ill-conditioned matrices?

~~~
mturmon
Yes, in some sense (see below) you do care about A.

But back then, everyone was worried even about well-conditioned systems. As of
the mid 1940s the best error bound on cumulative roundoff error was 4^n where
n is matrix order. That 4^n would be a multiplier on top of condition number
(which at the time had not been formalized, it was Turing who did that).

Anyway, the dependence on matrix size turned out to be order n^2, and in
practice more like O(n).

* * *

Also, the bounds on solving

    
    
      A x = b
    

for x, knowing b and A, work differently than solving for the inverse of A.

Sometimes for a given A and b, you can compute x above quite accurately, but
you can't compute inv(A) accurately and multiply by b to get x. It depends on
the how the "difficult directions" in A interact with b. This is one reason
numerical analysts always ask, "Do you really need to invert A?"

Because of this, the actual condition number of A (i.e., the norm of A times
that of inv(A)) does not enter into the bounds on solving A x = b. The bounds
depend on a different norm.

In fact, there are several forms of the error bound -- known as backward and
forward errors -- depending on what you need.

* * *

Incidentally, this is the kind of stuff that's covered in Golub and Van Loan,
or in Trefethen and Bao, but not in a standard Abstract Linear Algebra book.
For instance, Strang's book spends only _one chapter_ on numerical linear
algebra.

~~~
eru
Thanks. Incidentally, this is the kind of stuff that I should still remember
from university.

------
trentonstrong
Linear algebra is one of my favorite subjects. Of all the math I used while
still in physics, linear algebra ended up being the most practical for solving
everyday problems in other fields.

A person can do a lot of damage armed with a good grasp of linear algebra and
fourier analysis.

------
zeratul
It all depends how often you solve linear problems. If you are occasional user
any package for any language will do.

If, however, you process tons of sparse, dense, categorical, or continuous
data BE WARY: "lapack", "eispack", "linpack", "scalapack", and "lis" will give
you different solutions. Occasionally, crash on some machines. Compilers,
linkers, precision, and missing values matter here, too.

~~~
mturmon
linpack and eispack are vestigial. They were folded into lapack, which is the
"one true package" now (e.g., <http://en.wikipedia.org/wiki/EISPACK>). The
situation here is really not complicated.

scalapack is a different animal entirely. It's a parallel version of lapack
(via MPI) so should not be expected to duplicate lapack results because
partitioned algorithms will be used. But if you're using scalapack, you know
that already.

------
T_S_
I used Jim Hefferon's Linear Algebra to get back into the subject a few years
ago. Great balance of rigorous and intuitive. Free book available here
<http://joshua.smcvt.edu/linalg.html/>

~~~
dxbydt
Ok, so Hefferon was my textbook for 2 full semesters. Lots of problems with
this book. Hefferon is an algebraist, so the computational aspect of linear
algebra ie. the stuff you are most likely to use linalg for at your workplace,
has been given short shrift. Instead, the author gives fullblown treatment to
theorem proving in vector spaces. So you are spending a lot of time
constructing isomorphisms between subspaces....that shit is neat for a math
major, which I was, but now that I'm in the workplace, I find what the world
calls "linear algebra" isn't what Hefferon cares about.

eg. One of the basics you must absolutely know, or so I've been told, is how
to do a PCA, and why a PCA is not the same as regression. So if you've got
multidimensional data, say data in 7 dimensions ( risk ratio, fico score,
salary, age etc etc ), you want to know if you can reduce that whole set to
say 1 dimension. ie. can you transform the original set of 7 variable vectors
into a much smaller set of 1 variable vectors that captures 90% of the
information in the original ? If so, how ? Turns out you construct a linear
combination of correlated variables. Ok, but there's infinitely many
combinations. So how ? Well, construct it in such a fashion that the variance
of the combination is maximized. That component is called the Principal
Component, and its the single most used technique to reduce dimensionality in
the industry. So you've taken a 7-dimension space and nicely reduced it to a
1-dimension subspace with maximum variance that explains 90% of the original 7
dimension space. Its quite amazing actually.
<http://en.wikipedia.org/wiki/Principal_component_analysis>

Similarly, there are a whole bunch of very common stuff you'd actually use
linear algebra for in CG - translation, rotation in 3D, shear matrices and the
like. Hefferon doesn't do those either.

In portfolio theory which is my bread and butter
(<http://en.wikipedia.org/wiki/Portfolio_theory> ) almost everything I do is
very properly linear algebra. But again, Hefferon doesn't offer any insight
into how/why finding a vector with minimum variance can yield better returns
in a vector space of portfolios.

~~~
starwed
Your problem with the book seems to be that it is general, rather than
applicable to your particular field.

Why on earth would expect that it would be? I took and enjoyed linear algebra
using that text-- it didn't specifically address the QM I deal with as a
physicist, but since the author _can't predict each student's future_ , I'd
hardly characterise that as a problem with the text!

If it was used as a course on portfolios, ok, bad choice -- but directed
against a general linear algebra text, your criticism doesn't seem very valid.

~~~
dxbydt
>> rather than applicable to your particular field.

No, I'm saying Hefferon is not applicable to any field. He eschews application
in favor of formal theorem proving in vector spaces, in isolation of any real-
life examples. That is great, but that is not the typical use case of anybody
out there who actually uses linear algebra in industry, whether in computer
graphics or portfolio theory or multivariate statistics or what have you.

~~~
lotharbot
Being "not the typical use case... in industry" doesn't make it bad. It just
makes it a pure mathematics book. Formal theorem proving in vector spaces is
the typical use case of a pure mathematician; this book is for them.

If you're looking to do application, use an applied book. Gilbert Strang's
"Linear Algebra and its Applications" is quite good.

~~~
patrickc
I'll second that. Strang's book is by far my favorite introductory math book.
He addresses fundamentals quite well, and gives good treatment to applications
such as linear programming and game theory.

------
cheez
The biggest surprise for me was never having to compute the inverse. I never
saw such gymnastics to avoid computing things like I saw people avoiding
computing the inverse of a matrix (for obvious reasons.) It's really amazing
to work in the numerical computing field for a while. You leave it though
because you can't stand the code.

But you learn a lot.

------
Sniffnoy
Here I was expecting some actual surprising theorems.

------
NY_Entrepreneur
Sure. No surprises. All decades old stuff.

------
whatgoodisaroad
I remember taking linear algebra some time ago. I tweeted my one main takeaway
from the class. <http://goo.gl/gEu4y>

~~~
chollida1
<https://twitter.com/#!/whatgoodisaroad/status/26337970149>

The link the above minimized link goes to.

