
Eigenvectors from eigenvalues - bigpumpkin
https://terrytao.wordpress.com/2019/08/13/eigenvectors-from-eigenvalues/
======
bhl
There's a backstory to this paper from the r/math subreddit [1]. One of the
authors initially posted the question there, and stumbling upon a similar
question on math overflow emailed Terry Tao. And surprisingly, he responded.

[1]
[https://www.reddit.com/r/math/comments/ci665j/linear_algebra...](https://www.reddit.com/r/math/comments/ci665j/linear_algebra_question_from_a_physicist/)

~~~
leereeves
Not only responded, but responded in under 2 hours, which is amazing from a
busy professor like Tao.

~~~
mkl
Not only responded in under 2 hours, but "Tao’s reply also included three
independent proofs of the identity."

From the article we discussed earlier,
[https://www.quantamagazine.org/neutrinos-lead-to-
unexpected-...](https://www.quantamagazine.org/neutrinos-lead-to-unexpected-
discovery-in-basic-math-20191113/)
[https://news.ycombinator.com/item?id=21528425](https://news.ycombinator.com/item?id=21528425)

------
est31
See also prior discussion:
[https://news.ycombinator.com/item?id=21528425](https://news.ycombinator.com/item?id=21528425)

------
banachtarski
For those curious about applications:

Suppose you have a learning algorithm that requires the
eigenvectors/eigenvalues for a large matrix. Suppose new data is streamed in
that increase the size of the matrix. well, now you have an algorithm for
updating your existing solution instead of needing to compute the entire thing
from scratch. (consider updating principal components over time for example).

Suppose you want to accelerate the process of computing eigenvectors for a
large matrix. Well, you can now do parallel reduction relatively trivially
given the theorem.

*edit: wording.

~~~
mxcrossb
Can you go into more detail? I don’t see how that would work. In this
algorithm, you need the eigenvalues of all of the minors. So if I’m adding
rows/columns to the matrix, yes I have one of the minors from my last
solution, but I have to compute N more. And I also need the eigenvalues of the
new full matrix.

I think this result has some intriguing opportunities, and have been toying
with them since yesterday’s post, but I don’t see something so
straightforward.

~~~
banachtarski
I obviously haven't worked it out myself, but I think you can memoize the
results from the smaller minors to accelerate the computation of the big one.
So you would need to decide for yourself the speed/memory tradeoff.

------
xxpor
So it's been _checks notes_ 9 years since I've had to deal with eigenvectors
and eigenvalues. So the post went way over my head, but folks on twitter were
making quite a big deal over the post last night when it went up on arXiv.
Could someone explain this like I have a computer engineering bachelors
degree? :)

~~~
randomsearch
Don’t know how simple you want me to go, so apologies if required.

You’ve got a matrix A, ie a 2D array of numbers.

You’ve got a vector v, that is a 1D array of numbers.

You can multiply A by v, written Av, to get a new vector.

Matrices can therefore be viewed as devices for taking a vector and giving a
new one, ie as linear transformations. You can design a matrix to rotate
vectors, stretch them etc.

An eigenvector of a matrix is a vector that changes only by a constant factor
c, when you apply (multiply by) the matrix. So Av = kv where k is a number.
For example, if k=2 then applying A just doubles every number in v.
Eigenvectors are stretched or squashed by the matrix, no funny business like
shearing or rotating.

An eigenvalue is the “k” mentioned above, ie there’s at least one eigenvector
such that Av=kv. There could be multiple eigenvectors all with the same k, ie
all changed in the same simple way by applying the matrix.

Ok? Slight complication: matrix A may contain complex numbers, not just your
common garden variety real numbers. But the above all still applies.

“Symmetric matrices” are matrices that have a symmetry across the top left to
bottom right diagonal. So the number at A[i][j] is the same number as that at
A[j][i].

“Hermitian” matrices are the equivalent of symmetric matrices when we’re
dealing in complex numbers. Rather than A[i][j] being equal to A[j][i], if one
entry is a complex number then the other entry is the “complex conjugate” of
the first entry.

(In case you’ve forgotten complex numbers: we extend the real numbers to
including numbers of the form r + zi, where r is the “real part” (just a real
number) and “zi” is a real number z multiplied by i, where i = sqrt(-1).
Imaginary numbers were introduced to allow us to solve equations we couldn’t
otherwise solve, but have turned out to be super useful everywhere. The
“complex conjugate” of an imaginary number is obtained by flipping the sign of
z, eg (2 + 3i) -> (2 - 3i).)

Anyway, so Hermitian matrices are “symmetric” in this way and turn out to be
super useful in physics. They also have the property that when you multiply a
Hermitian matrix H by a vector v, the possible eigenvalues are real numbers,
despite the fact that H may contain complex numbers.

What the paper states is a relationship between the eigenvalues and
eigenvectors of a Hermitian matrix H and the eigenvalues of a smaller
“submatrix” matrix H’, where H’ is just the same as H but with a column and
row removed, where the index of the row and column is the same. This is neat,
because it can be used to speed up the calculation of eigenvectors (as
important as say array sorting in CS) in some specific situations. Commenting
further is beyond me.

~~~
akie
Thanks! It's been 15 years since I did any linear algebra or had any use for
eigenvectors or matrices, and I could follow this. Much appreciated.

------
j1vms
Somewhat related to some of the questions that have been raised here:

\- for the set of matrices that possess them ("transformation matrices that
only perform stretch/contract"), eigenvectors (with their associated
eigenvalues) play a role quite analogous to the role primes play in the
integer set. They provide a unique, identifying "spectrum" for said matrix.
This is made explicit by eigendecomposition (spectral decomposition).

\- with extension via singular value decomposition (SVD method) to _any_
square matrix (e.g. "transformation matrices that might also shear, rotate"),
certain operations such as exponentiation of the square matrix can performed
very quickly once eigenvectors/eigenvalues have been obtained via the SVD
method.

~~~
creative-coder
Wow... your comment is infact more useful pearl of wisdom for getting the
intuition of Eigen-values/vectors. Can you suggest some book/reference that
_best_ conveys this intuition ? Somehow, Gilbert Strang's Linear Algebra does
put me to sleep :/

~~~
nimish
Linear algebra done right by Sheldon axler

Linear algebra that focuses on matrices of numbers is missing the point and a
proper numerical linear algebra class will teach them better anyway

~~~
btilly
I would recommend _Down with Determinants!_ instead. Same author, and was the
inspiration for that textbook.

[https://www.maa.org/sites/default/files/pdf/awards/Axler-
For...](https://www.maa.org/sites/default/files/pdf/awards/Axler-
Ford-1996.pdf)

As a graduate student at the time, it opened my eyes about a whole bunch of
theorems that I thought I already knew. And it is a lot more approachable than
a textbook.

It won an award for the best piece of expository writing on mathematics in
1996. It was a deserved win. :-)

------
ur-whale
Before you excitedly jump to coding a new algorithm to produce eigenvectors
from eigenvalues, it's worth noting that:

    
    
        . This only applies to hermitian matrices and is therefore of limited scope.
    
        . The identity only produces the magnitudes of the coefficients of the eigenvectors. The signs of the coefficient sare still unknown.
    
        . Even if knowing only the magnitude of the coefficients is somehow useful, this is still very expensive to compute - just like Cramer rule which is of great theoretical interest, but has very little practical use when actually computing a matrix inverse or a determinant.
    

None of the above diminishes the importance of the result, btw.

~~~
gajomi
Came here to pretty much write these same notes. The fact that you need to
know the spectrum of all the minors really limits the usefulness for solving
general eigen* problems. But I can imagine that this could make a great
starting point for all kinds of approximate calculations/asymptotic analyses.

Regarding the first caveat that you bring up though, whereas the problem
statement says you need a Hermitian matrix I think the results should
generalize to non-hermitian matrices. In particular take a look at the third
proof of the theorem at the end of the article. The only assumption required
here is that the eigenvalues are simple (which does not preclude them being
complex/coming in complex pairs).

Protip: I had to read the second step in the proof a few times before I could
see what was going on. Explicitly what you do here is to (i) multiple from the
right by the elementary unit vector e_j (ii) set up the matrix vector inverse
using Cramer's rule (iii) notice that this matrix can be permuted to have a
block diagonal element equal to the minor M_i with the other block diagonal
entry equal to 1 and the corresponding column equal filled with zeros (iv)
Write the determinant in the block form, which simplifies to the expression
involving M_i by itself then finally (v) multiply by e_j from the left to get
scalar equation.

------
aj7
Since I’ve always been terrible at abstract linear algebra, I would request
that someone do a simple problem, say finding the eigenvalues and eigenvectors
of a 3 x 3 Hermitian matrix with three distinct eigenvalues. I’d also be
curious under what circumstances the above method leads to a computational
advantage.

~~~
aj7
Anybody?

------
lxe
For a second I thought this was going to be HN's darling "Eigenvectors AND
Eigenvalues" explained visually [http://setosa.io/ev/eigenvectors-and-
eigenvalues/](http://setosa.io/ev/eigenvectors-and-eigenvalues/)

------
pgt
I have a sneaky suspicion this finding can used to speed up expert system
matching (e.g. Rete networks), which would affect rendering and compilation
times for lots of software.

------
math_and_stuff
The entries of a matrix are eigenvalues of 1x1 minors, and clearly
eigenvectors are a function of the entries.

That the result is a function of (n-1) minors is first-order information and
would help clarify the default assumption that the result computes
eigenvectors from the full Hermitian matrix's eigenvalues.

------
kgwgk
"The real question is whether the discovery is important or interesting. And
whether it is surprising. [...] OK, why does Wolchover claim that it is
interesting or surprising? Because they gave that formula to Terence Tao, a
Fields Medal winner, for him to say something and he didn't know the formula
and even expressed doubts. [...] So that's how it works. If you want to
promote a new chocolate candybar, send a formula – or the nutritional values
from the package – to someone like Terence Tao, to a darling of the media. The
popular science writers will do the rest."

[https://motls.blogspot.com/2019/11/hype-about-formula-for-
ei...](https://motls.blogspot.com/2019/11/hype-about-formula-for-
eigenstates.html)

------
jl2718
I believe this is equivalent to the solution given by Gene Golub in 1973 for
the constraint vector required to produce a specific set of stationary values
in a linearly-constrained quadratic program where the constraint vector is
unitary.

[https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=9B...](https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=9B05FFCB5C07BC4A1A2A00C7FF2C2C68?doi=10.1.1.454.9868&rep=rep1&type=pdf)
(eq. 3.6)

I haven't looked at Tao's proofs to see if this is covered, nor do I have a
clear formulation of the equivalence, but it certainly is striking.

------
acd
When I see people who look similar I think of them as having similar
Eigenfaces!

~~~
jjgreen
Eigenfaces are a thing!
[https://en.wikipedia.org/wiki/Eigenface](https://en.wikipedia.org/wiki/Eigenface)

------
Noxmiles
Reading german words, still don't understand anything...

------
pgt
This can probably be used to speed up Transfer Learning.

------
angel_j
Does this mean you can test large neural networks with smaller ones?

~~~
c1b
Are your neural networks parameterized as Hermitian matrices? (No.)

------
rossmohax
Will it boost FPS in 3d games?

~~~
mkl
No, it doesn't speed anything relevant up. I'm curious as to what made you
ask, though.

~~~
ur-whale
I suspect he was joking.

------
auggierose
This is a good example of math being presented in an unnecessarily confusing
way.

Values lambda_i(A) are introduced, but then in the formula also values
lambda_i(M) are referenced for some M which is not A.

Basically, we have no idea what lambda_i(M) might mean from the statement of
the theorem. Is the order lambda_1(M), lambda_2(M) ... related to the order
lambda_1(A), lambda_2(A), ... How many lambda_i(M) exist? And so on.

Maybe this is all clear if you deal with minor matrices all the time, but when
I see a statement like that I am not even interested in its proof anymore.

~~~
banachtarski
Terry is being abundantly clear. Why are you trying to correlate indices in
two separate product series?

~~~
auggierose
He is not being "abundantly clear". He doesn't give a definition of
lambda_i(M) for the various M appearing in the theorem that are not A.

~~~
_hardwaregeek
Doesn't he say it here:

where {M_j} is the {n-1 \times n-1} Hermitian matrix formed by deleting the
{j^{th}} row and column from {A}.

~~~
auggierose
Yes, he says what M_j is, but he does not say what lambda_i(M_j) is.

~~~
chombier
He says that eigenvalues of A are the lambda_i(A), so it's pretty obvious that
lambda_i(M_j) is the i-th eigen value of M_j.

What might not be obvious is the ordering of the eigenvalues, which must be
the same for the formula to make sense.

~~~
FabHK
No, the ordering of the eigenvalues of M doesn't matter, because we take their
product (or rather, the product of their difference from a fixed number), and
multiplication commutes. It's maybe not obvious, and as such I can understand
auggierose's complaint to an extent, though it was maybe unnecessarily harshly
phrased.

~~~
chombier
You're right, my bad.

