I was blown away in my Digital Signal Processing (DSP) class that eigen "values" exist for certain systems in the form of "waves".
Basicaly you put in a wave made from multiple sine and/or cosine waves through some function f(x) and the output is STILL a wave, though its frequency, amplitude and phase might change.
Technicaly if I remember correctly this applies to all complex exponentials, since those can be rewritten in the form of e^(ix) = cosx + i*sinx.
This formula also beatifuly shows how rotations and the complex exponentials are connected.
So basicaly you don't just have eigen values, eigen vectors: you also have eigen FUNCTIONS (sine and cosine above are the eigen functions of f(x)).
DSP basicaly revolves arounds functions that don't "corrupt" wave-like inputs (wave in -> wave out).
If you're into eigenfunctions, pick up any textbook on quantum mechanics. The hamiltonian is a linear operator whose eigenfunctions are the stationary states of the system ("stationary" because an isolated system in a stationary state will never leave that state) and whose eigenvalues are the observable values of the energy of the system. In general, there is a correspondence between observable quantities and Hermitian operators on wavefunctions: "measurement" is the application of a Hermitian operator to the wavefunction, and the values you may observe are the eigenvalues of the operator. So, for instance, energy is quantized in some systems because their hamiltonian has discrete eigenvalues.
Small correction: eigenfunctions are analogous to eigenvectors, not eigenvalues. In fact they _are_ eigenvectors, in the sense that they are vectors in a vector space of functions (or some restricted set of functions, e.g. continuous functions or periodic functions).
Maybe I'm missing what's interesting about this, but a function like f(z) = 5z + 2 would output a wave with changed amplitude and phase when z = sin(x). That doesn't seem that interesting to me, so f(z) must have some other interesting properties?
I think what the comment is missing is that the class of linear f(z)'s is much larger than you might expect. Partly because it just is, and partly because we like it that way.
In the land of math abstractions of signal processing: differentiation, integration, finite-impulse-response (FIR) filters, IIR filters, frequency-domain equalization, etc. All linear. Remember, linearity is with respect to the full time history, so f(z) = z(t) - z(t-1) + z(t-2) is still linear.
So we already know the eigenvectors of that whole arbitrary pile of componentry! Given any box of the above components, we can exactly characterize its response to any input -- for all time -- by knowing a list of that system's eigenvalues -- one for each of the already-known eigenvectors.
That's the "frequency response" -- the eigenvectors are sinusoids, and the frequency response is the eigenvalue corresponding to each eigenvector (sinusoid). And of course the Fourier transform takes you back and forth from the time domain to the eigen-domain.
We liked this analytical framework so much that when we could fabricate nonlinear devices (transistors) easily, we purposefully arranged things so these devices were only used in a linear part of their response curve. Hence, amplifiers: f(z) = 11 z.
And then the musicians introduced distortion and f*cked it all up -- our system isn't in a linear regime, our old eigenvectors are meaningless, and we can't predict what will come out. Pure chaos.
All of linear algebra is based on linear functions. Of course that doesn't mean that you can't study affine functions with it, but it adds a layer of extra complication.
For example, linear functions over finite dimensional vector spaces can be represented with matrices which means that everything you can compute about matrices you can also compute about linear functions.
Honestly I forgot the details, but basical the ENTIRE field of DSP stands on this fact.
Basicaly there exist some functions into which you can feed in sound waves and the output is guaranteed to still be a sound wave.
If you'd feed in a sound wave and if the function would corrupt it you would not be able to do any digital signal processing, since the output must be a wave.
Sound(wave) in -> Sound(wave) out, guaranteed to always be true.
> there exist some functions into which you can feed in sound waves and the output is guaranteed to still be a sound wave.
That in and of itself does not seem like a particularly insightful observation. It's just obvious that such functions exist. I can think of three of them off the top of my head: time delay, wave addition, and multiplication by a scalar. There must be something more to it than that.
In math, the obvious things aren't always true and the true things are often not obvious.
Trivially, the identity f(x) = x satisfies the guarantee as well. What amounts to insightful observation is the definition and classification of these functions. In exploring their existence in various forms, we can begin to understand what properties these functions share.
So the interesting part is not that this class of function _exists_, because of course it does! Your intuition has led you to three possible candidates. But if we limit ourselves to only the functions that satisfy the condition _wave-in implies wave-out_ what do they look like as a whole? What do these guarantees buy us if we _know_ the result will be a wave? For example, f(g(x)) is also guaranteed to be _wave-in-wave-out_. Again, maybe obvious, but it's a building block we can use once we've proved it true.
Let me try. Let’s say you have some operator F = crazyweirdblackbox. Like, it exponentiates and does a bunch more complicated things. You can apply this operator to some input function and transform it into some output function. The programming analogy would be some piece of code that works on a lambda and spits out another lambda. The finite-dimensional linear algebra equivalent is a matrix. You would ideally want F to be invertible (or undoable) for all inputs. Just in case.
If g(x) is an eigenfunction of F, then h(x) = F(g(x)) is actually just a rescaled of g(x): h(x) = ag(x), for some constant a. No matter how complicated and hard-to-compute F is, it boils down to just one number a, when acting on some special function g(x).
So what? This only applies to special g(x), and not any choice for g. Let’s say that I have some special function y(x) that isn’t necessarily g(x). But I have a whole bunch of eigenfunctions of F called g_k(x). If I knew that F was linear (kind of a prerequisite for computing eigenfunctions anyway), then I can decompose some output function y(x) into a weighted sum of g_k: y(x) = sum([w_k g_k(x) for k in range(infinity)]).
Some abuse of Python list comprehension there.
So instead of evaluating F, which might be very hard computationally or numerically, we can instead do a for-loop over potentially easier functions g_k. And store some weights w_k that essentially describe how F transformed an input function into y. Easy-peasy.
And maybe I don’t want to evaluate the entire sum. So I could choose to evaluate only the “important” terms in the sum to approximate y(x).
The entirety of signal processing, much of quantum mechanics, much of electromagnetism, and many other partial differential equations can fit into this framework. And we use PDEs to describe F when we really have no idea how to even to write it down. But we can still compute eigenfunctions and therefore make progress in evaluating F since we know the effect of F on certain special functions.
It's been 15 years since I was in my EE degree, and I learned all this stuff to the point of intuition. Since then I've been a C-monkey at megacorp and not doing any real math beyond simple geometry.
In fact functions are just infinite-dimensional vectors. Almost all of the theory goes through unchanged. This is the basic idea of functional analysis.
Notably, one very important part which does not go through is that for mappings between infinite dimensional spaces linearity does not imply continuity.
(E.g. a series of functions bounded by a constant can have arbitrarily large derivatives)
A large part of functional analysis is dealing with that fact and its implication for PDEs.
If you want, but you can do better. I believe, for instance, that at least continuous functions on the reals have a countable basis. Might even be as strong as measurable, not sure about that. That's how, for instance, fourier transforms work.
The fourier Transform essentially means that every L^2 (the square of the function has a finite integral) is completely "described" by an l^2 series (a series of numbers whose sum of squares converges), which is about the greatest piece of magic in mathematics.
One very important piece here is that the term "function" is somewhat of a lie (since the result couldn't be true if it weren't).
The result for measurable functions (not almost functions) shouldn't be true (I think).
I am not even sure it is true for L^1 almost functions.
Functions satisfy the laws of a vector space (well, certain classes of functions, such as L^2 anyway -- I'm certainly no expert here). I.e., functions are vectors. And the derivative is a linear operator on that space: d/dx(af + bg) = ad/dx(f) + bd/dx(g). Therefore d/dx has eigenvalues, one of which you properly identified as e^x (and more generally, e^ax, as others have pointed out).
> (well, certain classes of functions, such as L^2 anyway -- I'm certainly no expert here)
For any vector space V over a field F, the set of functions V -> V is a vector space over F. This is essentially because the vector space properties are carried over "point-wise".
When you consider only certain classes of functions (such as L^2), the question of whether they form a vector space boils down to the question whether those classes are closed under addition and scalar multiplication. All other vector space properties are satisfied because they are in the larger vector space of all functions.
Another good example is that sinusoids are eigenfunction of a linear dynamical system. If you drive the system with a sinusoid, the output is also a sinusoid of the same frequency, but different amplitude and phase. In other words, the frequency response is the eigenvalue at a given frequency.
This is the best explanation of the eigenthings in my opinion. Too many people, this article included, reach for more advanced concepts to explain something as simple as "eigenvectors are how an operator transforms stuff and eigenvalues are how much."
Thank you - its too easy to get into the weeds with mathematics and lose the "why" of something. Connecting these higher level concepts with the algebra that _most_ people have experience with ends up being a more direct way to share knowledge vs diving into the higher level stuff that enthusiasts really enjoy.
God knows how I passed 4 years of engineering math without ever being able to describe to anyone what eigen* actually are, but I think that's actually the norm.
That's completely normal and very unfortunate. I only learned about the geometry of the eigen*s from an EE professor who loved math.
Unfortunately, the history of the eigen*s is rooted in the opaque and cryptic (but rigorous) linear algebra definition, and the non-rigorous but meaningful geometry came later.
Fixed directions. The operator operating on an eigenvector doesn't change the vector's direction, but can change its magnitude (i.e. length). (Except, if you change the length from positive to negative, then you kind of change, or flip, the direction to 180 degrees opposite.)
I was learning principal component analysis a few years back which uses Eigenvectors to reduce feature dimensions while minimizing information loss (sorry if I butchered this)
I was really struggling to grok what Eigenvectors and Eigenvalues were and found this video to be the best intuition primer. I wish I had 3b1b when I was in high school and college
I think anyone starting a lin alg course could do a lot worse than
watch all his "Essence of Linear Algebra" series before starting - then
watch the relevant (c. 15 min) episodes as you take each lecture.
One day, when 13 year olds can use VR to meaningfully bootstrap their understanding of differential equations, I’ll probably wish the metaverse was around when I was a kid.
Seriously, the course on the essence of linear algebra did more for me than my entire college linear algebra course. It's only a pity I discovered the videos at the end of my mechanical engineering degree and not sooner.
Something that frustrates me, and maybe I’m just confessing my stupidity, is the extra layer of indirection in any discipline when things are named after people and not the thing’s characteristics.
My doctor once told me “if you learn enough Latin, a lot of names in medicine will hint at what they are, so you have less to memorize.”
I find that these names often lend a sense of complexity to concepts that turn out to be rather simple. In high school this really contributed to my struggles.
Edit: apparently Eigen isn’t a person’s name so I sure picked an embarrassing moment to bring this up.
> is the extra layer of indirection in any discipline when things are named after people and not the thing’s characteristics.
"Eigen" in German has same English root as "own": "Eigenvalue" is Germanglish for "Own/inherent value", so meets your spec of naming a thing after its characteristics, as long as "naming" is allowed to be in multiple languages.
How I learnt/taught myself when I first studied this subject, was characteristic or character. People change when they come into money, or rather their character changes (in my head I called them characterless). People with character don't change. Characteristic vector/Eigen vector is something which doesn't change.
I recently embarked on a journey to come up with a math vocabulary for Toki Pona, a lovely little artistic conlang which deserves better than what I'm doing to it. In Toki Pona, words are build up from simpler ones to describe a thing as it is. A friend is 'jan pona', a person who is good (to me, the speaker). So I've had to come up with names which describe math topics.
It's awful.
You know how many same-xs there are?! Eigenvalue, eigenvector, homomorphism, isomorphism, homeomorphism, homotopic. Which one gets to actually be "same shape"? Worse are when well meaning mathematicians use descriptive names anyway. Open and closed are not mutually exclusive, giving rise to the awful clopen (and don't pretend like ajar helps. an ajar door is an open door!). Groups, rings, and fields all sort of bring to mind the objects they describe, but only after you know the archetypal examples. Math is the study of giving the same name to different things, and that gives rise to more names than there are short descriptions.
So do you know what I did? Whenever I could, I used a real person's name. It freed up a limited vocabulary, and gave enough wiggle room to translate most undergrad math without too much loss. I suspect a similar thing is in play with math. Maybe the category theory people have abstractions to usefully describe "same-functions" without confusion. But in general, things are named poorly because it's genuinely a hard task.
It is sometimes very hard to name things well.
The name either becomes so unspecific that it is just as useless, or it gets so long that nobody will use it.
This gets worse the "deeper" the math goes, but for me it never was a real problem, as you usually learn the definition together with the name.
For really simple compounds, names are more or less settled and consistent (with some exceptions).
But as soon as your compound starts to get more complex (think organic chemistry) all the sudden, it becomes nigh impossible to consistently name things. There are tons of compounds with the same chemical formula that are regionally named differently. Even worse, there are tons of compounds with the same chemical formula that are actually different things due to how the compound is arranged. (Good ole carbon chains).
A common language fosters research and common understanding.
In IT, that language is English. In diplomacy, before interpreters were plentiful, that language was French. And in many classical, medieval-era sciences, that language was Latin (as a commonly-understood language that came from it's ease of being learned by romance-language speakers and being rather relevant in the (then church-run) universities).
So, there's no indirection intended. It's just an artefact of the past - an artefact that helps Chinese, Spanish and American doctors communicate (in broad strokes) even today.
My favorite is the "Lemma that is not Burnside's". Also known as the orbit-counting theorem, the Pólya-Burnside lemma, the Cauchy-Frobenius lemma, and of course Burnside's lemma.
Does anyone know of an example of a simple physical system where eigenvalues have a physical interpretation? The examples I know of are all in quantum mechanics, which is a bit abstract for me.
I think a system of springs is a good example. I think having a bunch of springs hooked together is a bit abstract so let's instead think of a molecule and model the bonds between the atoms as springs. If you were to squeeze this molecule together or try to pull it apart and then let go, it would vibrate in some complex way. By complex I mean that it wouldn't just bounce back along the direction that you compressed or stretched it.
However, if you write down the matrix of spring constants for the system and solve for the eigenvalues and eigenvectors of this system you can do something special. If you compress or stretch the molecule along the direction of the one of the eigenvectors then let go, the molecule will continue to vibrate along that same direction. The motion will not spread out to all other degrees of freedom. It will also vibrate with a frequency given by the eigenvalue of that eigenvector.
Additionally, any complex vibration of the system can be broken down into a combination of these independent vibrational modes. This is a simple fact because the eigenvectors form an orthogonal basis for the space.
Get a sheet of rubber. Grab it in both hands and stretch it. Inspect your sheet and find a line on the sheet that you could draw in with a marker and when you stretched the sheet the line would grow and shrink, but would not change what it was pointing at (probably a line from one of your hands to the other, in this simple example) That is an eigenvector of your sheet stretching transformation. The eigenvalue is how hard you're stretching the sheet.
Take a linear map from some space to itself, and ask:
What lines (through the origin) are mapped back to themselves? Those are the eigenvectors, and the amount by which they're elongated or shortened are the eigenvalues.
So, if we talk about 3d space, and we rotate things - the rotation axis is unchanged. That's an eigenvector (with eigenvalue 1).
If we mirror things - any vector in the mirror plane remains unchanged, that's an eigenvector (with eigenvalue 1), the vector perpendicular to the mirror is unchanged, but flipped, so that's an eigenvector (with eigenvalue -1).
If we dilate everything along the x axis by a factor of 2, say, then the x axis is an eigenvector (with eigenvalue 2), while the y and z axis and any vector in that plane is an eigenvector (with eigenvalue 1). Any other vector is "tilted", so not mapped to itself, so not an eigenvector.
It means the eigenvalues will only give you information about the system relatively to the center of that system.
Before describing any system, it's up to you (your "convention") to assert where is the zero-point of your world and in which directions the axes (x,y,z) are pointing.
For instance, in the real world you can choose your 3D coordinate system such that your mirror, as a physical system, keeps the origin untouched (0,0,0) -> (0,0,0). If you decide the origin is a point on the mirror, the equations will be linear: mirror(X) = AX. However if you setup the origin some point far from the mirror, like the center of your eyes, the equations are no longer linear, but affine: mirror(X) = AX+B. Looking at the values of the "AX" part of the system would reveal you the mirroring plane, but now shifted by an offset of "+B" -- the distance between the mirror and your eyes -- because your choice of coordinates was not leaving the origin intact.
It means it doesn't matter where it is: you can choose the origin, ie the point you measure from, it is arbitrary. Or another way of saying that is you can move the system to a different set of coordinates and it works in the same way.
... which means it's probably an imaginary physical system.
Maybe a good physical example is a piece of cloth that warps in 2D, and shrinks, when washed? Eigenvectors would describe the warping (skew, say) and eigenvalues the shrinkage relative to the original warp and weft.
Steve Brunton on YouTube has really good videos on eigenvectors & eigenvalues in context of matrix algebra (and then applied to simultaneous differential equations); https://youtube.com/watch?v=ZSGrJBS_qtc .
With more than 2 masses, you don't need to arrange the masses on a line, but you can have a 2d or 3d arrangement, with interconnecting springs. I am sorry I failed to find an example image.
The theory is explained, for example, around page 479 in this Thornton and Marion Classical Dynamics textbook. But you need to read about Lagrangian mechanics (chapter 7) before it makes sense.
> Does anyone know of an example of a simple physical system where eigenvalues have a physical interpretation?
Yep, vibration modes. Vibration frequencies represent their eigenvalues while the shape that the structural system exhibits when subjected to said vibration corresponds to it's eigenvector.
If a structural system is modelled as a linear elastic system it's possible to apply an eigendecomposition of that system and represent it in terms of linear combinations of it's vibration modes/eigenvector, and consequently we can get very accurate representations by using only a hand-full of these eigenvectors.
You know swing sets? We would start to swing back and forth just by moving our legs in a particular frwquencey, and without much effort we could move more and more? It turns out the frequency we moved our legs was the system's vibration frequency/eigenvalue for the vibration modes/eigenvector representing the we swinging back and forth.
> Does this relate to the normal modes or eigenmodes of a system?
Yes. The eigenvalues and eigenvectors of an undamped harmonic oscillator are respectively the vibration frequency and vibration mode.
One major class of structural analysis techniques is modal analysis, which determines the vibration modes and corresponding frequencies of specific structural systems subjected to particular boundary conditions.
Moment of inertia tensor [1], and principal axes of rotation [2].
Principle axes are the axes where a weightless body can rotate around without "wobbling". These axes are orthogonal to each other. If a rigid body has I_1 < I_2 < I_3 moments of inertia, then rotation around the first and third axes is stable and rotation around the second axes is unstable.
The vibration of a bell (say) could be modelled by a matrix, with a state vector to represent position, velocity, and acceleration, and the matrix modelling the differential equations describing their evolution over time. The eigenvectors represent a basis of the system, so that we can describe any potential state vector as a sum of eigenvectors. If we do so, then each step of the system can be modelled by multiplying each of these eigenvectors by its corresponding eigenvalue. If an eigenvalue happens to be complex, then we can describe it in phasor form as the product of an amplitude and a angle. The amplitude tells us how it will decay (or amplify) over time. The angle tells us the frequency of oscillation, and thus the note that the bell will typically sound.
Theory of Vibration with Applications by William Thompson and Marie Dillon Dahleh.
Say you have two cars linked, with some spring constant;
| --^^-- [c1] --^^-- [c2] --^^--|
where '^^' is a spring and '|' is a wall.
The motion of these cars can be written using the spring forces in the system or, alternately, as the harmonic motion of the undamped system with some natural frequency.
Setting this up as two simultaneous equations (one for each car) and solving for the roots give you the eigenvalues. The natural frequency is the square root of the eigenvalue. In other words, the eigenvalues help you define the natural frequencies which can be used to characterize the motion of the cars in the more complicated spring-mass system.
Things that vibrate have natural modes of vibration. A particular vibrational pattern can be decomposed into a time-varying linear combination of these modes. The modes of vibration are eigenfunctions and the frequencies at which they vibrate are the square root of the corresponding eigenvalues.
You can look up a vibrating drum head (circular membrane) for a simple example.
That's more of a further packing than an unpacking. Although that totally should be an expression for things that go on for too long: "Can you pack this for me please"
If i recall correctly you can represent a harmonic oscillator as a linear differential equation with a 2x2 matrix. The imaginary part of the eigenvalues of this matrix correspond to the angular frequency of the oscillator.
I like this example because it gives a physical meaning to both eigenvalues and imaginary numbers. It also shows the connection between the sine and cosine and the complex powers of e comes from (since you can show that all three solve the differential equation).
Take an airplane... it’s dynamics are described by a series of differential equations. We want to know if it’s stable! If the wife values of the dynamics are real and greater than 1 it’s unstable. If the eigenvalues are complex and have a modulus greater than 1 it will oscillate instability. If one is equal to one, it will cause everyone to vomit.
The speed of sound is the eigenvalue of a particular matrix (the "flux Jacobian") in the Euler equations, the 5-component system of partial differential equations that describe gas dynamics.
There are a couple of other comments that have mentioned oscillation modes, vibrations, etc. The first 7 pages of this series on sound synthesis might help give an idea of where these might come from:
The third page in particular shows a plot of "amplitude" versus "frequency" to show the "harmonic spectrum of a sawtooth wave". The "frequencies" correspond to the modes of vibration (i.e., sine waves of different frequency), which are the "eigenvectors" in this case. The "amplitudes" are the relative contribution of those vibrations to the overall sound, and these correspond to "eigenvalues".
The article is talking purely about constructing sounds via synthesis, so there's not necessarily a linear system associated with it, but there is a connection. Wave equations represented by linear partial differential equations can often be analyzed as a linear system that has these "modes of vibration" (i.e., series of orthogonal sinusoids at different frequencies). If you were to, for example, model a plucked string (like a guitar), you can model the solution as a weighted sum of eigenvectors (in this case, "modes of vibration" or sinusoids of different frequencies). The "weights" would be the eigenvalues, which determine the spectrum and ultimately the timbre of the sound produced.
That might seem more involved, because it's an infinite-dimensional linear system (i.e., the vectors are functions on a interval, rather than finite lists of numbers). It turns out, though, that the finite-dimensional discretization of an infinite-dimensional linear system (i.e., a partial-differential equation approximated by a finite-dimensional linear system) will sometimes have eigenvectors / eigenvalues that have similar features as the infinite-dimensional case. For example, there are certain finite-difference operators that can be written in matrix form whose eigenvectors will work out to be sampled sinusoids.
I'm not totally sure of the history, but I think a lot of the interest in eigenvectors / eigenvalues as a topic in matrix theory originated from this are (i.e., numerical solutions for partial-differential equations that were used to model physical systems).
Most are introduced to the interplay between physics and linear algebra through the study of the mass - spring system where the type (real or complex), sign and amount of the eigenvalues determine its behavior and stability. For example, complex eigenvalues with positive real part indicate an unstable, or chaotic, in terms of amplitude convergence oscillation.
Not quite. For a Markov probability matrix, 1 is always an eigenvalue, and all other eigenvalues are less than or equal to 1. For each eigenvalue that is equal to 1 you get a long term stable state probability. These distributions contain disjoint subsets of the states, and the system will converge to one of those subsets, depending on the initial state. The eigenvalues that are strictly less than 1 do not add any information to the long term state of the system.
See Stochastic Processes and Their Applications, V4 (1976) pages 253-259. I wrote it while still in grad school.
The values associated with each vertex on the _dominant eigenvector_ (the eigenvector associated with the dominant eigenvalue) are the long-term stable state probabilities. That's from a single eigenvector, not "the eigenvectors".
I read an interview with Australian wire music composer Alan Lamb that a stringed instrument with multiple overtones vibrating on the string can be analyzed by breaking down the vibration into eigenvalues, but I've never found any reference material that explain that. I'm wondering if he was referring to FFT.
If you discretize the string into a bunch of tiny masses, linked together by a bunch of tiny springs, you can build a mass Matrix (diagonal) M and a Stiffness matrix K (element ij = stiffness of spring that links mass I and mass j).
I can't remember the next part exactly, you can look it up in a textbook, but you multiply the matrices KMK, or similar, and the eigenvalues of this are the natural frequencies of the string. The eigenvectors represent the mode shapes, ie the displacement of each mass element.
The same technique is used in Finite Element Analysis to find the modes and modeshapes of complex structures (a car frame, a bridge, etc)
Think of a fun house mirror that for the sake of this example make you look twice as tall but 20% skinnier. This can be modeled by a two-by-two matrix with eigenvalues of 2 and 0.8. (Indeed, it will have them on the diagonals which makes it easier to study.)
That's not true. The page rank is read from the eigenvector, and is the value associated with the given vertex (ie web page). There are as many page rank values as there are web pages, but only one eigenvector from which to read: the dominant eigenvector of the transition matrix, which is the one with the largest eigenvalue. So, only a single eigenvalue for the entire pagerank computation.
I like this explanation, which puts the concept of operator at center and has some nice computational relations, and is related to the physical world by the notion of an observable, with an eigenfunction being a special kind of operator-function relationship. At the end, there's a nice non-QM application (Fourier transforms).
> An observable is any dynamical variable that can be measured... in classical mechanics... observables are represented by functions* (such as position as a function of time), in quantum mechanics they are represented by mathematical operators... We shall not in general distinguish between the observable and the operator that represents that observable (such as the position of a particle along the x-axis)"
> "An operator is a symbol for an instruction to carry out some action, an opeeation, on a function...in certain cases, the outcome of an operation is the
same function, multiplied by a constant" [that constant being the eigenvalue]
> "An important point is that a general function can be expanded in terms of all the eigenfunctions of an operator, a so-called complete set of functions... then a general function can be expressed as the linear combination [a sum over a complete set of functions, each function having its own coefficient]"
Finally, we get to a practical real-world example (non-QM):
> "...for instance, the straight line g = ax can be recreated over a certain range by superimposing an infinite number of sine functions, each of which is an eigenfunction of the [differentiation] operator, d2/dx2. Alternatively, the same function may be constructed from an infinite number of exponential functions, which are eigenfunctions of d/dx."
Extending this general concept, we can go into the famous Fourier Transform, used widely for all kinds of classical problems in converting waveforms into frequency peaks, such as in musical analysis, electrical engineering, etc. The wiki page on Fourier transforms has a very brief mention of this view, i.e. The Fourier transform decomposes a function into eigenfunctions for the group of translations.
Here's what looks like a deep dive into this approach to the Fourier Transform (2008):
So my actual favorite first example is to do this with Fibonacci numbers as a linear recurrence relation, but that's not really a "physical" interpretation. Let me give you my favorite physical one:
The essence of special relativity is that acceleration is a bit weirder than you think. In particular when you accelerate by amount a in some direction x, even after accounting for the usual Doppler shifts you will find that clocks separated from you by that coordinate, appear to tick at the rate 1 + a x/c² seconds per second, where c² is a fundamental constant. Clocks ahead of you tick faster, clocks behind you tick slower (and indeed appear to slow down and approach a ‘wall of death,’ more technically called an ‘event horizon,’ at a distance c²/a. (This effect is called the ‘relativity of simultaneity,’ and it is in some sense the only real prediction of special relativity, as the rest of this comment will show—the other effects of ‘time dilation’ and ‘length contraction’ are second-order and can be derived from this first-order effect.)
This means that the transformation equations for moving into a neighboring reference frame are not the ones that Galileo and Newton proposed,
t' = t
x' = x – v t
but slightly modified to (to first order in v, so only considering small velocity changes)
t' = t – (v/c²) x
x' = x – v t
where w = c t is a measure of time in units of distance using this fundamental constant. How do we generalize and get the full solution? We can do it by looking in the eigenvector basis. Consider new coordinates p = x – c t and q = x + c t, given any (x, t) you can find a unique (p, q) which describes it and if you want to get back those values you would say x = (p + q)/2, t = (q – p)/(2 c). But feed these magical coordinates that come from eigenvectors into the above transform and it "diagonalizes",
p' = (1 + v/c) p
q' = (1 – v/c) q
and therefore if you want to make a big change in "velocity" c φ (here instead φ turns out to be "rapidity") out of N smaller changes, you can repeat this transform N times with little boosts by v/c = φ/N, and you will stitch together the full Lorentz transform out of little first-order Lorentz transforms:
p' = (1 + φ/N)^N p = e^φ p
q' = (1 – φ/N)^N q = e^{-φ} q
Transforming back and using the hyperbolic sine and cosine, sinh(x) = (e^x – e^{-x})/2, cosh(x) = (e^x + e^{-x})/2, the full formula is
w' = w cosh(φ) – x sinh(φ)
x' = x cosh(φ) – w sinh(φ)
where w = c t is a simple time-in-units-of-meters coordinate. Usually we denote cosh(φ) = γ, sinh(φ) = γ β, which gives this the more familiar form you'll find in textbooks, and the identity cosh²x = 1 + sin²x gives a formula γ = 1/√(1 – β²) for the latter... but this ‘rapidity form’ is in some ways more elegant. Anyway, point stands, from the "first-order" transform you can derive the "full" transform just by building any large velocity change out of an infinite number of infinitesimal velocity changes, and this is the source of the factor γ which describes time dilation and length contraction.
Okay, now for physical interpretation. You asked what physical meaning these eigenvalues and eigenvectors of the Lorentz transformation have, and the answer is this: the eigenvalues (1, 1) and (1, -1) of the Lorentz matrix represent light rays, the p/q description we came up with above was a description of spacetime in terms of light-ray coordinates where we identify an event at a particular place and time with the light rays that it casts, announcing that the event has happened, in the +x and -x directions. On the negative side, these are also the last light rays that were able to touch the event before it happened, so represent "everything it could have possibly known about" -- there is a space between these two "light cones" which is its "relativistic present," the things that anything which was there at the event cannot know about until the future.
The eigenvalues, exp(φ) = sinh(φ) + cosh(φ) = γ + γ β = √[(1 + β)/(1 – β)] and exp(-φ) = √[(1 – β)/(1 + β)], are the Relativistic Doppler shifts of those light rays. Indeed one can read them as e.g. exp(-φ) = 1/γ * 1/(1 + β) , here 1/(1 + β) is the standard Doppler shift formula from nonrelativistic physics and 1/γ is the decrease in frequency due to time dilation.
"By finding eigenvectors we’ll find axes of new subspace where our life gets simpler: classes are more separated and data within classes has lower variance."
No, it isn't. A star-autonomous cartesian category is just a preorder. Chu(Set, n) is star-autonomous but not a preorder, and therefore not cartesian. Or more concretely: cartesian categories are models for type systems with copying and deleting, but the Chu construction builds a model for a linear type system.
I went through Linear Algebra I & II where eigenvalues & eigenvectors played a central role many a time and I was left with no elucidation regarding their practical applications.
Now I know that eigen*s (at risk of egregious oversimplification) can characterize systems and transformations, they are fundamentally widely applicable. Think of stretching narrow an arbitrary 2D image on a non-cardinal axis; an eigenvector can be a key factor that describes that transformation.
A far easier to digest primer on eigenvalues is available from 3Blue1Brown [+]. His presentation format is undeniably approachable, so much so that I think you could probably use it to teach linear algebra and eigenvectors to 9-year olds.
It used to be called "proper value" in English (you can still find that in old textbooks), but the (semi-)German word has basically entirely replaced it.
The thing about Higham is that he's sort of a one-man Wikipedia of linear algebra. Many of the terms that he uses also have their own pages that (eventually) break the concepts down into comprehensible terms.
Often, papers or terse textbooks will list a definition like the first sentence without the added detail below. I think this is great for undergraduate students or folks who'd like to refresh their memory a bit on eigenvalues, how they're derived, and what they may imply. I certainly found it helpful.
Basicaly you put in a wave made from multiple sine and/or cosine waves through some function f(x) and the output is STILL a wave, though its frequency, amplitude and phase might change.
Technicaly if I remember correctly this applies to all complex exponentials, since those can be rewritten in the form of e^(ix) = cosx + i*sinx.
This formula also beatifuly shows how rotations and the complex exponentials are connected.
So basicaly you don't just have eigen values, eigen vectors: you also have eigen FUNCTIONS (sine and cosine above are the eigen functions of f(x)).
DSP basicaly revolves arounds functions that don't "corrupt" wave-like inputs (wave in -> wave out).