
Understanding the FFT Algorithm - linux_devil
http://jakevdp.github.io/blog/2013/08/28/understanding-the-fft/
======
j2kun
If anyone is interested in more of the mathematics, I have a few posts on
Fourier Analysis (starting with four lengthy posts here:
[http://jeremykun.com/primers/](http://jeremykun.com/primers/) ) and a follow-
up on actually deriving and computing the FFT here:
[http://jeremykun.com/2012/07/18/the-fast-fourier-
transform/](http://jeremykun.com/2012/07/18/the-fast-fourier-transform/) with
a (relatively misguided) application of denoising a sound bite.

My implementation is in almost pure python (except for complex number
arithmetic) and it shows.

~~~
achompas
Your primers look great! Do you have any sources for people interested in
learning more about the various topics?

~~~
j2kun
I think you mean about Fourier analysis, yes? Brad Osgood's Stanford lectures
(available on YouTube) give a very good (read: steady, unassuming) approach
and the lecture notes associated with that course are extremely detailed and
informative if you want a deeper dive.

~~~
achompas
Yep, definitely Fourier analysis, I haven't studied it formally at all.

Thanks for the pointer to Osgood's lectures. A co-worker recommended Stein's
"Fourier Analysis" after a bit of work on my part. Do you have any thoughts on
that text?

------
existencebox
Coincidentally, I was just reading through the wiki pages on Fourier
transforms the other evening, and having a fantastically hard time wrapping my
head around some parts of it. Maybe the wrong place to ask, but if anyone had
on hand some good intuitive (potentially directed more towards the programmer
than the data scientist) description of the Fourier transform?

~~~
KaeseEs
I don't know any data scientists, or even what one is really, but here's the
EE explanation: the Fourier transform maps a signal in the time domain to the
frequency domain, and vice versa. So if you have a sine wave in the time
domain [say something at 3khz: y = sin(3000x)], the Fourier transform would
turn that into a pair of Dirac deltas (spikes that are infinitely thin but
have an area of one) at omega = +-3000. Or, if you had a pulse, the Fourier
transform would map that to a sinc [sinc(x) = sin(x)/x].

The Fourier transform is related to the Laplace transform, which maps signals
in the time domain to those in the "S domain", a domain where locations are
normally described with complex coordinates which contains information both on
frequency of the periodic components of the signal as well as any transients
(starting conditions, in layman's terms).

------
coherentpony
Oh, he didn't do the bit reversal trick! That's the really cool part :)

------
James_Duval
I don't know Python, so I may be missing something, but:

I don't understand how the naive approach can be O(n^2) if it involves matrix
multiplication.

Surely at best it would be somewhere between O(n^2) and O(n^3), and most
likely O(n^3)?

~~~
sampo
The post means matrix-vector multiplication, which is O(n^2). You're thinking
of matrix-matrix multiplication, for which the naive algorithm is O(n^3).

He writes

    
    
      X⃗ = M⋅x⃗
    

and here only M is a matrix. x⃗ is a vector and X⃗ is the discrete fourier
transform of it, also a vector.

~~~
benjamincburns
Total tangent, but man do I love unicode!

Also for future reference, there's a superscript two (² at U+00B2) and three
(³ at U+00B3).

~~~
Dylan16807
Bah, those two are firmly in 8859-1 territory.

------
scast
The book by Cormen et al. has a fantastic explanation of the FFT too.

~~~
mtdewcmu
This is true. I looked up CLR's treatment recently, and it was surprisingly
readable (for CLR). It describes the FT as transforming a polynomial between
its coefficient representation and point-value representation (n point values
are sufficient to fully represent a polynomial of order-bound n). To yield the
point values, the polynomial is evaluated at the n complex roots of unity,
because these numbers have useful properties.

My background in math is thin when it comes to things like complex numbers,
but the explanation more or less made sense to me. CLR was focused on how the
FFT was useful for multiplication, so it didn't really go into all the things
I would have liked to know.

------
af3
Python is too slow for FFT. There is no reason to understand how FFT could be
implemented in python, if at the end of the day, C or Fortran will be used
[1].

[1] [http://www.freeware.vasp.de/FFT/fft-
pentium.f](http://www.freeware.vasp.de/FFT/fft-pentium.f)

~~~
Wilya
Read the article. Quote for you:

> "Though the pure-Python functions are probably not useful in practice, I
> hope they've provided a bit of an intuition into what's going on in the
> background of FFT-based data analysis."

There's very little to be gained by reading pure Fortran code, especially if
you don't know the mathematical tricks/optimisations used beforehand. The
Python code, on the other hand, stays (relatively) readable.

It's not extremely fast (though it stays decent, thanks to Numpy), but it's
not the point.

~~~
af3
> There's very little to be gained by reading pure Fortran code, especially if
> you don't know the mathematical tricks/optimisations used beforehand. The
> Python code, on the other hand, stays (relatively) readable.

Don't agree. Why trying to understand implementation in the language that will
never be used in the production? And, there is a lot to gain by reading
Fortran code.

~~~
Wilya
Because a good part of the optimizations are language-independent. And you
need to understand these optimizations before going further.

The computations which are redundant and can be done once and stored
afterwards don't change depending on whether you're coding in Python of
Fortran. The symmetries that you can exploit to compute less stuff don't
either. The math behind is always the same.

Once you have done that (and gained two or three order of magnitude in
computation time), and you're still not fast enough, then okay, it's time to
drop to C/Fortran and start playing bookkeeper with your memory allocations.
But not before.

~~~
af3
> Once you have done that (and gained two or three order of magnitude in
> computation time), and you're still not fast enough, then okay,

You are talking about the language (or better implementation) that is compiled
to bytecode. The speed increase that you gain by optimization is of the order
of magnitude smaller than by just using language that compiles to machine
code.

Once you understood how it's done in python, then you need to port it to
fortran/c, what is the point?

Fortran is not much different from python in syntax.

~~~
darkarmani
> Once you understood how it's done in python, then you need to port it to
> fortran/c, what is the point?

At that point, why would you port it to fortran/c instead of using FFTW? The
whole point of the exercise was to learn about and optimize the algorithm. If
you can learn the easiest using python, why wouldn't you do your exploration
in python, since going from python to your language of choice should be the
easiest part of the exercise if you choose to do that.

