I can't link with fftw3 since it's GPL. The other libraries that were permissive licensed had confusing APIs and the output binaries were almost as large as my entire project. CPU usage is high but because of one poorly designed part of the GUI (eqhover.cpp)
Why not license your software under the GPL? For end-user applications like these, there's no reason software shouldn't be GPL, and if fact it would be better for you since it prevents companies from exploiting your work that you intended to serve the greater good for purposes of their own profit.
Did you test against pulseeffects cpu usage?