I don't want each language providing its own package manager. Sure, CPAN was a great leap forward back before apt became popular, but now I want to just apt-get install libraries. Language-specific package managers always fall short of apt because they don't deal with external dependencies - cabal isn't going to install the sqlite libraries when I install the bindings, and even if it did, I doubt they're going to take stability and security updates as seriously as Debian does.
But for real, it's not that hard to integrate your language library distribution thingy into something like Debian. Using what the OP was talking about as an example, CL has had the Common Lisp Controller (http://www.cliki.net/common-lisp-controller) in Debian for a long time now (6 years? more?). This still doesn't help answer "which libraries should I use? which ones should I avoid?" For something like that, the problem with the CL community is that people are generally reluctant to criticize each other's work.