So in a sense, you are right: PA is not perfect, and neither were ALSA, OSS, and other earlier approaches. That is not the issue; the issue is, is PA actually better? Are we actually able to do things now that we were having trouble with before? From where I sit, PA makes it easier to use things like D-BUS and various "Kit" systems to duplicate the desktop user paradigm you see in Windows or Mac OS X, and nothing more (and that does nothing for me, but certainly gets in my way when I try to do things like multiseat setups).
I remember trying to get bluetooth audio working in the alsa-only days. Or network sound with esd/ssh. Or figuring out how to route audio nicely apps. Or saving my laptop battery. Or many other features that were simply damn difficult to pull off.
People don't seem to remember how crappy the audio situation used to be, for users and for ISVs. I respect Lennart and the current PA maintainers a lot for tackling the problem head on – even if they didn't or couldn't do it perfectly – instead of just complaining about the sad state of Linux audio, especially in the face of all the criticism they got over it.
In exchange for that I put up with confusing audio dialog boxes, high audio latency, high cpu usage when playing audio, skips and crackles when playing games, arcane commands to enable loopback, and the occasional necessary restart of the pulseaudio daemon when sound stops altogether. I guess that's a reasonable trade.
Actually, serious audio work on Linux is done with JACK. PulseAudio is completely inadequate for serious audio work.
Latency vs. CPU usage is always a compromise. To achieve low latency, you need very small buffers for "rendered" audio, and lots of well-timed copy operations to the audio hardware. To just play some audio files efficiently, you want large buffers, because then fewer copy operations are necessary.
Here's a better explanation by Lennart Poettering: