Actually, serious audio work on Linux is done with JACK. PulseAudio is completely inadequate for serious audio work.
Latency vs. CPU usage is always a compromise. To achieve low latency, you need very small buffers for "rendered" audio, and lots of well-timed copy operations to the audio hardware. To just play some audio files efficiently, you want large buffers, because then fewer copy operations are necessary.
Here's a better explanation by Lennart Poettering: