If I read your charts right, your app's single core performance was much better than pbzip2's, which is quite surprising. I thought these apps were severely optimized... Any comment?
Yup, it was quite a bit better on the upper end especially. Looking at the snoops on the bus using PAPI RaftLib does a better job at keeping the cache lines from bouncing.
The benchmarked version also has a dynamically resizing FIFO which uses utilization of the queue itself to guide the sizing. This means that the FIFO can better adapt to dynamic behavior found in most applications run on top of an operating system (most all these days outside of HPC). Looking at load stalls, the RaftLib version has fewer, but not quite enough to account for the results.
If you look at the single worker thread case, then jump to two threads..you can see a fairly big jump. RaftLib by definition is a pipelined programming system. The read file and compress are done perfectly in parallel. The bzip2 code doesn't quite pull it off in a perfectly pipelined fashion. It's close, but not quite. This results in less overlap of execution and communication. If I'd run on Linux (thread affinity on OS X is well, fun last time I checked..if not impossible to do manually), I'd also add thread affinity to the list which most people don't bother to optimize. Hot caches and synergistic cache accesses are quite beneficial.
