

Massive Speed Gains via Parallelized BZIP2 Compression - SnowLprd
http://hackercodex.com/guide/parallel-bzip-compression/

======
aphyr
_18.7 seconds for bzip2, and… wait for it… 3.5 seconds for pbzip2. That’s an
increase of over 80%!_

Er, not really. How about...

"pbzip2 reduced running time by 80%."

"pbzip2 took only 20% as long as bzip2 did."

"pbzip2 is five times faster."

~~~
SnowLprd
Quite right. Clearly not enough coffee. Updated the article with better
wording. Thanks!

------
wmf
BTW, bz2 is kinda over. Check out xz and the parallel version pxz.

~~~
vasi
I've also got my version of parallel xz: <https://github.com/vasi/pixz>

It doesn't require use of large temporary files like pxz, and the xz files it
produces can also be decompressed in parallel.

~~~
ak217
vasi, thanks very much for writing pixz, I've had a great experience working
with it and it sped up my code very significantly.

The only thing I wish for is more documentation and inclusion in xz-utils
(which I know is not up to you, but I'm hopeful).

~~~
vasi
Glad to hear it's been of use :)

The author of xz is working on a parallel compression implementation of his
own, which will hopefully be in future versions of xz.

------
th0ma5
Since our move to multicore over faster processors, I'm sure we'll see a lot
of this sort of thing, that is, people suddenly realizing that their code will
be some multiple faster if they can find a way to do operations in parallel. I
imagine that the compression itself might be slightly less optimal however
since similar blocks that could be compressed are on different threads? I
didn't dig into how this might or might not be a concern with this project,
however. Long of the short of it, however, parallel is the reality. In theory
one could arbitrarily split the file, and then compress each of the splits and
get a speed up that is roughly comparable?

~~~
stcredzero
_Since our move to multicore over faster processors, I'm sure we'll see a lot
of this sort of thing, that is, people suddenly realizing that their code will
be some multiple faster if they can find a way to do operations in parallel._

Reimplementation of things like compression algorithms seems very
math/algorithm-heavy and thus amenable to functional programming. How about
the Haskell/OCaml guys re-implement a bunch of Un^x style utilities for us?

~~~
th0ma5
in theory gnu parallel can be used to automate an idea in place like xargs

------
sciurus
For parallel gzip there's pigz (pronounced pig-zee).

<http://www.zlib.net/pigz/>

------
dguido
Parallel gzip, in case anyone wanted it: <http://zlib.net/pigz/>

I've used it to great effect during incident response when I needed to search
through hundreds of gigs of logs at a time.

------
malkia
"The results: 18.7 seconds for bzip2, and… wait for it… 3.5 seconds for
pbzip2. That’s an increase of over 80%!"

File cache effect? He should cold reboot first (not sure how you force the
file cache out on OSX/linux, on Windows I do it with SysInternals RamMap) and
try in different order.

It could still be faster, but he could really be measuring I/O that was done
in the first case, and not in the second.

It's also strange that .tar files are used, not tar.bz2 or .tbz (if such
extension makes sense)

~~~
sciurus
On linux, it's

    
    
        echo 1 > /proc/sys/vm/drop_caches
    

<http://linux-mm.org/Drop_Caches>

~~~
minimax
Awesome tip! Thanks!

~~~
malkia
Yup. Thanks scirius! Anyone knows what's the one on OSX? (Or some multi-
platform library that knows more about this stuff (Windows included))?

~~~
eridius
On OS X it's merely `purge`.

------
mattst88
I used to use pbzip2 before I learned about lbzip2 (<http://lacos.hu/>)

lbzip2 is able to decompress single streams using multiple threads, which
apparently pbzip2 cannot do. See the thread beginning with
<http://lists.debian.org/debian-mentors/2009/02/msg00098.html>

------
juiceandjuice
bzip2 has always been parallelizable. At one point a few years ago I was
working on a compressed file format with that included compressed block
metadata, because bzip2 is most efficient when it gets about ~900kB to
compress at a time. In effect, you split the file up into 900kb chunks,
compress them in parallel, and recombine them into one file at the end.

------
Inufu
Is there a reason this is not the default?

~~~
reidrac
I'm GNU tar user (I believe that's the version in most Linux distributions,
but I may be wrong), so I tend to use -z for gzip, -j for bzip2 and -J for xz.

That said, I guess using the "alternatives" framework in Linux it would be
reasonably easy (and transparent) to support the parallel version of each tool
as replacement to the regular one.

------
BrainInAJar
is there a pbzip2 that doesn't eat _all_ your memory ?

------
rorrr
A GPU implementation would be cool.

