First, this is not that much data (~180GB). Is there a particular reason not to just throw it on a hard disk on some machine that doesn't do too much during the night and write a trivial Perl script?
Secondly, (g)zip may not the best solution here. A quick unscientific test on ~3MB of Apache log data (in the default Common Logfile format): gzip or zip produce ~240KB of data, xz (formerly lzma) gets it down to ~80KB (using -9e) or ~96KB (using the default option).
In my quick unscientific test, xz can decompress data about half as quickly as gzip and about ten times faster than bzip2. It's very likely able to keep up with your disk.
First, this is not that much data (~180GB). Is there a particular reason not to just throw it on a hard disk on some machine that doesn't do too much during the night and write a trivial Perl script?
Secondly, (g)zip may not the best solution here. A quick unscientific test on ~3MB of Apache log data (in the default Common Logfile format): gzip or zip produce ~240KB of data, xz (formerly lzma) gets it down to ~80KB (using -9e) or ~96KB (using the default option).
In my quick unscientific test, xz can decompress data about half as quickly as gzip and about ten times faster than bzip2. It's very likely able to keep up with your disk.