OK, I'll bite. This article is overblown in its criticisms and has some issues of its own.
1. The article itself recommends that "a tool is supposed to do one thing and do it well". Well, xz is for compression, nothing else. If you want to check integrity, use a hashing tool such as sha256sum. If you want to recover from errors, generate parity files using par2 or zfec.
2. The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
3. It's not clear what the author means by "long-term archiving". Archives professionals (as in, people who are actually employed by memory institutions) will tell you of many factors (all unmentioned here) that have a bearing on whether a file format (particularly a compressed one) is suitable for use in data preservation.
4. The section on trailing data is particularly bizarre. By claiming that xz is "telling you what you can't do with your files", it seems that the author considers it perfectly reasonable to append arbitrary data to the end of a file and expect it to continue functioning. Is my normality detector off today or is this just a wacky thing to want to do?
5. The article is written by the author of lzip (which it compares favorably to xz), but does not disclose this. Admittedly it is hosted on the lzip site, but overall the article comes across as an opinionated hit-piece designed to sow doubt in a competitor.
> The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
No: that's why the LZMA compression format was chosen, and why the .lzma container format temporarily became popular as people started using lzma-tools. The move to xz, which results in slightly larger files for seemingly arbitrary reasons and which technically provides one benefit (seekability) almost everyone defeats (by compressing a tar file) while using the same relying on the same underlying compression algorithm is less-well motivated.
Were there many prominent open source projects using the LZMA format before the creation of xz? Debian, for example (which is where this particular flamefest originates), adopted[0] xz the year following the stable v5.0.0 of XZ-Utils[1].
My impression was that the whole 7zip->p7zip->lzma_alone->xz journey was concluded before any "xz hype" began.
I don't quite understand the timeline you are drawing here but the Debian Package Manager is a perfect example of a prominent open source project which used the LZMA format before the creation of xz. They recently shifted to "deprecate" the LZMA format, a change which I have removed from the negligible fork of dpkg I ship to the tens of millions of people running Cydia.
Regarding #4: Padding out a file is popular because of tape drives. You'll typically write the archive to the an appropriately sized tape raw, without a filesystem (which is just another format you have to maintain a parser for). So because the file will be smaller than the tape, there will be garbage at the end of the tape. Because we wrote the file to the tape directly, there is no way of knowing where it ends, and a tool reading the file will need to just deal with the garbage data.
Indeed. This is why the tar format explicitly allows garbage data at the end. So then people started pondering all of the nifty or clever things they could do with tar files. And they didn't want to give it up when they started compressing the tar files.
Tar is just bundling into a single file AFAIK. There is a slight benefit depending on your compression tool to tar and then compress, because (AFAIK again)some tools compress files individually and then write them into a hierarchical file(I guess this is what xz does as well, since it's searchable?). If you tar first, these tools will work better, since they encode patterns found in all files instead if doing it per file(which means e.g. if there is a header once per file, that will get compressed in the tar.comorpress, not in the . compress)
Making a backup with tar is done by typing something like that on bash:
> tar -c - dir1 dir2 dir3 > /dev/tape
That will (hopefully, I doubt I got the tar switches right) backup those dirs into the tape (that will actually have a weird name, not '/dev/tape').
Now, in practice Linux doesn't always know the size of a tape you inserted. But this is not the issue, if you accept the seeks needed for that, you'd better write at the beginning anyway.
Considering neither lzip nor xz is a commercial product, I don't think your final point stands. To me, it only makes sense that the person who wrote a "competing" file format would have a strong negative opinion of xz
When I use xz for archival purposes I always use par2[1] to provide redundancy and recoverability in case of errors.
When I burn data (including xz archives) on to DVD for archival storage, I use dvdisaster[2] for the same purpose.
I've tested both by damaging archives and scratching DVDs, and these tools work great for recovery. The amount of redundancy (with a tradeoff for space) is also tuneable for both.
I recently did the same, but used tar without compression. For starters most of the data I was archiving didn't compress well (like photos), but mainly if par2 fails, I can still recover at least some of the data.
With compressed files you usually need it to be perfect to recovery any data, but you can use fixtar [0] to recover some data from a corrupted tar archive.
I used BluRay as the media cost is about the same as DVDs per GB here (at least 25GB discs). Also there are HTL discs available, which supposidely have a greater longevity than DVDs.
I'm afraid my archiving and testing are not very systematic. I don't usually go back and test my burnt DVDs, except when I need to get data off of them, which is not very often. But so far I haven't run in to issues, and have been pretty lucky with the DVDs I've used. I don't think I can recall a situation where dvdisaster actually had to recover from errors on any old DVD. Of course, it's a good idea to transfer valuable data from old DVDs to new ones every now and then, as they won't last forever.
I just tried a DVD from 2013 (Verbatim DVD-R, 16x, 4.7 GB) and it read fine, without any errors, and dvdisaster found that the checksums on it matched.
I usually try to use at least 20% redundancy, but will settle for less if the data's not very important. If the data's really important I'll max out the amount of redundancy (up to dvdisaster's limit, which I don't remember off-hand). Sometimes I'll even burn an extra DVD with the same data, both with dvdisaster error correction on them.
As for par3, I only heard about it for the first time today in this thread. So I have no opinions on it except to say that if it's an improvement over par2, I'm all for it. Backwards compatibility with par2 also would be nice.
The article's right that you probably should not use xz (or even any similar formats) for archival purposes.
However what's totally awesome about xz as a container format is seekable random access to compressed files. I use it to store infrequently used disk images, which I can boot up without even decompressing them. (https://rwmj.wordpress.com/2013/06/24/xz-plugin-for-nbdkit/)
I agree completely, in fact it seems odd that the article is really complaining about the format being more complicated than bzip2 or gzip, when the comparison is not really apples-to-apples. Comparing it with zip would seem more sensible to me.
A compressed container format that allows quick access to individual files is very useful and actually improves data security - corruption in one part of the data will be far less likely to ruin the entire collection of files, whereas corruption in a compressed tar file may lead to the loss of everything (I know recovery tools exist, but they cannot prevent one file in the container being dependent upon data from another file.)
You are confused. xz does not provide the ability to find individual files inside of the format: it is not at all comparable to zip files, and like the other formats mentioned it compresses one file. The "seekable" property is that it lets you somewhat efficiently decompress arbitrary byte ranges inside of the file, which is why a compressed disk image (as used by your parent commenter) benefits from this property but the industry-standard usage of compressing a tar file (which is a file format which inherently makes it impossible to find individual files without reading the whole thing) totally throws away this benefit.
A tar file doesn't have a central directory like a zip, so you do have to search all the file headers. However, each file header contains the length of the file it describes, which lets you seek past the content of any files you don't care about if the tar isn't stored in a way that would prevent seeking.
Yeah: what they mean by "container format" here is similar to the usage in video compression file formats, where ".avi" doesn't imply any particular compression algorithms. An xz file is a container format which is designed for use to store another container format.
The article's point is, in part, that xz is more like zip or rar than gzip or bzip2, and use of it like those is incongruous (e.g. .tar.xz files). (Its other points are that xz has a number of systematic flaws, in its error detection and some "misfeatures".)
Ah - I presumed that the debian xz-based packages were using it as an archive, instead of the old 'ar'. But I just looked and you are right, the xz support is for the data.tar.xz that itself is still inside the 'ar' archive format. That is odd. I guess they chose this way for simplicity - far less code needs changing in all the .deb file processing utilities.
I can't speak for Debian, but I personally did not realize that xz was as I stated above before reading this article, so I wouldn't be surprised if some people simply didn't realize it.
Unsafe: 1 Likely to cause severe data loss even in case of the smallest corruption (a single bit flip). 2 Likely to produce false negatives.
It does seem like xz is somewhat overengineered, but I don't think that's a characteristic unique to it; any other algorithm with similar compression performance will yield similar behaviour on corrupted data, since the whole point and why compression works is to remove redundancy. I say use something like Reed-Solomon on the compressed data, thus introducing a little redundancy, if you really want error correction.
I haven't read the whole article yet, but gzip and bzip2 recover fairly well from errors. As they are divided into blocks, flipping a bit will make you lose, at most, one block. This is not related to error correction though, but a way of mitigating losses.
>> Just one bit flip in the msb of any byte causes the remaining records to be read incorrectly. It also causes the size of the index to be calculated incorrectly, losing the position of the CRC32 and the stream footer.
As they are divided into blocks, flipping a bit will make you lose, at most, one block.
What happens if bits in the block header are corrupted? If it can't find the start of the next block, the same thing will happen.
Also, breaking up the data into blocks will decrease compression, since each block starts with a fresh state. It is ultimately a tradeoff between compression ratio and error resistance.
> If it can't find the start of the next block, the same thing will happen.
This has been addressed in the article. "Bzip2 is affected by this defect to a lesser extent; it contains two unprotected length fields in each block header. Gzip may be considered free from this defect because its only top-level unprotected length field (XLEN) can be validated using the LEN fields in the extra subfields. Lzip is free from this defect."
> Also, breaking up the data into blocks will decrease compression
This has been tested very thoroughly. Larger block sizes give rapidly diminishing marginal returns (man bzip2). Now, the largest you can go with bzip2 is 900KB.
gzip usually references the data from the previous window in the current window. If you lose one block, you're likely to lose the following one. And the one following it. etc. until the end of the compressed data.
Par2 has inherent limitations that make it look distinctly historical (no Unicode etc.). There is a preliminary Par3, but the main reason why Par doesn't enjoy the popularity it should have is simple: Abysmal tooling.
I've tried all the usual implementations that are available on Windows recently, and they were all unusable. Like in "try to give it a hundred files, each in a single-megabyte range, and it will crash hard".
Why would par2 need to handle anything related to encoding? It's acting on the data independently of the actual contents of the data.
I'd agree about the tooling problems with it, it's "acceptable" on a linux/unix command line but I've never seen anything elsewhere that even looked halfway usable.
> It does seem like xz is somewhat overengineered, but [...] any other algorithm with similar compression performance will yield similar behaviour on corrupted data
You seem to confuse format and compression algorithm. From what the
article says, the format seems bad. Once any data is damaged, all the rest of
reading the file (as opposed to decompressing the data from file) goes out
of the window. No way to re-synchronize reading with data blocks after
corruption.
Of course, any archival strategy that leaves you vulnerable to the problems discussed in the article is defective. You ought to be backing up to multiple physically distant locations if you're really concerned about archival.
The argument makes more sense for the casual user who isn't paying for a professional backup service. But how many of those are using xz? They're more likely to be using zip files; and they'll be in a better place for it.
The zip (concatenation of compressed streams) vs tar.gz (compression of concatenated streams) distinction is useful in an archival context, if one is genuinely worried about bitrot and the risk of needing to recover partial data where the container is damaged. A concatenation of compressed streams will have worse compression especially for lots of small files, but it is far easier to recover from an error.
How does TAR mitigate the issues mentioned in the paper? Sincerely curious, my knowledge is limited. AFAICS, XZ can still "corrupt" the TAR stream and you'll need quite a bit of extra error correction to recover from that.
All of the arguments here seem to assume corruption (even extensibility is a form of it), and to treat it as a thing that can be fixed by the decompressor. It seems xz isn't ideal in that case, but none of the others are great either. As mentioned elsewhere in this thread, if integrity is your primary concern you should use PAR or some other error-correction mechanism. If you just need a binary integrity indicator you should have separate hashes published somewhere.
xz wasn't chosen for its archival integrity AFAIK, it was chosen for its size. Package distribution ought to spend as much resource as practical to reduce bytes over the wire. Size is why bzip2 "won" over gz despite being many times slower and requiring much more RAM. lrzip would be my first choice for maximum compression, but its downstream resource requirements are far too high vs any of the others, so from my experience xz would be the answer anyway.
> xz wasn't chosen for its archival integrity AFAIK, it was chosen for its size.
No: that's why the LZMA compression algorithm is being used. The switch from the original LZMA container format to the "improved" xz container format had nothing to do with size, as the old container generated slightly smaller files in addition to not having this combination of negative properties.
I remember the hype around LZMA2 and xz being that, because it uses blocks, you can do the compression and decompression on multiple threads. LZMA was generally considered too slow to use for regular usage (only for archival, basically) and multicore was the new way to scale so this was perfect.
The time for complaining about xz has well and truly passed. One might infer that the problems with xz are related to the attitude of other developers...
While I agree that you should use PAR-like error correction systems when doing archival, one of the points of the article is that xz has poor error detection and recovery, compared to any of the other formats available.
If it offered none, and explicitly stated that, it would be one thing, but it offers some that functions quite poorly in practice, which is rather another.
It's only a problem if you are relying in it to detect and correct errors. I've got SHA checksums and FEC and ZFS and cascading backups for that, which are deliberately not relying on one code provider not having critical bugs. Also, I may have crypto, and I need error detection and FEC on the outside of that. So it seems like a storm in a teacup really.
I think there is scope in xz to embed some FEC in the archive as a backwards compatible extension. Not such if the decoding scheme could be changed to reduce error cascades etc.
> This article started with a series of posts to the debian-devel mailing list [Debian], where it became clear that nobody had analyzed xz in any depth before adopting it in the Debian package format.
Debian source packages are cryptographically signed. Debian archives are also cryptographically signed. There's no error-correction, but you don't want that. You just want to verify integrity every time you make a copy, and re-copy if it didn't copy right. And all the tools do that.
I remember this discussion on debian-devel, and there were a lot of people pointing out that the author's guesses at Debian's threat model had little to do with Debian's actual threat model, at which point the author became frustrated.
The Debian file format also doesn't take advantage of any supposed benefit of xz over lzma, and so is simply taking in the cost of xz's larger file size for what I consider to be "absolutely no good reason" (and as a major downstream of dpkg I pointed this out to its author a month or two ago when I re-asserted non-deprecated status for .lzma Debian packages as now one of only two noticeable changes I maintain to dpkg).
I very strongly disagree with the author's complaint that "Xz is unreasonably extensible". The history of computers is littered by numbers and numbers of restrictive design choices that ended up biting us in the ass later. Let's design FAT12. Oops a 32MB file size limit is no longer sufficient? Let's make FAT16, err I mean FAT32, err no really NTFS... How about 7-bit ASCII, err 8-bit ISO-8859, err I mean 16-bit Unicode, err no I really mean 32-bit Unicode, now that should be sufficient, right?
So it was the right choice for the Xz format to use variable-length integer encoding. Not all integer fields will need to represent values up to 2^63, but it doesn't matter.
I don't think the variable-length encoding itself is the author's concern, it's adding additional filters later without an overall version number. Unlike long file names in FAT32, adding additional filters is not backwards compatible. With a version number, it's super clear what can decode a file. Without, your xz file is only decodeable if your decoder implements the same set of filters as the encoder, rather than the much simpler approach of gzip where any gzip decoder can decode any gzip file.
That's certainly an interesting read but if I really care about something I'll usually generate PAR files. At that point, does it really matter what archiver I'm using at the time or should I still be worried?
People who professionally care about long term archival usually don't care about heavy compression, as storage costs are always dwarfed by data governance and other costs.
Individuals may care about space more because the storage or medium is expensive for them. Even then, bulk data like photos and video doesn't compress well, so this argument is still pointless.
ZPAQ is a defined standard format which embedds the decompression algorithm for each block as bytecode into the archive. One can define a transformation (eg. LZ77 or color transformations of a picture) and compress this data through context mixing and arithmetic coding.
It uses checksums but I can't judge rightnow whether there is some issue implied concerning bitflips.
But of course for real world use cases one would choose a general compression algorithm or write a specific one for a certain type of data you want to handle (because it will perform better than a general algorithm).
Is there any actionable advice here for a Windows user? I mean let's say one is using 7zip which defaults to .7z format. Does that correspond to any of the formats discussed in the article? Would it be better to use a different format or a different program?
"the right way of implementing binary filters is to write a preprocessor that applies the filter to the data before feeding them to the compressor. (See for example mince)."
"We present a novel technique to boost the compression of sequencing that is based on the concept of bucketing similar reads so that they appear nearby in the file"
I'm still not sure I understand that fully, but guesstimate that it shuffles file content in such a way that it compresses so much better that it more than offsets the space needed to store the information on how to deshuffle the decompressed data. Is that correct?
Or does it require you to specify the parts and discards the reshuffling information, assuming you aren't interested in the order?
I love rzip but it is important to note that it is not a stream compressor. So it is unsuitable for most of the use cases of LZMA2 which can compress data on the fly or have data appended on the end of a file at a later date.
So rzip is probably excellent for archival situations (not counting any flaws the algorithm may have) but not so great for things like compressed tarballs (which are meant to be appendable).
> The only reliable way of knowing if a given version of a xz decompressor can decompress a given file is by trial and error. The 'file' utility does not provide any help:
I just put files in tars because everything is already compressed. Really, 99% of formats already have their own compression - you might get a couple % by recompressing them. Maybe a few text files that I have are not compressed but I can probably live with them taking up space.
I know it's not a compression format. My point is most formats that you put in tars are already compressed! Jpgs - compressed, RAWs - already compressed, any text format that isn't a plain-text file - already compressed. Compressing compressed formats is hardly worth it.
Not really related to the article, but would you look at that web page. This is what all of them should look like! Nothing fancy, just good content, 64.91KB in total.
That's because your browser sucks. Try Opera Mobile or something similar. Good browsers can reflow text to fit whatever screen size they display at. This website has zero CSS related to layout, so it is your browser's duty to render it nicely.
It boggles my mind why reflow is not one of the most important and most highly regarded features of any mobile browser.
The problem is quickly solved on Firefox thanks to its "Fit text to width" addon [1].
>it is your browser's duty to render it nicely
I think browsers should have the ultimate say on how to display websites, according to the user's preferences (think of text size and colors). Sadly, with websites gradually transforming into "web apps", users are losing control.
even after 'zooming in'?
it didn't create a new viewport for me after doing that that.
on opera mobile: are you truly comfortable to use a 'default on man-in-the-middle-ware' from china? (it proxies everything and compresses it, to save data bandwidth)
i stopped using it in August because of that. might've been too paranoid though.
Yeah, they should add the meta/viewport with scale=1 tag to the head, and it would improve readability on phones. I mean it't annoying that the web is in a state where that is necessary, but until it's solved, it's better (and not much more difficult) than nothing.
1. The article itself recommends that "a tool is supposed to do one thing and do it well". Well, xz is for compression, nothing else. If you want to check integrity, use a hashing tool such as sha256sum. If you want to recover from errors, generate parity files using par2 or zfec.
2. The article claims xz was chosen by prominent open source projects due to hype. No, it was chosen because of its favorable size/speed tradeoffs.
3. It's not clear what the author means by "long-term archiving". Archives professionals (as in, people who are actually employed by memory institutions) will tell you of many factors (all unmentioned here) that have a bearing on whether a file format (particularly a compressed one) is suitable for use in data preservation.
4. The section on trailing data is particularly bizarre. By claiming that xz is "telling you what you can't do with your files", it seems that the author considers it perfectly reasonable to append arbitrary data to the end of a file and expect it to continue functioning. Is my normality detector off today or is this just a wacky thing to want to do?
5. The article is written by the author of lzip (which it compares favorably to xz), but does not disclose this. Admittedly it is hosted on the lzip site, but overall the article comes across as an opinionated hit-piece designed to sow doubt in a competitor.