> So the theoretical minimum is ~1.2 MB. Zstd is only acheiving ~1.4 MB, which is nowhere near the theoretical limit.
> Interestingly, 7-zip compresses the data much better than zstd, reaching a size of 1,298,590 B.
This isn't surprising to me. Zstd is designed for real time compression at I/O speeds (~1 GB/s), so it probably would leave some entropy in exchange for speed, but LZMA (7zip default) is designed for good compression ratios.
His method for computing entropy is not so good. His method works for an independent and unpredictable sequence of bytes, but the actual data is a bunch of signed 16-bit residuals that presumably are normally distributed. Really, he should be measuring the entropy of the original heightmap, but instead he is measuring the entropy of the heightmap after he has applied a couple of confounding transformations (predictive filter, words->byte pairs, and a random shuffle) that act to increase the apparent entropy.
Calculating the exact entropy of data is at least as difficult as breaking encryption. Encrypted data should be indistinguishable from random (full-entropy) noise, but really its entropy should be the entropy of the plaintext data plus the entropy of the encryption key.
There are no comments on particular command line usage of respective compression algorithm frontends.
zstd has higher compression levels that near LZMA and you can additionally train a specific dictionary to be used in compressing particular kind files.
> Interestingly, 7-zip compresses the data much better than zstd, reaching a size of 1,298,590 B.
This isn't surprising to me. Zstd is designed for real time compression at I/O speeds (~1 GB/s), so it probably would leave some entropy in exchange for speed, but LZMA (7zip default) is designed for good compression ratios.