
How LZ4 works (2016) - codeismightier
http://ticki.github.io/blog/how-lz4-works/
======
wolf550e
LZ4 is good engineering work, better than Google snappy (zippy) which itself
was better than LZO (both have somewhat improved by looking at each other's
code since then). The current best general purpose compressor for
decompression speed is Oodle Selkie, but it is not open source. I don't if a
fast compressor for Oodle Selkie exists - since it was developed for game
assets, they don't care about compression speed.

------
nerdponx
It's amazing how something useful and innovative gets invented, and isn't so
much as documented, let alone patented or published in a journal.

~~~
userbinator
LZ4 is only one of the _many_ variations on the basic principle of the LZ
family of algorithms, which is to replace repeated sequences with references
to where they were before.

What does amaze me a little, is the fact that the rather more complex Huffman
algorithm was published and implemented decades before LZ.

The core idea of LZ, however, has been known for centuries:
[https://en.wikipedia.org/wiki/Iteration_mark](https://en.wikipedia.org/wiki/Iteration_mark)

~~~
hedora
This is true, there is low algorithmic novelty in lz4.

However, that's missing the point: lz4's decompressor is simultaneously simple
and also the fastest thing aroud, at least at the moment.

In fact, it is so fast, it can be used to accelerate local data transfers over
"slow" links like bonded 10GBit ethernet and arrays of PCIe SSD's.

~~~
kzrdude
Doesn't that also explain why it came later? The low compression ratio is
useful nowadays, with large throughputs.

~~~
hedora
Maybe. It might just be that the tricks it plays matter a lot on newer CPUs,
and older fast compressors played older tricks.

I think the ratio of CPU cycles to i/o bandwidth is what really matters.
Presumably the optimal tradeoff between CPU throughout and compression ratios
depends on that and varies over time.

------
faragon
LZ4 is beautiful.

~~~
snakeanus
Just like DEFLATE.

~~~
faragon
Yes, but faster.

