1. Use pretty standard LZ77 sequence matching to previously seen data.
2. A move to front encoding of the LZ77 references.
3. A canonical Huffman encoding of the MTFed references.
But as far as I understand, the canonical Huffman encoder implies that the compressor can only operate on files, not streams.
These sort of initiatives should be loudly celebrated; and I think they will be more common as the end of Moore's law progresses, and as hardware will become less and less generic.
My bet: in less than 10 years we're going to see an open source kind of chips like the following:
The JavaChip™, maintained by OpenJDK; available at your local cloud, provided by TSMC:
* Zipline acceleration for your data needs
* GC acceleration
* Code and data caches modes specialized for small threads
* UTF-8-based datagram dynamic parsing pattern
* Explicit prefetcher control
* The JVM's profiling data done in the hardware, more extensively
* Collections.sort() as a hardware bitonic sort where appropriate
* Facilities for predicate pushdown to the SSD / NIC
Basically: all the specificities of your very specific programming model down in the hardware.
I use it almost daily. It is a really nice Verilog simulator. I also use Verilator, ModelSim and other sims too . All of them find different issues with the code. The linter in Verilator is very nice to use to get rapid feedback.
I haven't tested, but Icarus Verilog should be able to handle the code.
But anyway, I am not a believer in this because a lot of data that I'm aware of flies to the cloud already pre-compressed. Not because it saves hard space, but because it saves bandwidth and time during transfer from user land. I've never seen any algorithm take a compressed file and smash it down to anything less than 98%. Maybe I'm a loon living in a niche area though.