
Stateless ZIP library – SLZ - marcopolis
http://1wt.eu/projects/libslz/
======
JoshTriplett
Every DEFLATE compression block can either supply its own Huffman table or
compress using a fixed table. You can emit a DEFLATE-compatible compression
stream without ever emitting a custom Huffman table or depending on previous
data. And you'll still do better than nothing for many types of data.

Given this, I wonder why HPACK invented a new compression scheme rather than
just carefully using DEFLATE.

~~~
andrewf
The HPACK spec [1] remarks on this. Google carefully used DEFLATE in earlier
SPDY versions [2] but HTTP/2 decided to be less clever.

Interestingly, SLZ's page comments on lack of history as a way to avoid CRIME
attacks, which is one of the things the HTTP/2 people worried about. That
seems like something it'd be easy for an implementation to screw up though
(just add some buffering somewhere before the compressor...)

[1]
[https://http2.github.io/http2-spec/compression.html#rfc.sect...](https://http2.github.io/http2-spec/compression.html#rfc.section.1)
[2] [https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-
dr...](https://www.chromium.org/spdy/spdy-protocol/spdy-protocol-
draft3#TOC-2.6.10.1-Compression)

------
doublerebel
Looks like SLZ hit 1.0 back in April [1]. Has anyone used it in production?
Are packaged binaries available to get it up and running quickly?

[1]: [http://git.1wt.eu/web/libslz.git/](http://git.1wt.eu/web/libslz.git/)

------
donatj
Very interesting. This appears to do much better something I've been working
on a PHP wrapper to do for a while now. I ended up currently opting out of
compression as a whole and just using STORE zips to send multiple files at
once without having to keep data in memory. This is absolutely worth looking
into. It's been a long time since something on HN has gotten me this excited.

------
striking
Absolutely incredible. I had no idea this was even possible. Defeating CRIME
while keeping compression? I can hardly believe it.

~~~
amluto
AFAICT this only avoids CRIME if there's never both attacker-controlled data
and a secret in the same SLZ compress call.

~~~
bonyt
But I think the idea is that the compress call takes such a small amount of
data that it is far less likely for there to be such data in the same call

~~~
mikeash
It goes beyond that, and suggests explicitly dividing up the data into
sensitive and potentially attacker-controlled pieces, and processing those
pieces separately. In short (and I could be wrong, but this seems to be the
basic idea), if you always start a new chunk after you're done sending headers
and before you start sending content, you're safe.

------
trippy_biscuits
How does it address quines?

[http://research.swtch.com/zip](http://research.swtch.com/zip)

~~~
X-Cubed
It's just a compressor, it doesn't provide any decompression support.

------
hinkley
So would this be useful for compressed virtual memory? It only does compress
so it would only halve the memory usage but that's something.

~~~
andrewf
If you're a webserver and want to serve clients which support gzip encoding,
you need to be DEFLATE compatible.

For something like virtual memory where you'd control both the compressor and
decompressor, there's little point in clever DEFLATE-compatible compressors.
Instead, just pick any compressor which has the compression/CPU tradeoff you
want.

------
logicallee
Ah yes, for that elusive "malloc-safe" spec requirement.

~~~
tcas
Having a fast malloc less compression library is extremely useful in resource
constrained embedded systems world, where either memory usage should be
minimized (you can still run modern Linux kernels on systems with 32MB of RAM,
probably less), or where traditional memory management isn't available, such
as a microcontroller.

In the microcontroller case you can get a 32bit ARM Cortex M0 for $2.00 qty 1
these days, allowing the bit-twiddling you find in compression libraries to
compile successfully. Having a fast malloc-safe compression library is
extremely useful. lz4 is an example of one which you can run on a
microcontroller successfully, but isn't as compatible as zlib is.

~~~
logicallee
hey, this answer was really interesting! I was actually making a (poor) joke
about the quixotic idea of requiring a desktop program to be just fine with
malloc failing, no biggie. your answer is much more interesting.

~~~
baruch
Large embedded systems also like malloc-less solutions. In such systems you
essentially have explicit use for all the memory you have and you don't like
to buy extra memory that is unused (reduces your margins). When you need to
account for all memory needs and you use allocation schemes you need to create
memory pools that are not too large (create waste) and not too small
(performance bottle or deadlock source).

One such example are large SAN storage systems. We can have gigabytes or even
a terrabyte of ram and still have very little spares (little by our standards
< 100MB for diagnostics and general OS nuisance).

------
warmwaffles
May not be useful to me, but it is still cool either way.

------
ladon86
How would one use this with HAProxy?

