
Smaller and faster data compression with Zstandard - jamesgpearce
https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
======
tmd83
I have been waiting for this to hit 1.0 and more importantly get popular so
that I can use it everywhere. I am really a fan of Yann Collet's work. These
are extremely impressive work specially when you consider that lz4 seems to be
better than snappy (by google) and zstandard from LZFSE (from apple). I think
he is the first one to write a practical fast arithmetic coder using ANS. And
look at how his huffman implementation blazes past zlib huffman though
compresses less than FSE [0]. I also like reading his blog posts. While a lot
of them goes over my head I can generally make a sense of what he is trying
and why something's working despite the complexity.

[0]
[https://github.com/Cyan4973/FiniteStateEntropy](https://github.com/Cyan4973/FiniteStateEntropy)

~~~
binarycrusader
> I think he is the first one to write a practical fast arithmetic coder using
> ANS.

I don't think he is the first; although RAD game tools has been cagey about
the details, many strongly suspect their recently announced Kraken, etc. use
ANS in some form and some of their previous products; see the discussion here:

[https://news.ycombinator.com/item?id=11583898](https://news.ycombinator.com/item?id=11583898)

...and here:

[http://encode.ru/threads/2492-Kraken-
compressor](http://encode.ru/threads/2492-Kraken-compressor)

[http://cbloomrants.blogspot.ca/2016_05_01_archive.html](http://cbloomrants.blogspot.ca/2016_05_01_archive.html)

~~~
tmd83
Yeah read about Kraken a while ago and found the encode.ru thread you pointed.
Its unfortunate that we can't compare that side by side. Those guys seem
another of those compression geniuses. Overall the world of compression is
moving very fast and well these days.

But even with that Yann might be the first. Yan's work on FSE is old [0] and
if I'm reading this correctly his work on FSE is a mix of his own work and
Jarek Duda. But it seems one of his failed attempt challenged Jarek to invent
another version of ANS called rANS [1] and as you can see in the encode.ru
comments from the RAD game tools that they seem to be using rANS at least for
Kraken. Regardless these are impressive works and these people are bouncing
ideas of each others, being challenged and inspired which is a very good thing
for the rest of us :).

[0] [https://fastcompression.blogspot.com/2013/12/finite-state-
en...](https://fastcompression.blogspot.com/2013/12/finite-state-entropy-new-
breed-of.html) [1] [http://encode.ru/threads/1821-Asymetric-Numeral-
System?p=360...](http://encode.ru/threads/1821-Asymetric-Numeral-
System?p=36050&viewfull=1#post36050) [2]
[https://github.com/rygorous/ryg_rans](https://github.com/rygorous/ryg_rans)

~~~
eln1
Before Yann, this ANS variant was implemented by Andrew Polar in 2008:
[http://www.ezcodesample.com/abs/abs_article.html](http://www.ezcodesample.com/abs/abs_article.html)
Here is a list of implementations: [http://encode.ru/threads/2078-List-of-
Asymmetric-Numeral-Sys...](http://encode.ru/threads/2078-List-of-Asymmetric-
Numeral-Systems-implementations)

------
levbrie
There is just so much awesome stuff in this article. Finite State Entropy and
Asymmetric Numeral System are completely new concepts to me (I've got 7 open
tabs just from references FB supplied in the article), as is repcode modeling.
I love that they've already built in granular control over the compression
tradeoffs you can make, and I can't wait to look into Huff0. If anyone outside
of Facebook has started playing with it or is planning to put it into
production right away I'd love to hear about it.

~~~
joseraul
Indeed ANS is difficult: it is the biggest innovation in compression in the
last 20 years. Its author has some nice but dense slides about it.

[https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf](https://dl.dropboxusercontent.com/u/12405967/ANSsem.pdf)

Not sure exactly when repcodes were invented. Igor Pavlov has already used
them in 7zip.

~~~
DannyBee
The best TL;DR of ANS is something like this (without being too wrong). It's
still too long:

Huffman requires at least one bit to represent any symbol, because it finds
unique prefix codes for every symbol by varying the leading bits.

Arithmetic coding encodes symbols as fractional numbers of bits, by using
binary fractions. It divides up the range to make this work. In the end, you
get one fraction per "message" that can be decoded back into the message (IE a
fraction like 0.53817213781271231 ....)

range coding is similar, it just uses integers instead of floating point. You
get one number that can be decoded back into the message (IE a number like
12312381219129123123).

Note that in both of these, you have not changed the number system at all. The
number of even numbers and odd numbers still has the same density.

Another way to look at the above is based on what a bit selects.

In huffman, a single bit is generally not enough to select something, the
prefixes are long. So you need to walk a bunch of bits to figure out what
you've got.

In arithmetic or range coding, a single bit selects a very large range. They
change the proportions of those ranges, but at some point, something has to
describe that range. This is because it's a range. Knowing you have gotten to
the 8 in 0.538 doesn't tell you anything on it's own, you need to know "the
subrange for symbol a is 0.530 ... 0.539", so it's a. So you have to transmit
that range.

ANS is a different trick. Instead of encoding things using the existing number
system, it changes the number system. That is, it redefines the number system
so that our even and odd numbers are still uniformly distributed but have
different densities. If you do this in the right way, you end up with a number
system that lets you only require one number to determine the state, instead
of two (like in a range).

~~~
vintermann
With regular arithmetic coding and Huffman coding you don't need to send over
a dictionary. Instead, you can have an adaptive model that learns to compress
the data as it goes (e.g. keeps running track of symbol frequencies), and it
will still be reversible.

I thought this wasn't possible with ANS. Or has this changed?

~~~
eln1
It depends if you have static or adaptive coder.

Static is much cheaper, uses the same probabilities for the entire data block
(e.g. 30 kB), probabilities are stored in the header - practically all Huffman
and tANS compressors (however, there are considered exceptions:
[https://en.wikipedia.org/wiki/Adaptive_Huffman_coding](https://en.wikipedia.org/wiki/Adaptive_Huffman_coding)
).

Adaptive can start with e.g. uniform probability (no need to store in header)
and learns on the way - it is more costly but gives better compression, used
with arithmetic coding or rANS. See
[https://fgiesen.wordpress.com/2015/05/26/models-for-
adaptive...](https://fgiesen.wordpress.com/2015/05/26/models-for-adaptive-
arithmetic-coding/) [https://fgiesen.wordpress.com/2015/12/21/rans-in-
practice/](https://fgiesen.wordpress.com/2015/12/21/rans-in-practice/)

~~~
vintermann
I know that's what adaptive coding means, I thought that was impossible with
rANS. All the descriptions of rANS I could find (back when I looked at it)
described it terms of a static model, and I ran into problems trying to
generalize it to an adaptive one.

Thanks for the resources.

------
cbr
The plot of compression ratio against speed for the various compression levels
is pretty helpful for understanding its performance:
[https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_9...](https://scontent.fsnc1-3.fna.fbcdn.net/t39.2365-6/14146892_944159239044397_638267599_n.jpg)

"The x-axis is a decreasing logarithmic scale in megabytes per second; the
y-axis is the compression ratio achieved."

I'd love to see a version of this chart that also included Brotli. (And I'm
somewhat surprised Brotli isn't mentioned at all.)

(Disclaimer: I work at Google, which made Brotli)

~~~
bryanlarsen
I thought that brotli was tuned for typical web workloads, that it contained a
dictionary tuned for web workloads. Our internal testing shows that it
performs very poorly for binary 3D vector data.

So a test between zstd and brotli would show brotli in a poor light if it used
a mixed corpus, but a test between zstd and brotli on a web corpus would give
an advantage to brotli...

~~~
gribbly
>Our internal testing shows that it performs very poorly for binary 3D vector
data.

Uh ? What format of data was this ?

I did a pretty large test (2gb+) on OBJ/STL 3d data and brotli compressed
within ~5% margin of lzma, and this holds true on other binary data I've
compared.

It also compressed better than zstd (as in compression ratio) on the same data
at their highest respective compression settings:

bro -quality 11 -window 24

zstd --ultra -22

So I find it baffling that you find brotli to be poor for any binary data,
could you share the data in question ?

~~~
haughter
What is the compression time for brotli to achieve compression ratios
comparable to LZMA on these types of data sets:
[https://github.com/google/brotli/issues/165](https://github.com/google/brotli/issues/165)

I understand that brotli is incredibly slow as compared to LZMA when using
these high ratio settings (q11, w24), so slow as to be impractical in
production even in a write-once, read-many scenario if you have any non-
trivial amount of data being regularly produced. I do not want to have a farm
of machines just to handle the brotli compression load of our data sets just
because it is 10x slower than LZMA.

~~~
gribbly
>What is the compression time for brotli to achieve compression ratios
comparable to LZMA...

Compression time is indeed the achilles heel of Brotli, which is why it's
something I would only use for compress once (and preferably decompress very
often) scenarios.

Compared to lzma at it's best compression setting for this particular data
(-mx9 -m0=LZMA:d512m:fb273:lc8), brotli took 6 minutes and 4 seconds to
compress, while the same data took 1 minute and 47 seconds for lzma.

On the other hand, brotli decompressed the same data in 0.6 seconds, while it
took lzma 2.2 seconds.

------
AceJohnny2
Note: this is from the same guy who created the popular lz4 compressor, Yann
Collet: [http://cyan4973.github.io/lz4/](http://cyan4973.github.io/lz4/)

[https://twitter.com/Cyan4973](https://twitter.com/Cyan4973)

------
ctur
Yann will be giving a talk on Zstandard at today's @Scale 2016 conference, and
the video will be posted. He can answer the most technical questions about
Zstandard, but I may be able to answer some as well; we both work on
compression at Facebook.

~~~
tmd83
I am really looking forward to this. I usually like to read more than vidoes
but for complicated topic with a good presenter it can actually be a
comprehensive starting point. Would the video also be posted today or we will
have to wait?

One thing I haven't figured out from either today's post or Yann's blog is
whether Zstandard is switching between huff0 and FSE depending on compression
level or is it somehow using both together? Also the post says its both OoO
friendly and multi-core friendly but the speed benchmarks are those in a
single core context or multi-core? Does only the format/algorithm multi-core
friendly or the standard cli can run multi-threaded.

~~~
ctur
I'm not sure when videos will be posted -- hopefully soon.

All benchmarks today are single threaded. The algorithm itself is single
threaded, but can be parallelized across cores. We will soon release a pzstd
command line utility to demonstrate this, similar to pigz, which accelerates
both compression and decompression.

Zstandard uses both huff0 and FSE together when it compresses -- it doesn't
switch between them based on the input.

------
AceJohnny2
The modern trend of compressors is to use more memory to achieve speed. This
is good if you're using big-iron cloud computers...

 _" Zstandard has no inherent limit and can address terabytes of memory
(although it rarely does). For example, the lower of the 22 levels use 1 MB or
less. For compatibility with a broad range of receiving systems, where memory
may be limited, it is recommended to limit memory usage to 8 MB. This is a
tuning recommendation, though, not a compression format limitation."_

 _8MB_ for the smallest preset? Back in the mid-2000s, I was attending a
Jabber/XMPP discussion, about the viability of using libz for compressing the
stream. It turned out that even just a _32kb_ window is _huge_ when your
connection server is handling thousands of connections at a time, and they
were investigating the effect of using a modified libz with an even smaller
window (it was hard-coded, back then).

I know Moore's law is in ZStandard's favor w.r.t. memory usage (what's 8MB
when your server's got 64GB or more?), but I think it's useful to note that
this is squarely aimed at web traffic backed by beefy servers.

~~~
caf
I don't think 8MB is the smallest preset, the text you quoted says that the
lower levels use "1 MB or less".

The concern I have is that this makes it sound like the compressor can choose
how much memory the decompressor will need to use. Does this mean that zstd
can't be used in a potentially adversarial environment? (Eg. is there a
denial-of-service vector here by forcing the server to use large amounts of
memory to decompress my requests?)

~~~
Cyan4973
The decompressor can receive a parameter to refuse compressed frames requiring
more than a selected amount of memory.

------
ohitsdom
I'm a complete dunce when it comes to compression and how it fits in the
industry, so help me out here. Say that everyone accepts that Zstandard is
amazing and we should start using it. What would the adoption process look
like? I understand individual programs could implement it since they would
handle both compression and decompression, but what about the web?

Would HTTP servers first have to add support, then browser vendors would
follow?

~~~
wongarsu
>what about the web

The browser sends the server a request header indicating which compression
methods it understands. Current Firefox for example sends

    
    
        Accept-Encoding: gzip, deflate, br
    

meaning the server is free to send a response compressed with either gzip,
deflate or brotli. Or the server can choose to send the data uncompressed.

This means the adoption path for the web would be an implemtation in at least
one major browser, which advertises this capability with the Accept-Encoding
header. Then any server can start using Zstandard for clients accepting it.

~~~
ohitsdom
Nice, so really browsers and servers can implement it independently. Seems
elegant, I like that browsers can advertise multiple compression formats.
Thanks for the info!

~~~
riboflava
It's also possible to implement a decompressor in javascript to support
browsers which don't do it natively. The performance would likely suck but if
you're truly bandwidth constrained and don't mind users having a bit of a lag,
it's an option...

~~~
hueving
Assuming the size of the decompressor isn't larger than the savings you gained
from using this compression algo over another...

~~~
cooper12
Well the decompressor would be cached by the browser so it would pay off for
repeat visitors. And if it gets used enough, the user will already have
downloaded the decompressor from another site.

~~~
acchow
And easily shared across sites if they all use a popular CDN'd version.

------
ktta
Some more benchmarks on this[0] page

Also, I actually discovered something _very_ interesting (to me at least). At
the bottom of the link mentioned below, the link attached says
[https://github.com/Cyan4973/zstd](https://github.com/Cyan4973/zstd) but then
redirects to
[https://github.com/facebook/zstd](https://github.com/facebook/zstd) . Anyone
know why?

[0]: [http://facebook.github.io/zstd/](http://facebook.github.io/zstd/)

EDIT: After a little bit of sleuthing, it looks like the author of zstd
(github.com/Cyan4973) is now contributing[1] to github.com/facebook/zstd

And the page layout for lz4[2] looks the same as zstd[0]

Anyone know if Yann Collet works for/with facebook on things other than zstd?

EDIT 2: In the time it took me to google a couple things, looks like the
children comments have already answered my questions.

Also, previous discussions on zstd (not that its completely relevant)
-[https://news.ycombinator.com/item?id=8941955](https://news.ycombinator.com/item?id=8941955)
[https://www.reddit.com/r/programming/comments/2tibrh/zstd_a_...](https://www.reddit.com/r/programming/comments/2tibrh/zstd_a_new_compression_algorithm/)

[1]:[https://github.com/facebook/zstd/pull/312](https://github.com/facebook/zstd/pull/312)
[2]: [http://cyan4973.github.io/lz4/](http://cyan4973.github.io/lz4/)

~~~
CyberShadow
> the link attached says
> [https://github.com/Cyan4973/zstd](https://github.com/Cyan4973/zstd) but
> then redirects to
> [https://github.com/facebook/zstd](https://github.com/facebook/zstd) .
> Anyone know why?

I believe this is just GitHub's standard behavior when a repository is moved
to another namespace. We recently renamed [https://github.com/D-Programming-
Language](https://github.com/D-Programming-Language) to
[https://github.com/dlang](https://github.com/dlang) , and e.g.
[https://github.com/D-Programming-
Language/dmd](https://github.com/D-Programming-Language/dmd) redirects in the
same way.

~~~
ktta
Oh, D language is dlang now? Looks like you guys did a redesign of the website
and stuff too! I hear good things about D, keep up the great work!

~~~
CyberShadow
Thanks! To clarify, we didn't rename the language, just the GitHub
organization. However, dlang.org is what we've been using as the canonical
address for the website, and "dlang" is a much more searchable term than just
"D". (Now, if only Google would stop auto-correcting "dlang" to "golang"...)

------
markonen
The goals sound similar to Apple's LZFSE (see
[https://github.com/lzfse/lzfse](https://github.com/lzfse/lzfse) for more).
Any comparison out there?

~~~
jdcarter
That was my first thought, too. I installed and ran both against a tar'd set
of PDF files totaling 435MB in size. My timings:

    
    
        lzfse  45 MB/s encode, 229 MB/s decode, 1.12 comp ratio
        zstd  181 MB/s encode, 713 MB/s decode, 1.13 comp ratio
    

The numbers are so dramatically different that I ran several different tests,
but those results showed the same rough results. I used default command-line
options for both tools, and both created very similar compression ratios.

Note that LZFSE has a somewhat different goal, however: it's designed to be
the most power-efficient compression algorithm out there, in other words on
mobile devices LZFSE optimizes for bytes-per-watt rather than bytes-per-
second. Zstandard, on the other hand, runs multiple pipelines and such--it's
banking on having a server-class processor to run on.

Edit: hardware is a 2013 MacBook Pro, pretty fast flash storage, and 2 cores/4
threads. I warmed cache before each run and sent output to /dev/null, so the
numbers above are best-case.

~~~
glhaynes
I use "fastest to complete == least power usage" as a rule of thumb because of
"race to sleep". I suppose that might be thrown off by power usage
characteristics varying based on number of cores working? How does one even
begin to write code that prioritizes power-efficiency over performance?

~~~
to3m
For short runs, any fixed-cost warmup/cooldown periods might dominate - a
latency/throughput tradeoff kind of affair. (I am just shooting my mouth off
and have no idea whether that's the case here.)

As for how to do it, I'm not sure... but if I were a valued customer of
various CPU suppliers, and were famous for the depth of my pockets, I'm sure
I'd be able to find somebody to explain it to me ;)

------
Twirrim
From the bits of testing I've done today, it's phenomenally fast on x86. Much
better than gzip (and pigz for that matter) in every metric I think I
generally care about: CPU Usage, Compression Speed, Decompression Speed,
Compression Ratio.

On other architecture the picture gets a bit murky, it seems to get handily
beaten by pigz through what at first blush I'd guess is just sheer
parallelism. It's got solid performance, and without a shadow of doubt faster
than vanilla gzip. If/as/when I get time, it'll be interesting to dig into why
performance is worse there.

~~~
Twirrim
Dug in a bit further. On the non-x86 architecture I use, it looks like it's
really just straight core performance that explains it. pigz's only advantage
there really seems to be the brute force parallelism.

In particular note the huge difference in branches between gzip and zstd on
decompress:

8959780663 branches # 143.024 M/sec

2969481781 branches # 64.454 M/sec

and on misses:

542158823 branch-misses # 6.05% of all branches

89060880 branch-misses # 3.00% of all branches

~~~
Cyan4973
There is now a pzstd implementation, if you wish to compare it to pigz

~~~
Twirrim
Had a shot. It's slightly buggy, but holy crap is it fast.

I'm not a C programmer, understanding what happened is a bit beyond me but: 1)
to compile on linux it needs the -pthread flag passed to it, Makefile is
missing that (compiles fine on OS X) 2) decompression over stdin appears to be
effectively impossible, still demands in input file. Compression over stdin
works fine.

------
ryao
This is an awesome blog post that is very well written, but the lack of
incompressible performance analysis prevents It from providing a complete
overview of zstd.

Incompressible performance measurements are important for interactive/realtime
workloads and the numbers are extremely interesting because they can differ
dramatically from the average case measurements. LZ4 for instance has been
measured at doing 10GB/sec on incompressible data on a single core of a modern
Intel Xeon processor. At the other end of the spectrum is the worst case
scenario for incompressible data where performance slows to a crawl. I do not
recall any examples in this area, but the point is that it is possible for
algorithms to have great average case performance and terrible worst case
performance. Quick sort is probably the most famous example of that concept.

I have no reason to suspect that zstd has bad incompressible performance, but
the omission of incompressible performance numbers is unfortunate.

~~~
Cyan4973
zstd goes at > 1 GB/s on uncompressible data. It has some fast heuristics for
such cases too.

------
morecoffee
A recent compression discussion I saw involved how do compressors fare on
uncompressible input? For example, suppose you wanted to add compression to
all your outbound network traffic. What would happen if there was mixed
compressible traffic along with the uncomressible kind? A common case would be
sending HTML along with JPEG.

Good compressors can't squeeze any more out of a JPEG, but they can back off
fast and go faster. Snappy was designed to do this, and even implementations
of gzip do it too. It greatly reduces the fear of CPU overhead to always on
compression. I wonder how Zstd handles such cases?

*Ignoring security altogether

~~~
nimrody
Dropbox say they can _losslessly_ compress JPEG files (~20% saving):

[https://blogs.dropbox.com/tech/2016/07/lepton-image-
compress...](https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-
saving-22-losslessly-from-images-at-15mbs/)

~~~
gcp
Known techniques. JPEG uses Huffman coding for entropy coding, you can replace
that with arithmetic coding. This requires knowing the format details, though.

------
esaym
>> "It is written in highly portable C, making it suitable for practically
every platform used today"

I love C, it is not the enemy everyone makes it out to be.

It's already in debian:
[https://packages.debian.org/stretch/zstd](https://packages.debian.org/stretch/zstd)
and judging by the small requirements,it is portable indeed.

------
xrstf
Quick benchmark on a 194MiB SQL dump:

    
    
        gzip -9: 27.574s, 48MiB output
        zstd -9: 14.182s, 41MiB output
    

Thanks, I'll gladly use zstd as a drop-in replacement for my daily backups. :)

------
tambourine_man
Isn't it a bit presumptuous to call your own thing "standard"?

~~~
partycoder
Like these guys:
[https://github.com/feross/standard](https://github.com/feross/standard),
which is permissive to the point it's completely pointless. You can follow it
to the letter and still end up with vomit code.

------
mrmrben
Really nice work compared to what I consider to be the quite bad Brotli -- an
incredibly slow compression standard that only ended up in browsers because it
was created by Google.

~~~
tessela
except for the PATENTS file =/

~~~
ac29
Curious what your issue is with it -- it basically says "if you dont sue us,
we wont sue you". Thats about as good as I can expect from a large tech
company these days with regards to patents.

~~~
gcp
Compare it to the Opus patent license (also including a retaliation clause),
which includes grants from Broadcom, Mozilla and Microsoft.

Using zstd gives Facebook a free license on ALL your patents.

Using Opus gives Facebook only a license on patents that apply to Opus.

So no, large tech companies can and have given MUCH better grants for
compression tech, than Facebook is doing.

~~~
tmd83
I don't understand these things well. But if true this would be really bad.
LZ4 is everywhere because it was completely free and I think most of
Zstandard's real work was pre-facebook by just Yann alone. Now to have its
hand tied because of his job at facebook is the worst thing possible.

I just read the Opus patent summary. It seems like if zstd followed the same
license using it wouldn't give facebook any license but if I sue facebook I
loose the licese to use zstd. Am I correct in that.

~~~
gcp
_Now to have its hand tied because of his job at facebook is the worst thing
possible._

If you accept the job you give up your work. It's not like there's no
compensation.

 _Am I correct in that._

Yes, the retaliation (and hence, implicit license to Facebook) is only for the
technology itself.

------
cromwellian
I think for typical JS/CSS/HTML sizes, and decompression times, probably
maximum compression ratio, followed by decompression speed is what I'd look
for. I don't care too much about compression speed, in the sense that if I
have to spend 1 minute compressing JS to crunch it by 10%, but I serve that
file a million times, then as long as decompression doesn't negate the gain in
network time saved, it's a win.

I guess the other factor for mobile is, besides memory and decompression
speed, how do various compression schemes fare battery wise?

~~~
nikkun
Regarding decompression times, I think it's much more important to save
transferred network data than to prioritize quicker decompression.

HTTP request times have a long tail; they can get really slow for the many,
many people limited to slow connections. Decompression times are going to be
much more consistent. Our aim here should be to improve the 10% slowest
requests, and you do that by optimizing the actual transfer.

------
erichocean
For this to become anything like a standard, Facebook would have to remove its
patent poison pill.

------
yread
If anyone wants to try it on windows there is a 7-zip install with support for
ZSTD

[https://mcmilk.de/projects/7-Zip-zstd/](https://mcmilk.de/projects/7-Zip-
zstd/)

------
mydeardiary
If facebook hopes the new compression algorithm to be a standard, why doesn't
it publish an IETF RFC draft? Will it follow OpenDNS way of dnscrypt by open-
sourcing the reference implementation without publishing any IETF RFC draft?

------
espadrine
The following link points to a fairly good benchmark / tool that showcases the
tradeoffs in real life: since (de)compression takes time, what is the fastest
way to transmit data at a given transfer speed?

[https://quixdb.github.io/squash-
benchmark/unstable/#transfer...](https://quixdb.github.io/squash-
benchmark/unstable/#transfer-plus-processing)

Spoilers: zstd wins at ethernet and wifi (and is among the best in 4G), lz4
wins at hard drive encryption… both were designed by the same author.

~~~
jacobolus
Charles Bloom’s Oodle new codecs handily beat them across the scale:

[http://cbloomrants.blogspot.com](http://cbloomrants.blogspot.com)
[http://www.cbloom.com/oodle_arm_report/](http://www.cbloom.com/oodle_arm_report/)
[http://www.cbloom.com/rants.html](http://www.cbloom.com/rants.html)

Caveat: proprietary commercial product.

------
anjanb
I was looking for a windows version of zstd. On the github page, I could only
get the version 0.81 version of the windows tool. Can someone release the 1.0
version of zstd for windows ?

------
nemo1618
How difficult is this new standard going to be to implement in another
language? It seems highly sophisticated -- which is great, of course -- but
the cost of that is relying on giants like Facebook to maintain their One True
Implementation. For software this is (usually) fine; for a nee standard, it's
a problem.

~~~
ctur
The format itself is documented
([https://github.com/facebook/zstd/blob/master/zstd_compressio...](https://github.com/facebook/zstd/blob/master/zstd_compression_format.md))
with the intention of other implementations and language bindings being
readily available. We also have a zlib-compatible API for easier porting to
applications already using Zlib. Our hope is that Zstandard is both easy to
use and easy to contribute to.

------
kstrauser
That's truly beautiful. Thanks, Facebook! I particularly love that you can
pre-compute and reuse dictionaries, say if you're regularly compressing
similar JSON objects.

------
mana_12
Zstandard is both a command line tool (zstd) and a library.

------
philplckthun
I'd love HTTP content encoding to support this and see a comparison to Brotli.
Looks like it might be yet another good alternative to gzip.

------
mozumder
Is this being pushed as being standard as part of the HTTP spec, seeing that
it comes from Facebook?

~~~
mrmrben
Just about anything is better than brotli.

------
bnolsen
turbohf claims to be 4x faster than zlib's huffman coding and 2x faster than
FSE and is a generic cpu implementation. Even if claims are only partially
true and turbohf is a clean dropin replacement for zlib and licensing were
friendly the appeal of zstd drops substantially in my book.

[http://encode.ru/threads/2276-TurboHF-1GB-s-Huffman-
Coding-R...](http://encode.ru/threads/2276-TurboHF-1GB-s-Huffman-Coding-
Reincarnation)

[https://sites.google.com/site/powturbo/entropy-
coder](https://sites.google.com/site/powturbo/entropy-coder)

------
partycoder
Unless they integrate it into software like web servers and web browsers it
will be hard to see it really flourish as a "standard".

But at least within the perimeter of your own systems you can totally profit
from this technology now.

------
faragon
Beautiful.

------
z3t4
Tables should be in HTML.

------
Grishnakh
Looks very interesting, however I'm not impressed by the name. "Zstandard"???
With ".zstd" as the extension? I don't like it.

They should have named it _letter_ -zip, along the lines of gzip, bzip, and
xzip, with the extension _letter_ z. "fz" would have been a good one since
they work at Facebook.

~~~
VanillaCafe
[https://en.wiktionary.org/wiki/bikeshedding](https://en.wiktionary.org/wiki/bikeshedding)

~~~
Grishnakh
This isn't bikeshedding. Bikeshedding is about quibbling over unimportant
details. Names are critically and absolutely important. Lots of great things
have been hobbled or ruined by poorly-chosen names. A terrible name can cause
something worthy to be ignored in favor of something inferior but with a
better name. And you don't need to even be competent in the inner workings of
a project to criticize its name or suggest better ones. The people who name
cars aren't the same people who design the engine-control algorithms for them.

If you disagree, what do you think of naming your kid "Adolph Hitler
[lastname]"? Most people agree that a name like that will cause great harm to
a child growing up because of the constant ridicule and ostracization he'd
inevitably face. That's an extreme example, but names _are_ important.

Note that I don't think this project's name is horrible, but I don't think
it's very good either, and could be a lot better.

~~~
Blaisorblade0
[https://en.wikipedia.org/wiki/Godwin%27s_law](https://en.wikipedia.org/wiki/Godwin%27s_law)
? :-D

------
ilostmykeys
How does this compete with PiedPiper?

~~~
bbcbasic
This is general purpose whereas Pied Piper, Hooli et al. focussed on video.

------
cristiandan
Awesome

------
bananaoomarang
Best middle-out in the game.

------
lasryaric
Whats their weissman score?

~~~
brickmort
I bet it doesn't even touch pied piper's.

------
f137
Probably nictpicking but "Smaller data compression" makes no sense really

------
kaushalp88
Should we start with the pied piper jokes now or later?

~~~
bbcbasic
Not until I know the Weissman Score of Zstandard.

------
DJ_Icebear
Should've named it "Pied Piper"

------
DJ_Icebear
They should've named it "Pied Piper".

