
Zstandard – Real-time data compression algorithm - josephscott
http://facebook.github.io/zstd/
======
tzs
> Zstandard is a real-time compression algorithm, providing high compression
> ratios. It offers a very wide range of compression / speed trade-off

This would have been very handy about 17 years ago when I was doing a virtual
CD-ROM driver, and needed to store compressed images, and the client wanted a
wide range of speed/compression options for the user--much wider than zlib
offered.

I ended up doing a hack that was either the most disgusting thing I have ever
done, or was brilliant. I have not been able to decide which.

My hack: I gave them a compression/speed slider that went from 0 to 100. If
the slider was set to N, I would apply zlib at maximum compression to N
consecutive blocks, and then apply no compression to 100-N consecutive blocks.
Repeat until the whole image is stored.

The client loved it.

~~~
derefr
Reminds me of how LEDs are dimmed using Pulse Width Modulation: "40%
brightness" actually means "on at 100% intensity for 40% of the length of each
refresh cycle, and then completely off for the rest."

~~~
szemet
Or playing PCM on the old PC speakers having only on/off beep function... It
seemed like a miracle at the time :)

~~~
vardump
> Or playing PCM on the old PC speakers having only on/off beep function

That _is_ PWM, pulse width modulation.

[https://en.wikipedia.org/wiki/PC_speaker#Pulse-
width_modulat...](https://en.wikipedia.org/wiki/PC_speaker#Pulse-
width_modulation)

~~~
szemet
Ok then: playing PCM with PWM...

------
ComputerGuru
zstd is incredible, but just in case the thought hasn't occurred to someone
here that may benefit from it: if you're in control of the send and the
receive, type-specific compression is hard to beat.

For example, if you know you're dealing with text, you can use snappy, if you
know you're dealing with images, webp, videos x264 (or x265 if you only care
about decode speed and encoded size), etc and then fall back to zstd only when
you don't have a specific compressor for the chosen file type.

~~~
Smerity
Agreed that if you have the control (and potentially the time depending on the
algorithm), type specific compression is the way to go. Having said that, zstd
beats Snappy handily for text ^_^

On enwik8 (100MB of Wikipedia XML encoded articles, mostly just text), zstd
gets you to ~36MB, Snappy gets you to ~58MB, while gzip will get you to 36MB.
If you turn up the compression dials on zstd, you can get down to 27MB -
though instead of 2 seconds to compress it takes 52 seconds on my laptop.
Decompression takes ~0.3 seconds for the low or high compression rate.

~~~
hokkos
Why not use a XML/XSD specific compression like EXI for that ?

[https://www.w3.org/TR/exi-primer/](https://www.w3.org/TR/exi-primer/)

~~~
cldellow
It really is mostly just text. It's not _quite_

<Article>... [30 kb of text] ...</Article>

but almost.

------
beagle3
I used to object to using zstd based on Facebook's onerous license and patent
policy; but now that zstd is plain BSD+GPLv2, I endorse it.

------
terrelln
Zstd recently added a long range mode that can find matches up to 2 GB in the
past using a specialized algorithm [1]. It can be enabled on the command line
with `zstd --long` for a 128 MB window, or `--long=windowLog` for a
`2^windowLog` byte window.

[1]
[https://github.com/facebook/zstd/releases/tag/v1.3.2](https://github.com/facebook/zstd/releases/tag/v1.3.2)

~~~
exikyut
I'm not sure if you'll see this, but I have a question. I have a situation
where I need to implement in-place compression with perfect rollback in case
of failure. The first approach I thought of was feeding LZMA input until I had
X amount of output, then finalizing and starting over, but then I started
wondering if I could simply periodically serialize the LZMA state machine data
every 30 seconds or so.

I haven't investigated either approach, but this is one of my next projects
I'm working on, so I'll be figuring it out either way. I was wondering, does
Zstd provide any ability to serialize/restore and continue where it left off?

(Context: Disk compression. Due to long-term factors beyond my control I've
had severe disk space issues for over a decade (with frequently only tens of
MBs free). Complex story ending in "cannot work". I recently realized that
some of my disks contained data I didn't immediately need, and began wondering
about ways I might be able to compress this data out "of the way" so I could
carefully partition the remaining space. This would need to be done in-place
as I would have nowhere with enough space to hold the compressed data, at
least not to begin with.)

~~~
mkesper
Lending/buying an empty (external) drive for starting seems to be much simpler
and safer.

~~~
exikyut
That would be a highly preferable option.

The last time someone gave me an old empty 320GB HDD they weren't using, one
of my own disks started clicking about a week later and I was able to save
everything on it. I still shake my head at that perfect timing.

Of course, this meant I lost all the additional free space, haha. One step
forward, two steps back...

------
loeg
We added zstd to largely supersede gzip use at work. It performs better than
oldschool gzip in both compression and speed.

That said, I'm not sure I'd call it a "real-time" compression algorithm. It's
still an factor of 2x slower than lz4 compress and a factor of 4x slower
decompress.

~~~
michaelmior
It's not clear exactly what "real-time" is supposed to mean, but I don't think
it makes sense to define it in terms of other algorithms. If someone developed
an algorithm faster than lz4, would lz4 cease to be real-time?

~~~
joosters
I thought that 'real-time' is not about how _fast_ a program is. Instead, it's
about being able to guarantee that a program will always complete in under a
fixed amount of time, under all circumstances.

I can't see how that applies to Zstandard either.

~~~
hinkley
It’s a definition not used much anymore but this relates to on demand
(streaming) versus ahead of time data processing.

With only a small delay, can I compress and decompress at least one second of
data per second, indefinitely.

Think live TV versus film.

~~~
chowells
That's usually called an "online" algorithm, in my experience. It does seem to
agree with the way it's used in this context.

~~~
foota
Online isn't quite the right distinction here imo, since that's really more
about possibility than speed. An algorithm could be online (doesn't require
all the data at once to run) but still not be fast enough to stream.

------
waterhouse
A curious fact I encountered: First, the zstdcat executable that ships with
zstd (actually a symlink to zstd itself, roughly equivalent to "zstd -cqdf")
can decompress .gz files. Second, zstd is _faster_ than gzip at decompressing
.gz files, taking about 2/3 as long.

I'm not kidding. I couldn't believe it myself, but subsequent testing
stubbornly bore it out—on one file that was 15 MB after compression, and on a
mix of smaller files. I tried compiling gzip from source, using the same
compiler I used for zstd, and the results were the same. strace seemed to show
zstd read and wrote in chunks 2x the size, but the number of syscalls didn't
seem to be nearly enough to explain the difference. It seems to have this
"zlibWrapper" subdirectory; its README has some benchmark numbers that I don't
fully understand, but some of them seem to match the 2/3 factor I got.

I'm wondering if this is a clever tactic to drive adoption—get people to use
the zstd executable even when they're still using gzipped files. ;-)

Also, the fact that it has support for reading (and, apparently, writing)
gzip, lz4, and xz on top of its own format really makes "z standard" an
appropriate name.

~~~
emj
A zlib feature is being patent free, that might very well affect speed. I do
not know if that is the issue you are describing.

~~~
waterhouse
So, the zstd executable links to libz.so (says ldd), and I don't think it
reimplements the core algorithmic part of gzip. I gather that it's simply more
intelligent in how it uses the API... e.g. it says some things about reusing
contexts, though that seems to apply only to handling multiple files.

------
cmurf
Supported on Btrfs since kernel 4.14. Like other compression options, it's a
mount time option.

Edit: Also in squashfs. Here's the git pull request which includes some
benchmarks.
[https://lkml.org/lkml/2017/9/11/369](https://lkml.org/lkml/2017/9/11/369)

~~~
terrelln
Upstream SquashFUSE also has zstd support.

[https://github.com/vasi/squashfuse](https://github.com/vasi/squashfuse)

------
desertrider12
I wonder how the RAD Game Tools compressors would fit into that benchmark
list. In their benchmarks Oodle Selkie has roughly the same ratio as Zlib but
is 3-4x faster at decompression than Zstd (not the same benchmark though).
[http://www.radgametools.com/oodle.htm](http://www.radgametools.com/oodle.htm)

~~~
blattimwind
Asset compression has a specific optimization function (Charles Bloom of RAD
Game Tools has a ton about this on his blog, cbloomrants): maximize output
bandwidth given this specific CPU and input bandwidth.

Compression speed, for example, doesn't really matter. Sure, it should be
something reasonable (e.g. not like PAQ), but if you can bump the above metric
by 10 % while compression takes 50 % longer: probably worth it.

~~~
desertrider12
Aha, he actually addressed zstd in a post. tldr is zstd usually lower ratio
and lower speed than mermaid but sometimes better ratio for text.

------
jdhawk
We've been using zstd for a while in production, switching over from
rar/gzip/7z for archiving huge XML files. It's speed to compression ratio is
really impressive. Hats off to the development team.

------
cjhanks
I use the C bindings for ZStandard, it performs very nicely as an algorithm. I
wish the documentation was a little easier to use. I am still not sure if the
provided examples really properly handle the degenerate cases of /dev/urandom
and /dev/zero when using `ZSTD_DStream{In,Out}Size` and friends.

In any case - thanks for releasing this, it's been very helpful to me.

------
bajsejohannes
Some good discussion in earlier postings too:
[https://hn.algolia.com/?query=zstandard&sort=byPopularity&pr...](https://hn.algolia.com/?query=zstandard&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

------
reacharavindh
Anyone using zstd along with an incremental rsync approach? Like gzip
--rsyncable ?

Say I have 1000 files. I want to compress them and let the cron rsync do it's
thing. Next day, if only one file had changed, rsync should pickup only the
differential instead of the whole archive.

Or is there a better way of doing it?

------
cakoose
In the "Compression Speed vs Ratio" graph, the Y-axis doesn't start at zero.
If it were changed to start at zero, it would be easier to evaluate the
magnitude of the compression ratio change at a glance. IMO, that's probably
worth the extra whitespace in the graph.

------
revelation
From that benchmark list I'd prefer lz4..

~~~
terrelln
lz4 and zstd are complementary, not competitive. lz4 targets applications
where extremely fast decompression speed (and compression speed) are required,
at the expense of compression ratio. However, many applications would prefer
to spend a bit (or a lot) more CPU in exchange for better compression, while
maintaining fast decompression speed.

------
lars_francke
Last time this was discussed Zstandard didn't have a splittable mode and by
the looks of it[1] they still don't. That doesn't make it a bad algorithm it
just means that it's not a good choice yet for Hadoop et al. As far as I know
no container format has implemented Zstandard yet.

Does anyone know any better? It seems like we could use a better alternative
to Snappy.

[1]
[https://github.com/facebook/zstd/issues/395](https://github.com/facebook/zstd/issues/395)

~~~
gopalv
> As far as I know no container format has implemented Zstandard yet.

ORC has a reserved future enum for ZSTD [1] (my guess is that FB already uses
it).

The original headache was the patent licensing scheme[2] for the actual
codebase, but now that hadoop has a ZStandard built into libhadoop.so, this is
much more easy to ship to Apache.

My branch is so far behind now that I probably would implement it faster if I
started over & used the libhadoop.so impl as the basis -
[https://github.com/t3rmin4t0r/orc/tree/zstd](https://github.com/t3rmin4t0r/orc/tree/zstd)

[1] -
[https://issues.apache.org/jira/browse/ORC-46](https://issues.apache.org/jira/browse/ORC-46)

[2] -
[https://github.com/facebook/zstd/issues/775](https://github.com/facebook/zstd/issues/775)

------
jakozaur
In a lot of big data workflows, zstd is no brainer replacement of gzip. Always
faster decompression, plus you can save tons of storage and data transfer.

------
jokoon
When one talks about general data compression, isn't that relevant that it
generally applies to text compression, and not video or audio?

I guess general data compression works on audio and video, but most of the
time you either choose to compress text, audio, video or you create a file
format that indexes your data.

------
teej
We switched many of our Redshift compression encodings to zstd. Works
fantastically on text and JSON.

------
karmicthreat
Not really totally this topic, but I deal with a bunch of 8 Gig images of
device firmwares. Is there anything that can make compressing these and de-
duplicating these images fast? I have used borg in the past, but archive
operations are just seem slow.

~~~
cjalmeida
You may want to store them uncompressed and rely on ZFS block level
deduplication. It's similar to borg but may be faster.

Also, you may want to play with borg options and use lz4

------
unixhero
So like 'middle out'

------
powturbo
see Compression Benchmark at
([https://github.com/powturbo/TurboBench](https://github.com/powturbo/TurboBench))

------
2pointsomone
That's the best Weissman score I have ever seen.

~~~
2pointsomone
Come on people, take a joke

------
natbobc
Why is this site an SPA?!?!?

~~~
akx
It's... not? It's a static page with some JavaScript-driven charts.

~~~
userbinator
It's a page with the majority of the content wrapped in an XMP element with
style="display:none;". I'd consider that pretty hostile.

------
DyslexicAtheist
TL;DR: Facebook acquires "Pied Piper" to gain exclusive rights on the Mean-
Jerk-Time (MJT) compression patent; renames it to Zstandard for political
correctness.

------
gateway7
That patent clause though. Stick to the fork from before Facebook bought the
man and the algorithm.

~~~
terrelln
It was relicensed in v1.3.1. It is now dual licensed under BSD without the
patents clause [1] and GPLv2 [2]. GPLv2 was added for inclusion in the Linux
kernel.

[1]
[https://github.com/facebook/zstd/blob/dev/LICENSE](https://github.com/facebook/zstd/blob/dev/LICENSE)

[2]
[https://github.com/facebook/zstd/blob/dev/COPYING](https://github.com/facebook/zstd/blob/dev/COPYING)

