
CDN77 Now Supports Brotli - clarinois
http://blog.cdn77.com/cdn77-now-supports-brotli/
======
jgrahamc
We took a very detailed look at Brotli performance (both speed and
compression) and concluded that for dynamic content it was only useful if the
file size was greater than 64k on slow connections and there's a very detailed
description here:

[https://blog.cloudflare.com/results-experimenting-
brotli/](https://blog.cloudflare.com/results-experimenting-brotli/)

If you want to experiment with it, it's enabled on CloudFlare's test server
[https://http2.cloudflare.com/](https://http2.cloudflare.com/).

Since we don't charge by byte the focus is on end-user performance and not
'saving money' and we concluded that it wasn't currently worth implementing
Brotli widely but will continue to experiment with it.

~~~
KurvaKde1334
There's a lot of stuff greater then 64k nowadays ;)

~~~
clarinois
Exactly :) If Cloudflare decides it’s worth several man-days to offer this
feature to their clients who deliver larger files than 64k, feel free to reach
out to us, we’ll be happy to share our experience :) devs@cdn77.com

~~~
jgrahamc
Thanks. We actually wrote and open sourced our nginx modifications for
choosing between Brotli and gzip.

[https://github.com/cloudflare/ngx_brotli_module](https://github.com/cloudflare/ngx_brotli_module)

------
ape4
I see its dictionary is optimized for web content. Seems a bit like cheating
;)
[https://en.wikipedia.org/wiki/Brotli](https://en.wikipedia.org/wiki/Brotli)
Unlike most general purpose compression algorithms, Brotli uses a pre-defined
120 kilobyte dictionary. The dictionary contains over 13000 common words,
phrases and other substrings derived from a large corpus of text and HTML
documents.[6][7] A pre-defined algorithm can give a compression density boost
for short data files.

~~~
NelsonMinar
What's wrong with cheating?

~~~
kevingadd
It leads people to believe this will be a compression improvement for all
their content, but after they do the work to integrate it they may discover
that it's only a slight improvement because their content isn't identical to
the content its static dictionary was designed for.

It's at least a very good codec, though, so it's still a win for other data.
Just smaller than you might expect.

~~~
xnyhps
Also note that the dictionary is strongly biased towards English. Sure, there
is some Russian, Chinese, Arabic (and probably some other scripts in there
which I don't recognize), but there seems to be more English words in there
than all those others combined. If you're compressing small documents in any
other language than English, it might not be worth it to use Brotli.

Edit: They wrote this about it in
[http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf](http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf)
:

> Unlike other algorithms compared here, brotli includes a static dictionary.
> It contains 13’504 words or syllables of English, Spanish, Chinese, Hindi,
> Russian and Arabic, as well as common phrases used in machine readable
> languages, particularly HTML and JavaScript. The total size of the static
> dictionary is 122’784 bytes. The static dictionary is extended by a
> mechanism of transforms that slightly change the words in the dictionary. A
> total of 1’633’984 sequences, although not all of them unique, can be
> constructed by using the 121 transforms. To reduce the amount of bias the
> static dictionary gives to the results, we used a multilingual web corpus of
> 93 different languages where only 122 of the 1285 documents (9.5 %) are in
> languages supported by our static dictionary.

~~~
xnyhps
Here's the list of words, by the way. I couldn't find it anywhere in non-
hexadecimal form:

[https://gist.github.com/xnyhps/677f7c1b444f346bef99](https://gist.github.com/xnyhps/677f7c1b444f346bef99)

(I cleaned it up a bit to remove newlines and tabs, and a couple that are
entirely of unprintable characters.)

------
bhouston
I find that Brotli is over sold for the case of compressing generic binary
files.

The claims that it is comparable to xz/lzma for generic binary data are not
accurate.

In my real-world tests of compressing 3D data it far underperformed xz/lzma
although it was still better than gzip:

[https://github.com/google/brotli/issues/165](https://github.com/google/brotli/issues/165)

~~~
kevingadd
It's fairly competitive with LZHAM, at least, even if it's way slower to
compress.

You will get much better results out of Brotli if you restructure your data to
be more compressible, and that will also improve your lzma and gzip (
_especially_ gzip) compression ratios, to a tremendous degree. Have you done
any of this? If not, ping me, and I can explain some techniques to apply.

~~~
detaro
> _You will get much better results out of Brotli if you restructure your data
> to be more compressible, and that will also improve your lzma and gzip
> (especially gzip) compression ratios, to a tremendous degree._

This sounds interesting, I'd like to read some examples/links/explanations.

~~~
EdHominem
I imagine a lot of it is segregating your content. All strings in one file,
etc.

It'd be an interesting test to take our eight-language set of localization
strings and compress them in UI order and language order and see if there's
much of a difference. (UI order is all languages for one dialog element, then
all eight for the next, etc. Language order is all the English first, then
...)

I'd definitely like to hear Kevingadd's tips though.

------
grandalf
Like Facebook's Dragon, Brotli is an algorithm that is optimized for typical
usage patterns.

Similarly, an entire IOS device could be fabricated as a single ASIC, and (for
example) uikit could be fabricated as part of that ASIC.

There is always a tradeoff between generic optimization and usage-specific
optimization which comes at the expense of flexibility.

Google can do a statistical analysis of all the data it serves compressed with
gzip, and determine exactly the characteristics of a compression algorithm
that would save the most money.

These are small, evolutionary optimizations that save tons of money by
incrementally increasing efficiency in a large system.

------
sremani
Is it true that Brotli wanted the .bro file extension and moved away from it,
because it was deemed offensive?

~~~
Bud
Yes, that's confirmed.

"In late September, Google released a compression algorithm called Brotli and
gave files it makes the extension “.bro”.

But last week the extension was changed to “.br”.

The reason for the change is threads like this one, in which posters suggest
that “'bro' has a gender problem” and “comes of[f] misogynistic and
unprofessional due to the world it lives in.”

[http://www.theregister.co.uk/2015/10/11/googles_bro_file_for...](http://www.theregister.co.uk/2015/10/11/googles_bro_file_format_changed_to_br_after_gender_politics_worries/)

~~~
plugnburn
Symbian app installation files have ".sis" extension. Why can't we have ".bro"
for parity?

~~~
SwellJoe
Because no one has used Symbian in 10 years and the vast majority of us have
never seen or heard of a .sis file? Not really parity if .sis is
extraordinarily obscure and .bro becomes pervasive.

~~~
plugnburn
Don't speak for the entire world ("no one"), I personally know two guys that
still use Symbian-powered smartphones and don't plan to change them to
anything.

~~~
SwellJoe
I know more people than that who still use a Commodore 64 on the regular. But,
it doesn't make it a viable platform.

------
jakubstr
Nice. Hope to see it in FF soon. And in Internet Explorer 87.

~~~
TheRealPomax
[http://caniuse.com/#search=brotli](http://caniuse.com/#search=brotli) tells
us FF already supports this, and IE has it marked as being under
implementation consideration (which one would expect to turn into active
development once they finish their current WOFF2 support work; WOFF2 has a
hard dependency on a working Brotli implementation)

~~~
KurvaKde1334
Try nginx & FF45 -> doesnt work as FF uses deprecated (and now obsolete &
removed) API

~~~
bzbarsky
Reference for this, please? Because I see no bugs filed on FF regarding brotli
apart from
[https://bugzilla.mozilla.org/show_bug.cgi?id=1222541](https://bugzilla.mozilla.org/show_bug.cgi?id=1222541)
and
[https://bugzilla.mozilla.org/show_bug.cgi?id=1207234](https://bugzilla.mozilla.org/show_bug.cgi?id=1207234)
neither of which seems to match your problem description. I'm happy to get a
bug filed for you so whatever problem this is can be fixed, but I need a bit
more to go on here...

I did look around for known brotli+Firefox+nginx issues, but the only one that
comes up is
[https://bugzilla.mozilla.org/show_bug.cgi?id=1215724](https://bugzilla.mozilla.org/show_bug.cgi?id=1215724)
which was fixed in shipping Firefox months ago, so I'm assuming that's not the
one you're talking about.

~~~
KurvaKde1334
Sure, there is no official bug for this. The only way to see it is to check
the dates when the new API was applied - its far too late after releasing
versions already supporting brotli, but this commit was not backported
anywhere so in a real life, its broken in FF. Just get an nginx, nginx brotli
module from google and use FF 45. Good luck :)

[https://hg.mozilla.org/releases/mozilla-
aurora/log/10e1774de...](https://hg.mozilla.org/releases/mozilla-
aurora/log/10e1774de80e/modules/brotli/dec/decode.h)

~~~
bzbarsky
> but this commit was not backported anywhere

Which commit are you talking about, exactly? Bug 1242904 was backported to
Firefox 45, and everything else I see at [http://hg.mozilla.org/mozilla-
central/filelog/tip/modules/br...](http://hg.mozilla.org/mozilla-
central/filelog/tip/modules/brotli/dec/decode.h) as of today (which is the
same as <[http://hg.mozilla.org/mozilla-
central/filelog/ea6298e1b4f7/m...](http://hg.mozilla.org/mozilla-
central/filelog/ea6298e1b4f7/modules/brotli/dec/decode.h>)) was checked in way
before 45....

~~~
bzbarsky
Ah, looks like the part of bug 1242904 that was backported is just the
security fix, not the entire library version update and the version update is
in 46. So what you could be seeing is something like
[https://bugzilla.mozilla.org/show_bug.cgi?id=1254411](https://bugzilla.mozilla.org/show_bug.cgi?id=1254411)

Of course the version update wasn't supposed to change the on-the-wire
behavior... or so the library authors claimed. :(

------
frewsxcv
If anyone wants to check out a Rust implementation:

[https://github.com/ende76/brotli-rs](https://github.com/ende76/brotli-rs)

It's currently in use in Servo

------
pmlnr
Are there any memory & CPU consumption graphs?

I also don't understand why not xz, though I know that requires significantly
more resource than gzip.

~~~
TheCondor
Brotli actually has levels of compression that are more dense than gzip and
compress/decompress faster. There is also a spec, xz has a reference
implementation but no spec.

~~~
nacs
> more dense than gzip and compress/decompress faster

If you're saying brotli compresses/decompresses faster that gzip, it doesn't
according to Cloudflare tests [1] (see table near bottom).

Even the fastest compression level of Brotli is slower than the
highest/slowest compression level of gzip in most cases.

[1]: [https://blog.cloudflare.com/results-experimenting-
brotli/](https://blog.cloudflare.com/results-experimenting-brotli/)

~~~
cornstalks
> If you're saying brotli compresses/decompresses faster that gzip, it doesn't
> according to Cloudflare tests.

It doesn't always _compress_ faster, but as far as I can tell they didn't
measure or say anything about _decompression_.

------
cromwellian
I think decompression speed is more important. I'm willing to burn compression
time precompressing resources. For more resources, especially JS and images,
you can achieve precompression. Or, if you compress on the fly, you can cache
the result.

------
superiphone77
Heh, Brotli + http2, it looks so Fast:
[http://www.http2demo.io](http://www.http2demo.io)

------
Joky
So since this is a "dictionary" based algorithm, what about specializing the
dictionary per content (content/type=javascript would have a different
dictionary than html for instance). Also to embed Brotli for my custom
application format, what should I expect by specializing the dictionary for my
use case?

------
GigabyteCoin
Title should be changed to "...25% improvement over Gzip"

re: "Brotli should bring 25% reduction in data size compared to Gzip for the
most common assets like Javascript and CSS files. For HTML, Brotli promises up
to 40% difference (with median around 25%)."

------
KurvaKde1334
Official comparison by google

[https://cran.r-project.org/web/packages/brotli/vignettes/bro...](https://cran.r-project.org/web/packages/brotli/vignettes/brotli-2015-09-22.pdf)

------
charlesju
Is this a Silicon Valley TV show joke or is this real?

Not sarcasm, real question.

~~~
detaro
What made you wonder about it being a joke?

------
SureshG
Do we have any implementation for Brotli encoder/decoder on JVM or JNI/JNA is
the best option available right now for using Brotli on jvm apps?

