
‘You can't use Brotli for dynamic content’ and other misconceptions - nailer
https://certsimple.com/blog/nginx-brotli
======
linsomniac
We just had tried brotli and decided not to implement it yet.

One of our devs just returned from a conference super excited about brotli.
I'm the ops guy and want to try to support excitement, so I spent the better
part of 2 days setting up an experiment, but I was skeptical. I'd been keeping
my eye on brotli for years.

I first went out and looked for some existing evaluations of brotli, and found
years old write-ups, one from Cloudflare from 2015, and another but I can't
remember the details there. They both concluded that brotli wasn't ready yet,
but they were also old...

Our stack uses Apache on Ubuntu 16.04, and the first stumbling block was that
the Apache version on 16.04 is just a bit before the mod_brotli was added. So
I used Ansible to spin up an 18.04 web server which did have the right version
of Apache. But, apache is built without brotli support. So I ended up having
to build my own custom packages, which went fairly smoothly.

We have one particular asset that right now we would like to speed up. It is a
big JSON data structure. The dev did some testing and found that his 350K
data-set (with gzip) ended up as 320K with brotli.

Maintaining our own custom Apache packages, and possibly having to upgrade our
entire stack from 16.04 to 18.04, plus resolve known problems with OS packages
that we've seen in 18.04, for a <10% compression gain, for our cherry-picked
"best example for brotli wins"?

Our decision was that if we could have just turned it on via changes to the
Apache configs, we would have done it. But as it is, it's not worth it for us
to do.

YMMV.

~~~
rgbrenner
None of that has anything to do with brotli. Blame the distro or the package
maintainer.

~~~
orev
2.5 is the development branch. That should not be used in production.

~~~
linsomniac
The versions I was speaking of were Ubuntu 18.04 = 2.4.29, Ubuntu 16.04 =
2.4.18, mod_brotli was introduced in 2.4.26.

------
jbergstroem
If you want to attempt building nginx with brotli support, let me save you 15
minutes: Google's original module is more or less abandoned. Look here
instead:
[https://github.com/eustas/ngx_brotli](https://github.com/eustas/ngx_brotli)

Another alternative is exploring h2o/caddy as a web server replacement.

~~~
nailer
(deleted, I see your point now)

~~~
icebraining
But that PPA is using Google's repo as upstream; what's the point of the OS
alerting you to updates, if there are none?

------
Solar19
The CertSimple article completely ignores CPU and memory usage. That is where
Brotli can become completely unacceptable for dynamic compression – it uses
vastly more CPU and RAM than gzip.

Brotli is fine if you can precompress static files on a prep server or
desktop.

It's unprofessional and unscientific to evaluate compression codecs without
measuring how much CPU and memory they use. Brotli is a disappointment, coming
20 years after gzip. In 20 years we should expect something better than
Brotli, something that is better optimized for modern CPUs, vectorized by
default, and with a much better dictionary.

~~~
d33
How about zstd?

[https://github.com/facebook/zstd](https://github.com/facebook/zstd)

~~~
joosters
I find zstd to be almost twice the speed of brotli when compressing files (my
simple test case is a 55M directory of small files - about 2k each)

However, zstd doesn't have the browser support that brotli does. So you can't
easily replace brotli with zstd for web use.

~~~
JyrkiAlakuijala
Zstd compresses 5% less densely. Zstd compresses slower to a given density.
Decoding zstd may use 128 mb of ram whereas Brotli may use up to 16 mb.

------
thecompilr
FWIW at Cloudflare we were running brotli for dynamic content for a while now.
However the Cloudflare gzip library is much faster than brotli. You can find a
benchmark in here: [https://blog.cloudflare.com/arm-takes-
wing/](https://blog.cloudflare.com/arm-takes-wing/)

------
halayli
Why not standardize zstd? it seems to beat brotli in ratio,comp/decomp based
on the benchmarks below:

[https://facebook.github.io/zstd/](https://facebook.github.io/zstd/)

~~~
klodolph
Just get browsers to implement it, I think by the time Zstandard was open
sourced, there was some browser support for Brotli already (probably because
Google controls a major website, a major browser, and the standard library for
a major OS).

Timeline: Zstandard was released August 31, 2016. By that point, Firefox,
Chrome, and Opera already had Brotli support. Safari and Edge got it shortly
afterwards.

~~~
khaled
I think browser support for Brotli is mostly a side effect of supporting WOFF2
font format (which uses Brotli).

------
noname120
Worth noting for backend engineers: every major browser supports Brotli
compression, and 85.22% of users run a browser that supports it[1].

[1] [https://caniuse.com/#feat=brotli](https://caniuse.com/#feat=brotli)

~~~
WorkLifeBalance
I wish it wasn't, but IE11 is still a major browser, at least too big to just
dump completely in my experience. The UK is worse than most countries in that
regard though with higher than average IE11 usage.

~~~
dictum
Indeed, but this is not a problem for this enhancement specifically. IE11 asks
for gzipped files, and the server can still send them when the client doesn't
support Brotli, based on the Accept-Encoding header.

------
jgrahamc
Mirrors our experience: [https://blog.cloudflare.com/results-experimenting-
brotli/](https://blog.cloudflare.com/results-experimenting-brotli/) but we
chose to use Brotli level 5.

~~~
jbergstroem
Does Cloudflare pass a dictionary when compressing for your clients? Would be
interesting to hear about excursions in testing dictionary sizes/variations at
that scale.

~~~
jgrahamc
I'll suggest to someone internally that they write up all the details of how
we support Brotli.

------
nailer
Edit: the title was changed again to it's current form(a moment ago it was
'brotli can be smaller and faster than gzip'). Thanks HN mods.

Original post below.

>

Author here. This was submitted with:

    
    
        'You can't use Brotli for dynamic content' and other horses**t (2018)
    

Odd the title was changed, you can see from the timestamp I spent yesterday
updating it for 2018 (brotli is more of a big deal now iOS users can use it).
Also the original title highlighted the very specific point that a lot of what
is written about brotli online is incorrect. Most HN users are grown ups and
have handle a starred-out swear word. Oh well.

~~~
paulie_a
I never understood the starring out thing. If the forum is inappropriate for
that word, then using stars is also inappropriate.

------
paulddraper
Brotli is amazing. But...

> Brotli can compress faster than gzip and still produce smaller files

Huh?

> For dynamic content, we'll use 4, which still produces smaller responses but
> takes less time to compress than gzip or brotli on a higher setting.

Oh if you compare to gzip max. But who runs gzip max? Gzip 6 gives decent low
cost compression. Gzip 9 is barely smaller but much slower.

Algorithms have their sweet spots. Brotli (small), LZMA (fast), gzip
(somewhere in between).

Knowing nothing about my clients, bandwidth, or dynamism, I can turn on gzip
without really a second thought.

------
therealmarv
This reminds me I have to develop and send sometime a PR for Axios to support
Brotli. We turned brotli on and figured out we only get garbage because Axios
itself also needs to understand on how to handle brotli
[https://github.com/axios/axios/blob/cb630218303095c0075182b5...](https://github.com/axios/axios/blob/cb630218303095c0075182b542ccb2f72d20dd9d/lib/adapters/http.js#L156)

------
dvfjsdhgfv
> Modern websites in particular often have large JavaScript bundles - the
> front page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed!

What puzzles me though... How much of this 1.1MB is really needed? You can do
form validation with HTML5. I don't see any other candidate on their front
page. So instead of getting a 14% advantage, what about pushing it to 100% or
so?

~~~
Klathmon
So you looked at their front page and concluded that they should be able to
get rid of all javascript?

CertSimple is a full dashboard and application that does a lot of crypto in
the browser which requires quite a lot of code.

Sure, there are probably gains to be had with code splitting and lazy loading,
but those get significantly more complex and can cause issues for codebases
that aren't setup to take advantage of it from the start.

And improvements are improvements. This was a blog post showing you how and
why to implement brotli, not a request for how to improve page loading speed
for their application specifically.

~~~
dvfjsdhgfv
I looked at the front page because they specifically mentioned it in the
article:

> the front page of CertSimple is 242 K gzipped, and would be 1.1MB
> uncompressed!

Of course the full app is a completely different thing.

~~~
nailer
> Of course the full app is a completely different thing.

Author here: no. The app is the front page.

/blog is a seperate AMP based site and tiny. Search 'discify' in this thread
for a breakdown of what's in the module.

------
cm2187
For those who wish to put Brotli support on Microsoft's radar:
[https://windowsserver.uservoice.com/forums/310252-iis-and-
we...](https://windowsserver.uservoice.com/forums/310252-iis-and-web-server-
role/suggestions/33465607-support-brotli-compression)

------
tzahola
>Modern websites in particular often have large JavaScript bundles - the front
page of CertSimple is 242 K gzipped, and would be 1.1MB uncompressed!

Mike, I’m afraid you have bigger problems than inefficient compression...

~~~
nailer
CertSimple is a fairly complex web app with a bunch of ractive components that
use webcrypto (requiring ASN1, PKI libraries etc) and URLs (using a WhatWG URL
polyflll) which takes up a large chunk of the bundle.

We use [https://github.com/131/discify](https://github.com/131/discify) to
analyse this and should be able to cut down a little further once the URL spec
is more widely implemented.

~~~
k__
Are you doing code splitting? If not, this would be my first target for gains
:)

~~~
nailer
Seperate bundles for different parts of the app, but not doing lazy loading or
anything like that. Might look at it again if/when ES6 modules ever become
popular (I just built a smaller project
([https://mikemaccana.com](https://mikemaccana.com)) using ES6 modules and
right now the lack of ES6 modules on npm makes any gains from tree shaking not
worth the productivity loss).

Perf is always fun to optimise, but we're got features customers want that we
need to work on first!

------
powturbo
See [Static/Dynamic web content compression benchmark]

One of the best benchmark about web compression including brotli + different
gzip compatible libraries:

[https://sites.google.com/site/powturbo/home/web-
compression](https://sites.google.com/site/powturbo/home/web-compression)

------
rocqua
Afaik combining compression and dynamic content has a tendency to weaken
encryption (BREACH).

Compression ratios give information about the contents of a message. Hence, it
seems to make sense to only compress static content from a security
standpoint.

------
mkesper
Sane defaults matter. Why does brotli do maximum compression by default?

~~~
klodolph
There really is no such thing as a sane default for compression in general.
There are three common use cases and each one demands different tradeoffs.
Maximum compression is often suitable for compression once and decompression
many times. Lower compression is useful for compression/decompression once. A
middle value is useful for compression once/decompression less than once. Then
you choose different values based on the relative costs of CPU, memory,
network bandwidth, disk space, etc.

“Maximum” is good because it handles a common use case. When you’re running a
CLI tool, you’re much more likely to be processing static assets which are
served many times. So “maximum” is a good guess of what the user wants.

------
mozumder
Is there a good Brotli library for Python?

~~~
klodolph
Have you tried searching Pypi?

[https://brotlipy.readthedocs.io/en/latest/](https://brotlipy.readthedocs.io/en/latest/)

------
wildpeaks
I wish there was a simpler way to use Brotli with Nginx: at the moment, either
you have to compile from sources, either rely on a PPA to use a special build
of Nginx (and an extra module) instead of the one from the LTS packages, even
in 18.04.

I'd especially like to use Brotli because it has good support in modern
browsers (pretty much everything except IE11) and would be good replacement
for gzip (which should be disabled for HTTPS).

~~~
chrisweekly
> "gzip (which should be disabled for HTTPS)"

?

Please explain.

~~~
evil-olive
BREACH and CRIME [0,1] are attacks on the combination of HTTPS and
compression.

There are workarounds / mitigations with various effectiveness, but disabling
compression when TLS is used is the simple way to prevent the attack.

0:
[https://en.wikipedia.org/wiki/BREACH](https://en.wikipedia.org/wiki/BREACH)

1: [https://en.wikipedia.org/wiki/CRIME](https://en.wikipedia.org/wiki/CRIME)

~~~
emj
I thought the TLS length hiding extension was supposed to fix this, I wonder
why it did not progress. Maybe it is just easier to disable compression for
dynamic content.

[0] [https://tools.ietf.org/html/draft-pironti-tls-length-
hiding-...](https://tools.ietf.org/html/draft-pironti-tls-length-hiding-02)

------
bestboy
The fact that there is no native JAVA encoder for Brotli is an obstacle for
the adoption in the enterprise segment.

