
Nginx module by Google for rewriting web pages to reduce latency and bandwidth - nikolay
http://ngxpagespeed.com/
======
jdmarantz
Hello from the PageSpeed team.

Image compression and device-sensitive transcoding are important for mobile
and emerging markets, and are somewhat challenging to deploy on your own. For
larger sites we acknowledge there is some tuning & thought that must be
applied, and we'd be happy to help via the discussion groups (mod-pagespeed-
discuss, ngx-pagespeed-discuss). We are also working on a new release that
will make this easier. For smaller sites, things should work out of the box.
As with all PageSpeed issues, please let us know if they don't, or you think
there's something specific to improve.

We've heard user concerns about SEO, and are happy to report that this should
be improved with our latest beta release (1.0), which adds 'link canonical'
attributes to image response headers. This allows search engines to point back
to the origin image.

Please give us a shout on our discussion group or bug-list and we'll try to
help sort through any others issues that you see:
[https://github.com/pagespeed/ngx_pagespeed/issues?q=is%3Aope...](https://github.com/pagespeed/ngx_pagespeed/issues?q=is%3Aopen)

~~~
sandGorgon
its really tricky for people to figure out how to build this and deploy it
production.

I have a nginx build script with pagespeed that I use for my docker images. I
dont think I leverage pagespeed enough - it would be great if you could point
out what needs to change (and others could learn from it)

[https://gist.github.com/redcarpetup/e52cbddb907effe70bd0](https://gist.github.com/redcarpetup/e52cbddb907effe70bd0)

EDIT: in fact that the first bug I see on your bug list is
[https://github.com/pagespeed/ngx_pagespeed/issues/1106](https://github.com/pagespeed/ngx_pagespeed/issues/1106)
"Nginx cannot find module when building inside Docker container"

~~~
blfr
Better yet, an nginx .deb repo with pagespeed already compiled in. Like the
one nginx provides[1].

I'm very reluctant to use anything that doesn't come packaged. Compiling your
own software only really works when you're constantly maintaining the same
project and its stack. In every other situation, it's a hassle, and I always
fall behind.

[1]
[http://nginx.org/en/linux_packages.html](http://nginx.org/en/linux_packages.html)

------
Theodores
Seems as if people have had a few negative experiences of 'pagespeed'. Why is
this?

Well, not every Pagespeed feature works with every site. However that tends to
get learned the hard way. Those lazy loading images were working a treat on
Friday, come monday - ah, no Apple users could see any images, consequently in
an instant Pagespeed gets removed.

To take an automotive analogy, Pagespeed is a bit like doing things to a car
for improved 'performance'. Let's save weight, let's put some active
suspension in there, let's remove a few bits to improve the aero, let's put
some 100 octane fuel in the tank and chuck in some twin turbo. All these
improvements - let's keep going there are 50 more tweaks we can do in the
garage, why not try 38 of them at once? What could possibly go wrong?

So, the main problem with Pagespeed is not how it works it is just that it
encourages people to keep going with it, layering on more 'improvements' until
the website does the computing equivalent of leaving the road to spontaneously
burst into flames.

~~~
jakejake
That sums up my experience pretty well. I tried installing it for a couple of
sites and seemed really cool at first. But just weird things started to happen
with styles and such and it seemed like it was going to take a lot of research
and debugging to get everything dialed in. Not really having time to do that,
I disabled it and put it on the list for later. I wouldn't call it a
"negative" experience necessarily, just not the easy win that I was hoping
for.

I do think it would be awesome for some legacy code if it was critical to get
the performance tuned up and you have a day or so to get everything
configured.

------
notnarb
> Recognizes common JavaScript libraries and redirects each to a canonical
> URL.

> Inserts Google Analytics javascript snippet.

While these are useful (and what appears to be optional) features, it leaves a
bad taste in my mouth to think that a business reason for Google to support
this library could be to lower the barrier of entry for developers to allow
Google to track their users.

~~~
zongitsrinzler
I would argue that adding a Google Analytics JS snippet is the easiest thing
possible for an average user while adding an Nginx plugin is infinitely more
difficult.

~~~
de_dave
In most cases yes. I have a feeling there's a customer somewhere running an
old CMS or has a big jungle of HTML documents but not the resources to modify
them all accordingly.

~~~
nikolay
Yes, this is one the main utilities of PageSpeed - you can optimize and tweak
content on pages on which you don't have control over.

------
ComputerGuru
We ran this in production from its initial release over two years ago. Last
month, we moved to a new datacenter and decided to forgo ngxpagespeed
entirely.

The results have long been mixed - it does a great job minifying content and
reducing requests, but the net results are highly questionable. While we
reduced our network transfers and round-trips, the amount of time it took to
serve requests was significantly increased to a point where the net gains were
minimal. We played around with enabling and disabling the various filters and
changing the settings and configurations to a great extent, and filed quite a
few issues with the (ngx)pagespeed teams as we came across them.

If you do use this module, don't enable image recompression. It'll use up a
lot of CPU, but worse, it'll also add significant latency to your requests.
The caches on disk will always be full, and your disk IO will increase across
the board. My personal favorite filter was the CSS merger which pulls various
CSS resources into a single file. My favorite feature that always let me down
was the "prioritize critical CSS" filter which (in theory) inlines CSS needed
for above-the-fold content to display immediately. It never worked reliably
for us, and broke our CSS more times than I can count. We always went back
every few months to try it and always came away disappointed.

ngxpagespeed does not play well with caching (it must be enabled on the outer-
most node) as it relies on realtime metrics collected via JS from served pages
to make decisions about content, caching, inlining, and more. It also doesn't
work well for SEO without significant manual tuning as it rewrites content
URLs (I hear there is a way around this on Apache, but not on nginx) to pass
back information to nginx, appending .pagespeed.xxxxx to your non-HTML URLs,
including images that will be picked up by Google's bots. If you disable
pagespeed later, all these will be broken. There's also no guarantee that
these links will remain valid across pagespeed versions or if you
enable/disable certain ngxpagespeed modules/filters, though I can't say I've
been bitten by that (solution: you need to hack your nginx configuration to
always include a Link...canonical header in your outgoing requests).

There are open issues galore regarding HTTPS and ngxpagespeed - it's an
ongoing battle to make it play nicely with HTTPS upstreams for caching. It
also basically proxies all requests made to your site back to your site once
more so it can cache and optimize them, adding an entire roundtrip to initial
or uncached requests. By default it won't cache HTML, so all HTML pages get
proxied an extra time. There's an option to disable this for static resources
by loading them from disk instead, but good luck getting that to play nicely
with multiple upstreams and overlapping paths taking advantage of rewrite
rules.

In the end, it just wasn't worth it for us. We've configured our webapp to
pull in, minify, and inline its own (entire) CSS on first page load (about 20
lines of code w/ the Yahoo minify library doing the crunching), then
configured nginx to reduce latency (http2, ssl_buffer_size, ssl session
caching, smart keepalives, gzip, ssl_cipher optimizations (hint: bigger isn't
better), proper expire headers, MTU optimization, etc) and the results have
been better than what ngxpagespeed was giving us.

Sorry for how unstructured this comment is, I've been remembering stuff as I
go along. By all means, ngxpagespeed is an awesome effort. As a primarily non-
web developer, I think this is what browsers and http servers should do by
default: act as a compiler of sorts for your content and find and implement
optimizations where possible because your job isn't to figure out how to
minimize network requests but rather to write code that works. However, it's
the early days of "web compilers" and the intelligence and logic is far from
being any good just yet. Just like 30 years ago no compiler could come close
to even an average developer's hand-crafted (not even tuned!) assembly, that's
how I see the state of "optimizing" proxies today. Give it another decade or
two, I suppose :)

~~~
Fradow
Thanks a lot for this comment!

For me, this really underline how an already optimised website won't benefit
much from it (especially the image recompression bit), and it may even be
harmful.

On the other hand, it could work great by being properly configured for small
websites that don't get much optimisations, and could really benefit from
generic optimisation.

~~~
nikolay
You still could benefit, because the optimizations are user agent-specific,
and if you aggressively use caching with Nginx, then you don't need to make as
many live calls to the backend to segment the cache per user agent type.
PageSpeed is written in C and highly optimized so it has performance benefits
in your case as well, I think. For example, there are user agents for which
PageSpeed may decide to deliver your CSS as inline and small images - as data
URIs. It also automatically (but optionally) does lazy loading of images, etc.
It has beacons that are also implemented with it. It's perfect for CMS
websites like it is in my usecase. It's not a panacea though.

------
therealmarv
Why is this so much upvoted? This is old news. IMHO ngxpagespeed is a good
option if you have a lot of old legacy stuff which cannot be converted for a
good web compression nowadays.

But compiling your own nginx is not trivial and I would not recommend it for
everyone.

It would be great if there would be a great precompiled nginx with newest
security fixes for my Ubuntu which can enable ngxpagespeed or the new brotli
compression: This is missing in the open source world.

~~~
cbr

       compiling your own nginx is not trivial and I would
       not recommend it for everyone
    

Yes, this is frustrating as an nginx module developer as well. Apache has a
loadable module system and we can ship binaries, but to add a module to nginx
you have to recompile.

Nginx's developers have been hinting that they're going to add support for
loading .so modules, and I really hope they do!

~~~
tuananh
tengine (a fork of nginx) fixes this problem.

~~~
nikolay
But is, unfortunately, a few generations behind.

------
theli0nheart
Cool to see this featured. If anyone's interested in actually playing around
with ngx_pagespeed, I wrote a pretty straightforward tutorial on my blog to
get up and running (granted, I last updated it in 2013, so some things may be
outdated).

[https://dlo.me/archives/2013/05/14/nginx-pagespeed-
cloudfron...](https://dlo.me/archives/2013/05/14/nginx-pagespeed-cloudfront/)

------
yyin
This made me chuckle. I've been doing this type of stream editing for years,
on the client side. Of course I strip out Javascript. In addition to blocking
google-analytics, doubleclick, etc. client-side, via DNS. Google reminds me of
Goldman Sachs: trying to take both sides of a bet. But Google can never
deliver the best experience for the user, because the best experience does not
include redirects to set cookies, tracking, ads and other nonsense that serves
no benefit to the user. Either Google is for advertisers or it is for users.
Users do not pay Google. Draw your own conclusions.

~~~
gouggoug
What made you chuckle?

ngxpagespeed and its apache brother mod_pagespeed are absolutely incredible
tools for improving performances of any website. It automates lots of work
that used to have to be coded manually in your application, and moves the
responsibility to the webserver. It, in the end, allows you with a dead simple
configuration file, to minify, shard, optimize, and make many other
performance optimization, without you ever worrying about implementing those
yourself.

> Either Google is for advertisers or it is for users.

What does that have to do with ngxpagespeed?

~~~
lifeisstillgood
I think the OP is confusing strategic and tactical decisions by Google. On a
strategic board-level view, yes Google is an ad company and reducing web page
size without stripping ads is a bit disingenuous. Strategically Google will
never drop ads, so the next best step for this project probably won't happen

However tactically Google benefits from paying clever people to make great
products irrespective of their strategic fit - and this seems a good product.

I had never heard of it, and will build it in my pipeline RSN.

~~~
true_religion
> Strategically Google will never drop ads, so the next best step for this
> project

I'm curious what this step would be?

NGINX pagespeed is a server side module that you run before you send data to
the end-user.

Many people who own servers want to extract revenue from their end-users, and
yes, that means ads.

If pagespeed stripped ads from a person's website, I think they'd be very
unappreciative of it.

------
dclaw
Pagespeed is absolutely one of the worst plugins in the world. mod_pagespeed
for apache is garbage, and permanently breaks some websites when removed. I
see no reason why the nginx version would be any better.

~~~
cbr
I'm sorry it didn't work out! Permanently breaking a website when removed is
very surprising, since if you uninstall the module it's not there anymore to
have an effect. Would you be up for explaining more?

(I work on mod_pagespeed, and if we're breaking sites on removal that's
something I want to look into.)

~~~
gouggoug
This kind of answer from Jeff (and other people) from the MPS team is in part
what made me enjoy using MPS. Should you have _any_ issue with MPS, if you
send an email to the mailing list, you will always receive a kind answer, even
in those cases were it is obvious you didn't read the doc.

I "worked" for _months_ with Jeff and his team to find a real tricky bug to
catch (well, they mostly did the work, I sent the bug reports) and they went
above and beyond to find and fix that bug. (Remember the #1048 issue?)

I definitely recommend MPS, it will not "break your website".

~~~
nikolay
I agree, Jeff and Otto have an amazing ethic and truly care for the project. I
wish more project are lucky to have such leadership. Otto is no even with
Google but has spent endless hours trying to help me and never tried to pitch
consulting services although it seems that this is his business.

------
Rezo
Does image spriting and js/css concatenation still make any sense if you're
already using HTTP/2 in Nginx? The requests should already be multiplexed.

------
richardkeller
Are there any precompiled packages for Nginx for this? Looks like an awesome
project, but I'm not a huge fan of compiling from source because it makes
maintainability difficult in future. Having to compile Nginx from source just
so I can use PageSpeed isn't something I look forward to.

~~~
vbtechguy
just script the compilation so it's easier over time. That's what i do for my
LEMP stack's installer and source compiled nginx. There's a shell based menu
option i can run to just recompile nginx for upgrade, downloads or recompiles
[http://centminmod.com/menu.html](http://centminmod.com/menu.html) :)

automation is the key ^_^

------
nubela
Ahh, Kloudsec [0] uses a fork of this in our free CDN service for our users.
The issue with something like this is that

a) most website owners who want to get things done will find it hard to
recompile, optimize, and deploy in production.

b) most websites in the web are really just WP sites hosted on X hosting
provider

And yet we truly believe that sth like Pagespeed is a worthwhile investment
(if done by 1, deployed on many). So we built Kloudsec to make optimizations
like this available for free in our CDN layer (which is also free).

\---

Most importantly, it is super easy to deploy. No programming/dev-ops required.
Register an account, add your domain, then update DNS records. (You get to
keep your fav nameservers)

[0]: [https://kloudsec.com](https://kloudsec.com)

------
gouggoug
I've been a heavy user of mod_pagespeed (the Apache version of Google's
plugin) for about 5 years now. It's been an absolutely incredible tool.

I am now, however, wondering what the future of MPS is with the advent of
HTTP/2? A lot of the work MPS is doing, and was designed for, is optimizing
for HTTP/1.1 shortcomings, and most of MPS's work seems irrelevant in the
HTTP2 world, and even detrimental in some cases (take URL sharding for
example).

~~~
nikolay
It still can add tremendous value. For example, I do all my CDN offloading
with it, domain sharding, making all paths relative, etc. It has a growing
number of filters. Combining assets is just one of its most basic feature. It
has a lot. Minification is still gonna be useful with HTTP/2.

------
vbtechguy
Also don't forgot about ngx_pagespeed's admin console lots of useful stats
there too
[https://developers.google.com/speed/pagespeed/module/console](https://developers.google.com/speed/pagespeed/module/console)
&
[https://developers.google.com/speed/pagespeed/module/admin](https://developers.google.com/speed/pagespeed/module/admin)

screenshot examples [https://community.centminmod.com/threads/admin-stats-
console...](https://community.centminmod.com/threads/admin-stats-console-
revamp-in-ngx_pagespeed-1-9-32-1-beta.1669/)

------
gonyea
Beware if you're using this BEHIND a separate caching layer such as Varnish.
That's not well-supported (and helped contribute to an outage for us. we used
auto-scaling groups and that shot us in the foot when first trying out
pagespeed).

In itself this module is pretty cool, but you're actually better off simply
internalizing the various practices within your own code & infrastructure,
with a CDN in front.

None of this is to knock the Pagespeed plugin. It IS really cool, and in an
organization with many silos (like Google) it has many benefits.

~~~
nikolay
Actually, it is supported and this is how I've been using it. It's documented
here [0].

[0]:
[https://developers.google.com/speed/pagespeed/module/downstr...](https://developers.google.com/speed/pagespeed/module/downstream-
caching)

~~~
gonyea
"Note: this feature is currently experimental" and that's a true statement.

~~~
nikolay
The whole thing is a beta. I've been using Downstream Caching in production
for years.

------
avelino
Reduces latency and greatly increases the disk writing and reading

------
xfalcox
We used this a lot when we had a old project to maintain and it's great!
Today, having a new version where everything is optimized before we turned it
off.

------
rmdoss
Does anyone know if that works with sites behind a nginx proxy? I have Apache
and Nginx and if that works for proxied content.

~~~
me_bx
yes, it does :)

~~~
rmdoss
Thanks! So it works on the post-cached content? If I choose the minification
or CSS unification.. Or is that on the fly?

Anyone familiar with that?

------
pulina
Is there any standalone program that use PSOL library to post process static
page? I only read about modules for http servers.

------
alexcasalboni
Wow. Is this compatible with AWS, Load Balancers, etc.?

------
jimjag
Ugg... Apache httpd module been around for awhile, no news.

Nginx module re-announced, and big news.

Holy crap... is everything Nginx-related somehow important and note worth and
everything Apache httpd related delegated to the trash bin?

Lest anyone forget: Nginx doesn't even have _loadable modules_ so this
requires rebuilding Nginx

------
gnu8
Anything that impacts the readability of the markup sent to the client is an
atrocity - not that Google would care about this since they are notorious
abusers of js obfuscation. Don't be evil indeed.

~~~
cbr
If you're visiting a website that runs PageSpeed and you want to see the
original markup, add ?PageSpeed=off to the url.

For example, compare
[http://www.jefftk.com/index](http://www.jefftk.com/index) to
[http://www.jefftk.com/index?PageSpeed=off](http://www.jefftk.com/index?PageSpeed=off)

