
Google Chrome 51 disables HTTP/2 on most Linux distros due to old OpenSSL - Mojah
https://ma.ttias.be/day-google-chrome-disabled-http2-nearly-everyone-may-15th-2016/
======
heavenlyhash
And right about here is where the claims that dynamic linking is better for
everyone's security update schedule just falls apart, for this reader.

Dynamic linking does not make it easier to update things. There, I said it.

Updating things confidently and safely requires -- as the article wisely
references -- testing those things, and the entire matrix of other things they
interact with.

Static vs dynamic linking simply doesn't register. In _either_ case, the newer
versions need to be built, and then tested together.

The only thing dynamic linking does is make it possible for distros to
hamstring themselves[^] by tossing _everything_ into one big cauldron, so
_nothing_ can be upgraded because it requires _testing too much at once_.

And here's the endgame. "Is OpenSSL getting updated on $distro?" "Nope."

Sigh.

\---

[^] And this is a red herring, in turn: dynamic linking isn't the real enemy
either; it's the global sloppile approach to sharing dynamic linked libraries
that really causes the mess. NixOS and others like that still use dynlinking,
and yet have successfully immunized themselves from these problems by using
content-addressable library versioning. So let's stop with the "dynlinking is
better for security updates" claims. It's not. It's either unrelated at best,
or actively a ball-and-chain making things harder and slower, and I submit as
evidence OpenSSL versions across these distros.

~~~
stingraycharles
I would be the first to agree that dynamic linking is evil, but how is this
related with the issue at hand, which is client-to-server communication? No
amount of static linking of Chrome would magically upgrade the SSL version
being used to package Debian, for example.

What am I missing here?

~~~
heavenlyhash
Chrome did not previously rely on a dynlinked system library for the
negotiation feature. Now, evidently, it does.

In making this transition, Chrome experienced a massive practical regression
for many users, precisely because the dynlinked library is not sufficiently up
to date for many people.

I'd say dynamic linking is related to the issue at hand, yeah.

~~~
slrz
No, it's not. The article is about missing support on the server side, not in
Chrome.

------
nteon
The Chromium team's blog post about this suggests that > 99% of HTTP/2
connections are _already_ negotiated with ALPN [1]. I imagine all of the CDNs
and other large providers already use ALPN, so this mostly affects the long
tail of small sites. It seems like the claims in the above blog post are
overblown.

1 - [http://blog.chromium.org/2016/02/transitioning-from-spdy-
to-...](http://blog.chromium.org/2016/02/transitioning-from-spdy-to-
http2.html)

~~~
mcguire
I have this strange suspicion that they could push the change to Google's
servers and get ~30%. Add reddit, CNN, and ESPN and they're pushing 90%.

~~~
coldtea
They might reach 30% with just Google -- but Reddit, and even more so CNN and
ESPN are insignificant as percentages of web traffic, whether in the US or
(even worse) globally.

CNN is not even in the top-50 websites, and ESPN is not even in the top-100
(e.g. in the Alexa index).

Heck, pornsites like xhamster have far better placements than either.

~~~
mcguire
I'm old.

Google, YouTube, and Facebook.

------
pfg
If you're stuck on a OpenSSL version without ALPN, and don't want to deal with
upgrading OpenSSL (or rather statically linking your web server to a newer
release), consider using caddy[1]. It's a HTTP/2 web server written in Go,
which comes with its own TLS library (crypto/tls) and thus doesn't have any
dependency on OpenSSL. It even supports automatic deployment of a publicly-
trusted certificate through Let's Encrypt.

[1]: [https://caddyserver.com/](https://caddyserver.com/)

~~~
pfooti
I'm not a golang user, but I am curious, how hardened is the crypto/tls
library? One of the reasons I use openssl is because I don't want a DIY crypto
layer, I want something a lot of people have banged on. I'd hate to move to a
new server, just to find out it is susceptible to a whole new set of Ye Olde
Underrun Bugges that openssl patched ages ago.

~~~
Matt3o12_
It is maintained by Google and the whole go community since it is in the
standard library. So, I think it is fairly secure (probably not as well tested
as OpenSSL but the attack surface is smaller, bugs and security risks are
still there, though).

What is important is that the library is not hardware accelerated and it thus
uses a software accelerated AES cipher. So, I think it is a bit slower, but
you would really have to measure it yourself.

Upgrading your Load balancer should not be that hard, though.

~~~
vetinari
It supports AES-NI on x64 architecture, if available.

------
chemodax
Support for OpenSSL 1.0.1 will cease on 2016-12-31 [1], so distributions need
to update anyway.

[1]
[https://www.openssl.org/policies/releasestrat.html](https://www.openssl.org/policies/releasestrat.html)

~~~
jnky
Just because something is no longer supported upstream does not mean that the
distributions won't support it anymore.

As far as I can tell it is more likely the package maintainers will backport
fixes to their old version in those cases.

~~~
chemodax
Sure, package maintainers may backport fixes to their old versions. But they
need to fully understand _all_ upstream source code and follow _all_ commits.
Otherwise they can miss important fixes: following only security fixes for
supported branches is not enough. Sometimes project developers fixes
bug/security problem in the code, but doesn't flag it as CVE because current
code usage doesn't trigger it. But code in old branch could.

~~~
viraptor
That's the current reality. That's how it was for years. Especially CentOS and
other RH-based systems are more happy to patch than to upgrade. This caused
the kernel 2.6.32-573 situation where lots of patches (over a hundred?) were
applied by the distro.

------
smithclay
I struggled with getting HTTP/2 working with ALPN on Ubuntu 14.04 back in
February. After building nginx from source [1] I ended up switching to Docker
base images that support OpenSSL 1.0.2 [2].

1: [https://www.clay.fail/posts/ubuntu-http2-in-mere-
hours](https://www.clay.fail/posts/ubuntu-http2-in-mere-hours) 2:
[https://www.clay.fail/posts/hip-http2-using-
docker/](https://www.clay.fail/posts/hip-http2-using-docker/)

~~~
leesalminen
I actually found your blog post [1] on Google earlier this morning after
seeing this thread. It was hugely helpful. Thank you very much for writing it.

Was there a bug or technical limitation that made you switch to a Docker image
or just preferences/ease of updating?

~~~
smithclay
The main reason I switched Docker is there's an nginx image (on alpine) that
supports ALPN. No compiling from source required, which simplified things (in
my mind, at least).

~~~
leesalminen
Gotcha. As a heads up, there's a confirmed bug with HTTP/2 implementation on
nginx 1.9.15 and 1.10.0. It seems to affect Safari and iOS (native and
browser) markedly.

For me, the first HTTP/2 POST request to a server running the affected version
of nginx would never leave the browser and Safari left a less than descriptive
"Failed to load resource: Could not connect to the server." console message.

I'm having luck with 1.9.12 + OpenSSL 1.0.2h

[https://trac.nginx.org/nginx/ticket/979](https://trac.nginx.org/nginx/ticket/979)

[https://trac.nginx.org/nginx/ticket/959](https://trac.nginx.org/nginx/ticket/959)

------
Skunkleton
HTTP2 is a relatively bleeding edge technology at this point. Seems fine for
it to depend on a recent version of Open SSL. It's not like HTTP 1.1 is going
anywhere any time soon.

------
bkeroack
Somebody tell me again about the supposed benefits of dynamic linking, if we
have to recompile and redistribute all the dependent packages for every
OpenSSL upgrade anyway?

~~~
acdha
> if we have to recompile and redistribute all the dependent packages for
> every OpenSSL upgrade anyway?

Because we don't – look at how many updates Debian, Red Hat, etc. have shipped
for OpenSSL vs. the number of times you've had to recompile dependent
packages.

OpenSSL is also something an outlier because it's both critically exposed for
security and has a somewhat unusual development model, so it's also important
to remember that even if it _was_ true that we have to rebuild OpenSSL callers
regularly, that's manifestly not true of the hundreds of other libraries where
dynamic linking saves time and memory.

~~~
fixermark
It's honestly so mission-critical to interactions on the internet---and
computers are so increasingly assumed to be on the internet---it's a bit of a
wonder that no-one to my knowledge has tried to roll it into the kernel as a
core and necessary function yet.

~~~
predakanga
It's definitely been discussed[1]. I last read about it when investigating the
use of splice(2) in nginx; seems there are a few use-cases, but nothing really
compelling.

[1]: [https://lwn.net/Articles/666509/](https://lwn.net/Articles/666509/)

------
jb613
> We knew, since the end of 2015, that this change was coming -- we were given
> 6 months time to get support for ALPN going, but by the current state of
> OpenSSL packages that was too little time.

This hints at part of the problem - simply not enough resources. It's not that
OpenSSL folks are slow or lazy (absolutely far from it) - but rather they've
got tons on their plate already.

One day (hopefully) corporations will recognize that everyone can benefit by
helping open source (either providing resources or $funding).

~~~
bluejekyll
This is the package maintainers, not OpenSSL, right? Or does the OpenSSL group
maintain all their packages on all these OSes too?

------
pilif
In our case, we were building our own nginx anyways, so the fix was relatively
simple to just statically link it against libressl.

This only affects the nginx we use on the frontend for SSL termination and the
rest of the OS remains unaffected, linking against the OS OpenSSL.

------
ryanobjc
As usual, OpenSSL is screwing things up.

Been delving into hairy problems at work and yup, it's openssl's fault. More
specifically it's the misguided way it handles multithreaded, and memory
management. Yay.

~~~
fixermark
This has definitely been a known issue for awhile, frustratingly.

At one point, a team I was working with had to code their own RNG replacement
for OpenSSL's; the one in the codebase was behaving badly in a multithreaded
environment. I don't know if the change was ever accepted back upstream; if I
recall correctly, the initial response was something to the effect of "you're
using it wrong" (as if multithreaded applications are some new and fascinating
beast, and not increasingly just the way you do it to leverage all the
performance a modern computer offers in hardware).

~~~
mwpmaybe
> At one point, a team I was working with had to code their own RNG
> replacement for OpenSSL's

I recently released a RubyGem that does just that. It's unfortunate.

------
mrsaint
> Upgrading OpenSSL packages isn't a trivial task, either. Since just about
> every other service links against the OpenSSL libraries, they too should be
> re-packaged (and tested!) to work against the latest OpenSSL release.

Not necessarily. You could simply statically link the newer OpenSSL library
with your web server and everybody is happy.

------
angrygoat
Chrome was no longer accessing my site via http/2 – turns out that the docker
nginx image has OpenSSL 1.0.1k. I switched to the Alpine image and that fixed
it – a few nginx.conf changes needed, notably to change the user from www-data
to 'nginx'. UID is 100 and GID is 101, which differs from the non-Alpine
image.

------
krzyk
Why disabled? More like "will disable", unless you are on some island on the
pacific - for the majority of Earth May 15th will come in more than few hours.

------
alexchamberlain
Potentially, big sites won't be (shouldn't be) relying on system binaries for
their web server etc. That would give them the freedom to remove unneeded
modules, add custom modules and upgrade at their own speed.

------
mwpmaybe
And TLS False Start, which reduces the average time of an SSL handshake by
30%, and can be enabled independently of SPDY and HTTP/2.

Fortunately, it was pretty easy to redeploy my HAproxy config to Xenial and
enable ALPN.

------
dang
Can someone suggest a more neutral title for this?

~~~
yuhong
Google Chrome 51 disables HTTP/2 on most Linux distros due to old OpenSSL

~~~
dang
Ok, we'll use yours. Sorry for the delay—I forgot about this one over the
weekend.

------
HeadlessChild
> Debian 8 (Jessy)

