
Nginx doesn't suck at SSL after all - seiji
http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/ur
======
tptacek
In case you're wondering what "Perfect Forward Secrecy" is: SSL/TLS, like most
protocols, uses (expensive, dangerous) RSA to exchange (cheap, simple) AES or
RC4 session keys; bulk data is encrypted with session key.

In the normal protocol, if you lose the RSA key, an attacker can retroactively
decrypt the session keys, which are protected only by that same RSA key.

In ephemeral DH mode, instead of encrypting a session key with RSA, both sides
run the Diffie Hellman protocol to exchange a key†. DH allows two unrelated
parties who share no secrets to exchange a secret in public; it's kind of
magical. But it's also trivial to man-in-the-middle. To get around that
problem, ephemeral Diffie Hellman mode in SSL/TLS signs the DH exchange with
the RSA key.

The win here is that losing the RSA key now only allows you to MITM future
SSL/TLS connections. This is still a disaster, but it does not allow you to
retroactively unwind previous DH exchanges and decrypt earlier captured
sessions.

† _DH is unbelievably simple; go read the Wikipedia page._

~~~
shrikant
Diffie-Hellman and Boyer-Moore are two algos that completely blew me away with
their simplicity when I first encountered some theory on them.

[http://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exch...](http://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange)

[http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_sear...](http://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm)

------
ccollins
From the article, to find out what your website is doing:

openssl s_client -host HOSTNAME -port 443

I ran this for my own website and a few bigger websites

    
    
      openssl s_client -host www.gusta.com -port 443 (My site, hosted on Heroku)
      Cipher    : DHE-RSA-AES256-SHA
      
      openssl s_client -host www.google.com -port 443  
      Cipher    : RC4-SHA 
      
      openssl s_client -host www.airbnb.com -port 443  
      Cipher    : AES256-SHA
      
      openssl s_client -host www.facebook.com -port 443
      Cipher    : RC4-MD5
      
      openssl s_client -host www.paypal.com -port 443
      Cipher    : AES256-SHA
    
      openssl s_client -host www.amazon.com -port 443
      Cipher    : RC4-MD5

~~~
rythie
Presumably Amazon, Facebook and Google are using RC4 for speed reasons, though
it's not really thought to be secure anymore.

~~~
tptacek
RC4 is fine in SSL/TLS.

Nobody likes it, but until relatively recently, the AES ciphersuites were all
CBC mode, which means they burned a couple bytes of padding for every record.

RC4 is also faster than AES, which was, until very recently, an issue for
server performance.

We have AES-CTR ciphersuites now, but I'm not sure how widely deployed they
are.

~~~
newman314
I was under the impression that RC4-MD5 was no longer recommended but RC4-SHA
was okay (comparatively). Is this incorrect?

~~~
yuhong
The hash algorithm in TLS is _HMAC_ -hash. Some uses of MD5 like secret suffix
are now insecure (which is why usage of MD5 for certificate signing ended long
ago), but HMAC is not one of them.

------
benblack
An article about configuring SSL that doesn't 1) discuss trade-offs of
security vs. resource consumption, 2) how to figure out your performance
requirements, and 3) indicate the author really understands implications of
decisions about crypto is an article you should probably disregard. Modern
CPUs are so ridiculously good at crypto, and most sites have such ridiculously
low connection rates, that optimizing for maximum performance at the expense
of security is a fool's game in most cases. Instead, focus on measuring your
real performance requirements first, and things like sane configuration of
SSL, for example by explicitly listing ciphers instead of using the
impenetrable +aNULL:-yourMom syntax.

Here's my vintage code for scanning SSL configs:
<https://github.com/b/tlscollect>

Here are a couple of must read posts from someone who really knows his SSL
business:

[http://www.imperialviolet.org/2010/06/25/overclocking-
ssl.ht...](http://www.imperialviolet.org/2010/06/25/overclocking-ssl.html)

[http://www.imperialviolet.org/2011/02/06/stillinexpensive.ht...](http://www.imperialviolet.org/2011/02/06/stillinexpensive.html)

It's great to learn.

Lil' B

~~~
tptacek
Does any of this have anything to do with Matt's post? Adam's first post says
the same thing Matt's does: DHE is expensive.

The "tradeoff" in security vs. performance you're referring to irrelevant to
almost everyone building on nginx. If you've lost your RSA key, you are well
and truly fucked. DHE is interesting, but sniping at people for not using it
(in your case, implicitly) is unfair.

~~~
benblack
Adam's post is rather more thorough and nuanced, which makes sense since he
actually understands SSL and benchmarking. While you might summarize them both
as "DHE is expensive", I don't know why you would. Here is each post on DHE:

Adam - "However, with a pure RSA ciphersuite, an attacker can record traffic,
crack (or steal) your private key at will and decrypt the traffic
retrospectively, so consider your needs."

Matt - "Unfortunately, it also includes a very computationally intensive
cipher using an ephemeral Diffie-Hellman exchange for PFS. Sounds scary
already, doesn't it? ... The problem cipher is DHE-RSA-AES256-SHA [b]."

The first is factual and straightforward. The second is muddled and clearly
skewed towards blindly disabling DHE. I believe we are in agreement that it is
irrelevant to almost everyone building on nginx: their connection rates are so
low they will not notice the overhead introduced by DHE.

I am sniping at enthusiastic ignorance and encouraging others to behave
similarly. I hope that is all quite clear now.

Hugs and kisses, Lil' B

~~~
tptacek
Are you a little worried that you come off sounding like "Adam is one of the
cool kids and Matt isn't"? Matt's conclusion is ultimately correct.

And we apparently disagree completely about DHE, because you appear to be
saying you'd recommend it to web startups, despite the fact that the bank that
clears those startups transactions isn't even using it.

Especially weird given that Boundary, your startup, doesn't do DHE.

~~~
kelnos
I think benblack's argument is that Adam can recommend disabling DHE because
he knows what it is and what it does and can make an informed decision about
whether or not your average SSL-enabled site needs it.

Matt simply says "I messed with my settings and leaving this one out makes it
faster", without knowing whether or not turning DHE off is safe (or if he does
know, clearly he's making it seem like he doesn't). The fact that it _is_ safe
-- in this instance -- isn't particularly relevant. The point is that someone
who doesn't understand the security implications of something is making a
recommendation about security, just cloaked in a recommendation about
performance.

Anyway, I don't know any of the people we're talking about here, just trying
to help clear up what I believe benblack was trying to say :)

~~~
tptacek
Right is right. Wrong is wrong. Pants aren't shirts. It's clear Ben doesn't
think Matt is qualified to write the post. But he should have holstered the
impulse to gripe about it until Matt wrote something wrong.

~~~
kelnos
Well, Matt did write something wrong. The original post about nginx "sucking"
at SSL was wrong. Maybe it sucks for SSL in its default configuration (is that
even that case, or was Matt's config copy/pasted from elsewhere?), but saying
it sucks in general is incorrect and link-bait'y. You can presumably configure
other web servers to suck just as much at SSL by enabling DHE ciphers and
providing DH params.

~~~
tptacek
We're commenting on _this blog post_. As was Ben, who didn't comment on the
previous post, but did single this one out here and, as I recall, on Twitter.

~~~
kelnos
It sounds like you're implying that when someone posts something on the
internet, when you're evaluating the usefulness of that information, you
should ignore anything else they wrote previously. Frankly I don't care all
that much about Ben's motivations behind calling out the author here, but I
read both blog posts as they came out, and the lack of attention to detail in
the first post definitely affected my opinion of the second post. I don't
think a lack of participation in the first HN discussion means you're
disqualified from participating in the second one using information from the
first.

------
thirsteh
So Nginx got unwarranted hate for having the most secure defaults. That sucks.
I hope the user nginxorg -- whom I assume is Igor Sysoev -- who dropped by the
previous thread (<http://news.ycombinator.com/item?id=2752136>), sees this
post too.

Either way -- based on his attitude in the first post, I'm really surprised by
how Matt owned up and did his homework for this one. (He should have done it
from the beginning, of course, but none of the people bashing him in the
previous thread actually provided anything to support what they were saying.)

~~~
tptacek
It's not the "best/most secure". All the other servers can trivially enable
EDH as well; it's OpenSSL that implements it, not nginx. Reasonable people can
disagree on what the right default is, but plenty of financial institutions
have made the studied choice not to enable it.

~~~
masklinn
> All the other servers can trivially enable EDH as well

Unless I'm reading OP wrong, that's not the case for the servers he uses for
his tests: Stud can't enable it at all:

> stud doesn't have at all.

and in stunnel you have to compile it in for support, it's not just "not
enabled by default", it's not compiled in:

> stunnel has it as a compile time/certificate configurable option.

~~~
jamwt
From stud -h:

    
    
        Encryption Methods:
           --tls                    (TLSv1, default)
           --ssl                    (SSLv3)
           -c CIPHER_SUITE          (set allowed ciphers)
    

Edit: Though, you'd need to set DHE params, as another commenter said below.
Stud doesn't do this atm, but I'm open to a patch!

~~~
tptacek
I think 'seiji's right, and OpenSSL won't actually do DHE handshakes unless
you give it parameters, which is another 2 lines of code that aren't actually
in stud.

------
jeremyw
Unlike the above post, this fellow actually did some broad cipher testing
(<http://zombe.es/post/4078724716/openssl-cipher-selection>), particularly
around AESNI instructions in recent Intel chips.

With AESNI, use AES-128, AES-256, RC4-SHA, CAMELLIA-128. Without AESNI, use
RC4-SHA, AES-128, AES-256, CAMELLIA-128.

In nginx, this looks like:

    
    
      # (wo/AESNI): ssl_ciphers RC4:AES128-SHA:AES:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2
      # (w/AESNI):  ssl_ciphers AES128-SHA:AES:RC4:CAMELLIA128-SHA:!MD5:!ADH:!DH:!ECDH:!PSK:!SSLv2
    

You eliminate weak ciphers. You retain RC4 for compatibility and speed. You
order by performance. (Note that AES-128 is still ranked as secure through
2030 [at least]. You don't need to prefer AES-256.)

------
cbetz
I can't tell if this is an apology or a non-apology. It seems to have elements
of both.

Clearly the moral of the story is: "Don't claim that X _sucks_ unless you are
are damn sure".

Saying something sucks is fightin' words. Don't expect to people be nice if
you are wrong.

~~~
tptacek
Why would someone apologize for writing an informative blog post? I'm glad he
wrote both, even if he had to walk the first one back a bit.

~~~
SnowLprd
I found both posts informative, and yet I'm with cbetz on this one. The issue
isn't whether Matt's first post was informative but is instead whether it was
fair|wise|necessary to say "Nginx sucks at SSL" instead of, say, "My initial
testing, which needs to be investigated further, is showing Nginx SSL
performance lower than other alternatives." The first headline is more likely
to grab folks' attention, which is probably why Matt chose that headline. It
is probably _that_ choice that cbetz finds objectionable, and if so, I agree
with him.

~~~
pyre

      > "My initial testing, which needs to be investigated
      > further, is showing Nginx SSL performance lower than
      > other alternatives."
    

You keep using that word ('headline'), but I do not think it means what you
think it means.

~~~
SnowLprd
Okay, I'll bite. Here you go: "Initial Tests Show Slow Nginx SSL Performance"
Wasn't that hard, was it?

~~~
pyre
My point was that you seemed to have purposely made that 'headline' that you
thought he should have used needlessly verbose, to the point where it couldn't
be considered a headline.

------
tlrobinson
_"Final feeling: Twitter is better than HN in all social dimensions of
engagement, kindness, and authenticity."_

Ouch.

~~~
michaelbuckbee
There is some selection bias and context that the OP had via Twitter (people
following already kind of know him and his work, etc.) whereas HN evaluates
his work solely on this one post.

~~~
AltIvan
Yeah, the correct stament should be something like:

Final feeling: My long time Twitter Followers & Friends are more engagement
and kind than the anonymous programmers on HN.

------
clintjhill
It's great to follow up. Especially with such a particular detail.

It's also good to have thick skin. HN can be aggressive. But for good reason.
I'd be willing to bet this tiny sting will result in more rigor in the future.
I know it has worked that way for me.

------
WestCoastJustin
changes: slow ssl encryption ciphers on by default, keepalive

before: nginx (ssl) -> haproxy: 90 requests per second

after: nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per
second

------
grandalf
Any serious article ripping on the performance of something should at least
link to the config file used.

------
dfc
This is why security is such a wierd/nice/confusing/irratating line of work to
be in. Newsflash SSL is not a one size fits all secure you against anything
technology. I did not see the original article so I won't pretend that I knew
the answer ahead of time. I just hope that I did not accept SSL as being a
onesize fits all completely uniform technical conmponent.

There is a Dave Chapelle joke about cops sprinkling crack-cocaine over a crime
scene in order to make the case quick and easy. Too many developers trest SSL
like magic pixie dust for security.

Or as ptacek says "thanks in advance for putting my kids through college."

------
giberson
I use a similar directive in my apache2 configuration. Would I see an
performance improvement in removing the DH option from the cipher suite? Or is
this only directly related to ngix and how it implements its ssl protocol?

Secondly, by removing the DH method do I restrict any browsers from connecting
my site? Ie, are their any browsers, or security settings on browsers that
prevent the site from being trusted if DH isn't available?

~~~
blumentopf
It's an openssl configuration option, so it does affect performance for Apache
with mod_ssl. It's not specific to nginx.

If you do not allow DH ciphers, you'll probably just lose users who
deliberately configure their software to use only strong ciphers.

What you can do is put DH ciphers at the end of the list. That way, weaker
ciphers are preferred but you're still supporting strong ciphers.

------
beachaccount
I ran this against the slowest SSL website I know of. This site absolutely
kills my phone web browser and I've been wondering about this problem for
years. The site: manager.skype.com. The result? DHE-RSA-AES256-SHA. No wonder!
Fix this guys!

In regards to this post, if this is the default configuration of Nginx then I
agree that Nginx sucks. This is not a good default configuration for the
Internet.

~~~
bdonlan
There's nothing wrong with using DHE algorithms, particularly if you're going
to be transferring financial secrets around. If your phone can't keep up,
well, then it probably should have a better entropy generator.

~~~
tptacek
Are you sure the problem is entropy generation and not just extra bignum math?
Also: there are plenty of major financial apps that are not _allowed_ to use
DHE, because DHE makes it impossible for the provider to monitor its own
connections ("conventional" SSL/TLS allows for middleboxes that monitor and
archive sessions by holding a copy of the server's RSA key).

~~~
burgerbrain
What is best for security in regulated fields and what is _mandated or
forbidden_ for security in regulated fields are often almost disjoint sets.

Anyway, whether your phone has trouble coming up with the entropy, or
preforming the math, you should probably be using something more substantial.

~~~
tptacek
You're suggesting that people should pick a different phone so they can get
PFS with the small fraction of SSL servers that support it?

~~~
burgerbrain
I'm suggesting that if for some reason a site thinks that that sort of
security is necessary, they shouldn't change their mind for the sake of people
using their telephones.

Particularly since at the current rate of development, the average phone will
be able to do it just fine in a few months.

------
alexkon
If you are exploring and testing SSL, the SSL Labs tools come in handy. For
instance, see what Gmail and Github are doing:

<https://www.ssllabs.com/ssldb/analyze.html?d=mail.google.com>

<https://www.ssllabs.com/ssldb/analyze.html?d=github.com>

------
newman314
Guess I was right about the cipher used.

<http://news.ycombinator.com/item?id=2753903>

------
mmaunder
Thanks to Matt and everyone who contributed to figuring this out - and then
figuring it out again. Comments on the nginx mailing list (which I encourage
you to subscribe to if you're a user):

<http://forum.nginx.org/read.php?2,212229>

------
ynniv
Ah, if only we had an occasional second upvote! This is far more useful than
the original post, which was already well above average. If you are deploying
nginx with SSL, you need to know about the configuration details in the
article.

------
lanstein
keywords in footer: nginx, openssl, ciphers, DHE-RSA-AES256-SHA bad,
AES256-SHA good, hn, twitter, glee soundtrack

~~~
sthatipamala
Look at his other posts. He just has joke keywords in all his posts.

------
epi0Bauqu
So what should ssl_ciphers be? Can't you just move that one to the end
somehow?

------
ldar15
So, are these the new numbers? I copied the numbers from the original and the
new post:

    
    
      haproxy direct: 6,000 requests per second
      stunnel -> haproxy: 430 requests per second
      (OLD) nginx (ssl) -> haproxy: 90 requests per second
      nginx (AES256-SHA) -> haproxy: 1300 requests per second
      nginx (AES256-SHA with keepalive 5 5;) -> haproxy: 4300 requests per second
    

Did other things change or is nginx more than twice as fast as the next best
solution?

------
alnayyir
Well at least he followed up. Most people don't bother to correct their
mistakes.

