
Nginx sucks at SSL, here's what to do instead - seldo
http://matt.io/entry/uq
======
randomstring
Without any reference to the nginx options

    
    
      ssl_session_cache
      keepalive_timeout
      keepalice_requests
    

and whether or not the other webservers mentioned support such options and if
they are enabled in the tests. There is no way to know if this is alarming or
just FUD. Also missing from this comparison is a vanilla Apache SSL benchmark
(which will suck, but would serve as a reference point).

Given nginx's track record, I prepared to give nginx the benefit of the doubt
and assume that it's a invalid test.

That said, I'm going to run some benchmarks of my own.

~~~
greglindahl
Mr. randomstring, your test results before blekko launched were much faster
than this guy's number. Want me to quote from your email?

------
pilif
If I see such bad numbers, usually, before I write a post about it, I try to
investigate and share the results of that with the post.

I'll try to reproduce these results tomorrow, but if I had to guess, I'd say
ssl_session_cache was left to its default (off) which means that every
connection has to do the expensive SSL handshake.

~~~
yid
Sigh. Sometimes posts like these make me want to go back to academia (it's not
perfect, but they generally believe in this little thing called rigor).

From TFA:

 _I tested nginx as a proxy, serving static files, and serving nginx-generated
redirects. I tried changing all the relevant ssl parameters I could find. All
setups resulted in the same SSL performance from nginx. I even tried the setup
on more than one server (the other server was quad-core nginx got up to 75
requests per second)._

So "all the relevant ssl parameters I could find", no details about what those
involve, and the surprising result that it made no difference.

In the same situation, I might think I was doing something wrong...

And then this overarching statement:

 _Never let nginx listen for SSL connections itself._

~~~
seiji
Rigor is an interesting point. Do we prefer to have a flawed, but slightly
useful, post now - or - do we prefer to wait a month or two for a squeaky
clean post with all issues worked out?

~~~
whirlycott1
In two years, people will still be saying "nginx + ssl = bad" because of this
post even though the problem may well be fully addressed. Google will continue
to surface this article even though it may be totally wrong at some future
date. That sucks.

~~~
vog
If it was really that easy to spread this questionable message for years, it
would be as easy to spread other articles as well.

So it wouldn't need more than a few articles, like "nginx + ssl = works like a
charm", or "nginx has better SSL support than Apache". It would not matter
whether those were actually correct, just a single article of questionable
quality would be sufficient.

~~~
vog
... and here it is:

"nginx does not suck at ssl"

<http://news.ycombinator.com/item?id=2759596>

------
mmaunder
Thanks! I didn't know nginx sucked at SSL. You may have increased our revenue.
Many businesses like us have our conversion pages on SSL. Our front-end server
is doing 2000 to 4000 http requests per second and we get over 3 million
uniques on the main site where we sell stuff via SSL. If SSL is this slow, it
probably impacts performance on our secure pages which affects revenue. Where
do I send the beer?

On a 4 core Xeon E5410 using ab -c 50 -n 5000 with 64 bit ubuntu 10.10 and
kernel 2.6.35 I get:

For a 43 byte transparent gif image on regular HTTP:

Requests per second: 11703.19 [#/sec] (mean)

Same file via HTTPS with various ssl_session_cache params set:

ssl_session_cache shared:SSL:10m; Requests per second: 180.13 [#/sec] (mean)

ssl_session_cache builtin:1000 shared:SSL:10m; Requests per second: 183.53
[#/sec] (mean)

ssl_session_cache builtin:1000; Requests per second: 182.63 [#/sec] (mean)

No ssl_session_cache: Requests per second: 184.67 [#/sec] (mean)

The cache probably has no effect because each 'ab' request is a new visitor.
But I'd guess the first https pageview for any visitor is the most critical
pageview of most funnels.

~~~
newman314
Choice of cipher as well as openssl version + features used make a difference
too. See <http://zombe.es/post/5183420528/accelerated-ssl> for some examples.

Use "openssl speed -elapsed" to test performance on your system.

------
stock_toaster
The post didn't mention if ssl_session_cache was enabled in the nginx config
or not. In fact, I didn't see any configs posted. :(

Also, the article author apparently added support to stud (in his own fork)
for x-forward-for. I don't think this is required any longer, due to this
fairly recent stud commit:
[https://github.com/bumptech/stud/commit/9d9b52b7d3ce90fa84c6...](https://github.com/bumptech/stud/commit/9d9b52b7d3ce90fa84c67eca261b087336233775)

~~~
jamwt
I'm reading over Matt's work carefully, but my initial inclination is not to
merge the bulk of this into stud mainline. I'd rather keep stud simple and
protocol-naive and have HAProxy do the HTTP work.

~~~
jamwt
Which is to say indirectly that I think the right answer is for nginx (and
daemons generally) to support the PROXY protocol, or some other agreed-upon
standard for a naive upstream proxy to indicate host/port information.

~~~
sciurus
Could someone link to a description of the PROXY protocol?

~~~
stock_toaster
<http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt>

------
tptacek
I don't know what the limits are in the nginx HTTP parser Matt's using, so
this is probably moot, but code that does things like "realloc(ptr, size +
newsz)" or "malloc(size + 1)" expecting things to be fine gives me the howling
fantods.

~~~
__rkaup__
I'm quite new to C programming. What's wrong with doing those things?

~~~
tptacek
Numbers can't grow indefinitely; they wrap (usually at the size of the
register).

------
nginxorg
It would be nice it Matt provided full details on the testbed, including the
client. In a test scenario it is very important to understand what gets tested
in the end. I liked that "Russia" tag too :)

------
greglindahl
For a different point of view, check out:

[http://web.archive.org/web/20090619214443/http://www.o3magaz...](http://web.archive.org/web/20090619214443/http://www.o3magazine.com/4/a/0/2.html)

The author benchmarks nginx at 26,590 TPS on a quad-core 2.5 ghz amd system.

~~~
spydum
That benchmark is using an SSL Accelerator card and nginx. Can honestly say I
have seen very few people swing for SSL Accelerator cards.

~~~
greglindahl
The "open source SSL Accelerator" mentioned in the blog posting is a quad-core
server running Linux and nginx.

~~~
spydum
whoops! my apologies -- the capitalized "SSL Accelerator" set my mind into
thinking dedicated hardware device.

------
cshesse
A friend of mine worked on some large scale SSL deployment, he wrote up the
results of his tests here: <http://zombe.es/post/5183420528/accelerated-ssl>

He's concerned about the raw speed of the SSL calculations, not requests per
second, but if you're actually concerned about SSL speed and you have enough
requests per second to justify optimizing SSL speed, it could be pretty
useful.

------
piotrSikora
No configs, no methodology, no graphs... Great "benchmark".

~~~
wpietri
If only there were some way you could create those things that you want and he
didn't care about doing!

~~~
damncabbage
Calling out a tool as having some terribly negative characteristic places the
burden of proof on he or she doing the calling.

------
auxbuss
I needed to use SSL on nginx and got great results from following a number of
pieces of advice. I jotted down my noted here:
[http://auxbuss.com/blog/posts/2011_06_28_ssl_session_caching...](http://auxbuss.com/blog/posts/2011_06_28_ssl_session_caching_on_nginx/)

It made a significant performance difference to me.

------
fexl
I've seen good results with the "pound" front-end:

<http://www.apsis.ch/pound/>

I haven't done extensive benchmarking on it, but very knowledgeable people
vouch for it.

~~~
getsat
Pound consumes an absurd amount of memory if you have lots of concurrent
connections (due to thread stacks).

~~~
fexl
Interesting, thanks -- I'll watch for that. Until now I haven't paid much
attention to it since I'm not responsible for that part of the configuration.

------
ominous_prime
I'm not sure why he would be getting numbers that low. The only setup I have
at the moment which would give useful numbers for ssl req/sec is a small
single core VM, running one nginx worker process, and that pumps out 135 new
req/sec. Add a few cores, workers, put it on real hardware, and I don't see
how this couldn't push well over 400 req/sec.

This is using nginx strictly as an ssl termination, where I need to do some
header manipulation that I couldn't do in stunnel/stud.

~~~
ominous_prime
Self reply, since I waited too long to edit:

I remembered I had an older 8 core server sitting unused at the moment. I
configured nginx with 8 workers, and ran `ab` against it. From a single (VM)
host, I can get 680 connections per second (maxed the cpu on the host running
the test). From 4 hosts, each host got > 290 connections per sec, so I got
nginx up to over 1190 new connections per second, and can likely push it
further.

[EDIT] got it to peak at 1535 requests per second with 4 hosts testing.

------
sigil
Where's the benchmark code? I'd like to see how ucspi-ssl [1] performs.

[1] <http://www.superscript.com/ucspi-ssl/sslserver.html>

------
foobarbazetc
Not to rain on the parade here, but we handle several thousand connections per
second on nginx + SSL per 8 core Westmere machine.

The article needs way more detail.

------
ck2
If anyone does benchmarking, please include litespeed as I am curious. I
suspect it's much faster than nginx at ssl. Even with the connection limit on
the free version I suspect it will still be feasible for testing.

------
nginxorg
more nginx ssl testing
<http://nginx.org/pipermail/nginx/2011-July/027960.html>

------
schiptsov
Let me guess - it is not nginx code which is slow but one from openssl? ^_^

~~~
schiptsov
I won -
[http://matt.io/technobabble/hivemind_devops_alert:_nginx_doe...](http://matt.io/technobabble/hivemind_devops_alert:_nginx_does_not_suck_at_ssl/ur)

------
ams6110
Quoting:

    
    
        (on an 8 core server...)
        haproxy direct: 6,000 requests per second
        stunnel -> haproxy: 430 requests per second
        nginx (ssl) -> haproxy: 90 requests per second
    

Yet Matt Cutts tells us that SSL is not computationally expensive anymore.
Based on these results it's still an order of magnitude slower.

