
The Architecture of Open Source Applications: Nginx - larsvegas_
http://www.aosabook.org/en/nginx.html#
======
sinzone
Lua support in nginx is phenomenal, especially when combined with LuaJIT. It
basically allows to transform nginx into an application server and run
arbitrary code that can pretty much do anything.

I would like to mention agentzh and his team that did an amazing job in
releasing OpenResty[1] which makes it easy to extend nginx with custom Lua
functionality, which also happens to be the backbone of CloudFlare
architecture, and the core technology being used by projects like Kong[2] when
it comes to microservices management.

[1] [http://openresty.org/](http://openresty.org/)

[2] [https://github.com/Mashape/kong](https://github.com/Mashape/kong)

~~~
kator
+1

I built a nginx+luajit RTB bidder that did 168k qps on 8 cores. It smashed the
C eventlib version and of course it crushed the Java version. Golang came in a
close second but at the time with 8 cores I had to do crazy things to get it
to pin to CPU's since the golang thread scheduling didn't seem very scaleable
beyond 4 cores.

~~~
lpgauth
168K/s? Did you actually decode the bid request (json) or just return no bid?
That seems too good to be true.

~~~
fit2rule
I also would love to know details about this .. very impressive stats!

~~~
royjacobs
These bidding platforms typically don't provide you with JSON but with some
headers or query parameters that are very quick to parse (especially for the
native nginx code), so your Lua code can basically boil down to a few table
lookups.

~~~
lpgauth
Uh, look up OpenRTB. Except for Doubleclick Adx almost all exchanges encode
bid requests in JSON.

------
nisa
I really enjoy nginx, as it's more flexible to configure but I've never
understood why Apache got this slow label...

FTA: In February 2012, the Apache 2.4.x branch was released to the public.
Although this latest release of Apache has added new multi-processing core
modules and new proxy modules aimed at enhancing scalability and performance,
it's too soon to tell if its performance, concurrency and resource utilization
are now on par with, or better than, pure event-driven web servers. It would
be very nice to see Apache application servers scale better with the new
version, though, as it could potentially alleviate bottlenecks on the backend
side which still often remain unsolved in typical nginx-plus-Apache web
configurations.

I'm using Apache 2.4 with mpm_event + mod_proxy_fcgid and it's doing fine -
99% of the work and time spend is done in the FastCGI application anyway and
for static content mpm_event is good enough. I wouldn't run a dedicated static
CDN box on Apache but for everything that can run on a single server Apache
can also do the job... even HTTP/2 with mod_h2 works fine as of 2.4.17

A problem with nginx is to figure out what matches in a complex config... it's
not straightforward. .htaccess is nice and simple for a shared server with
lot's of users.

I really like nginx but I guess most people just don't really need it.
Migrating to 2.4 and mpm_event should be good enough.

~~~
twothamendment
I've used Apache forever and have always been able to tune it to get the job
done. The thing that made me switch to nginx was the ability to use more than
one SSL cert on the same IP. Now that I was "forced" to try it, I'm pretty
happy with it.

~~~
rgbrenner
there was never a time when nginx supported sni and apache did not.

Nginx requires an openssl version that supports sni -- that was added in
0.9.8f on Oct 2007 (if you compiled openssl manually to enable it), and was
enabled by default in 0.9.8j on Jan 2009.

Apache supported SNI using mod_gnutls since 2005; and in 2.2.12 (July 2009)
using openssl.

~~~
librvf
Technically correct, however for users that install from OS packages it's
possible that they're getting a version of nginx that had SNI enabled and a
version of Apache that did not.

------
evincarofautumn
No matter how many times I see “nginx”, and know that it’s supposed to be
“engine X”, I always pronounce it as [ŋɪŋks] in my head!

The way nginx handles requests and responses in an implicit event loop reminds
me of a recent talk by Brian Kernighan, in which he mentions the ubiquity of
the “pattern–action” model in many domains. I think it’s a very useful
architectural pattern to have in mind when you’re designing a configuration
system or a DSL.

I also liked this quote:

> …it is worth avoiding the dilution of development efforts on something that
> is neither the developer’s core competence or the target application.

~~~
skissane
Until I learnt the correct way of pronouncing it, I always thought it was "en-
ginks".

Sometimes there can be a long gap (even many years) between first reading
about something and first using its name in conversation. A long time for
mispronunciations to stew away in my brain (assuming the person I'm talking to
even knows how to pronounce it themselves.)

~~~
hueving
I spent an embarrassing amount of my life thinking epitome was 'epa tome'
instead of 'a pit oh me'.

~~~
billbrown
Mine was "hitherto." I pronounced it "hit-hurt-oh" once. Once. (It's "hither
too" if you didn't know.)

~~~
creshal
Better than hit-her-too, I guess.

------
arca_vorago
While I like nginx over apache because I've been burned too many times by
strange apache configs, I have recently found and am growing to love Hiawatha.
It's a GPL webserver focused on security, uses PolarSSL which jut got bought
out and is now mbed, and it's pretty fast. All the benchmarks I've seen show
it comparable to stock nginx, apache, but once you tack on some of the
optimizations nginx will beat Hiawatha. It also has a very easy config syntax.

Just for anyone interested: [https://www.hiawatha-
webserver.org/](https://www.hiawatha-webserver.org/)

The dev doesn't do much advertising, so word of mouth on a place like HN
really helps.

~~~
faeron
What optimizations are you talking about? And how do those beat Hiawatha?

~~~
arca_vorago
Well, I could have sworn I remember seeing a benchmark comparing stock vs
basically tuned (config files) nginx, hiawatha, and a few others, where stock
hiawatha won but tuned nginx beat it out, but I just spent ten minutes or so
looking for it and couldn't find it, so I'm going to have to retract my
earlier statement I suppose.

Either way, I think Hiawatha is a great webserver that should get more
attention. Especially since I am GPL proponent and Hiawatha is one of the only
currently maintained GPL webservers.

As a bonus here is a benchmark of nginx and hiawatha under attack. (notice the
service drop on nginx)

[https://www.hiawatha-webserver.org/weblog/64](https://www.hiawatha-
webserver.org/weblog/64)

------
alexandrerond
Every "new" webserver tries to fix apache configuration syntax mess, and imho
they all fail. Yes, setting up a reverse proxy looks simpler with nginx or
haproxy for that matter. Now when it comes to complex configurations they all
suck, and I'm not sure a json/yaml config format is going to fix that as long
as webservers have such wide scopes (from serving static pages, to proxying
traffic, authorizing, authenticating, encrypting...). At least apache is very
modular on this regard and some credit should be given to it having survived
and evolved along with all the newer options.

~~~
sanderjd
The better choice is not a new webserver program with a different, possibly
better, configuration language, it's HTTP server _libraries_ that can be used
from actual programming languages to do whatever you want. Many languages ship
with really good libraries these days.

~~~
therein
They indeed are shipping with really good libraries these days but no matter
how good they are, there is some level of doubt that they fail to alleviate
and they get deployed behind nginx or apache httpd anyway.

~~~
sanderjd
It seems more like they are deployed behind load balancers, because load
balancing is a specific job that it makes sense to do with a separate piece of
software. Admittedly, that separate piece of software is often nginx, but
there are other popular options, and I don't think very many people use Apache
that way (though I could be wrong).

~~~
SSLy
Every third telecom around the world uses apache to load balance requests for
specific application.

Source: I have to support that SW stack.

~~~
sanderjd
Interesting, thanks for the info!

------
davidmichael
I wish there was more input as to why Apache 2.4 isn't suitable. It's been 3
years since its event driven model was released and it is a perfectly
acceptable web server even for static content.

~~~
Karb
Apache 2.4 isn't event driven. The Apache's Even MPM only handles keep-alive
connections asynchronously, while the whole request processing is still
synchronous. It solves only one problem of Apache, but it still doesn't make
it as scalable as nginx.

~~~
pquerna
Hi, I'm one of the main authors (along with others, it is a community driven
project) of the Event MPM for Apache.

On Benchmarks and timelines: I agree that 2.2+ was not widely available for
many years due to the update cycle of linux distributions, but at the time
nginx wasn't in the distros... so it become a thing where people would yum
install apache2, get a 2-5 year old version, and they would benchmark that
against an ngnix from their latest dev download.

The original work for the Event MPM started around 2004:

[http://mail-archives.apache.org/mod_mbox/httpd-dev/200411.mb...](http://mail-
archives.apache.org/mod_mbox/httpd-dev/200411.mbox/%3C41868F73.9010602@force-
elite.com%3E)

The version in 2.2 was mostly focused on Keep-Alive requests. Apache 2.2.0 was
first kicked out on December 1, 2005.

To go beyond Keep-Alive requests, is a set of features/patches called "Async
Write Completion". Much of this work was done in 2006-2007 by Graham Leggett:

[https://mail-archives.apache.org/mod_mbox/httpd-
dev/201510.m...](https://mail-archives.apache.org/mod_mbox/httpd-
dev/201510.mbox/%3C28641C04-8936-4004-BA40-C889C8B7846A@sharp.fm%3E)

Timing wise, most of that work did not find its way into a stable release
until 2.4, which came out February 17, 2012. This is the date the article
references.

~~~
mwpmaybe
Thanks for your work on Apache. Do you recommend any published benchmarks of
modern Apache vs. modern Nginx?

~~~
Karb
There are a few benchmarks of Apache 2.4: 1\.
[http://www.eschrade.com/page/performance-of-
apache-2-4-with-...](http://www.eschrade.com/page/performance-of-
apache-2-4-with-the-event-mpm-compared-to-nginx/) 2\.
[http://mondotech.blogspot.ru/2012/02/apache-24-vs-nginx-
benc...](http://mondotech.blogspot.ru/2012/02/apache-24-vs-nginx-benchmark-
showdown.html) 3\. [http://blog.zhuzhaoyuan.com/2012/02/apache-24-faster-than-
ng...](http://blog.zhuzhaoyuan.com/2012/02/apache-24-faster-than-nginx/)

~~~
mwpmaybe
Those look to be a few years old. :-)

------
davidw
> Although Apache provided a solid foundation for future development, it was
> architected to spawn a copy of itself for each new connection

That's not really correct:

[https://httpd.apache.org/docs/2.2/mod/prefork.html](https://httpd.apache.org/docs/2.2/mod/prefork.html)

Nginx makes it easier to handle a bunch of concurrent connections, but it's
not as if Apache simply forks for each new connection.

~~~
lobster_johnson
You're quoting from a paragraph that's summarizing the history of Apache, and
the statement was true about Apache 1.x.

Back in the day, Apache 2.0 took a long time to gain substantial market share,
for various reasons. The situation was not too dissimilar from the current
Python 2/3 split.

Edit: But I guess Apache never forked for every connection, even in 1.3. It
only forked if it needed a new child and the existing ones were busy.

~~~
davidw
> the statement was true about Apache 1.x.

Apache 1.3 did not fork a process per connection. It forked to handle
additional concurrent connections, but that's very different than forking for
each and every connection.

------
jrochkind1
> In February 2012, the Apache 2.4.x branch was released to the public.
> Although this latest release of Apache has added new multi-processing core
> modules and new proxy modules aimed at enhancing scalability and
> performance, it's too soon to tell if its performance, concurrency and
> resource utilization are now on par with, or better than, pure event-driven
> web servers.

When was this written? Is it still too soon to tell? 2.5 years seems like
enough time to tell?

~~~
mwpmaybe
Math? 3.75 years?

------
maattdd
More impressive in the same book, how another server (Warp) written in Haskell
(GCed lazy functional language, supposedly way slower than C over epoll)
achieves the same performance as nginx!

[http://www.aosabook.org/en/posa/warp.html](http://www.aosabook.org/en/posa/warp.html)

------
gtrubetskoy
Back when I was tinkering with mod_python performance[0] there was this web
server called nxweb[1], which out-performed nginx consistently by quite a bit.

[0] [http://grisha.org/blog/2013/11/07/mod-python-performance-
rev...](http://grisha.org/blog/2013/11/07/mod-python-performance-revisited/)
[1]
[https://bitbucket.org/yarosla/nxweb/overview](https://bitbucket.org/yarosla/nxweb/overview)

------
anowlcalledjosh
Why do web servers always seem to invent their own config file format? Whilst
nginx seems to do it slightly more sanely than Apache, it still doesn't use
something like YAML or JSON; is there a good/obvious reason for this I'm
missing?

~~~
rdtsc
JSON doesn't allow comments. So it is an ok-ish config format. Yaml would have
been better. But they probably wrote everything from scratch to be optimized
and decided if they are writing a parser for a config file might as well
invent a config file format as well :-)

~~~
CrLf
Not allowing comments doesn't make JSON an "ok-ish" config format, it makes it
completely _unsuitable_ for it.

~~~
rdtsc
But it is used though pretty oftne. That's why it is an -ish format ;-)

You can do things like

    
    
       {
         "comment": ["blah blah comment message"],
         "k1" : "v1", "k2" : "v2", ... 
       }
    

Or some silliness like that.

------
losvedir
Does anyone know how this architecture compares to cowboy? I know erlang is
known for concurrency, but I'm assuming erlang isn't as fast as the custom-
tailored C here. OTOH, I feel slightly less concerned about security issues
with erlang.

~~~
rdtsc
Well cowboy is an application library. So it is a bit in a different league.
With cowboy you can start writing your business logic or application code
directly and go to work. Nginx can do some of that with Lua integration but it
is nice for serving static files and proxying to back-ends.

I could see for example nginx in front of cowboy. It would strip away ssl,
serve static pages, maybe authorization/authentication and then proxy
connections to cowboy servers in the backend for application logic.

------
DonnyV
I love using nginx but it seems that all the new features lately are only
accessible through their enterprise service.

~~~
gamesbrainiac
Nginx is a solid piece of software with a lot of people working on it.

I find it sad that people expect good services built on top of it to be free
as well. Without an enterprise/paid offering how else do you suppose people
fund nginx? Right now the state of open source funding is abysmal.

~~~
nileshtrivedi
His point is that all this funding is not leading to any new features in the
open-source version.

~~~
Karb
Are you joking? Check the changelog of the open-source version.
[http://nginx.org/en/CHANGES](http://nginx.org/en/CHANGES)

~~~
pnommensen
It's a common question. As you correctly noted in the changelog, the vast
majority of features have been placed in the open source version. The
commercial product has gained great visibility and adoption but open source
NGINX is very core to our company. In fact, development and feature releases
to open source NGINX have rapidly increased since the inception of NGINX, Inc
(the company) because it can support the development efforts :)

(disclaimer: I work at NGINX.)

------
ferbivore
Sort of unrelated, but I would love to see one of these for Unreal Engine. If
there's an internal architecture overview somewhere, I haven't found it.

------
jsprogrammer
>These days the Internet is so widespread and ubiquitous it's hard to imagine
it wasn't exactly there, as we know it, a decade ago. It has greatly evolved,
from simple HTML producing clickable text, based on NCSA and then on Apache
web servers, to an always-on communication medium used by more than 2 billion
users worldwide.

The Internet[0] has a much richer history and larger ecosystem than just the
World Wide Web. The Internet started nearly six decades ago, the web has only
been around for a bit more than two.

[0]
[https://en.wikipedia.org/wiki/Internet](https://en.wikipedia.org/wiki/Internet)

~~~
scott_karana
Yeah, I thought that was pretty damn breathless of the author too.

Architecturally _and_ usability-wise, we're more or less at the same place as
a decade ago:

In 2004, both nginx _and_ Gmail were released, and people were going ape about
exciting "Web 2.0" technologies like DHTML and AJAX, whose paradigms more or
less still underpin all modern development. There have been a _lot_ of
additions and streamlinings, but "dynamic pages/apps in the browser without a
pageload" were the modus operandi then, and are the MO now.

~~~
nkozyra
It's not readily apparent, but this item was written in 2012.

