
Thread Pools in Nginx Boost Performance 9x (2015) - ithacadream
https://www.nginx.com/blog/thread-pools-boost-performance-9x/
======
tyingq
This message should probably be repeated at the bottom of the article:

 _" The page cache works pretty well and allows NGINX to demonstrate great
performance in almost all common use cases...So if you have a reasonable
amount of RAM and your working data set isn’t very big, then NGINX already
works in the most optimal way without using thread pools"_

So using thread pools in NGINX isn't a general recommendation. The article
suggests "a heavily loaded NGINX‑based streaming media server" as the type of
situation where it makes sense.

------
dumbfounder
The moral is that threads and asynchronous programming are not mutually
exclusive.

No matter how much you can do with a single thread, in a highly parallel
environment (such as a web server), on a cpu with multiple cores (the norm),
you can (almost always) do more with multiple threads.

~~~
brianwawok
Sure for threads up to cores / 2\. Past that it is likely a throughput and
latency loss.

~~~
virmundi
Does anyone have numbers to back this claim? Seems like for IO focused
software you'll have a lot of dead CPU time. If you have more threads than
cores, you can take advantage of that.

~~~
jorangreef
"Seems like for IO focused software you'll have a lot of dead CPU time. If you
have more threads than cores, you can take advantage of that."

Yes, see here for more discussion on this:

[https://github.com/ronomon/crypto-async#adjust-threadpool-
si...](https://github.com/ronomon/crypto-async#adjust-threadpool-size-and-
control-concurrency)

------
Neil44
As long as you're not using Apache in pre-fork mode then the rest boils down
to a few percent in terms of http server overhead. Unless you're trying to
serve tons of little static object with either no resource or big scale then
there are better places to spend your time optimizing.

~~~
cagenut
and even then it only matters if you don't have something in front of it
handling the connection-pooling/spoon-feeding/static-stuff like a load
balancer or a cdn or varnish or pretty much _anything_.

------
nwmcsween
I think the main takeaway is that Linux is broken as far as async file reading
goes.

------
fasteo
It seems quite similar to libuv's work queue [1]

[1]
[http://docs.libuv.org/en/v1.x/guide/threads.html#id1](http://docs.libuv.org/en/v1.x/guide/threads.html#id1)

------
skyde
So the main problem seem to be that libaio on linux does not use the
filesystem cache because it's forcing the use of O_Direct. Any reason linux
could not support AIO on regular file like FreeBSD ?

------
csense
TLDR, the Linux non-blocking filesystem API requires you to bypass the OS file
cache (!!!)

So the whole webserver serving clients might occasionally block waiting for
disk if you're serving online videos or some other application where your
whole website is too big to fit in the OS cache. But if you have multiple
threads, then if one of them blocks, it isn't a big deal as the other threads
will continue to unqueue requests.

> The asynchronous interface requires the O_DIRECT flag to be set on the file
> descriptor, which means that any access to the file will bypass the cache in
> memory and increase load on the hard disks

------
avip
Looks similar to apache worker MPM (release ~2005)

~~~
scott_karana
Different paradigms.

Worker does use process pools with their own thread pools, but they're still
not evented/async.

Apache 2.4 (2012) saw the release of 'event', an async variant of worker,
which is like the new nginx model.

[https://httpd.apache.org/docs/2.4/mod/event.html](https://httpd.apache.org/docs/2.4/mod/event.html)

------
majke
Ok, I understand why you might "offload" read() from _disk backed files_ file
descriptors. But why "sendfile"? I thought sendfile is properly async in
linux... Unless I'm getting something wrong.

"aio_write" -> wasn't the idea of AIO operations to actually be truly async?
Maybe it's because it requires XFS. Anyway, I'm confused.

~~~
jsn
From what I can see, sendfile isn't really async, it still returns the number
of bytes written or error status, and nginx probably needs to wait for that to
e.g. log the results, handle short writes, or proceed with whatever next
operation must be done to the socket (like closing it). No idea why aio_write
needs offloading, though.

~~~
taf2
They say why in the article

"The asynchronous interface requires the O_DIRECT flag to be set on the file
descriptor, which means that any access to the file will bypass the cache in
memory and increase load on the hard disks"

They even go into how no interface has been surfaced yet that allows
determining whether a file is in cache or not. Also how FreeBSD's aio
interface doesn't have the same limitations

~~~
dom0
I never tried, but mmap and mincore could work (or not, as one of the many
special cases in this area). I expect that's not a viable approach for this
application, anyway :)

------
ioquatix
It would be interesting to compare this with coroutines/fibers, and it would
also be interesting to know how latency is affected. In my experience, thread
pools introduce latency.

~~~
nemanjaboric
Trick is that nginx uses thread pools for non-blocking reading on the "fast"
devices - you can't use coroutines/fibers, because on Linux is not reliably
possible to read the regular file in a non-blocking way, as far as I'm aware,
so that means that `read` in a single fiber would block entire thread until
the data is fetched from the disk.

In the storage engine I'm developing/maintaining, I've been using fibers, for
both network and disk IO, until people started noticing that, if number of the
client grows, and if they ask a lot of data (generating a lot of disk
traffic), the performance drops severely. I've moved to the thread pools,
dispatching my disk IO operations there (although it's much problematic for
the reads, as writes would be performed by kernel anyway, `fsync` and `close`
would be operations that you still want to do in the separate thread):
[https://github.com/sociomantic-
tsunami/dlsnode/blob/master/s...](https://github.com/sociomantic-
tsunami/dlsnode/blob/master/src/dlsnode/util/aio/AsyncIO.d)

edit: used right link for the freshly opensourced repo.

~~~
vbezhenar
Why files can't be read without blocking with Linux? There's certainly an API
for this.

~~~
nemanjaboric
Which API are you referring to? If you're referring to `aio_` that's just
glibc's user-space thread-pool emulation of the POSIX's asynchronous IO + many
operations needed for the truly async IO are not supported (such as `stat` and
`open`). There's `io_` syscalls family, but that's not in the ready-to-use
state yet, judging by the manpages and various patches being submitted every
month.

~~~
vbezhenar
I'm referring to `fcntl(fd, F_SETFL, flags | O_NONBLOCK)`. `stat`, `open` are
blocking, yes, didn't think about that. Also according to [1] file operations
will always block, even in non-blocking mode, so I was wrong, sorry.

1:
[https://www.remlab.net/op/nonblock.shtml](https://www.remlab.net/op/nonblock.shtml)

~~~
signa11
> ... Also according to [1] file operations will always block ...

not quite :) look at aio(7)

------
signa11
this approach sounds very similar to SEDA (staged event driven architecture)
that was done a while back, iirc, circa 2001.

------
scrp
Interesting read, however a 2015 tag would be nice, since the post is from
June 19, 2015

------
stephen123
must need some monads =P

------
acdjuiamadfn
This is not the right place but I failed to find an answer elsewhere so would
somebody answer?

Is it possible for me to use lighttpd as a proxy but at the same time serve
static pages for a subset of urls?

~~~
dsr_
I don't know, but you can definitely do that with nginx.

You should ask here:
[https://redmine.lighttpd.net/projects/lighttpd/boards](https://redmine.lighttpd.net/projects/lighttpd/boards)

