
Nginx 1.7.1 adds syslog support - kolev
http://nginx.org/en/docs/http/ngx_http_log_module.html#access_log
======
akurilin
Why is logging to syslog as opposed to a separate file an important use case?
Genuinely curious.

~~~
quonn
It's useful if you want to log to a centralized logging server. This helps to
have all your logs in one place and also keeps the logs safe, if someone
breaks into your server.

~~~
nrr
That's a pretty weak argument considering that syslog is entirely UDP and is
bound to drop log data, sometimes en masse, most likely even silently. Not a
good idea.

Why not use something like multilog or svlogd and wire up a tiny processor for
it to kick logging data over someplace using something like rsync?

To boot, syslog is annoying to tune, depending on your particular
implementation. rsyslog has a default buffer limit of 2k, whereas other syslog
implementations (IIRC, syslog-ng and Solaris syslog at the very least) have
default buffer limits of 1k, and this might not be obvious until you're
running up against that and make the shocking discovery that you're losing
data.

On an nginx server that services 2TB/mo worth of transit (which is distinctly
possible since I've got infrastructure in production that does this), there's
a good chance that you'll be stretching some of these limits a bit.

~~~
lobster_johnson
In addition to what danudey says, modern sysloggers also support Unix domain
sockets, which are reliable. Typically this is /dev/log; on Linux, I believe
the GNU C Library's syslog() uses this by default.

On Linux, sending UDP to localhost is very reliable and fast, essentially
going through kernel buffers with very little overhead. You will only see
dropped data if the system is extremely overloaded. I did some testing, a few
years back, and was not able to induce packet loss on localhost.

The usual way to set up centralized logging with syslog is to have each node
run a local syslog daemon (eg., RSyslog), which then buffers the data and
streams it to a central syslog daemon using a more reliable protocol such as
RELP [1] over TCP.

[1]
[http://www.rsyslog.com/doc/omrelp.html](http://www.rsyslog.com/doc/omrelp.html)

~~~
IgorPartola
While I agree with you on the whole, one minor point about UDP/datagrams: they
are not reliable even on localhost under some circumstances. The point of
datagrams is that they are allowed to be lost without a trace if the consumer
(syslog) is not consuming fast enough. For example, if process A starts
spewing 10,000 log records (UPD or datagram UNIX socket packets) a second at
syslog, and syslog can only handle 5,000, then the other 5,000 records will be
lost. Any other process will also get its records lost as they will not be
guaranteed to be processed. The rate of loss will be controlled by how large a
datagram buffer the consumer's kernel has. Moreover, the processing will not
be uniform: the buffer is LIFO, so older records will be processed while newer
ones will be lost.

On the other hand if you use stream sockets, the producer will either block or
be told that the consumer is not ready to read any more data (beauty of TCP).
In either case, TCP produces enough overhead compared to UDP to slow down the
actual useful part of your application, which is often not desirable.

Neither one of these is a good solution as either your consumer or your
producer needs to keep their own very large buffers to accommodate spikes in
traffic. Ideally, you do this anyways to ensure that you hold onto all the
packets you received.

Having said that, I don't know exactly what rsyslog does so I cannot say if
this would actually be a problem for it.

~~~
noselasd
While it's certainly true that UDP is unreliable even on localhost, unix
datagram sockets arn't, they have flow control.

~~~
IgorPartola
I have never seen a reference to this. Can you link a source?

~~~
lobster_johnson
AF_UNIX with SOCK_STREAM will behave like TCP, in that a "full" stream will
block. See [http://stackoverflow.com/questions/1478975/unix-domain-
socke...](http://stackoverflow.com/questions/1478975/unix-domain-socket-is-
there-such-a-thing-as-a-busy-signal), for example.

------
mfjordvald
Especially considering that it's been available as a community patch since 0.8
days.

[http://wiki.nginx.org/3rdPartyModules#Third_party_patches](http://wiki.nginx.org/3rdPartyModules#Third_party_patches)

~~~
kolev
True, it's been available, but having it in the core means that it will get
into Linux distros, i.e. people don't have to compile.

~~~
nailer
Distros have been patching things like this for a long time.

~~~
kolev
True, but I can't recall any having the syslog module out of the box.

------
ishi
What does Nginx do if the log message is longer than the 1KB limit imposed by
the syslog protocol?

~~~
viraptor
That's an implementation defined limit in practice. rsyslog for example
defaults to 2K (apparently to be compatible with upcoming RFCs), but it's
configurable to higher values. If you want to be compatible with other
software though, it may need to stay at 1K.

------
Andys
How much buffering does it do, if any? If not sufficient, it can result in
either reduced performance or dropped log messages..

~~~
peterwwillis
Why would you buffer udp packets?

------
kolev
Here's also the error_log:
[http://nginx.org/en/docs/ngx_core_module.html#error_log](http://nginx.org/en/docs/ngx_core_module.html#error_log)

And the changelog: [http://nginx.org/en/CHANGES](http://nginx.org/en/CHANGES)

------
staunch
They had to add a checkbox to [http://nginx.com/products/feature-
matrix/](http://nginx.com/products/feature-matrix/)

------
nailer
Any systemd experts know if there would there be any benefit to supporting
journald directly, rather than through the syslog API, on Linux hosts?

~~~
gegenschall
Have a look at this[0] blogpost. Also, I'm just guessing, once kdbus is in the
kernel using direct calls to journald will (or could potentially) be more
efficient.

[0] [http://0pointer.de/blog/projects/journal-
submit.html](http://0pointer.de/blog/projects/journal-submit.html)

------
Thaxll
Finally... it worked well with logstash but a native implementation is always
better. Is that TCP only, I don't see any UDP settings?

~~~
peterwwillis
It's UDP only. They also hard-code the port instead of looking it up with
getservbyname, which is a little weird. If you want to use TCP, send syslog
messages to localhost, and have a local rsyslog/RELP forwarder send messages
remotely.

~~~
stonogo
getservbyname under glibc uses dlopen, effectively working around static
compilation. while this is a non-issue on ubuntu, the nginx devs are aware of
the number of people building against uclibc or musl to produce static
binaries for embedded use. I've seen nginx running on bare metal -- just it
and libc, no kernel underneath. kudos to them for making this easier than they
have to.

------
livid
This is very useful when you need to consolidate error_log from hundreds of
Nginx servers.

------
napsterbr
Did someone manage to setup remote log using the buffer option? I get a
"parameter "buffer=32k" is not supported by syslog" but I'm pretty sure I'm
doing something wrong.

------
atmosx
It was about time, although I won't make any use of this feature.

ps. I always found metalog so much better than syslog.

------
kolev
I hope they also add cache purge into the community version as well.

------
jesalas
Hooray! Whiny complaining on the Internet works sometimes!

