
Syslog: Complete System Administrator Guide - SCHKN
https://devconnected.com/syslog-the-complete-system-administrator-guide/
======
beefhash
It seems worth noting that the "Syslog message format" was introduced in RFC
5424; the article cites RFC 3164 for message delivery (and should be citing
RFC 5424 based on the format).

RFC 5424 defines a more formal and more complete version of the protocol.
NetBSD understands and emits it, FreeBSD is still in the process of gaining
support for it and OpenBSD seems to be perfectly content sticking to
traditional BSD syslog as recorded in RFC 3164.

~~~
DominoTree
And let's not forget that there are non-standard "syslog" implementations that
were being used for _decades_ before RFC3164 happened. Cisco devices and AIX
are the first two big ones that come to mind.

It's also worth mentioning that newlines are significant in TCP syslog (for
batching log messages) but not in UDP syslog (because of frame size limits)

~~~
astrobe_
> It's also worth mentioning that newlines are significant in TCP syslog (for
> batching log messages) but not in UDP syslog (because of frame size limits)

More because of the D in UDP, I believe. UDP is a packet (datagram) protocol;
TCP is a stream protocol.

------
caro_douglos
this is a great summary of the protocol. I recently needed to debug some weird
router behavior. An rsyslog change saved me from having to deal with the
crumby router debugging GUI. All that said journald really is clutch.

sudo journalctl -o json -f | jq .

~~~
mooreds
Ooh, I didn't know that journalctl could output json.

Thanks!

------
nailer
Note that most current Linux distributions use journald, rather than syslog.
Journald has plugins for the syslog wire protocol though. Personally I prefer
app names vs LOCAL4 or the other syslog facilities (UUCP is still in there)
but YMMV.

~~~
exabrial
I was actually going to ask a question around this. On Ubuntu systems, I see
both journald and rsyslog. How are they related? Is journald forwarding to
rsyslog?

~~~
yrro
/dev/log is probably a symlink to /run/systemd/journal/dev-log on your system.
'systemctl list-sockets' will show you that the socket is part of the systemd-
journald-dev-log.socket unit. So when a program calls the syslog(3) function,
it ends up sending the message to journald.

If configured to forward messages to syslog, I think systemd does so via the
/run/systemd/journal/syslog socket, which is part of syslog.socket. But I
believe rsyslog will these days read messages directly out of the journal
rather than relying on journald to forward them.

~~~
chupasaurus
You've got it right besides the last sentence. Copying from _systemd-journald-
dev-log.socket_ comment:

 _Note that journald internally uses the this socket both for receiving syslog
messages, and for forwarding them to any other syslog_

edit: formatting

~~~
yrro
That's a but confusing and I don't know how it would work. But last time I
looked into how ForwardToSyslog= worked I discovered it was disabled in the
default build configuration these days because rsyslog prefers to ingest from
the journal rather than being fed messages via a socket...

~~~
JdeBP
See
[https://unix.stackexchange.com/a/294206/5132](https://unix.stackexchange.com/a/294206/5132)
.

------
grepthisab
Does the PRI formula seem nonsensical to anyone? Facility number * 8 +
severity level = PRI. At first looking at the example, I was thinking it would
be an easy way to look at the last digit to determine severity (0-7), but
since the result of facility number * 8 could contain any final digit, so
that's a no go. Since severity is 0-7, and one multiplies facility by 8, no
numbers would overlap, so one could eventually get used to the ranges and
understand that, for example, PRIs 32-39 have something to do with auth (auth
is facility 4, times 8 = 32, + 0 for emergency, up to +7 for debug), with 32
being an emergency and 39 being debug, but it seems unintuitive unless you're
a sysadmin that sees these frequently.

It seems like simply concating the facility and severity level would give a
much more intuitive at a glance understanding of what's going on, and make it
easier to awk all, say, emergency severity levels since they'd all end in 0,
or awk for all cron facilities, as they would start with 09, or all FTP errors
(FTP facility 11 + error 3, 113). Range would be 000 (for kernel, emergency)
to 237 (for local 23, debug). Of course, I don't know anything about syslogs
except what I learned in this article, so maybe this is totally off base.

~~~
knd775
I think it's because they aren't really supposed to be human readable. It's
trivial for an application to do (PRI % 8) to get the severity, and ((PRI -
severity) / 8) to get the facility number. I'd assume that's why severity only
goes to 7.

~~~
grepthisab
This could make sense, but I don't see why being human readable wouldn't
necessarily be a benefit. I do a lot of log shaping with sed, grep, awk, etc.
and human readability has been a plus for me on things, but I may not be the
norm.

Regarding the second point, I believe the 7 is so ranges don't overlap. So for
example facility X, severity 7 doesn't equal facility X + 1, severity 0.

~~~
loeg
> I don't see why being human readable wouldn't necessarily be a benefit

That's the wire protocol, right? When rendered, this might show up as
"2019-08-05T08:12:22Z <4.3> hostname ..."

------
newaccoutnas
Interesting TCP is recommended with pain old Syslog. I've seen that take out
systems due to not being able to dequeue. There are better protocols that you
can use for reliable delivery

~~~
reacharavindh
What would be an alternative option?

~~~
newaccoutnas
There's RELP[1] but if you can don't use syslog's protocol. You could use
something like the beat protocol (if using ELK stack) or just a plain old
message queue. Things like logstash (and fluentd etc) can have multiple
input/output targets.

You'd have something like filebeat or fluentd reading the logs locally and
then shipping via that protocol to a central system where they'd be ingested.
For application logging, definitely use structured data (like JSON, for
example) over log-lines. It's easier to parse in the long-run.

1:
[https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protoco...](https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol)

~~~
irishsultan
RELP also uses TCP, so how would that help in a situation where Syslog over
TCP doesn't work?

~~~
newaccoutnas
REPL != Syslog TCP. They're different things. It's not TCP that was the issue
but the implementation of syslog on top.

