

Linux syslog on the way out? - jfruh
http://www.itworld.com/it-managementstrategy/227291/linux-syslog-may-be-way-out

======
dredmorbius
My first (shallow) read of the proposal is that it solves one relatively high-
risk, low-probability case by throwing a great deal of complexity at standard
operations on which a great deal of existing infrastructure exists: log
analyzers, summarizers, rotation systems (broken as they may be), and just
simple shell tools.

Immediately prior to launching myself into Linux, I briefly toyed with Windows
NT 4.0 WS, and among the bigger disappointments of it was the system logging
facility and its user interfaces. While *nix logging has its bugaboos, the
ability to chain simple tools together to perform tasks (example: grep, cut,
and uniq -c on a postfix logfile to give a per-hour summary of mail delivery
status by various classifications) provides tremendous flexibility. As
compared with the single GUI interface Microsoft provided. I'd hope that
Poettering et al won't repeat this bone-headed mistake, but the simple
observation that their sample log event lacks ... a date stamp??!!! ...
doesn't inspire me.

If there's a useful element from this it's the list of horribles against
current syslog weaknesses. Many of these could be addressed by amending syslog
and/or providing a bolt-on solution. I don't see a need for a total
replacement.

As to the observation that there are currently several (incompatible) logging
systems on Linux systems, ob xkcd: <http://xkcd.com/927/>

~~~
rbanffy
A second read makes me believe it solves no problem except disk space usage.
The fact it's binary and signed protects against nothing - whoever has control
of the machine also controls the signing keys of the log and can generate
whatever log they want, with whatever event they want.

Are we doomed to repeat every stupid mistake Microsoft made? Is the "if you
don't know Windows you are doomed to reimplement its bugs" the new thing?

~~~
dredmorbius
The solution to integrity, IMO, is distributing the log (multiple independent
logservers). You can hash individual line items if you want, or sign them, or
do any number of other things to make it easier to find discrepancies. There's
the problem of log lossieness in general, which is a tough nut to crack. Some
form of log entry UUID might also be useful (though hostname +
millisecond/jiffy timestamp comes close).

And count me in among those who find the bug-for-bug reimplementation of
Microsoft's fatal errors just a tad annoying.

~~~
rbanffy
> The solution to integrity, IMO, is distributing the log (multiple
> independent logservers).

Isn't this already solved by rsyslogd?

~~~
dredmorbius
Yes.

------
maratd
This is a waste of time. Syslog is the standard for logging on *nix. You will
bork everything if you replace it with something incompatible. Everybody knows
this. So whatever you replace it with must be Syslog compatible. If that's the
case ... there are numerous alternatives to choose from. Like rsyslog. Which
is already standard on RHEL, nevermind Fedora. Rsyslog has this nifty feature
where you can immediately transfer all logs via TCP or UDP to another host.
Hacking problem solved. Stop fixing things that aren't broken.

~~~
naner
I disagree. A lot of these old *nix tools are showing their age and could use
a refresh based on known deficiencies. Creating new tools won't harm the old
ones so I don't see a problem.

~~~
rhizome
What are a few examples of other "old *nix tools" that are showing their age
and have "known deficiencies?"

~~~
alextingle
CVS?

~~~
oneweekwonder
I believe this is a bad example. Because CVS is only under maintenance there
is no new feature getting added and the last stable release was in 2008. Also
anyone that does a bit of research on version control software would realize
CVS is only around for legacy purposes and there is enough paths to convert to
SVN or a more "newish" DVCS.

~~~
alextingle
CVS uses a text-based format. SVN is an "improved" version that uses a binary
format. I think the parallels are quite striking.

(FWIW I don't think that syslog is broken, but then I'm a programmer. Source
control is a key tool in my craft, so I'm keenly aware of CVS's limitations.
If I were a ststems administrator, perhaps I'd feel the same way about
syslog?)

------
msumpter
On a compromised system you can't trust any file stored on the system. Be it
text log files or binary log files. The person who compromised the system will
just dump the entire binary store instead, which is normally what happens to
text log files.

I use many commands (grep, tail, head, etc) to view and parse log files daily.
And storing the files will just add complexity to a very simple system. Now I
don't disagree that syslog needs some further advancements for access
control/management. I may just be not thinking a head and what else could be
possible; but I just see this interrupting my normal workflow for debugging
issues.

~~~
dredmorbius
Cryptography, properly applied, allows you to verify file _integrity_. The
real problem becomes one of key management however -- if the Journal proposal
solves this issue (and it's not a simple one) then it's possible that the "has
the integrity of my system logs been preserved/maintained" question can be
answered.

Managing keys properly is highly non-trivial, especially on a system in which
you want those keys to be available/accessible automatically at boot. Since
the proposal works via signing (hashing) rather than decrypting, PKI could be
used, but I still smell issues.

~~~
jerf
Cryptography allows you to verify that somebody who does not possess your
private key did not modify the file. A hacker on the same system as a
syslogger has the private key of the syslogger. They can "simply" (for varying
values of simply) replay the logs again, only with whatever events they want
filtered out or modified, and it will pass the crypto checks just fine. You
have to offload your logs somewhere else, at which point the whole encryption
in the logs themselves are sort of redundant.

Managing keys in this case isn't just "highly non-trivial", it's flat out
impossible.

~~~
dredmorbius
Using PKI, this isn't necessarily true. Even constructing a system with simple
one-way hashes might be sufficient.

If the logging system is based on a hash of the current and prior records, you
get chaining and integrity validation _prior to the attack_ (one problem of
attacks is wiping out immediate or all prior history). If you've got remote
storage / logging, particularly on multiple systems with independent trust
models (compromising system A doesn't necessitate compromise of B and/or C),
you're in better shape still.

A signature which required periodic updating of a key from a key provider via
a mechanism that prevents later key reuse would mean that past records
couldn't be fabricated or modified after the fact. This might require passing
data through two or more systems to accomplish the signing in a trusted
fashion (not particularly amenable to high-rate logging).

The verification key need not be on the logging system at all (and ideally
wouldn't be). You'd access and verify logs from a standalone hardened and
highly trusted system (say: booted from known good static media).

Much of which is good for the extreme case, but as I note in my other
comments, isn't particularly practical for day-to-day needs. Which means it's
also likely to be unfamiliar to technical staff, rarely practiced, rarely
exercised, and buggy.

~~~
jerf
If they are on the box and you don't have your logs offloaded, they can at the
very least destroy the logs, and barring your complicated key updating system
which would itself require lots of careful review, probably forge anything
they want. But destruction is a pretty useful capability itself.

If the logs are being offloaded off the system, then you don't need any of
this either because they are already being put somewhere the hacker can't
touch. (Of course if they hack _that_ system then you're in real trouble, but
that's just moving the problem around; ultimately if your hacker owns
_everything_ , you've really, _really_ lost.)

There's no inbetween state where all this complexity solves any problem.
Either you've got off-box storage and you don't need it, or you don't and
you've already lost if a hacker gets root.

~~~
dredmorbius
I agree with your points here.

I don't see the proposed journal solution really solving on-system log
integrity.

There are a few other points (processes impersonating other processes, e.g.,
or extended logging formats) which might be better supported in something
other than a traditional syslog. But I'm absolutely not sold on Journal.

------
viraptor
I'd love to see something like that implemented - for the benefit of having
semi-structured messages... Oh wait, there are lots of daemons which already
do that. And they use properly documented formats, like GELF. And they support
forwarding over network. And they store data inside a proper database, instead
of yet-another-file-format.

I'd much rather they simply blessed one of the existing solutions (greylog2,
fluent, loads of others) and created a compatible minimal daemon which can be
included by default in a distribution. Currently they're neither providing new
good solution, nor solving the structured log problem (flat k/v? why not just
go json/bson?). They're also trying to solve single-host security, which seems
pretty close to impossible without moving logging into something like TPM -
just send the logs outside, so that the logging channel is append only.

There is also one point which makes me really worried about what the hell are
they thinking... "The timestamps generally do not carry timezone information,
even though some newer specifications define support for it." Simple solution
is to log in UTC, always, everywhere. Run your servers in UTC too - unless
it's your local desktop, there's no reason not to.

------
saljam
Plan 9 has an extra permission for files; "append." When set, this means
programs are only allowed to append text to the end of the file. Both
simplifying logging (just open the file and write) and making it more secure
(no arbitrary writes.)

I wonder if Lennart and Kay would accept a solution that simple...

~~~
wmf
Linux also has append-only files, although as usual root can override it.

~~~
ciupicri
Are you referring to _chatattr +a log_? The problem with these files is that
they can't be renamed:

    
    
        # mv log log.old
        mv: cannot move `log' to `log.old': Operation not permitted

------
tincholio
Do these guys even have an editor? From the article:

 _...nature of the syslog, which basically excepts text strings in whatever
form the application..._

 _...one daemon may send information about an even in one way, and another
daemon in a completely different way..._

------
siculars
Just because you make something "binary" does not mean it is inherently
secure. As pointed out elsewhere, syslogd is a pretty reliable, simple and
flexible foundational piece of infrastructure. Changing this with something
un-tested and not entirely understood would, imho, be a Very Bad Thing(TM).

------
jleader
It makes me smile sadly when one of the problems cited is "Syslog is only one
of many logging systems on a Linux machine", and the solution proposed is the
addition of another logging system.

~~~
Karunamon
<http://xkcd.com/927/>

------
spc476
There is no method currently defined to forward logs to another machine, so
the journal is definitely prone to a local attack wiping it out (you typically
have to be root to wipe log files anyway). The only method mentioned to move
logs off-system is scp or rsync or some other method of copying a file over
the network, which, to me, is a non-starter. I keep logs both locally and
forward them to a trusted host, just in case.

I'm neutral on the binary format of the log; I can see both sides of the
argument.

------
plq
Syslog is definitely not on its way out. It's been in use for 30 years because
it _works_. Perhaps this is rather Redhat trying its hand at the embrace &
extend game by introducing proprietary tools? Forcing staff to train on such
technologies (with pretty yet dysfunctional guis) will make migrating to other
systems harder.

As for the free-form text "problem" this solution claims to address: Different
daemons produce logs in different format simply because one's data won't fit
in another's schema. And you can actually live with postfix and qmail
producing logs in different formats about the same event because you have
strictly one of them installed in your server. And in my experience, switching
from one to another is a huge undertaking where rewriting your regexp hacks is
a drop in a bucket.

------
inopinatus
Mm. So many nails in this coffin.

Many times I've had a need to grep or tail a log in a hurry. This kind of
change compromises the entire modus operandi of good Unix admins.

There must be tens of thousands of Unix-based tools out there that use the
syslog(3) interface and I'm damn sure that most of them will not change Just
Because Someone At Fedora wants them to.

I don't believe a binary format is any more inherently secure than a textual
one. Want non-repudiation of logs? Sign them and send them off-box.

They would be much smarter to introduce incremental improvements i.e. extend
the existing syslog protocol to accept a structured data format; introduce a
log signing journal for integrity checks.

------
pstoneman
The bit I really don't like is that it enforces systemd - and it sounds like
it's part of it. I think one of the strengths of Unix/Linux is the
architecture of many simple tools, each doing one job well, and one job only -
which this sounds like it violates? Ick.

------
pjscott
They're going to need really good import and export tools. If they can provide
those, then most of the objections to the change go away. If they can't, then
I doubt it will gain traction.

------
jcmhn
This is the same sort of thinking that is trying to add a registry to linux as
a replacement for /etc

------
Ziomislaw
Is it just me? Or do They want to make un*x more windoze-like?

~~~
viraptor
Depends on what do you mean by that. Eventlog has both good and bad parts. If
they integrate only the good ones, I call it improvement, not "more windoze-
like".

------
lanstein
If this means that syslog-ng is in the way in, great!

