
“I wish systemd would get over its thing about syslog” - zdw
http://utcc.utoronto.ca/~cks/space/blog/linux/SystemdAndSyslog
======
vezzy-fnord
I think journald is by far the buggiest component systemd has. Log
corruptions, the dubious Forward Secure Sealing mechanism, journal file
fragmentation, the gateway daemon, coredump hooks. There's been several CVEs
for it, too. Of course, you cannot disable it without source code
modification.

Quoting from the systemd wiki [1]:

    
    
       Note that it is now the journal that listens on /dev/log,
       no longer the BSD syslog daemon directly. If your logging
       daemon wants to get access to all logging data then it
       should listen on /run/systemd/journal/log instead via
       the syslog.socket unit file that is shipped along with
       systemd. On a systemd system it is no longer OK to listen
       on /dev/log directly, and your daemon may not bind to the
       /run/systemd/journal/syslog socket on its own. If you do
       that then you will lose logging from STDOUT/STDERR of
       services (as well as other stuff).
    
       ...unless your syslog implementation is socket
       activatable many services will not be able to log to your
       syslog implementation and early boot messages are lost
       entirely to your syslog implementation.
    

So the fact that keeping your regular syslogd wasn't as simple as merely
symlinking syslog.service was publicly known, but often overlooked by
proponents during discussions.

[1]
[https://wiki.freedesktop.org/www/Software/systemd/syslog/](https://wiki.freedesktop.org/www/Software/systemd/syslog/)

~~~
Qantourisc
That's just bad, this is breaking API's ... suddenly other logging deamons
need to change their code because systemd is squatting the log file ?!?

------
mrmondo
SystemD makes me feel weird.

When I boot Debian 8 (which has systemd) I'm amazed how quickly it boots.
Systemd units are nice. SysVinit was a mess.

I wish I could interrupt stuck processes when booting / shutting down:
[http://i.imgur.com/a4ImjAk.jpg](http://i.imgur.com/a4ImjAk.jpg)

When someone comments about how overly complex and controlling systemd is for
an init system, I think of all the hideous shell scripts that don't talk to
their neighbors which end up forming most the sysvinit boot process on many
distros (in the past?).

I wish there was something [lighter|simpler|more portable| _more unix_ ] than
systemd.

Every time I seed something [good|bad] about systemd it makes me want to
[comment|upvote|walk away].

~~~
cturner
> SysVinit was a mess.

Not just SysV - every attempt to move beyond BSD-style init has been a mess.

> I wish there was something [lighter|simpler|more portable|more unix] than
> systemd.

I think the trick with systemd is to realise that it's not unix. But it's
something else that's really interesting in its own right.

Init/scheduling is a much more difficult problem than it first appears, and it
has always and everywhere been tacked on to its host OS as an afterthought:
init scripts, cron, windows services, system v init. All are poor, partial
solutions. Yuck.

To fill the void, there has been a proliferation of application containers,
each of which does a different 70% of the general problem. All of this stuff
could just be done in the platform. And that's what systemd is.

There's going to be two separate camps at the end of this: a group led by the
BSD platforms will stick to convenitional unix goals. On the other side, the
linux kernel plays second-fiddle to systemd. The systemd service graph is a
new operating system tradition.

[New, but not without precedent: in the debian world, the 'apt' package
management system has become more significant than the kernel. You can create
things that are not linux, but which are still debian. Both systemd and apt
use a sophisticated data-graph to contain a difficult problem.]

There are exciting opportunities for developers on a systemd operating system:
you'll be able to quickly put together application stacks using microservices
in a way that hasn't been practical until now. It /is/ unixy in the sense that
it gives the console operator a powerful new general-purpose tool.

~~~
ChuckMcM
_I think the trick with systemd is to realise that it 's not unix._

I keep coming back to that. It seems to me that there is a tension in the
Linux world about people who want an open source UNIX and people who want an
open source Windows. Systemd is much more a "Windows" approach to system
configuration management than a "UNIX" approach.

That isn't "bad" per se but it makes for some confusing things. I would be ok
if Linux became the open source windows flavored OS and Freebsd became the
open source unix flavored OS. I believe such an explicit step would help folks
move forward with "where" a certain idiom of OS management should land in the
open source world. Of course I don't expect that to happen :-).

~~~
anonbanker

       I would be ok if Linux became the open source windows flavored OS and Freebsd became the open source unix flavored OS.
    

This mindset should terrify everyone in the GNU foundation. systemd is the
begining of the march away from both the GNU and GPL principles, into an
incompatible fork of POSIX that uses RPC backdoors in order to circumvent GPL
protections.

why doesn't anyone else view systemd as the direct assault on GNU that it is?

~~~
fortytw2
RPC Backdoors? Care to elaborate on this for me?

~~~
anonbanker
Perhaps you could ask yourself "how, as a non-free software developer, could I
use systemd's RPC system to bypass the LGPL and GPL's requirements that all
related software and derivative software must be licensed under the same
license?"

The answer will come quickly, I bet. it's how you figure out magic tricks, as
well. :)

------
liveoneggs
You can sub syslog with any component systemd has cannibalized and write the
exact same article.

That's what the whole argument was all about.

~~~
feld
The people who say "it's not that bad" are those who haven't interacted with
these extremely reliable and integral parts of a Unix system for years and
years and understand the ramifications of bad behavior.

~~~
_delirium
The author of this article has done so, and seems to think that until this
syslog interaction it hasn't been too bad:

> _One of systemd 's strengths until now has been that it played relatively
> well (sometimes extremely well) with existing systems, warts and all._

------
icebraining
So systemd has a few bugs that affects its interaction with syslog. I see
nothing to indicate this is specifically because of some ill will towards it.

Is there a bug report with a dismissive reply? Or one being ignored?

(I do agree that it shows that systemd is too immature for critical production
environments)

~~~
JoshTriplett
> Is there a bug report with a dismissive reply? Or one being ignored?

That's always my reaction to blog posts like this as well. A blog post is not
a bug report. Given the mentioned bug in the systemd/syslog interaction that
made it into RHEL, there should be a bug in the RHEL bug-tracker, and I'd hope
it'd be a high priority one.

~~~
cpncrunch
I think the issue is just stability in general. I just moved to a new server
recently, and I decided to go with CentOS 6 instead of 7, as people were
reporting that CentOS 7 is just a little less stable in general than 6.
(Although systemd probably isn't _entirely_ responsible for the lower
stability in CentOS 7, it probably is one of the major changes in CentOS 7 so
it likely takes a good share of the blame).

You always end up with bugs in released software, even CentOS. Some bugs only
show up in certain situations, and might be difficult to track down. The only
way to give a good guarantee of stability is for a large number of people to
use the damn thing for years on end!

------
feld
So this means any logging/auditing software that requires the data to be
sourced from syslog is unreliable. This kind of behavior would make an
auditor's head spin.

~~~
richardwhiuk
If you are auditing systems based on systemd (or sysvinit or makefiles or
upstart or launchd) then you are probably doing it wrong anyway

~~~
unethical_ban
If you build a custom logger into a startup/scheduler instead of using the
existing logging component, you're doing it wrong.

Really, if you're not using the logging mechanism to check a system, what are
you using?

------
erikb
I only watched part of the systemd wars from the side line and to this day I
haven't come up with a reasonable solution. Both sides have reasonable
arguments.

Saying that a tool is unsupportive towards another tool because in the
interaction two irregular bugs happen sounds like a weak claim though. Don't
all tools have bugs? What makes these two special?

------
digi_owl
And the PR division is already in action in the comments...

~~~
pavanky
The PR division is the entire userbase of Fedora, Arch and a few other
distributions who have been using it for years without any issues.

~~~
deelowe
I use arch and still despise it. The reduced boot time isn't worth the
headache the overly complex systemd suit of tools has introduced. BSD might
have some issues, but it worked very well for my use-cases.

~~~
heinrich5991
To me systemd is not the reduced boot time but the simplified service
management.

~~~
thescribe
To me, as an Arch user, sytemd has been responsible for a series of btrfs
kernel panics. I'll probably switch to Gentoo just to not have to deal with
working around journald the next time I have a choice of distribution.

------
LinuXY
While I agree that systemd does not play nicely with syslog, you quickly
realize that journald can be exported as JSON, filtered, described as time
ranges, chunked and easily imported into ELK. My users (mainly developers)
with less sed/awk fu have come to rely on these features heavily. I'm not
really sure what there is to miss about syslog or syslog-ng when you have a
robust ELK+journald system.

~~~
rwmj
Unfortunately it's also very slow. Text files don't have the many advantages
you outline, but in everyday use they are much faster than the C journal API.

------
RexRollman
The whole blog is interesting to read. Thanks OP.

