I didn't know that systemd also provides a syslog functionality.
But it makes me wonder:
- the wiki pages mention that the default syslog should be removed because it makes no sense to have two solutions
- the pages also say that for more features you should install a "real" syslog solution (rsyslog, syslog-ng)
Question: Why does systemd then come with syslog functionality in the first place?!
This is a honest question, not trolling, what will i gain from systemd journal? Everything points to a) pain to switch and b) less functionality. Whatever that is..
After reading a bit, it appears that this journal is some binary data.. Call me old fashioned but i really like my easily transferable and everywhere readable plain text logs.. why is it binary?!
The journal has been introduced as the natural way to manage logs now that systemd takes care of all the output of daemons.
> If you are wondering what the journal is, here's an explanation in a few words to get you up to speed: the journal is a component of systemd, that captures Syslog messages, Kernel log messages, initial RAM disk and early boot messages as well as messages written to STDOUT/STDERR of all services, indexes them and makes this available to the user.
While syslogd and Co. just take care of moving the debug statements from RAM to the disk or to a network device, the journal is meant from the beginning as an interface for querying the logs, for example
Call me old fashioned... but what's wrong with grep? If you need more features like searchable metadata, the other logging solutions are already pretty full-featured.
I know this goes back to the old holy war of modular versus... not modular(?), but personally I prefer the modular approach.
The other answers give some of the advantages of journal, but imho fail to capture the philosophical underpinnings; or the real reason it's there.
Traditionally, unix systems have had a very distinct and different processes for managing the system during bootup and runtime. For hardware, there initially were only the bootup systems and hotplug was added haphazardly a piece at a time, and it never really worked properly in all corner cases. Non non-systemd systems, it still doesn't.
A lot of the justification of systemd is to unify the bootup and runtime systems, so that there is only one, and there should be no difference to a daemon or a driver if it's pulled up during bootup or when it's plugged in or by the user from console. This unification makes the system simpler, saner, and much more robust. However, it means that there has to be large-scale changes in system architecture. One of those changes was that logging interfaces need to be available before init forks the first time, and among other things, before filesystems. This meant that old logging daemons just couldn't do the job.
I could believe that some people buy into this explanation. However I get the impression the real reason those older systems bother Poettering is that they weren't written by him.
I would say that Poettering could have gotten all of this go a lot smoother had he just contacted the rsyslogd devs and worked out something with them before embarking on writing a new logging daemon. Not doing so probably cost him 6 months of adoption, and was probably caused by a personality flaw.
However, the technical reasons why a new one was needed are very valid.
According to the design document [0] the primary reason is this problem with syslog: "Binary data cannot be logged, which in some cases is essential (Examples: ATA SMART blobs or SCSI sense data, firmware dumps)"
Also, instead of lines of utf8 strings, the systemd format is extensible. This means applications can store additional meta data without some syslog-grepping script breaking.
I can totally relate to wanting to extend log output without (too much) fear of breaking consumers with every change. I was excited to read http://tools.ietf.org/html/rfc5424 where section 6.3 describes structured data storage in syslog. However, I have yet to see anyone using this in production, and I haven't had the opportunity to try it myself.
I didn't know that either. From reading the linked thread in OP, it seems like the idea is that the systemd journal comes in at a lower level (started from initrd? kernel module? no idea...), so it helps with debugging early boot process before /var is mounted. But I also think there should be standard syslog for reasons I just mentioned in another comment. Seems like there are good reasons to have both.
They wouldn't have to give that up by writing a text log. They'd just be presenting more than one source of truth about the the log, from less secure to more secure.
Could somebody please remove Lennart Poettering? This systemd thing is a mess that tries to do way too much. I've had a huge pain rewriting couple dozens of init.d scripts with custom logic to service files. Friends of mine complained on that too. I'm really glad that systemd is not adopted Debian so I won't have this pain on servers. Also journalctl is not a convinient tool, less and bunch of files were working much better for me.
That's beyond insane. Leave the man alone. Disagree all you like, but this "Ah, that guy that ruined my life with Pulseaudio is now going to take over other stuff" whining, this crappy attitude _really_ needs to go.
Start arguments against the technology, take part in the projects (-> here, Fedora) and vote against decisions you don't like, but don't start running around with pitchforks.
You consider 'Remove that person please' a software discussion? In what world?
Did you even discuss software? I see just an attack against a person and some vague anecdotes, "friends complain too", waive hand.
Note that your post is infuriating, but the submission is worse. This thread is why title moderation has a right to exist (for future reference, given that I hope someone's going to fix the title: It was "Lennart Poettering wants to remove syslogd; use systemd journal instead").
Would you tell me what title you'd expect? I honestly don't know what you think is wrong with it.
I don't think ranting against the title without providing a reason and/or better suggestion is very constructive.
"F20 System Wide Change: No Default Syslog" would be a good choice: It's the subject (both 'the mail header called subject' and 'the subject to discuss') of the thread you linked to.
Rephrasing that a la "Proposal to remove a syslog from Fedora default installations" would be fine.
Dragging a person into the title is just asking for trouble and attacks and serves no purpose. The proposal is technical, the discussion should be technical.
You deliberately put his name into the title. Why? And why just him, not both owners:
Change owner(s): Lennart Poettering, Matthew Miller
You had an agenda, even if it wasn't a conscious decision. That's just wrong (and not constructive either ;-p)
- systemd isn't relevant to the discussion. Fedora already uses systemd (and the journal)
- why in the world would you attack a person for writing software and lobbying for its usage? Even if you don't want to use it? I guess you could attack half of HN in that case
And finally:
- the linked thread is a proposal to remove a single package from the default installation of all Fedora distros. The package is still available for installation, just won't be installed automatically.
That's the topic. Not the person behind the proposal, not the software that is already running on all current Fedora systems. That's just not the point.
I'm a systemd fan, not that happy about journald/journalctl (mostly, because I'm not yet used to it and really just use journalctl -b or journalctl -f, which .. gives not enough benefits to make it 'nice' for me). So - should I insult Lennart, call for his 'removal'?
It's low, because it distracts from the technical subject. Imagine your next proposal at work would lead to a 'Can someone please remove pbsdp' comment. Is there _ever_ a time when such a statement is acceptable? Would that count as taking part in a discussion?
Again: journal is already part of Fedora, enabled by default and doing its job. The whole discussion is whether a separate syslogd is going to be provided by default. Most people here _love_ Chef/Puppet etc. -> Just override the default and make sure that a syslogd is installed. Done.
Nobody 'attacks' anything. Proponents of one module want to remove a redundancy. That's not an attack.
Anecdote:
I happen to know a thing or two about Tomboy (note taking app, written for Mono). Someone thought that Mono is 'teh ev!l' and created GNote, a more or less line by line port of Tomboy to a different language. Some distributions replaced Tomboy with GNote, after being convinced that it's the better choice. That's not an attack, it's potentially lobbying - or just plainly a matter of choice.
journald was enabled in Fedora very recently, on the premises that no one would feel a thing, just an implementation detail of systemd. Now systemd's syslog compat is seen as unnecessary overhead, and at some point the goalposts will move again and alternatives to the systemd journal won't be supported.
I'm writing this because I've been seeing the same sort of shifting promises when systemd assimilated udev. At first it was a trivial repository merge, just something to make the developers more comfortable, no impact to anyone else. Now building udev without systemd is unsupported, and some “cleanups” are made to tie udev to systemd and kmod. This doesn't benefit systemd but it hurts anyone who uses an alternative.
The same kind of breakage of alternatives is pushed on gnome via logind, and the kernel via cgroups. It would be myopic to focus on promises made in a particular thread and ignore most of systemd's history.
systemd includes journald (and - as the thread that this whole discussion is about explains - journald is really a ~hard~ dependency for systemd: That's where early boot messages go, for example).
Systemd is in Fedora for quite some time already [1].
2) "Now systemd's syslog compat is seen as unnecessary overhead"
NO! Not at all! That's going to stay, that's not even mentioned anywhere in this discussion. Really, the thread even talks about improved integration between journald and for example rsyslogd (the latter moved from the simple 'receive log messages from journald' interface to the 'let me ask journald for messages and import them as good as I can' it seems).
No one, certainly not in that thread, wants to remove the syslog compatibility.
The discussion is just about _not_ installing a syslog daemon by default. It's still supported, the compatibility (from systemd/journald) is there and going to stay. You just don't run a syslog daemon without installing it explicitly.
3) The trend, what you perceive about the slippery slope of things
This is hard to argue about. There are no hard facts to prove/reject. You don't seem to like a number of recent changes in the linux ecosystem. That's - okay. Natural. But I still think that you should take a step back and reconsider your opinion on the developers' goals: Are they really trying to 'attack' stuff or are they offering something they genuinely consider superior? Are they single-handedly changing the stuff around you, or are boards/committees convinced that these ideas are worth pursuing?
> 2) "Now systemd's syslog compat is seen as unnecessary overhead"
I can't believe rsyslog compat won't suffer, well-meaning reassurances from a third-party can't trump the documented attitudes of core developers who aren't bound by them.
> 3) The trend, what you perceive about the slippery slope of things
I've illustrated a lot, I could easily find more.
I do in fact believe Lennart et al are developing and promoting a system (distinct from the Linux ecosystem) that they consider genuinely superior. It's what being an open-source developer is all about. It doesn't mean they aren't engaging in some ends-justifies-the-means politicking on the way to world domination. I'm fed up with the latter, and with having the bad forced with the good.
> less and bunch of files were working much better for me
Just because it works for you does not mean it works for everybody else. The mess of log files below /var/log has been a problem for a long time. The format is not descriptive enough, there timestamps are second resolution only, no timezone information, and yet the times are in local timezone.
Please don't hold on /var/log/messages just because it has been like that for ages. That should not be the justification for it existing.
> I've had a huge pain rewriting couple dozens of init.d scripts with custom logic to service files.
That's funny, because my experience has been that writing init.d scripts is painful and even package maintainers sometimes get it wrong.
>The mess of log files below /var/log has been a problem for a long time.
How so? It follows the *NIX "everything is a file" philosophy, so they can easily be processed by other utilities that everybody already knows.
>The format is not descriptive enough
You mean the standard filepath? /var/log/(daemon name).
You mean the actual textual format of a syslog message? Timestamp, daemon, facility.severity, message. If your messages are useless, isn't that a function of what's creating them, rather than what's capturing them? What additional data do you want that you're not seeing?
>no timezone information, and yet the times are in local timezone
What other timezone should they be in? Personally I think it's easier to look at the system clock (actually, I usually don't even have to do that, since I have a clock in the corner of my screen session) and make a mental note what timezone you're in rather than add two bytes to every message.
>Please don't hold on /var/log/messages just because it has been like that for ages. That should not be the justification for it existing.
It's the thing that wants to change the way it's always been done that has to justify itself, not the other way around. What does the systemd journal give us that's worth throwing out decades of knowledge and muscle memory?
Configuring syslog isn't exactly a great feat of skill either.. pick your favorite daemon, it still follows the general format of (messages from this thing) (with this characteristic) (go here)
It's the thing that wants to change the way it's always been done that has to justify itself, not the other way around. What does the systemd journal give us that's worth throwing out decades of knowledge and muscle memory?
That's how modern systems are developed. That's all the justification you need, really. Define the use cases you and your CADT buddies considered as "modern" and everything else as "legacy", then you can say "That worked fine in the 1970s, but modern systems don't work like that anymore."
> Just because it works for you does not mean it works for everybody else. The mess of log files below /var/log has been a problem for a long time. The format is not descriptive enough, there timestamps are second resolution only, no timezone information, and yet the times are in local timezone.
Have you ever tried to actually configure your syslog?
> Please don't hold on /var/log/messages just because it has been like that for ages. That should not be the justification for it existing.
Seriously, did you ever try to configure it?
> That's funny, because my experience has been that writing init.d scripts is painful and even package maintainers sometimes get it wrong.
For simple cases init.d script is mostly overkill, personally I had to write generator for this stuff. Solved 99% of probles in simple cases. But in complex cases where you split configuration (hello /etc/{conf.d,default}/*) or need to make custom actions (no matter which), service files lose. Now you have to write a wrapper that will be actually started by systemd. Who were saying what about redunant entities?
systemd is somewhat better than upstart on some fronts, and equally bad or worse in others.
For instance, I like upstart init files much better that systemd "services". Their ini-like syntax is very limiting if you need embedded scripts. I'm looking at you, PreExec. Wrapping everything through "sh -c" looks hideous, and it's not an advantage.
upstart init files with logrotate's syntax are way more readable. The "script" blocks avoid the "sh -c" escaping madness for you.
both upstart and systemd don't "get" live system debugging. I complained loudly both on debian and on the systemd mailing list, but people don't seem to grasp the concept:
If my system doesn't boot (no console), I'm f$#@. The daemon is waiting for some event to occur, but I have no clue which one.
With standard init, there are very few commands to respawn. The system is much more robust. getty on a local tty is part of them, and as such I never had a system where I couldn't debug startup issues.
With upstart/system, if I have an issue with udev (very common between updates), I do not get anything, even though the system is ready enough to start one.
What I need is a sysrq like binding for "force-start a getty now please". I need to be able to query init and ask "what even target are you /going/ to, which events are requisites and still not met? There are the obvious things you need to debug dependency-based systems.
But you don't have them.
No. But instead you can kill the running system, and reboot with "verbose boot". Because a scrolling terminal with 20 lines/sec helps apparently (hint: it doesn't - the system will still hang). So after that you can reboot once more, with init=/bin/sh and boot yourself.
The same issues go for shutdown.
I have nothing against systemd. Dependency-based boot is great. But both init systems seem to miss that a critical piece like init needs to be small, robust and debuggable.
I'm running systemd on unstable. I've lost count of the times that the system didn't boot properly because of an udev change that incurs in a dependency loop. This should never happen. Shutdown doesn't work since two months if you have a cryptsetup system, again because of a stupid unmet dependency.
Plus, again, systemd ini-file scripts are fugly. You need external scripts just to run Pre/Postexec.
Many of the design decisions behind Lennarh are, IMHO, regrettable. This goes way beyond systemd in general.
Fortunately systemd seems to be going forward somewhat, thanks to debian mostly, and like I said before, dependency-based boot is nice.
Could somebody please remove all the people that hate Lennart Poettering? This systemd thing is precisely what we all need! sysvinit is archaic and upstart is quite buggy and lacks good tools.
Have you even seen the systemd and Journal tools? I've wanted that for years.
It's far from decided, and controversial to say the least. The systemd package maintainers are certainly acting like they're going to be the default, but it isn't their call.
- The journal has limitations, like what files it stores the journal in and the [default] size of the journal, which is only 10% of the filesystem size. Which sounds retarded to anyone who has ever created a /var partition for logs, but anyway. (Configuring multiple syslog files is an efficient hack to spread log data load across filesystems, which AFAIK systemd does not support)
- All applications that were developed specifically with syslog support (such as using the glibc syslog functions, or the command-line tools, or the flat log files) will break. The fix is to reinstall syslog and configure it to use a journal device that systemd exports [once you configure it to].
- Any devices that this host was collecting syslogs for, or any syslog collection servers this host was sending its logs to, will break. (Syslog is the only de-facto standard [enterprise or otherwise] for sending or collecting logs on a network, besides maybe snmp, but that's not really logging)
- Systemd's programming-oriented control over log journals means you can only do things it lets you do. For example, there is built-in log rotation based both time and disk space used, but no function (that I know of) to tell it to rotate on-demand, which is a common sysadmin task. You are removing all the previous flexibility of syslog logs by using these binary journals.
- If you support more than one operating system, or hell, more than one Linux distro, you have to install syslogd and configure systemd to use it, or you'll be doing twice the work just to support Fedora's mandatory new logging system along with every other system's syslogd. (Even if every distro switches overnight you have legacy systems and code to deal with)
The basic proposal seems to be: I have this new logging system, so remove your old system because mine is better. Which is no better an argument than: I have this old stable logging system that works great, so your new system needs to chill the fuck out until it supports all the legacy stuff.
It doesn't seem like there is any need to remove syslog. The guy just wants it to disappear.
Which sounds retarded to anyone who has ever created a /var partition for log
Oh, but don't you know? You shouldn't need more than one partition; they're getting rid of separate /var (I only half joke; after they broke separate /usr by linking boot/system things to it, they declared separate /usr to be "deprecated").
If you support more than one operating system,
Another one of Poettering's favorite ideas is that nothing besides Linux matters. I may agree with him for my own personal computing setup, I think it's flat out unprofessional to eschew portability and cross-platform standards. Even ignoring the obvious benefits, making things portable can reveal bugs and unstated assumptions in your design which will trip you up later.
It doesn't seem like there is any need to remove syslog. The guy just wants it to disappear.
If there's one thing that strikes me about Lennart (and others, such as those working on Wayland), it's what appears to be a severe case of myopia, or just flat out willful ignorance, of systems that have come before. The egotism is astounding.
Granted, sound under Linux was far from perfect (so I've heard; it has always just worked for me), and PulseAudio eventually got stable and efficient enough to be be acceptable. That being said, it does feel a whole lot like "here's how we're going to do it, shut up and take it." It would be more palatable if they identified problems, proposed solutions, and then canvassed the community. Hell, even just proposing solutions as patches or wholly separate projects and letting the community vote with their feet would be preferable to this dictatorship mentality.
EDIT: Just to be clear, I distrust Lennart (and Sievers') approach to system software because they don't seem to think of the bigger picture or be aware of the past. Do you know why so many people are attached to the "legacy" "deprecated" ways of doing things? Because they work, across multiple platforms, multiple OSes and multiple paradigms (embedded, server, desktop, cluster, mobile, and future ones we can't even imagine). Because other ways have been tried (eg, binary logging on Windows) and failed. Yes, some things could be improved (dependency based boot is a good idea). But other things, such as the core modularity principle in UNIX, are there for a reason, and should not be lightly tossed aside.
My understanding is that it's fine to have a separate /usr, it just needs to be mounted by your initrd. Trying to ensure that everything that could possibly be needed to mount /usr is in / turned out to be a lot more difficult than building an initrd with the relevant tools in it.
There may be ways to do it right now, but it's hard to fight the fact that Lennart doesn't want it to be supported. He's already thinking of deprecating initramfs builders that don't have systemd running inside (which causes trouble for Arch). If systemd continues to clash with initramfs tools it takes human resources away from working on things Lennart doesn't care about.
It sounds like systemd journal is a good thing, especially for early boot debugging. That said, I don't get the proposal to remove the standard logger. Sure, the logs are duplicated...but having important data stored redundantly seems like a good thing. Plus, you still keep the "everything's a file" philosophy, so you can continue to access the log file in situations where accessing a binary is unfeasible or impossible.
The example of a box where /usr is hosed and you only have access to in-memory programs like the shell, is maybe not the best example. Perhaps a more practical one is the need to monitor / alert on certain log entries in a legacy or upstream system that expects to open and watch a file.
As all distros I know of include a log file rotation facility that deletes old system logs after some set period, having duplicated data in the journal and in traditional system log on disk seems like it would be a problem only in special cases.
Cost-to-benefit, it seems to me there should still be a traditional syslog provider included by default.
I would wager that 99% of users would fall into one of two categories:
1) The journal is completely adequate for their needs; or
2) No default configuration could ever be adequate for their needs, since they need to configure rsyslog to ship logs off the box.
I'm curious as to what sort of cost-benefit analysis would lead you to conclude that everybody in group (1) should be forced to install rsyslog as well so that the perhaps 1% of people who "need to monitor / alert on certain log entries in a legacy or upstream system that expects to open and watch a file" can be spared the horror of having to type "yum install rsyslog".
OK, fair enough. Going with your figure (which I think is pretty low), what is the benefit to 99% of users from breaking tools for %1 of users? As far as I can tell it just a bit of additional storage space. Do 99% of users actually benefit from that extra storage?
Good point, but not my example.[1] I found it a bit weak tbh. I think a more common and compelling case is where you need to treat the system log as a file input to some existing tool / system. Sure, you can install a traditional logger if you need it in such a case, but what do you gain by not including it? A little bit of disk space?
I wonder if there's an unstated motive behind the proposal to promote a more pervasive use of systemd. I mean, being honest, admins aren't going to go to the journal unless they need something the traditional log file doesn't provide (which would be rarely if rsyslog and ilk are pulling everything they can out of journal anyway).
Nobody is going to switch to journalctl | fgrep OMGERROR unless they have to (well, they won't have to, but admins are lazy, and if that's the default way, then they will use it rather than installing a new package). Just look how long it took (is taking) for admins to move to ip vs ifconfig/route/etc...
Yes, Arch is using systemd. So is Fedora. The argument that is linked wants to remove a syslogd implementation from the default installation.
Right now, the default is journal (cannot be removed, if you're using systemd as explained in that discussion) _plus_ syslogd. The proposal asks for journal w/o syslogd by default (but available as a package of course).
That said: My Arch systems don't run a syslogd either, so .. maybe they switched 'completely' in that announcement?
When systemd became the default in Arch a syslogd implementation became optional.
Just after a few weeks after installing systemd and running it side by side with syslogd I completly switched to a pure systemd + journald without the syslogd.
That's how I remember it at least, I may be wrong in some point.
---------
EDIT:
Ok, I went and searched a little more, here is what I found related to systemd + journald + syslog discussion:
Can I just point out, for all the Lennart-haters, that he doesn't have the final word in what gets implemented in a distro. It's not like he's shutting down any syslog projects either, or threatening to kill the maintainers' cats if they don't use his software. Distro maintainers move to his software because they think it's an improvement over what they had before.
Case in point: Lennart works for Red Hat, but Arch is actually further along than Fedora in adopting the Journal.
Personally, I removed my syslog daemon about two months ago, and haven't missed it. I also switched to Fedora back in the new year (from Ubuntu) specifically because I wanted to try out Gnome 3 and Systemd. Loving it so far :).
While I'm not a Fedora user, I wish them good luck, and hope the change goes through. It would be of great benefit for other distributions aswell, and perhaps we could get rid of traditional syslog for good at some point... (wishful thinking, I know).
Never said it would be leading. But more distributions switching to journal-only is a good thing, because that makes it that much easier for the rest to follow suit.
Component B) Role: log dispatching/receiving/sending/filtering
And now what?
A component which partially does in a non _standard_ way, incompatible with all third party tools (in the sense the tools needs to be modified or adapted).
What is next, systemd will replace cron too, in powerpoint format?
Reading the whole thread, I have a new concept in my mind... "UEFI cookies", didn't know about them.
Really does not remember to me my first impressions with cron :)
I don't doubt of the quality of this guy and it's ideas and solutions, I cannot say I'm better. And I've seen many "weird for me" things across the years coming from upstream to my systems.
I'm tired of fight with abstraction layers to try to make things simple. But in my work I use to be condemned to use whatever upstream picks across different versions.
I haven't seen it in the discussion, but systemd's journal has horrible read performance. Apparently it's poorly indexed, or the disk layout has issues.
They're not even reinventing Unix poorly - they're reinventing the Windows event logger badly, and I'm not aware of anyone that ever really liked that in the first place, especially Windows admins.
- the wiki pages mention that the default syslog should be removed because it makes no sense to have two solutions
- the pages also say that for more features you should install a "real" syslog solution (rsyslog, syslog-ng)
Question: Why does systemd then come with syslog functionality in the first place?!
This is a honest question, not trolling, what will i gain from systemd journal? Everything points to a) pain to switch and b) less functionality. Whatever that is..
After reading a bit, it appears that this journal is some binary data.. Call me old fashioned but i really like my easily transferable and everywhere readable plain text logs.. why is it binary?!
* https://fedoraproject.org/wiki/Changes/NoDefaultSyslog
* https://fedoraproject.org/wiki/Features/systemd-journal
Edit: Thanks for the explanations on the binary format, much appreciated.