I'm one of the people who jumped ship to OpenBSD in the last year or so. I think I first touched a Linux machine in 1997 or 1998. systemd has been part of the reasons I did that for precisely the same reason.
There is a lot of development in the ecosystem dominated by Freedesktop.org, Gnome and (to some degree) large commercial players like RedHat and SUSE. It's more efervescent than it's ever been. Fifteen years ago, I'd have been ecstatic to see so much development being done.
Yet for all this fierce code churning, there's absurdly little innovation. Even in systemd, there's a lot of focus on non-issues like booting without needing a shell, while still inventing service startup notification types because none of the gazillion startup notification types already existent can actually solve precisely the problem systemd claims it solves -- managing complex startup chains.
In the last three or four years, my Linux systems have become increasingly blackboxed, and consequently harder and harder to configure, let along debug when they stop working. Over the span of six months, a tool you just got to learn how to work with (mostly because the documentation seems to have been written by very bored and very verbose monkeys who had only a typewriter to pass the time with) becomes obsolete or is revamped to be unrecognizable. Which would be fine if that actually solved any problems. Instead, it replaces the old set of problems with a new, even more complicated set of problems that developers claim you shouldn't have, while insisting that the old problems were simply due to the old architecture and we'll have none of that now that we have this shiny new thing.
Ironically, systemd is actually one of the better pieces of software to come out of that ecosystem. No, seriously. It's decently documented and as long as you don't need anything more complex than "here, start this daemon after this other daemon started", it does its job.
If you want to peck at something, have a look at Polkit, where someone brilliantly figured out that the best way to configure policies is to write basically undebugable JavaScript code in .rules files.
Polkit is a nightmare. And I agree with you on the churn. There was a time when Linux administration changed less frequently than Windows, round about the 95-XP-Vista era.
Polkit is a great idea, don't get me wrong. However:
* Like pretty much anything from freedesktop.org, the documentation is a joke.
* The security system implemented through it is next to impossible to debug. It's very easy to add "allow <this thing> to happen", but I can't do things like "ok, tell me who can mount filesystems" or "give me a list of all things that users in the wheel group are allowed to do". Normally this would be trivial to implement by parsing the rule files, except that the rules are written in JS (I have no idea why) and parsed in a particular, cumbersome order.
* Probably also as an artifact of having to execute JS code to add rules, many errors are silent, incomprehensible, or occur only when a particular action happens.
* polkit doesn't have a textual interface, it only works through D-Bus. This makes it useless for scripting.
> polkit doesn't have a textual interface, it only works through D-Bus. This makes it useless for scripting.
polkit(8) and pkcheck(1) describe how to use the pkcheck program. It returns 0 on success (access granted).
> I can't do things like "ok, tell me who can mount filesystems"...
I think that's the nature of the problem domain. polkit seems to be for enforcing arbitrary rules, when the built-in rules of something like sudo can't cope. Therefore: arbitrary code snippets as rules, therefore, no introspection.
However...
> ...or "give me a list of all things that users in the wheel group are allowed to do".
pkaction | grep mount
org.freedesktop.udisks2.filesystem-mount
org.freedesktop.udisks2.filesystem-mount-other-seat
org.freedesktop.udisks2.filesystem-mount-system
org.freedesktop.udisks2.filesystem-unmount-others
Gives a list of actions. Use pkcheck to probe for access.
# pkcheck --action-id org.freedesktop.udisks2.ata-standby
pkcheck: Subject not specified
<What the hell is a subject?>
# man pkcheck
/subject
<???>
It means to say that you have to specify a PID for which to check that. But the word "subject" doesn't even appear in the pkcheck manpage.
The synopsis also lists three possible forms for --process. However, only one of them is valid and (the developers currently think) presents no race conditions. There's a note warning that the other two are buggy, but they're still accepted for compatibility reasons, because those two (buggy) methods have been standard for years.
> I think that's the nature of the problem domain. polkit seems to be for enforcing arbitrary rules, when the built-in rules of something like sudo can't cope. Therefore: arbitrary code snippets as rules, therefore, no introspection.
I don't think it's the nature of the problem domain. The "arbitrary rules" can have a user- or group-base restriction. It's an architectural limitation that the system cannot go through the list of all currently enforceable rules and return those that match a particular user.
> However, [...] Gives a list of actions. Use pkcheck to probe for access.
The problem is that pkcheck verifies whether or not a certain process can execute a function. You can sort of hack it to do that, but it's not always effective.
By the way, the reason I need to be able to do it is the following gem:
> There is no guarantee that a function registered with addRule() or addAdminRule() is ever called - for example an early rules file could register a function that always return a value, hence ensuring that functions added later are never called.
This is extremely unpleasant, because you can (I have) run into problems of the following sort:
* I insert a rule with a particular priority that gets registered and runs. Ok, no problem.
* Later on, <some buggy software> inserts another rule which returns a value, thus terminating the rule chain before my rule gets to run.
Now all I find is that my own rule is not enforced, but I can't tell why that happens. There's no trivial way to trace the rule chain.
It also doesn't help that errors in rule files seem to go unreported. If you type "heel" instead of "wheel" for a group name, the whole thing will happily carry on, or at least I haven't yet figured out how to look for those errors.
Fine-grained access control needs domain-specific understanding
This also applies to the user trying to configure it. For all the reasons listed in "weland"'s sibling comment, this is hard to learn. If something doesn't work in polkit, and you're not completely familiar with it, it's very hard to make it work.
The difference appears to be that the sudoers file supports the finite set of access checks the authors of sudo anticipated. If you need something more elaborate than that, write arbitrary javascript to the polkit api.
> There is a lot of development in the ecosystem dominated by Freedesktop.org, Gnome and (to some degree) large commercial players like RedHat and SUSE.
Yeah, i've been thinking the same recently.
The current development above the kernel seems to either be from the desktop down, or from the point of view of managing 1000s of headless instances in a "container" farm.
This with a side-order of Windows-isms demanded from corporate/government clients looking for a way to get away from MS while retaining their centrally controlled fine grained access control.
> I'm one of the people who jumped ship to OpenBSD in the last year or so.
Ha. FreeBSD here.
And so long as you get the right hardware, I find BSD to be far simpler than linux distributions out there.
Ports? Beautifully elegant and simple. Configurable. Magnitudes easier than configuring rpm/debs.
Getting a package approved in debian? Alioth? No offense, the wiki, all these pages on the debian website - they have no concept they have of "Handbook", in the sense FreeBSD has. It's so adhoc, they're healthy, but they're in denial about how hard they make it for new people. By the time you've taken to time to learn Debian's dev warts, you're apart of the convent and no longer care about fixing the onboarding problems.
But to be fair. I have day to day gripes with FreeBSD just because we're short on the manpower Linux distributions have in excess:
On FreeBSD 10.1, chromium crashing easily, settings doesn't work. Firefox just clogs up and stops working. No skype (CURRENT will be in a better position to since epoll support has been added). No Dropbox (as of now, we don't have inotify, and kqueue may not be able to handle it, don't quote me on that).
I do my 90% of daily routines simpler and better on FreeBSD. If only Google / Mozilla (with all the resources they have) gave someone who could work to fix this stuff.
I've occasionally used {Net|Free|Open}BSD over the past few years (in fact, I think my Jornada in the back of the closet still runs NetBSD). I kept sticking to Linux partly because I was using it at $work, partly because it was simple enough and I'm kindda lazy. But yeah:
> I find BSD to be far simpler than linux distributions out there.
The way OpenBSD handles WiFi configuration is a good example of this. Want to connect to a wifi network? ifconfig <interface> <ESSID> <passphrase>. If you need some other fancy flags for those 0.1% of hotspots in the world, you can do those, too (also through ifconfig) but it's that simple.
In contrast, after everyone agreed that wpa_supplicant is probably what happened after Stalin's ethereal spirit had mad sex with Hitler's ethereal spirit and their ethereal baby was incarnated, what did we do to alleviate things in Linux land? Why, we wrote NetworkManager, of course! Conservatively, 90% of today's deployments of NetworkManager exist because no one can figure out how to set up wireless without it, and we put up with its crap because wicd is waiting for us at the other end.
Better yet, this is all nicely documented in the manpage. If you search for "wireless" in ifconfig's manpage, you find out all you need to know. Just for fun, compare that to man networkmanager.
> On FreeBSD 10.1, chromium crashing easily, settings doesn't work. Firefox just clogs up and stops working. No skype (CURRENT will be in a better position to since epoll support has been added). No Dropbox (as of now, we don't have inotify, and kqueue may not be able to handle it, don't quote me on that).
I haven't had crashing things on OpenBSD, but yeah, a bunch of things don't work. Fortunately, they're things I don't use. If I ever need them -- OH THE HORROR AND THE BLASPHEMY -- I have a Windows laptop, which for all intents and purposes is by now about as closed and as hard to wrestle with as Fedora, but at least it doesn't hopelessly break every six months. There are Linux-only things that I sometimes need because at some point between 2003 and 2013, "portable" ended up meaning "we can probably work if you have bash in a non-standard location but that's about it" (eh, Yocto?) for which a cheap VPS is more than enough. When it breaks, I don't have to spend days trying to find out exactly what part of PolicyKit broke or where my devices are mounted this week -- I just wipe everything and reinstall.
While I don't disagree that wpa_supplicant has been spawned from hell, I find it strange that the existence and popularity of NetworkManager is given as a clue for why {Net|Free|Open}BSD is better. You know, connecting to a WPA protected Wifi network is not the only problem users have. For example, another one would be connecting to a VPN, a task that NetworkManager greatly simplifies, assuming that such integration is available of course (e.g. if you think connecting to a Wifi network is problematic, try connecting to a L2TP over IPSec VPN).
There's also the issue that the simplicity exposed in your example is a little misleading. Will the system persist that command and re-execute it on boot? Will it remember my previously used Wifi networks and connect automatically should I want that? How can I view the available Wifi networks in range? Being able to connect to a Wifi network using a simple ifconfig command is great, but the notion that your 80% of use-cases is the same as everybody else's 80% is misguided at best.
> If I ever need them -- OH THE HORROR AND THE BLASPHEMY -- I have a Windows laptop, which for all intents and purposes is by now about as closed and as hard to wrestle with as Fedora
And therein lies my problem with your entire comment. It's not that I disagree with your usage patterns, but I disagree with your assertions in light of these usage patterns. So you're saying that xBSD is simpler / better, yet you can't use it on your laptop, as it lacks things you need. Well, no shit.
Watching Linux distros push essential functions into the Desktop layer (power management, network management, etc) without good CLI equivalents limited my window manager choices in the past. This was mostly because the Gnome/desktop folks at RHT were willing to do the work, and there wasn't anyone else to do it, and then that's what happened.
What this did though is pretty much make it where Gnome was the only usable option, or you had to fight a good deal to make something else usable.
as Fedora's desktop choices seem particularly rough, and have pretty much left me with using Linux in VM environments from Macs.
This is very quickly becoming the default for most developers I know, with a few exceptions here and there.
I don't think systemd hurts too bad in the service statup environment once you rememer the commmands are all different, but I don't like what it's doing with creeping into other areas, like what it does with logging.
I don't know about power management, but I've used nmcli for network management without much difficult, and nm-applet works outside of Gnome as well. I've never felt constrained to use Gnome, which is just as well, since I run Awesome launched by "startx".
> For example, another one would be connecting to a VPN, a task that NetworkManager greatly simplifies, assuming that such integration is available of course (e.g. if you think connecting to a Wifi network is problematic, try connecting to a L2TP over IPSec VPN).
That's because there is equally no uniform and simple way to connect to use those, either. The problem is that the "simplification" that NetworkManager provides is illusory. Yes, it provides an interface that is easier to access, and all the complexity is moved in the back, where it's handled by NetworkManager.
Unfortunately, the mechanisms through which it does that are fiercely complicated, too. I've had NetworkManager break on me countless times (I worked for a company that developed networking equipment for a while, so I've interacted with NM a lot), for a lot of silly things -- making me end up with slightly incorrect configurations or reapplying settings at weird times for no apparent reasons.
The end result is anything but simplified: not only do I have to deal with the inherent complexity of a quirky VPN connection -- now I also have to understand how NetworkManager reasons about it and figure out where it's wrong.
I also think it's misguided to have a single tool handle things so disparate like VPN and WiFi. It's not like you're ever going to be able to expose those through a unified interface -- NetworkManager's own D-Bus-based API being a prime example of that.
> There's also the issue that the simplicity exposed in your example is a little misleading. Will the system persist that command and re-execute it on boot? Will it remember my previously used Wifi networks and connect automatically should I want that? How can I view the available Wifi networks in range? Being able to connect to a Wifi network using a simple ifconfig command is great, but the notion that your 80% of use-cases is the same as everybody else's 80% is misguided at best.
You scan for networks with ifconfig <interface> scan.
The system will not persist commands (arguably, a good decision) but persisting those settings is pretty easy -- you just add the same parameters you pass to ifconfig in /etc/hostname.<interface name>. (The same parameters as in argv[1:n]).
The only thing it won't do out of the box is automatically connect to previously used wifi networks. I suspect none of the OpenBSD developers want that. I only needed a feature like that once, a few months ago when I moved to a new apartment and had no Internet connection of my own (cable company is shit) so I had to snoop from the free wireless of nearby cafes. The fact that the interface to ifconfig is so simple and uniform meant that the whole thing was a ten-line shell script to scan for, and connect to previously known networks, and a shell alias to save the current network to the list of known networks.
There's an even easier way, using virtual networks, but I didn't know about it at the time.
> So you're saying that xBSD is simpler / better, yet you can't use it on your laptop, as it lacks things you need. Well, no shit.
I have a Windows laptop, which I also had when I ran Linux. My day-to-day laptop runs OpenBSD.
> The only thing it won't do out of the box is automatically connect to previously used wifi networks. I suspect none of the OpenBSD developers want that.
That's not true. OpenBSD developers want this. There are several ideas floating around about making the kernel cache wifi passwords, scan and re-connect automatically when it lost link to the AP. This topic keeps coming up during hackathons.
For various reasons it's not as easy to do as one might expect.
The net80211 layer and wifi device drivers don't keep track of link state to the AP at present. This would need to be implemented first. It is essentially client-side roaming support.
One way of doing this is to count missed beacons and assume the AP is out of range if no beacons have been seen for a while. Some devices provide such counters in hardware, some don't.
The biggest part of the challenge is to make this work seamlessly with _every_ driver.
Then there's a risk of feature bloat. Add too much automation and you end up with something opaque like network manager which just pisses many people off. Add too little automation and you don't cover many use cases. And networking is not just about wifi, it's at least about ethernet and VPN as well. Some tools already exist to handle aspects of this (e.g. ifstated(8), trunk(4), interface priorities managed with ifconfig(8), a ton of ipsec tooling), and should be properly integrated in the design.
Another question is how to integrate with desktop wireless tools that come from Linux (e.g. wifi network pop-up menus for KDE4 and Gnome3) which usually require a backend using Linux-specific ioctls or dbus/NetworkManager/systemd APIs to function. But seamless integration with desktops would be outright awesome and make the system much easier to use for non-technical users.
There's probably a couple of gsoc projects worth of work in this. If someone is seriously interested ("no talk without diffs attached" kind of serious) in pursuing this I might be available to act as a mentor.
I hope my reply didn't come pff as a passive-aggressive "those bastards don't care" or anything like that -- I didn't remember seeing anything about this on the mailing lists, so I just figured out no one wants it. I, for one, can't say I do, for several reasons -- ranging from "I'm a control freak so nothing connects automatically to anything on any computer I own" to "I know of no clean way to implement something like this so maybe we should learn to live without it".
Since it's not something I'm interested in, I probably just forgot that I read about it :).
Network Manager still has problems. I've had a lucky streak for a few years where it hasn't given me problems, but the other day the GUI decided not to save my settings anymore. It looks like it does, but it doesn't. If I kill the beast and tap in the commands it just works, but if I run the daemon it keeps feeding the backends with old settings.
How do I debug this? There's just sockets and layers of GTK. I'll probably end up doing whatever voodoo I need to keep it away and give it another few years.
In case you're actually interested in fixing that particular problem, it's probably polkit that's gone bad. To examine the issue further, nm-connection-editor is a separate binary that you can run from the command line, and it'll print errors on stdout - including what it's unable to do.
I'm not the parent commenter, but I personally run OpenBSD on a pretty wide variety of hardware, most recently (among laptops, that is) a PowerBook G4 and a Dell Latitude D830. Most things work out-of-the-box, but there are a few exceptions:
* Power management is nonexistent on PowerPC hardware, so I currently have to shut down my PowerBook between uses, since I can't put it in standby. This is annoying, but workable.
* Hibernation is buggy on the Latitude. Suspend works fine, though.
* Both the PowerBook and the Latitude required the use of fw_update to install nonfree firmware (for the Broadcom-based Airport wireless NIC and the Intel PRO Wireless 3495abg-or-whatever wireless NIC, respectively). Also slightly inconvenient, but - again - still workable.
* Some things don't really work on PowerPC, most notably Firewire. A slight bummer, but hopefully this will be resolved someday.
From what I understand, IBM/Lenovo ThinkPads are - just like they are with GNU/Linux - among the best laptops to use with OpenBSD; IIRC, a lot of the OpenBSD devs dogfood pretty heavily on them, so they get a bit more attention. I'd avoid anything with Nvidia graphics, whereas AMD/ATI or Intel graphics should be decent.
It's an HP 4720s. I didn't buy it on purpose -- it used to be my girlfriend's. When it broke, I gave her my (much tinier) ThinkPad and after I fixed it, we never swapped them back.
I think the only thing that doesn't work out of the box is the weird WiFi adapter that has problems even on Windows. I just use a tiny USB adapter.
I also have OpenBSD installed on my even older laptop, which is an iBook G3. Power management and wifi don't work there but it is 15 year-old hardware, I don't really care.
> The way OpenBSD handles WiFi configuration is a good example of this. Want to connect to a wifi network? ifconfig <interface> <ESSID> <passphrase>. If you need some other fancy flags for those 0.1% of hotspots in the world, you can do those, too (also through ifconfig) but it's that simple.
Hmm, I'm surprised OpenBSD supports this, since it will cause the passphrase to get stored in your shell history in plaintext.
I generally run it from a script that then prompts for the passphrase. I'm working on a little tool of my own to work as kind of a minimalist OpenBSD-specific network manager that'll store a list of encrypted passphrases and let me connect to a network without having to enter the passphrase and SSID each time.
This has long been my experience with BSD (back before jumping to OSX for some years, and now back to OpenBSD). There is a willingness to update core pieces of functionality where needed to reduce inessential redundancy.
Another example: console / keyboard config. In OpenBSD, I set this up through wscons. If set here, my keyboard settings get propagated in a robust, non duplicative way to X11. Contrast this with Linux, where you need to set up your keyboard config in at least two places to have, say, Dvorak in X11 and the console.
It's a minor thing, and one that Linux distros "solve" during installation by setting up the relevant configs. But it's more elegant in OpenBSD. Maybe that's because OpenBSD writes documentation that answers questions like, "How do a change my keyboard layout?" In OpenBSD, it's a simple answer. If the situation were the same as Linux, the answer would be longer. Maybe seeing that inessential redundancy in print is what spurs the project to action.
This is indeed annoying; fortunately David Herrmann is working on a new user-space terminal emulator for Linux called KMSCON which uses the same keyboard input code as XOrg (libxkbcommon) so there is hope for improvement.
Ok, looking at the recent activity (or the lack thereof) in the github repo lets me be somewhat less enthused... but what's your objection to "KMSCON"?
Seriously, we need Prolog infused user interfaces and components. Logic assistant to compose subsystem. Most user facing things are crippled (good thing sometimes) views on it, fuzzy matching, completions etc etc
actually, wpa_supplicant isn't that hard to configure as long as you only use the GUI. just scan for APs, double-click on the network, and type in the PSK. no documentation-reading required.
I use wpa_supplicant directly on my desktop (running Slackware64 14.1) and have no problem doing so via the command-line and conf file editing alone, though in my case it required adding a snippet to /etc/rc.d/rc.local to get it to connect to my wireless network on boot.
> What are some things on OpenBSD that don't work for you?
The sort of things that you mentioned -- Skype, Dropbox. I also can't use OpenBSD for PCB layout (Eagle doesn't run on it; don't know about Kicad, but I have very little experience with it), but I don't really mind, because I have access to a licensed installation of Altium at work.
> What sort of hardware are you running?
Nothing special. It's a fairly high horsepower i7 box with 16 GB of RAM. The only component that didn't play nice with OpenBSD was the nVidia graphics card. I just bought a reasonable ATI card and it works fine.
> What about Adobe Flash on OpenBSD?
I haven't used the browser plugin ever since youtube-dl came out, not even on systems where it's well-(ish)-supported. But if you need it in the browser, I think you're pretty much out of luck.
I used FreeBSD for a while because it had ZFS, and found Ports confusing and fragile. Maybe I was doing it wrong, but the handbook wasn't much use: "here's 4 different ways to do it, we're not saying which is a good idea". Every upgrade took ages, required reading a load of documents, and broke something.
There didn't seem to be an equivalent of Debian's Stable, everything was apparently the latest release.
I have OpenBSD, FreeBSD, and Red Hat 6 (government wanted some software run and guess who gets the $) running in my server room.
Red Hat seems to be fine with yum, and I haven't had a problem yet. FreeBSD's pkg (pkg-ng technically) is ok, but I keep running into problems on upgrading because someone split a pkg or renamed it. I search but don't find a simple solution. I haven't had any problem on the OpenBSD side, but that could be because every 6 months do an upgrade with the new release. Patches are a bit harder on OpenBSD unless freebsd-update does something weird. the whole src thing in the freebsd-update bit me.
I also was frustrated with how poorly the handbook explained ports/packages. But now that I understand how they work, I feel like it’s an elegant system that works well. I just wish it didn’t take so much trial and error to figure it out.
That's what the mailing list and the forum is for where you'd most likely told to just use 'portmaster' which is the easiest. Not that the others aren't easy, too, but it keeps tabs on dependencies for you but its best to pick one method and stick to it.
I've been playing around with Void Linux recently (http://www.voidlinux.eu/). Unfortunately, even there, it's hard to escape a lot of the baroque nonsense of the Linux userland, and I eventually caved in.
To be fair, chromium crashing is due to the increased complexity of chromium and nothing to do with FreeBSD as the maintainers have noted in their comments. They have found it increasingly difficult over the past couple of years to keep up with it though more manpower would help I'm sure. But such issues are fixed within a couple of weeks or days. the issue you speak of has to do with javascript in GMail and settings and how chromium interacts with that; not chromium directly.
I don't have the issue with Firefox, and never had any issue using the latest hardware, and I don't think the lack of Skype is FreeBSD's fault either but I forget.
Steven, if you're reading, you got your history slightly wrong. System V got its init handling from Sun's SunOS 4.x. It was negotiated as part of the integration between SunOS and System V which would become the "one UNIX for everyone". Unfortunately for all of us, AT&T really really loved "run levels" and Sun really really liked BSD's way of configuring the system. Which got you the unholy love child you saw in Solaris 2.0 and System V release 2.0.
For the other Sun folks from the systems group who quit rather than submit to the demands, you were right, I was wrong. We couldn't convince them of the error of their ways.
You are correct it was System V, release 4 that merged with SunOS. It was also the release that Sun and AT&T "combined" the BSD init/rc system with the System V init/run-level system.
It got the job done by making the 99% majority of use cases more difficult in order to make the 1% minority simpler. This is a design pattern I think is being repeated in systemd.
I kinda had the opposite impression. I first encountered systemd when I started messing around with arch for a couple little vps projects. At the time I was doing sysadmin that involved both Ubuntu and CentOS, and I definitely found day-to-day tasks easier with systemd (better built in monitoring, consistent syntax, etc). For simple cases, writing a unit file sucks a lot less than writing an initscript. The warts in systemd only started to irritate me when I was first setting up my personal machines (one laptop, one tower) to run arch [1]. Even then, the bloat in the design irked me more than actually using it. So, I'm in favor of systemd, but not thrilled (dbus everywhere is annoying).
Then again, I don't maintain any packages with strange and/or complex daemons. If anybody's had a really hard time getting their stuff to work with systemd, I'd be interested to hear stories.
1: Not completely true. I want my damn log files back.
This is why I suspect that most of the divisions in the sysadmin community regarding systemd are generational. For people (like me!) who have been writing init scripts forever and can do it in their sleep, they'd prefer the flexibility and familiarity of init scripts to a new system that at first glance appears more limited.
For people who are less experien^H^H^H^H^H^H^H^Hcrusty :-), init scripts look like a horrifying way to manage a startup process, and just about anything -- including Yet Another Config File Format -- looks preferable.
I think this is also why so little progress is being made on resolving conflicts over systemd. There really isn't a whole lot of room for compromise; one group simply wants things to stay more or less the same (or evolve incrementally), the other group simply wants to throw it all out and start over.
I bet there would've been a lot less argument and vitriol about systemd if Debian hadn't made the decision to default to it, basically leaving all the crusty sysadmins homeless. Once Debian switched, there was a near guarantee that all of Linux was going to move to systemd. That suits one group just fine, another one not so much.
On the plus side, the BSDs should be seeing a really nice upswell of installations over the next couple of years, which is great, because they've been quietly building some pretty great operating systems that deserve more love.
It's perfectly possible to run scripts with systemd unit files, so there's no real "flexibility" lost anywhere. The primary improvement is that systemd handles by default most of what used to be boiler-plate in every script, on top of calculating dependencies automatically. Then you get integrated process supervisor capabilities so that the init system actually knows whether services are running or not.
Speed was certainly a major motivator as well, considering the increasing popularity of containers and on-demand virtual machines.
To me it seems that "gradual evolution" was exactly what they did, and it wasn't really going anywhere. Lennart came up with systemd and convinced enough people that it's worth the pain of migration to get a clean start.
I don't know how to write sysvinit scripts any more than I know how to write fizzbuzz. What I do know is how to program and specifically how to write and maintain and debug shell scripts. I NEED that for other automation purposes and cannot get rid of that mandatory requirement so it used to dovetail quite well with init scripts. There are corner cases and security issues and tradition, which means copying a known good init script and editing nothing but the executable names is a great start, but its not really a serious problem. I don't care if its verbose, as long as its simple and understandable and most importantly, predictable.
Now instead of applying a language and tradition I know to a simple task like init, I get to learn a whole new ecosystem and technology to do about the same thing I always did except vastly more complicated, and I can't even shed the old knowledge because its needed elsewhere. That makes overall systems harder to use, slower, buggier, and less reliable. And thats defined as "progress" because its new so it must be better.
Honestly, I don't think that argument holds much water.
Init systems are not "just shell". If you go and try to write a sysv init script with only shell skills, chances are your init script will be broken in some obscure way. Perhaps because you didn't specify LSB headers correctly, or your PID file handling is subtly broken, or you're missing a check somewhere and the script is racey. I can't really say how much reading it would take to catch all the corner cases with sysv init scripts, but for simple cases, you will be able to write a correct unit file after about 15 minutes of reading the manuals.
Knowing shell helps, but you'll still need the tool-specific knowledge.
I don't think it's really fair to compare writing unit files to being able to program in shell.
True, knowing the systemd unit options is not a reusable skill, but it's not a huge effort either: the syntax is trivial (INI-inspired, the same as .desktop files) and the options themselves are rather well documented.
And for sure, these options are immensely more reusable when writing a new unit file than mixing and matching shell snippets taken from different init script using different styles and conventions.
It was around before the Debian Technical Committee made its decision. It wasn't one of the options to choose from, but it was an existence proof, for those making the decisions, that systemd unit files will not lock them in and that there's a route for Debian packages that have systemd units on Debian kFreeBSD and Debian Hurd.
Too bad it is not the unit files one should worry about, but the continued scope creep of the systemd project.
by the time one want to move from systemd to nosh, or anything more limited in scope, one may have face the rebuilding of Linux user space from the ground up.
It's not the unit files as a potential lock-in that one need worry about now, because of the nosh project. But it most definitely was one thing that people thought about then.
Possible sure, but it the systemd compatiblity with sysv scripts are listed as 99%. Meaning that complex scripts are liable to break, and it is exactly those that you want the compatibility for.
Indeed. But at that point you have a DSL which is no longer obvious when one only knows about shell scripting (eg. I have no idea what `pexp="smtpd: \[priv\]"` does), so using a completely declarative language with a trivial syntax doesn't seem a huge issue either.
One may think about systemd's unit files as a set of shell variables that gets sourced by the real script.
That's not a DSL, that's a shell script. As for what `pexp="..."` does - it's just setting the pexp variable. If you'd like to know what that does, you can just look up the bash source of `/etc/rc.d/rc.subr`, another bash script.
No DSL knowledge required, at all. Just shell knowledge, and the willingness to look at sourced shell scripts.
I mean, I know that it sets a variable, I can't even guess what are the effect of setting that variable.
And "look at the source" is a sad answer, no matter if the source is a shell script or C program.
The point is that with your example you need domain knowledge, knowing bash is not enough: what does the pexp assignment do, what rc_reload=NO means, should flags be specified in the daemon variable or that's meant to only contain the binary, does that path be full or it takes in consideration a standard PATH?
And now you got two things to know: the bash (or even strictly posix sh) language (which, admittedly, isn't the best one in the world) and the domain specific functions/variables.
This isn't much different than with systemd: the language is much more simple and far less surprising being INI-like (the same as .desktop files), and it's basically just setting a list of rather well documented variables.
pexp
A regular expression to be passed to pgrep(1) in order to find the desired process or to be passed to pkill(1) to stop it. By default this variable contains the daemon and daemon_flags variables. To override the default value, an rc.d script has to redefine this variable after sourcing rc.subr.
Thanks, that wasn't the point. What I was trying to say is that if you use those helpers you're looking up stuff in a man page anyway, so knowing shell scripting isn't enough any longer.
My personal opinion is that having a simple, INI-derived language is better than keeping the weirdness of the shell language just to assign a couple of variables (and you can still launch shell scripts from units anyway).
That you can import shell scripts into shell scripts is one of those things that catch many off guard when first encountered. But when grok-ed, really demonstrate how far shell script can go.
Yep, my point is that knowing shell scripting alone is no longer enough and it's not different than running `man systemd.service`, only that you're keeping the weirdness of the shell language just to set a couple of variables.
Except that rc.subr is shell scripting alone. It's literally a shell script that gets sourced into another shell script (in this case, your initscript), which is - again - ordinary shell behavior. Yeah, you have to learn a couple of shell functions and variables, but that's not really that big of a deal for someone with even cursory knowledge of shell scripting.
The bigger point, though, is that if your only goal is to have terse daemon configuration with a minimum of boilerplate, you can already do this with shell scripts. Even if you decide that shell scripts are the spawn of the devil and ought to be burned on the stake (which, honestly, isn't all that unreasonable of a decision), it's quite possible to implement the same functionality using Ruby or Python or Perl or C or Go or Rust or Erlang or Lisp or PHP or COBOL or MIPS assembly or whatever-the-hell-else you feel like writing init "scripts" in (with some exceptions; FreeBSD and NetBSD, for example, apparently launch initscripts with 'sh' instead of following the OpenBSD and GNU/Linux approach of calling initscripts as executables).
The biggest point, though, is that because of this, if your only goal is to have terse daemon configuration with a minimum of boilerplate, you can have that without having to learn at minimum two languages (one to write your initscripts and one to do all the rest of your day-to-day administration tasks). You just learn shell and you're done. With systemd, you still have to learn shell scripting for even systemd-related tasks (let alone non-systemd-related tasks) plus a new DSL that looks like INI but isn't quite INI. I guess that's tolerable coming from, say, the Windows world (where you're learning DOS batch language and PowerShell and VBScript and INI and Lord-knows-what-else), but that's double what a seasoned Unix admin requires in a non-systemd environment, which is pretty significant.
Again, "read the source" is a sad answer. I can read systemd C sources too, but given that it comes with proper documentation there should be no need to do it.
My point is that either one has to get some domain-specific knowledge or they have to reverse engineer some code. Given that I believe that the second one is a bad choice (people will rely on the implementation rather than on the intended interface) if you have to get some domain-specific knowledge it does not matter if you're using posix sh syntax or a INI-style one.
> your only goal is to have terse daemon configuration with a minimum of boilerplate, you can already do this with shell scripts
Literally nobody ever claimed that systemd "only goal" is to have terse daemon configuration. It was one of the many goals. Sure, it could have been done with posix sh syntax, nobody disputes that, but since backward compatibility was being dealt in other ways (using LSB header), switching to a INI-style format seems a good choice if one wants to enforce a declarative-only style.
And if someone claims that init script are needlessy complex and redundant it is because that's the situation of most distributions before the switch to systemd. Switching to OpenRC would probably have fixed that aspect, but systemd solves many more problems that distributors faced and OpenRC did not address, faring more positively in a costs/benefits analysis.
> Again, "read the source" is a sad answer. I can read systemd C sources too, but given that it comes with proper documentation there should be no need to do it.
rc.subr also comes with proper documentation, in the form of the rc.subr manpage, which comprehensively documents its behavior.
Barring that, you're delusional if you think that C is anywhere near as readable as shell in this sort of context.
> My point is that either one has to get some domain-specific knowledge
Arguably less of it in the case of rc.subr, since said knowledge is already part of basic shell scripting knowledge.
> Literally nobody ever claimed that systemd "only goal" is to have terse daemon configuration.
Nor did I. That doesn't change the fact that "OMG initscripts are so verbose with so much boilerplate; my unit files are so much better" is a commonly-cited reason for people to prefer systemd over other init systems, never mind that said reasoning is plainly false.
> the fact that "OMG initscripts are so verbose with so much boilerplate; my unit files are so much better" is a commonly-cited reason for people to prefer systemd over other init systems, never mind that said reasoning is plainly false
Indeed, I agree with that. The only advantage is that the restricted syntax of unit files is more suitable for error checking rather than having the shell complain more or less randomly, but compactness is not an intrinsic advantage of it.
That said, not every distribution standardized on such kind of helpers (ie. Debian), so in those cases compactness was one of the benefits of switching to systemd (not exclusive to it).
You can still use plain shell-based LSB-compliant init scripts just fine, which is what Debian is still doing as they didn't want to have flag days like all the other distributions who switched.
The nice thing about initscripts, though, is that you can get the best of both worlds by sourcing in common functions, as is the default and recommended practice on OpenBSD. For example, here's the initscript for dovecot on my (OpenBSD-running) mail server (with some comments and newlines omitted for brevity):
Pretty straightforward: you set $daemon to the path of your daemon's executable (in this case, dovecot), source in rc.subr, and call rc.subr's rc_cmd command with $1 (which represents a command, like 'start', 'stop', 'restart', etc.).
rc.subr, meanwhile, holds all the complexity one will ever likely need in an initscript, in the form of various shell functions encapsulating things like starting, stopping, etc. And if that's not enough for whatever reason, it's easy to add more functionality either to the individual initscript or to rc.subr, since it's shell scripts all the way down. In my experience, reasoning about this sucks a lot less than writing a systemd unit file, let alone an initscript on a system without such niceties.
Slackware's a bit of a peculiar case. It does indeed use /etc/rc.d in a somewhat-BSD-style manner, but there are two key differences:
* Slackware doesn't use a BSD-style /etc/rc.conf. Instead, daemons are enabled/disabled by setting the executable flag on each initscript. While this loses some of the features of /etc/rc.conf, I personally like this method better.
* Slackware doesn't include an equivalent to /etc/rc.d/rc.subr, which means there's a lot of boilerplate in its initscripts.
But yes, systemd's primary objections to shell-based init systems in general seem to stem from a limited scope.
The author mentions that he believes the biggest problem of systemd is repeating some mistakes of SysV init, but I think it's actually pretty important that familiarity is preserved to help people upgrade. For example, the author mentions babysitting the users (systemctl edit) and config/daemon reloading as things that could be made automatic, I'm assuming with e.g. inotify change events.
But you often want some confidence that the change you're making to your system is what you expect, and we've had them all along:
- visudo(8) sanity checks, for atomicity and syntax, to help prevent you from bricking your authorization
- an explicit SIGHUP to reload the configuration file for your daemon, after you've checked the config, rather than accidentally saving your file while you're editing it and bringing down your daemon.
You could expect everyone to adopt new practices for editing the configurations, but the behavior would be very surprising to anyone used to the old way.
So I have not yet had to use systemd in any serious way, so I may be misinterpreting here, but I think this:
> ... and config/daemon reloading as things that could be made automatic, I'm assuming with e.g. inotify change events.
is a misinterpretation of the author's complaint. It sounds like with systemd, if you tell it to restart a service it will continue to use the old definition until you run a separate command to 'reload' it (and, presumably, all other configurations). On top of that, it indicates its knowledge that you've changed the file by telling you to run the aforementioned reload command.
No inotify seems to be involved in what he wants, which is that the reload command use the new configuration. Hell, be clever and use the old configuration's stop mechanism and the new one's start mechanism or complain if the restart mechanism differs or something, but a message telling you you didn't do something you almost always want to do is a pretty obvious smell for a UI problem.
I deeply dislike the similar behaviour of upstart, fwiw. In order to reload an upstart init script you have to stop and start it, restart uses the old one. I have spent annoying amounts of time wondering why it was doing it wrong.
Because systemd is event-driven, you can't actually be sure the service file won't be re-used in the running configuration - or that it won't trigger other service reloads/starts/stops due to its dependency specifications. Ergo systemd requires an explicit reload to commit the state of the unit files once you're satisfied they are as intended.
This is very important when you consider that unit files can kill other units on their conditions - I've used this feature in particular to make sure nginx goes down if a mountpoint becomes unavailable or the servicing process dies (and systemd can't restart it). This is extremely useful because it means a load balancer won't send requests to a webserver in an improper state. But it also means that while editing those config files, I wouldn't want systemd auto-reloading everytime I saved.
And in the end i will gladly trade a few seconds longer boot for a system i can actually reason about rather than treat as some kind of black box voodoo. The latter is why i moved away from Windows in the first place.
It's definitely more then a few seconds (systemd boots fast), but more importantly reasoning about unit files is a lot easier then reasoning about the init-script ordering.
With a unit file I can just declare the conditions under which it should run or stop. If the system doesn't achieve those conditions, then I can solve that problem - rather then having my unit run before the system is ready to do so.
You're not really reasoning about your system because you can say "this runs after this" if you don't also express all the dependencies and failure conditions.
Aren't you confused with Upstart, which is event-driven? Systemd is mostly goal-driven (e.g. on startup it will start default.target, which involves recursively starting its dependencies).
What I think your getting at is that essentially you need a mechanism to commit your changes before the thing that's reading those changes receives them. In essence, an atomic operation for the change. Completely agree.
Version control systems and transactional databases are built over atomic operations. Transactions in particular refer to having multiple operations that either succeed or fail all at once, making in essence atomic operations composable (they normally aren't).
Editing a file and saving it multiple times is not an atomic operation, but reading a file can be, so having the user issue a reload command only after he's finished editing a file is an entirely reasonable approach that has been used in Linux since forever and it only shares some basic building blocks with version control or transactions, but there's nothing fancy about the approach. Fancy and very error-prone would be to apply configuration changes automatically, which is what the article seems to suggest.
The Windows Registry is a dump on disk of some half-assed structures defined in a memory heap, poorly specified, being in essence a file-system as well. Also, only individual operations have been atomic and one update in the Windows Registry does not equal one update of an entire configuration file on Linux. Transactional support has been added, but late and interactions with it are still problematic. The Internet is filled with complaints and pieces of advice on corrupted registries.
And it is being obsoleted, with the elephant in the room being that the Windows Registry needs to be thread-safe because it's a central store of all settings and this was a really bad idea. I'll also have my nginx.conf over a registry entries file any day of the week.
That's true, that's why I said that it "can be". And if you have a special edit command, like "systemctl edit", then it can take care of the details for you, which is I presume the reason for its existence (much like visudo).
I assume the next version of systemd will include git.
Actually, it's not that bad an idea:
cd /home/efaref/config/servername
vim blahd.service
git commit -m "Update blahd service file to increase widget count"
git push live master
Then have something else watching the master branch of the live repo do a git pull, systemctl daemon-reload. It doesn't even have to live on the same server. If one of the config files is a puppet script, you can maintain everything.
I agree with most of this, especially the fact that complexity has simply been shifted to other places. It's much easier for me to read a systemd unit file than it was for me to read a complex set of init scripts. The problem is that when I have 10 or 20 units with interlocking dependencies, it's an absolute nightmare to try and untangle because I have to jump between so many different files. Having to specify After=X and Requires=X seems like a prime example of optimizing for the 1% usecase. 99% of the time, when A must start after B, it's because A requires B. Having to specify both is something that is easily and often missed, and a hard problem to track down if you didn't read the documentation carefully. There's also the problem that it seems to be metastasizing and taking other parts of the OS. The longstanding /var/log/messages is now replaced by journalctl tool, and the logs are actually in binary. The venerable "hostname" command is now "hostnamectl" because it must go through systemd. They have also replaced chroot with systemd-nspawn, and made it so you cannot even use systemd while in a normal chroot!
Have you tried the systemd tooling that graphs depencies for you? I haven't worked on that area yet, but my colleagues made heavy use of that and actually could resolve problems with these quite fast thanks to the diagrams that systemd can create for you.
I feel like systemd's tools and Emacs are on a race to achieve the title of "only program(s) necessary in one's path to have a fully-featured operating system". :)
> The problem is that when I have 10 or 20 units with interlocking dependencies, it's an absolute nightmare to try and untangle because I have to jump between so many different files
How that's different from sysvinit?
> The longstanding /var/log/messages is now replaced by journalctl tool, and the logs are actually in binary.
`journalctl` prints plain text on its stdout, so you can think of it as a glorified `cat` if you don't want to use its extra features. On the contrary, if one needs its extra features on plain text log files one usually resolves to using fragile parsing hacks.
> The venerable "hostname" command is now "hostnamectl" because it must go through systemd
They do different things, `hostname` only reads the current hostname. `hostnamectl` is able to set the hostname, standardizing what used to be a per-distribution custom mechanism. Despite being shipped with systemd (to make sure everyone can rely on it) it doesn't actually depend on systemd and in no way goes through it.
> They have also replaced chroot with systemd-nspawn, and made it so you cannot even use systemd while in a normal chroot.
`chroot`'s purpose has never been the ability to run an init system inside it, no matter if it is sysvinit or systemd. On the other hand `systemd-nspawn` creates isolated containers, I'd be surprised you can't run sysvinit inside it.
Again, you're comparing apples to whole orange trees as the systemd utilities do much more than the traditional ones you mentioned.
In sysv I can just make a list of startup tasks. A -> B -> C -> D. Easy. 10 things that must start in specific order? No problem, 10 things in list. There is no easy way to do this in systemd. I must create 10 different unit files, and then create a bunch of different dependencies (A depends on B and C, B depends only on D, can C then start at same time as D?, etc). When those dependencies change, I have to go look through a whole bunch of files or look at a graph tool to figure out what is even going on.
> They do different things, `hostname` only reads the current hostname.
That's... not true. From the first line of the hostname manpage: "Hostname is used to display the system's DNS name, and to display or set its hostname or NIS domain name."
> chroot`'s purpose has never been the ability to run an init system inside it
That's fine, but it worked under sysv and now it doesn't. When upgrading or working with different install roots, using chroot is common. Now I can't modify services during that time, unless I migrate everything over to use nspawn.
> the systemd utilities do much more than the traditional ones you mentioned
If I lived in a blue sky world, where I was fresh installing one new server with a simple set of services that I had built in systemd, I think I be quite happy with it. The problem is that I have thousands of servers, and a decade or more of accumulated infrastructure that relies on those traditional tools. Then systemd comes barging in and breaks a bunch of stuff, and that is frustrating. I'm not saying I hate systemd, in fact I agree that many of the new tools are quite nice. But it's kind of like the loud hipster that shows up at your party and starts talking shit about the music you are playing, and making fun of your choice in beer. Maybe my music does suck, and maybe these clothes aren't cool anymore, but we were sitting around having a fine time until he came around.
And in the end you hit a sore point square on the head.
Systemd is not built for traditional server farms, it was initially built to scratch a desktop bootup itch that Poettering (and perhaps Sievers) had that then got extended to deal with booting containers.
To them, and others working on systemd, the traditional server way is history and they are building for the containerized present/future.
Oh, sure, sequentially starting up a fixed list of services is probably easier with sysvinit, given that's the only thing it does.
With systemd you probably want to set Requires=foo and After=foo in bar.service to have foo running before bar. It does not seem overly complex, or you meant something else?
> I must create 10 different unit files, and then create a bunch of different dependencies (A depends on B and C, B depends only on D, can C then start at same time as D?, etc).
Wouldn't you create 10 different init script for sysvinit, each with its own LSB header listing the dependencies such that tools like Debian's insserv could correctly schedule them?
I'm quite sure that it's far easier to express complex dependencies with systemd than with sysvinit, if that's your concern.
Correct me if I'm wrong, your dependencies could be handled like the following:
[A]
Requires=B
Requires=C
[B]
Requires=D
If A needs B to be running before A even starts (ie. they cannot be run in parallel) just add After=B to A. With sysvinit I'm not sure if there's a way to say that A needs B but that they can be started in parallel.
I don't get your question "can C then start at same time as D": if they do not depend on each other, I don't see the problem. If they have a dependency, just add After= to the one that needs to be launched later in the process. If you don't know, add After=C to D and you've got a slightly slower but sequential boot as with sysvinit.
> That's... not true. From the first line of the hostname manpage: "Hostname is used to display the system's DNS name, and to display or set its hostname or NIS domain name."
Oh, indeed, sorry. I then guess `hostnamectl` only provides notifications and some extra info on top what `hostname` already does.
> If I lived in a blue sky world, where I was fresh installing one new server with a simple set of services that I had built in systemd, I think I be quite happy with it. The problem is that I have thousands of servers, and a decade or more of accumulated infrastructure that relies on those traditional tools. Then systemd comes barging in and breaks a bunch of stuff, and that is frustrating. I'm not saying I hate systemd, in fact I agree that many of the new tools are quite nice. But it's kind of like the loud hipster that shows up at your party and starts talking shit about the music you are playing, and making fun of your choice in beer. Maybe my music does suck, and maybe these clothes aren't cool anymore, but we were sitting around having a fine time until he came around.
I totally understand your concerns. I guess you don't have many options: you can stick with sysvinit for a while (eg. Debian still officially has full support for it) and in the meantime start making sure your scripts run fine with the LSB compatibility support in systemd (if you're just starting services they should work transparently with no effort).
But consider that in any case, staying with sysvinit has never been an option: during the Debian debate, the contenders for the default were systemd and Upstart, sysvinit has always been kept only as the baseline to test compatibility.
With that in mind, the LSB support in Upstart (and in every other contender) is far less mature than the one in systemd. As the incredibly smooth migration done by the Debian systemd team proves, systemd has the best compatibility support of the pack.
(I mention Debian as that's what I use and for the fact that the systemd maintainers poured a big deal of effort to provide the best systemd/sysvinit compatibility)
I don't "hate" systemd either, I just think it's a bad approach. It flies in the face of the "open architecture" concept and stifles innovation. It discourages users from learning about how their operating system works, and it unnecessarily puts too much control of a fundamental component of the OS in the hands of a too-small number of people.
None of these reflect the Linux I grew up with. Thus, I am not a fan of systemd. #notmylinux
IMHO systemd is taking much of the hacks and weird glue that has historically held distributions together and replaced it with an active upstream project. Maybe some of the solutions they've come up with could need some improvements, but I have much higher confidence those improvements would happen now compared to how things used to be.
Nowadays you can code something up and come to the systemd mailing list and get feedback and get it integrated. Previously you'd have to convince one or more not-really-connected distro hackers that your new fancy thing is worth rocking the boat for.
Unfortunately, the systemd development team is not that easy to work with, which is part of why a lot of people distrust systemd the way they do.
In theory, yes, it should be the way you mention it. In practice, even bug reports end up unacknowledged and labeled as features after a few philosophical debates.
No one who has used more than one Linux distro would disagree that a unified system to control init services is a good idea. It's this particular implementation that's problematic, not the principle.
One lovely example i have seen is Poettering completely misunderstanding a perfectly valid use of su, and instead recommending that people ssh back to their own box if they wanted a shell under a different user.
This came up in relation to a systemd set environment variable that would break software by overwriting one user's config files with that of another's.
The variable was apparently set that way because of pulseaudio.
The really worrying part was any lack of "mea culpa" from Poettering once someone outside the bug report told him that pulseaudio could be changed so that the variable was no longer needed and Poettering implemented that change.
Instead he just left the bug report hanging, and others had to report on the changes and declare the issue sorted.
But I think you could find a vocal minority (or several) to oppose ANY particular implementation. One of the reason the anti-systemd crowd has such a hard time gaining traction is because the only thing that unites all of them is opposition to systemd. They don't agree on what should be in its place instead -- some want to hold onto sysvinit, some want Upstart (although that's a dead end now), some want OpenRC, some want dmd, some want runit, some want something else or something that doesn't even exist yet.
Here is the thing. The problem is not systemd as init, it is systemd as, well, systemd.
The constant addition of daemons and functionality to the overall systemd codebase by either subsuming or replacing existing, INDEPENDENT, projects.
As such, who cares what init is used. The thing is that until systemd the other daemons/services started by init didn't particularly care what init it was.
Previously you could piecemeal your changes. Didn't like syslog? Grab something else that talk the lingo and slot it into place. Sorry, but journald demands that the init is systemd.
Want something to handle seats/sessions? Sorry, but logind only works with systemd as init.
Etc etc etc!
Going on and on and on about inits is a smokescreen.
It's very easy to agree that nuclear pulse propulsion is a bad idea for a vehicle, even among people who otherwise have very different preferences on what kind of engine they like on their vehicle.
Unfortunately, yeah, the whole debate there is unproductive. Instead of focusing on real problems, it gets stuck into an endless debate on Unix philosophy, carried among a bunch of people who aren't really sure why it's a problem if something doesn't follow it and another bunch of people who insist that they're following because saying otherwise would be bad PR.
But sadly that didn't stop various Linux developers from bullying the systemd developers in that bug report instead of trying to find some common ground.
The actual bugs (assertion failure creating debug output in systemd, and inadequate buffering in Linux's /dev/kmsg) could have been fixed much faster without the pointless flame fest.
Your link supports the assertion that systemd doesn't play well with the community -- Linus goes on to say:
It does become a problem when you have a system service developer [Kay] who
thinks the universe revolves around him, and nobody else matters, and
people sending him bug-reports are annoyances that should be ignored
rather than acknowledged and fixed. At that point, it's a problem.
Which was ultimately caused by a bug in the kernel, and the systemd developer ultimately accepted the proposed changes just to avoid any kind of confusion (to me, it looked perfectly acceptable to parse the kernel command-line, multiple applications still do for other purposes).
Arguably, the tone of the discussion resembles more the usual LKML tone rather than the one observable on the systemd mailing list.
However I would not characterize that as "Poettering completely misunderstanding a perfectly valid use of su".
It seems a quite complex issue about a workflow that is only half-working even without systemd. A systemd patch has been committed that fixes the most blatant error, but of course it doesn't solve all problems since most of them existed before systemd.
I don't know in what corner of the universe was that "only half-working". It was working absolutely fine. It has been working fine since... I don't know, v7, when it was first introduced.
The few times I tried to run GUI programs through su I always got plenty of warning about DBus and stuff. But I don't do that very often, so that's all I know.
It worked before, it stopped after a change in the kernel, another change made the problem disappear: the root of the problem was definitely a bug in the kernel.
That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use, and indeed it seems that this is how things ended up. Note that there are multiple userspace applications parsing the flags on the kernel command line: the most striking example when discussing the "debug" flag is the "quiet" flag.
> That said, it's perfectly fine for the kernel developers to clarify that a flag is for their own exclusive use
Oh for fuck's sake. It's a parameter passed to the kernel. Programs shouldn't parse it for precisely the same reason why Firefox doesn't parse sshd's arguments. I.e. because they're passed to sshd, not to Firefox.
It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself.
The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter.
The way the systemd development team chose to do that (widely seen in many other projects from the same lineage, like Pulseaudio) was ha look at that we can see the kernel's arguments so how about we use those and what do you mean "this can have unwanted side-effects", it works perfectly fine on my system.
The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug. It's also expected that a division by zero will crash your program. It doesn't mean that blindly accepting any input values is not a bug, nor that the correct way to solve "your program crashes if I pass 0 as the first argument" is "well stop passing 0 as a first argument ya evil bastard."
Just like "quiet", which is totally meant to affect userspace programs.
> It's a ridiculous clunky hack that stinks of large company coding practice: what they actually needed was a way to pass arguments to systemd itself.
Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose.
> The logical way to do that in an OS that has to be portable and flexible is to implement a mechanism for the kernel to pass some of the arguments it receives to init. Because there's no reason to assume that init is going to be systemd, that there's going to be a procfs ready for it so as to be able to read the kernel's parameters, or that the init process can read it for that matter.
The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now.
> what do you mean "this can have unwanted side-effects", it works perfectly fine on my system.
It worked perfectly on every system until the kernel bug triggered the feedback loop.
> The fact that the developers (again, it's a known pattern) decided to play the PR game and call it "expected behaviour" doesn't make it less of a bug.
It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.
> What userspace programs interpret the quiet parameter?
> Genuine question. AFAIK, quiet does (or should do) nothing other than disable some of the kernel logging.
At least plymouth and probably every init system out there.
Linus said that "we very much expose /proc/cmdline for a reason. System services
are supposed to parse it, because it gives a unified way for people
to pass in various flags. [...] And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature."
> The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they!
You're right that Kay was wrong to close the bug immediately instead of trying to help finding the real cause, and you're wrong when you say that userspace should not parse the kernel command line, see Linus' remark above.
> There's no reason to even assume procfs will always be there
To the contrary, dropping it would be an ABI break so it's really safe to assume that it will be always be there.
> who's been programming for more than an year
Please, don't lecture others about their job. I'm not assuming you're an incompetent fool, please don't assume I am.
> if a program did that on Windows
And can we try to stay relatively on topic? Windows is definitely unrelated.
(I'm not otherwise replying in detail because you assumed that parsing the kernel command line was not ok, while Linus officially blessed its usage. Also we reached the max thread depth on HN, which should probably suggest something.)
> Just like "quiet", which is totally meant to affect userspace programs.
What userspace programs interpret the quiet parameter?
Genuine question. AFAIK, quiet does (or should do) nothing other than disable some of the kernel logging.
> Having a global "debug" knob would have been useful when one does not know if the problme lies in the ramdisk, plymouth, systemd, udev, lvm, whatever. Having a systemd-specific flag defeats that purpose.
Yes, and it would have been very useful if the systemd team had offered to implement such a feature, instead of silently pretending to have it.
That thing was news to everyone. Except Kay Sievers, apparently, who chose to share the big news with the world when closing the bug report.
> The kernel command line, readable from procfs, has been used for this purpose (see the "quiet" flag) for a very long time now.
By what non-broken programs? There's no reason to even assume procfs will always be there, let alone that the kernel will expose its boot parameters through it. AFAIK, quiet doesn't affect anything but the kernel's logging.
With obvious exceptions (e.g kernel debugging tools and the like), that shouldn't happen.
> It worked perfectly on every system until the kernel bug triggered the feedback loop.
No no. It worked perfectly because the systemd team never bothered to see what happens when the kernel actually tries to use the debug flag, too (i.e. when an assertion actually fires). It should be obvious to anyone who's been programming for more than an year or so that, when you start parsing arguments that aren't yours (which is bad enough in the first place!) you should at least ensure that you don't break the program that uses it.
Yes, the rate limit for /dev/kmsg was increased when it turned out there was a potential to incapacitate the system by flooding the kernel with data from userspace. For what it's worth, if a program did that on Windows, every half-assed heuristic detection engine would have promptly flagged it as malware.
> It was "expected behaviour" in the presence of a kernel bug, which is why it was obviously not generally "expected behaviour" as you seem to imply.
How the hell is "I'm passing an argument to the kernel and a program from userspace decides to interpret it" expected behaviour? If I pass rw, too, should I now expect mount to mount every system as read-write?
That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag, which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature, so go whine to the kernel folks about the thing of theirs that we broke and don't feel like fixing", as if the systemd folks have become one with the Microsoft).
Whether or not there are other underlying issues stemming from such misbehavior is irrelevant to the fact that the misbehavior exists. It's like sticking your hand in a fire, then blaming your hand instead of your own stupidity for your third-degree burns.
> That says nothing about why the systemd folks are entitled to hijacking a kernel flag instead of (more sanely) namespacing their own debug flag
"No, we very much expose /proc/cmdline for a reason. System services are supposed to parse it, because it gives a unified way for people to pass in various flags. [...]
And yes, that does include "quiet" and "debug". Parsing them and doing something sane with them is not a bug, it's a feature."
> which is the actual bug that was filed (and in turn met with "it's not a bug it's a feature,
And that is the appropriate response to the bug that was filed, with the title "Do not parse "debug" command line parameter".
The bug report makes it immediately clear that this is not really about a technical problem that ought to be solved - no, the dirty user-space programmers should get their stinking paws off of our good kernel command line flags, how dare they!
Kay pointed out in his first comment that there is already a different parameter that turns on debug output only for the kernel; the reporter said he's too lazy to start using it.
And this is not about "entitlement": there is a clear administrative use-case for systemd using the debug parameter: if your system doesn't boot you want a simple and obvious way to turn on debug logging in all of the relevant plumbing code, whether it's in the kernel, in the initrd or during service startup. You shouldn't have to remember five different parameters because then you'll forget one and your outage lasts 10 minutes longer.
If the bug report had been something more like "I've enabled debug and my system failed to boot" without prescribing a particular fix and leaving that at the maintainer's discretion, it wouldn't have been resolved as NOTABUG but fixed quickly.
Of course, a wiser and more socially skilled maintainer than Kay Sievers would have immediately filed a follow-up bug report, or re-titled the bug report instead of lowering himself to the level of the reporter and playing resolve-reopen ping pong. (Fortunately the systemd project also also has some more patient maintainers like Tom and Zbigniew.)
Read the rest of the email you linked, where Linus clarifies that systemd's use of the 'debug' parameter seemed to assume that systemd was the only user of it, thus going against the whole point of a readable /proc/cmdline. Thus, feedback loops were caused and pain was created and flamewars were started.
Again, if you shove your hand into a fire, don't blame the hand (or, for that matter, the fire) for your own arrogance and stupidity. Common bloody sense.
If you discount the usual Linus style hyperbole from the rest of that mail, the only thing that remains is that Linus is unhappy with the fact that Kay closed that bug report instead of trying to find the root cause of the reporter's boot failure.
Honestly I find that a single global "debug" option would have been useful, instead of having to know which component (the kernel, the ramdisk, sysvinit, sysv-rc, systemd, upstart, udev, whatever) is at fault.
Unfortunately such mechanism introduced a feedback loop in the presence of a bug in the kernel. I understand that it would be difficult for the kernel to guard from such loops, but my sincere impression is that they threw out the the baby with the bathwater.
I agree that such a flag would be useful, too, in which case the systemd folks could very well have consulted with the kernel folks in advance to work out a deprecation path for the kernel-only behavior and an introduction path for a generic behavior that activates the respective kernel-specific and systemd-specific flags.
The problem is that such collaboration didn't happen, and the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned", thus causing breakage and flamewars. Had they taken the 10 seconds to write up an email on LKML or somesuch saying "hey, we think the 'debug' flag should apply to everything, not just the kernel; y'all game?", the vast majority of the fallout would have never existed.
> the systemd folks assumed that they were the center of the universe and decided "well fuck the kernel, we're going to use the 'debug' flag for ourselves, everything else be damned"
Note that even Linus said that "parsing [the 'debug' and 'quiet' flag] and doing something sane with them is not a bug, it's a feature". So the issue revolved on what people believe is "sane", not about userspace using kernel bootparams (systemd still does and Linus is ok with it). The systemd folks argued that the kernel should not affected if something broke and started throwing too many messages at it during early boot (indeed, the kernel was fixed to rate limit in those cases). The kernel people instead said that it's not ok for userspace to flood the kernel, not even in broken setups (indeed, systemd was fixed to stop sending messages to dmesg as soon as journald is up).
I believe that sending emails here and there to ask "it's ok to do that?" would not work. LKML is already high traffic as is, such messages would just get ignored until things break.
Once upon a time, we were obsessed with efficiency and speed of code. We created powerful, but simple systems like sysvinit, and we designed to optimize speed in 80% of use cases. The Linux you grew up with used to mostly run on x86 machines, except the majority of Linux devices now run on ARM with different power requirements, hardware, drivers, etc. than x86 ever had to deal with. The increasing complexity in the use cases of Linux is driving systemd, not some desire to stifle innovation (which IMO is exactly the opposite of what systemd is doing).
Our CPUs got fast enough and our use cases so diverse and network dependent (leading to lots of idle CPU time) that we stopped worrying about raw performance above all else. In nearly every case where you're not doing something glaringly stupid, performance is a function of the complexity of your problem set. Things like power consumption and safe asynchronous communication are more important than raw speed, and these things need to be coordinated through the same interfaces on every Linux machine (though not necessarily through the same methods).
Systemd doesn't so much fly in the face of "open architecture" as it does introduce architectural concepts that are sufficiently abstract that much of the community doesn't fully understand them. But this is complexity for the sake of flexibility -- which is far easier to deal with than complexity for the sake of performance. Best practices around specific use cases will eventually become common knowledge.
Your implication is that sysvinit was designed for speed. I can't seem to find much evidence for that.
In fact, sysvinit's purely synchronous nature and its reliance on the patterns of symlink farms to represent system states in the form of "runlevels", along with the tendency to write bloated shell scripts for it because no one ever bothered to write a proper rc/service runtime framework until much later (in contrast to the BSDs), meant it was slow.
IIRC, one of the motivations for some distros to symlink /bin/sh to dash was to reduce boot times. Then we got ugly hacks like startpar for asynchronous script executions, insserv for parsing dependency directives inside scripts, and Debian had all the cruft behind sysv-rc just to make managing the runlevels easier.
The whole thing descended into an unmaintainable mess that performance improvements in systemd and other alternative systems were unsurprising.
I think it was originally designed for speed (or at least for simplicity) because it only dealt with a handful of use cases that worked well. Granted, they were the more general use cases, but writing a dependency solver into the init system in 1980 would have negatively impacted speed to an unacceptable degree. Applications back then weren't modular enough for it to make a ton of sense anyway. So they traded flexibility for deterministic execution that was "fast enough". That's not a tradeoff you have to make anymore; CPUs are fast enough now that you can do all the thread syncing you need without worrying about the overhead too much.
But flexibility and modularity is more important now than maintaining compatibility with decades old init scripts. Some of those modules may need low level system access for a variety of good reasons. The kneejerk reaction is to apply network server use cases to every Linux feature, but Linux is a whole lot more than a network server these days.
There is something epically beautiful about OpenBSD's consistency over the last 10 years. I swear that a 2004 OpenBSD sysadmin would be close to 100% efficient with a circa 2015 OpenBSD system, despite the absolutely massive number of improvements and changes that have been made with the operating system.
Once systemd starts to get traction, it's cognitive overhead is going to become a real problem for sysadmins who want to be able to model the entire boot process and run-state in their head.
For better or worse (I honestly don't know which, btw), you are going to have to start to trust the operating system to manage its boot process/run-state correctly rather than having it be a direct implementation of your mental model. (I would argue other *nix Operating systems, like OS X, have, to some degree, been doing that successfully for a while, so it's certainly possible - just not clear whether you want that type of behavior on your server systems. I guess time will tell.)
I'm not a professional sysadmin, but as a developer with a bunch of virtual servers systemd has only made things far easier.
Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...
Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.
> Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...
This.
I'm not saying systemd is the best answer, but I really think sysadmins could learn a thing or two from developers. Nearly every custom init script I've ever seen has bugs, and isn't portable between systems. Where are the unit tests? Code reuse? Focus on maintainability/readability?
Shell scripting used to be amazing, but as a language these days it's behind the times.
Sysadmins would say that developers could learn a lot from them. Like, documentation, maintaining backwards compatibility, sane upgrade processes ...
Don't throw stones in glass houses and all that. We all could stand to learn things from other people.
Sure, sysadmins could stand to be better programmers, but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem, so for them, certain best practices in software development aren't all that attractive.
> but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem
Which is exactly why developers don't get to write any documentation other than the bare minimum: software for the client is just a tool, once it is running they want developers to move on to the next problem, not wasting time on any additional documentation (which, arguably, is not strictly a developer task, more a QA one).
> Shell scripting used to be amazing, but as a language these days it's behind the times.
It was never amazing, was given far too much credit, used too widely, and brought up a generation of people thinking that this is the way to manage a complex web of dependencies.
On the other hand, constraints like the above tend to foster creativity, so perhaps that's why people have been so nostalgic for this way of solving the system startup problem.
You should take a look at modern init scripts in FreeBSD. Common functions are sourced in, and then most "scripts" are just a series of variables that define various aspects of a daemon. Scripts are about the same length as systemd scripts, yet you have all the power of shell scripting right there if you need to do something special. In contrast, the systemd method means that if you need to do anything extraordinary, you have to call a shell script, and you've suddenly lost all the functionality given by the init system. So, for all intents and purposes, systemd ends up giving you the worst of both worlds if you need to call out to do housekeeping on startup/shutdown.
>I'm not saying systemd is the best answer, but I really think >sysadmins could learn a thing or two from developers.
>Nearly every custom init script I've ever seen has bugs, and >isn't portable between systems.
Nearly every custom init script is written by a developer for the product you desperately have to run because development wanted it, not bothering to give us sys admins time to look into the: packaging, configurability, operability and if a custom module/cookbook/playbook needs to be written.
>Where are the unit tests? Code reuse? >Focus on maintainability/readability?
If we were given the time, it would be there. My deadline for almost all work I do is "yesterday".
A lot of us sysadmins are or were developers too, it's a lot easier to maintain a system you understand. Plus if it can be automated in code, it will be automated in code, we're lazy like that (well some of us, if I have to do the same thing more than once, I'm not doing it by hand the second time) and end up writing non-trivial amounts of code. Nearly every piece of software I've ever seen has bugs, and what do you mean I can't run the software without xyz framework, abc runtime and libraries a-zzz being installed on the system. Where are the performance specs? Resource reuse? Documentation? Focus on deployment/maintainability? Security audits? (I'm only mostly sarcastic)
Shell scripting is still amazing... when you use it for shell scripting.
Now I do like systemd more, but not because init scripts are bad, I just prefer how it does things, and can always call out to a script if need be anyways. And I suppose making it more friendly to developers is going to make my life easier too, the atrocities I've seen written in every language imaginable to replicate something that would be trivial to do in an init script. I didn't think it was possible (well possible yes, but why, WHY) to essentially rewrite the userland tools and the job of the init system in (very poorly written) php, but I've been proven wrong. Even something like forever for nodejs doesn't really have a need to exist.
But point is developers and sysadmins are both more than capable of writing shit code, lacking tests, code reuse, and all the rest of the best practices. At least we have to deal with the fallout from our code ourselves ;)
The shell has always been awful. It is underpowered and overfeatured with a deeply crappy API. It is more like Windows 3.1 than any sane development environment.
The good news is that you don't have to implement initscripts as shell scripts. I recall at least one GNU/Linux distro that used Python for all its initscripts, for example. Really, as long as the "script" is executable, a SysVInit-based or BSD-rc-based system doesn't really care (unless it's doing something strange like calling a specific interpreter on each script - i.e. calling `sh /etc/rc.d/rc.*` or somesuch).
Actually, a FreeBSD or NetBSD rc system does indeed care, for that very reason. rc.d scripts are sourced by a /bin/sh shell, using the . command. See run_rc_script on the rc.subr(8) manual page for details. Ironically, it's daemontools and daemontools-like systems where what you say holds the most widely. Tools such as execline can be and are in practice used in "run" programs.
It doesn't look like OpenBSD has such a requirement (the start_daemon() routine in /etc/rc just calls each configured daemon's initscript as-is from /etc/rc.d without specifying sh or ksh to execute with); I had (incorrectly) assumed that this was a general BSDism rather than an OpenBSDism.
If your needs are simple, the resulting unit file is trivial and you can basically copy and paste from any example without looking at any documentation.
If your needs are complex, you have better chance by mixing different options in the completely declarative syntax of systemd than mixing shell snippets from different scripts using different styles.
In any case systemd happily launches LSB-compatible init scripts, you literally don't lose anything.
> Complex needs are better solved with source code rather than special blobs of $system.
Arguably, reinventing multiple complex wheels is often a not particularly good choice.
So either you're implementing something really novel in your init script (which does not sound like the best place for it) or it's something that may benefit from a single, shared and tested implementation.
And, again, you can still fallback to LSB shell scripts if you really need something incredibly special: my point was that, bloat notwithstanding, you don't lose any feature but gain many.
In the GNU/Linux world, yep, this is the case. Lots of copypasted boilerplate that becomes a nightmare to debug.
In the BSD world (at least with OpenBSD, where I'm most experienced), this isn't the case at all; OpenBSD ships with /etc/rc.d/rc.subr, which one can (and should) source into one's initscripts to make life much easier than even (in my experience) systemd's unit files. Set the $daemon variable, source in rc.subr, call rc_cmd, done. Sure, you might want/need to read a couple manpages, but this is the case with systemd, too.
Back to the GNU/Linux world, the complexity of SysVInit is not a hard dependency of using shell scripts for daemon management; there's very little stopping a GNU/Linux system from adopting a BSD-style rc system, rc.subr, etc. and making life significantly easier for sysadmins. Alternately, there's very little stopping a GNU/Linux system from implementing shell scripts in something significantly less horrendous to use, such as an interpreted scripting language (like Perl or Python or Ruby). SysVInit and BSD-style rc-based init both have a distinct lack of hard requirements on how initscripts are implemented, so long as said scripts are valid executables.
> Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.
Would you not want to be able to model the startup process and run-state of the software you write in your head? Why should the underlying operating system be any different? If you have to rely upon external tools just to be able to reason about the execution flow of something - your server, your program, your recipe for lamb vindaloo, whatever - chances are it's way too complicated and needs significant refactoring.
...and that's awesome if you don't need 99.9% uptime and thus need to have absolute faith in your system's stability. However, for mission-critical applications that do it's hard to argue that systemd is a better solution than writing all your scripts by hand. Unless you want to be able to blame systemd if it all goes pear-shaped but I doubt your boss will buy it =)
I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means being able to predictably spin up new instances that you can trust.
I'm also a developer and for example one problem I had with my servers on AWS was that my process sometimes happened to start before my volume meant for logs was mounted, which meant that all logs were going to my primary volume, with the server crashing at some point due to out of space errors. Another time I was using a local database meant for local caching and again, sometimes it would fully initialize only after my main process was started.
And because I was using ELB with auto-scaling, it meant that these incorrectly started servers would be connected to the load-balancer, because the main process was suddenly responding to /ping, even though it wasn't ready yet. And then it would receive thousands of requests per second per server, at a moment in time in which the cluster badly needed fresh instances, which led to an unstable cluster. But only sometimes ;-)
Except that there's no reason for systemd's model to "get traction" outside of desktop-focussed Linux distributions, since a "better user experience" is it's current raison d'etre.
So no, I shouldn't need to have to "learn to trust the operating system / systemd" on my servers -- it has no reason to be there!
The model where services do not daemonize and write a pid to a file is ...
... the one that people who moved away from System 5 rc scripts years ago have been pushing for with some degree of success, for years. systemd is not its source at all. systemd has merely been reaping the benefit of all of the programs over the past 18 years that have gained a --no-fork or a --foreground or a -D or a -F or whatever option in response to those who wanted to run services properly without bodging things under daemontools, or runit, or s6, or perp, or freedt, or daemontools-encore ...
The per-unit cgroup and the ability to kill it is rather nice too, without delving in the container-related features where the systemd server-side support really shines.
Many systemd features are explicitly tailored for server workload (eg. resource management), and a particular effort is being put over container-based deployments.
CoreOS has been quite successful in basing their own infrastructure on systemd and with rkt they are moving to use systemd even more.
> Try to find the answer to "which units are going to be started on next boot?".
Isn't this just `ls /etc/systemd/{user,system,network}//*` (bah HN formatting is messing this up)
Also the other states:
> Speaking of which, why is systemctl daemon-reload even a thing?
Because configuration changes while things are running. Otherwise we wouldn't need apache2 reload, nginx reload, haproxy reload, etc. Also, according to the man page:
> After the units have been edited, systemd configuration is reloaded (in a way that is equivalent to daemon-reload).
As for systemctl edit, it allows you to add diffs to the unit file. For example, let's say I am using redis and I want to ensure it always restarts (for whatever reason). Rather than manually editing the unit file (which will have issues if an upstream change happens), I can simply use systemctl edit to add a diff for my change. This way I can still maintain compatibility with the upstream unit file instead of having to deal with potential conflicts. It's not so much babysitting the user as it is a potentially better experience.
> Isn't this just `ls /etc/systemd/{user,system,network}//*` (bah HN formatting is messing this up)
Nope.
There are many virtual units for which there are no files. One example is the on-demand getty units. There are two for "plymouth" boot-splash stuff (I see these attempted-and-failed units on archlinux where plymouth never exists). There's some autofs thing. There's auditd.
I'm actually not booted into linux right now so I'm struggling to remember the specifics... but if you also count all the simple mounts and devices which clutter up the output of "systemd --all" as "units which are not files in /etc/systemd" then the count is around 100.
Couldn't/Shouldn't this be solved with some script that traces the dependencies of units?
I'm not familiar enough with systemd to understand how to trace dependencies and create a complete list of all units that will run at next boot but I expect this to be doable by someone with that expertise.
The resulting dependency diagram may look complex, but consider that there is no equivalent functionality for SysV init scripts despite the fact that they have the same level of inter-dependent complexity.
The complexity of the dependency graph produced by system-analyze is not present in most sysv init systems. That complexity is mostly a reflection of the efforts to make system services boot in parallel. Most sysv init scripts were blocking and executed serially, which is what allowed the simple filename prefixes to be used for ordering.
What to see your entire sysv init, as it will run every single time?
ls -1 /etc/rc3.d/[SK]* /etc/rc.local | xargs -- cat | less
Whether using a dependency system at all for the purpose of ordering and expressing relationships between daemons is the best approach is beyond the scope of this reply, but I just want to say that most modern sysvinit deployments did in fact have service dependencies, through the use of LSB initscript headers that were parsed by insserv(8).
Previously on arch linux I had a list of 7 to 10 services in rc.conf that started serially in less than 2 seconds. That was pretty damn simple. Besides that there was inittab which just had the gettys and the main startup script.
This is the first anti-systemd article that I've read that focuses exclusively on the technical issues with it (well, mostly: the last one kind of gets touchy on the politics). And this article is a good summarization of them.
Linux in the server world is rapidly turning into a thin platform hosting containers (docker, kubernetes, etc) of "fat" Linux distros. Systemd makes a lot more sense if you consider it in that context: a container can use systemd to have the host start a monitoring process whenever any processes are executed, etc. Sure, this probably could have been grafted on to sysvinit, but it would have been an ugly hack just like every other thing that's been grafted on to sysvinit.
Linux everywhere else but the server world is all about low power usage and tight integration with custom hardware. Embedded devices, mobile phones, tablets, wearables, etc. All of these use cases demand an init system that is aware of and able to manage the system power state as a gradient, not a binary value. Again, this could probably be hacked on to sysvinit or the kernel itself, but would require big kernel changes and likely a fork for "Mobile Linux" and "Server Linux" as the market begins to demand power-aware schedulers, memory mappers, etc.
If the computing world were to remain in a paradigm where we run small physical or virtual servers dedicated to tasks, then sysvinit would be fine. But that's not where we are today nor where we will be, and sysvinit simply isn't flexible enough for a lot of the use cases where Linux is seen today. The *BSDs exist if you want a rock-solid and simple x86 Unix system for hosting traditional applications in a client-server setup; but if you want a flexible platform for building arbitrary products, Linux is the way to go.
Some of the complaints resonate with me. In particular, the limited power of systemd's unit file DSL is why I think it's better to allow services to be written using a general purpose programming language. GNU dmd, while still not developed enough to do cool things like socket activation, gives the user the full power of Scheme to configure their system services as they like. I run it as PID 1 on two of my machines, and I also run it as an unprivileged user to manage my gpg agent, emacs daemon, offlineimap, mpd, development web servers, etc. on all of the computers I use, including at work. I think people that aren't happy with systemd, that are also interested in rolling up their sleeves and hacking on an init system, should give dmd some love.
One of the more sensible ^rants^ I've read against systemd (it is KISS for some weirdly convoluted definition of KISS). Too bad these naysayers are not/will not be taken seriously.
The naysayers are taken seriously, but they're in the decision-making minority. Linux distro maintainers wanted to move on from SysV Init, and systemd looked promising. Just because the naysayers didn't get their way (and on some distros, they did) doesn't mean that their criticisms aren't noted.
Are you seriously saying that most distribution maintainers were happy with sysvinit?
For what is worth, all the Debian maintainers I know, even those who are not systemd supporters, felt that sysvinit had to go in a way or the other (eg. during the default init debate the options were systemd and Upstart, sysvint was doomed to be replaced and only kept as a compatibility option in any case.)
Which distros? Slackware is the only near-mainstream Linux distro I know of that is even possibly not moving immediately to systemd, and even then, the Slackware folks have said that once it becomes too difficult to maintain upstream packages, they'll switch too. They're just being pragmatic about it.
It's worth noting that "too difficult" is a very high bar for a distro that to this day has yet to implement things like PAM; if you want an authentication system more advanced than /etc/{password,shadow}, you'll need to compile PAM from source and install it manually (there might be a Slackbuild for it nowadays; I don't really remember).
And gentoo had to jump through hoops (to the point of forking udev) to be able to continue to do so.
The whole systemd vs sysv back and forth is a smokescreen, because while sysv stops at init systemd continues on to cron, networking, sessions, dhcp, dns, /dev, and soon to be ttys etc.
And while the kernel is maintained under the rule that change do not break user space, systemd developers seem to do changes with a permanent "deal with it" meme.
I thought there were others in addition to Slackware, but that pretty much all the distros are jumping ship shows the naysayers don't have enough political oomph.
Those that has adopted systemd are either precursors for commercial distros, said commercial distros, or based of same.
The really early adopters seems to have been offshots of Fedora. And i do wonder if Debian was a holdout until systemd started sprouting containerization features, and Canonical (Ubuntu) was feeling the heat.
Agreed --- this is the first rant against systemd I've seen that didn't talk about Lennart at all, and the first one from which I've ever learned anything. It's a great example of how people ought to argue on the internet. While I find that systemd is great for my purposes, there's plenty of room on the internet for Steven and the BSDs.
With System V and Debian, one added some new apache or bind configuration and then called invoke-rc.d daemon reload to inform the daemon to reread the configuration files.
With systemd, you do the exact same thing and then call systemctl reload daemon. Is the author making an argument here based some other distributions way of handling System V?
No. The author is not getting reloading the daemon's configuration mixed up with reloading systemd's configuration. You, however, are. The systemctl manual page warns about this mistake. See what it says under daemon-reload.
The systemd war is probably the best thing that could have happened to ensure widespread adoption of systemd. It allowed the supporters to lump anyone with any concerns or objections in with the frothing-at-the-mouth haters, and completely ignore them.
I don't hate systemd. I like what it's trying to do, and I don't hate the big-picture implementation either. But I, and many others, have a bunch of concerns around some specific details of the system, that got ignored and swept under the rug in the tide of vitriol between the two sides of the "war".
If the discussion had been more civil, maybe the more rational objections might have got heard, and we might have been able to pressure Lennart and Kay into not rolling their own, un-reviewed, crypto for journald[0]. We might have been able to avoid some of the feature-creep[1] that seems to have grown out of a NIH-mentality, and a better relationship with the community might have made the core developers more willing to play nicely with others[2][3].
Why is resolved feature-creep? It's rather nice to have working name resolution out of the box when coupling it with networkd and firing up a container.
I mean, it's not my standard usecase in this moment of my life, but I see how it can be useful.
(Also, your last link reports that "This bug is for livecd-tools which has been fixed", I don't know what's the point you're trying to make.)
About a year ago, we started switching over new servers to systemd instead of openrc at work. I knew sysv/openrc/bsdinit and I liked them so for the first month or two I utterly despised systemd. (Plus as a sysadmin it seems like I have to hate it or risk losing my linux card) I just wanted my damn scripts back instead of dealing with the .ini-inspired config file crap. If I had a choice at the time, I would've been willing to move to busybox-init just to get away from systemd. Had I spent less time complaining to my coworkers about how much it sucked, I probably would've written this post myself. But little by little it grew on me.
Oh, so if an nfs mount fails on boot, it's not going to just sit there and hang? I guess that's pretty cool, saves me from having to throw a console on it while muttering under my breath. And it didn't start nginx either, so it's not going to serve a half broken site and tell the loadbalancer all is good? Maybe it's not all that bad. No no, what am I thinking, this is still crap, I could've written all this in to the init scripts (but I didn't).
This client's service is irritatingly unreliable, maybe I should write a wrapper to supervise it. Someone must have done something like this for syst... Restart=OnFailure... well that's nifty.
Ugh, monitoring daemon ate up all the memory again, why can't the vendor get their shit together or the client switch to something that isn't useless. Guess I'll google some ideas for a watchdog script. Hmm, looks like I can just throw MemoryLimit and Restart=onfailure in the unit? No way it's that easy. (it was)
After a couple months of this, systemd actually started to seem useful, I even started having a little bit of dread when hopping on an older server and having to deal with openrc. All of the complaints here really seem off base to me now that I've had a chunk of time to figure it out. Init scripts pissed me off when I didn't have much experience with them too. Now, thats not to say I like everything about the systemd switch. netctl is utterly worthless for anything other than a static ipv4 address (and reenable, really?). I still forward all my logs to syslog, because I prefer text files (and all the awk/sed/grep/cut/sort/column/etc goodness that comes with them) for anything more than running `systemctl status <service>` and seeing that I accidentally the config file for something.
I also don't buy the whole "It should be on desktops, not servers" because I don't see much of an advantage for desktops, other than maybe the trivially faster boots. Wifi is a pain in the ass with netctl and netctl doesn't seem play well with anything else. But for servers, I've found it to be pretty awesome.
To address a few things straight from the article that I disagree with:
>At least one Debian package (memcached) is already invoking a wrapper script from ExecStart.
This is exactly why I don't understand the whole argument around it being inflexible. If you need to call a script that does some setup that's incompatible with systemd, call the script? It's no different from helper scripts that many packages used with init scripts. If you still want to do some checks and call start-stop-daemon, go right ahead, nothing is stopping you. It seems pretty strange to consider it a fault that you can't use a bash script, and then complain that you'll have to use a bash script.
>Here, let me helpfully tell you that you should run systemctl daemon-reload, but perform the action you requested using the old configuration anyway.
This really seems like a trivial complaint. I guess you could always alias systemctl="systemctl daemon-reload; systemctl" if it's that much of an issue. Personally it's saved me after an accidental :wq (damn that muscle memory). "File has changed on disk" uhhh it shouldn't have. crap. I guess it's a matter of preference. Also I didn't even know systemctl edit was a thing. That does seem kinda pointless, but it also seems kinda pointless to be bent out of shape over it. Do you also cat `* * * * * command >> /var/spool/cron/crontabs/$USER` because crontab -e is babysitting you? (I used to do that, but because e and r are too close together and reconstructing a crontab from logs is painful. Now I just alias crontab="crontab -l; crontab")
>Now, systemd does rely heavily on D-Bus
I've never run into issues with it, but I do agree with you on that point from a philosophical standpoint.
>And yes, you can avoid the whole issue just by masking any units you don't want to start, but that's missing the point.
The only point I see is that it's different. Of course whatever you're used to is going to be more effecient to you than something new.
>I believe it creates more problems than it solves. Sadly, many of those problems will take years to become apparent, by which time all the major Linux distributions will have fully integrated it.
Well it definitely seems to solve more problems for me than it creates, and saying those problems will only be apparent years down the road is not only impossible to argue against, it's a hand wavy cop out of "Trust me, its just bad! Because reasons!" However this does touch on what is probably my biggest gripe with systemd, and that is that ALL the big distros seem to be switching. Granted I don't mind from a having-to-use-it angle, since I personally prefer it and would probably stick with distros that do anyways, but it does give me a certain unease, it may be my choice but it shouldn't be the only one, and I'd have the same problem if it was sysv that everyone switched to.
All in all, these complaints really just seem like complaining that it's different. And it is different, but that doesn't necessarily make it better or worse, just takes time to get used to.
You either must have gotten in late, or been exceedingly lucky, as failing NFS mounts is one of those things systemd downright hates.
People have had whole boxes fail to come up because of a NFS mount that didn't work, and therefore the unit file didn't come up roses, and so systemd decided to abandon ship and present the admin with a borked minimal console.
One early fix, a brute force 30 second kill timer on every unit file.
Went over well with one Fedora Alpha (lucky for them) where said timer tripped on the post distro upgrade house cleaning unit.
I don't care about systemd, as I want to use OSes that embrace modern architectures, unikernels, containers, safer programming languages, improved UI stacks,....
This rant is a good example how luddite users rather stay stuck in a UNIX System V world with their green phosphor VT100 terminals.
Thing is, Linux used to be flexible. This because of some stuff it inherited from Unix about user space being made out of loosely coupled parts with solidified interfaces between them.
Meaning that it could accomodate both the latest and "greatest" coming out of the labs, and the tried and tested stuff that had been sitting in production for a decade.
systemd instead gloops it all into a lightly coupled shoggoth of a project that is chasing the future come hell or high water.
Well said. This reads like someone complaining about the metric system because you can't easily divide metres into twelfths.
Those kinds of people will always exist - they've spent many hours learning it the good old hard way, and I'm damned if you kids are going to make it any easier! You lazy so-and-sos.
It seems that the author completely disregards the modern tendency to treat boxes as commodities. While pondering a "what's going to start on this particular machine" question is interesting in the "old world", it is hardly a question anyone asks when your fleet starts to be measured in tens of thousands of cloud instances.
I think OP's issue is with the low contrast between yellow and black; it is suboptimal for reading. It was readable but could have been better with some slight color changes.
Why would you punish yourself with various *BSDs when you can have OpenRC and more advanced ports on Gentoo? Do people really associate systemd, Gnome and other nuisances with Linux?
Punish ? .... BSD is extremely user friendly. It just seems to be picky at who it calls friends. They ONLY reason I still use Linux anywhere is because I have to keep my skills for it fresh since its where I make a living.
The metric I like to use for adopting new processes / services is pretty simple. I have _____ problem. What tool solves that problem for me.
In my case systemd solves nothing and creates more.
True (as I know firsthand, thanks to several Slackware-running machines that I own). My remark was just in response to the question of systemd being tied to Linux.
Don't you have to build every package from source with Gentoo? I can see where that would be useful, but isn't it really time-consuming (or punishing, if you prefer that word)?
This page's horrible readability is a perfect reflection of the kind of folks that dislike systemd.
Yeah, newcomers to the Linux land without the expertise you guys acquired through 10 years of using Linux, would apparently appreciate how easy it is to manage a server by invoking some lengthy commands they learned through reading the often overly-dense manual instead of `systemctl start/stop x`.
Nowadays some of those so-called "hackers" tend to ignore the part of work that makes the user experience better. They claim that they want simplicity and avoid bloat but most of the times they are just being lazy and not sympathetic to their users.
I'm not saying that things should be dumbed down to clicking GUI buttons on Linux. But if you want your program to become mainstream you need to think about providing good and practical docs and making your program easier to use, even for text-based programs.
Accept that most people would like Bootstrap'ed sites more than yours:
body
{
background-color: #080808;
color: #ffff70;
font-family: sans;
}
That's why I opt for Ubuntu 15.04 instead of 14.04 LTS on my VPS, since 15.04 has systemd.
Are we really using website design choices as a valid target to discredit someone's opinion on the technical merit of an init system?
Can I use Lennart's inability to buy a properly sized polo as a reason why systemd is bad?
Yeah. The guy has an ugly website. But he has specific points about mistakes that systemd didn't learn from and is repeating. Did he list 'it's too user friendly' as one of those mistakes? In fact, one of his core arguments is that there will actually be an increase in complexity and reduction in user friendliness due to design choices that have been made.
I'm saying that, if people keep responding to those less-experienced Linux users who complain init scripts too complex by saying "Quite. Deal with it." like this author (http://www.steven-mcdonald.id.au/ugly.shtml), I fail to see how systemd is gonna fail.
My comment is not about design choices but attitude and sympathy to users. This is not intended as an ad-hominem.
The author seems to think systemd "babysits" its users, but as a user I find systemd "cares" about its users.
It might not look fancy, but the readability is way better then on most sites these days. It loads fast, it has enough contrast, there are no distracting elements like images or animations that make it hard to focus on the content, scroll works as expected, the content is not split into useless pages, it does not make my fan spin up because some elements use insane amounts of computation power, content is not squeezed into some tiny box as it often is on other websites...
If the average website would let me read as comfortably as this one, that would be a huge step forward.
Fwiw I like the website. I would have toned down the yellow. And I would probably have had dark on light.
But it's really great to have a page that loads nearly instantly - the ratio of information provided : loading time is very good.
Other pages take an age to load (and Chrome on iOS has an annoying "feature" where the address can't be selected and copied until the page has almost finished loading).
Agree, it's a pleasure to use a site that isn't burdened by tons of ads, tracking scripts, and tries to re-implement scrolling (badly) if you visit from a mobile device.
See theverge.com for an example of a site that doesn't even scroll smooothly on my android phone.
There is a lot of development in the ecosystem dominated by Freedesktop.org, Gnome and (to some degree) large commercial players like RedHat and SUSE. It's more efervescent than it's ever been. Fifteen years ago, I'd have been ecstatic to see so much development being done.
Yet for all this fierce code churning, there's absurdly little innovation. Even in systemd, there's a lot of focus on non-issues like booting without needing a shell, while still inventing service startup notification types because none of the gazillion startup notification types already existent can actually solve precisely the problem systemd claims it solves -- managing complex startup chains.
In the last three or four years, my Linux systems have become increasingly blackboxed, and consequently harder and harder to configure, let along debug when they stop working. Over the span of six months, a tool you just got to learn how to work with (mostly because the documentation seems to have been written by very bored and very verbose monkeys who had only a typewriter to pass the time with) becomes obsolete or is revamped to be unrecognizable. Which would be fine if that actually solved any problems. Instead, it replaces the old set of problems with a new, even more complicated set of problems that developers claim you shouldn't have, while insisting that the old problems were simply due to the old architecture and we'll have none of that now that we have this shiny new thing.
Ironically, systemd is actually one of the better pieces of software to come out of that ecosystem. No, seriously. It's decently documented and as long as you don't need anything more complex than "here, start this daemon after this other daemon started", it does its job.
If you want to peck at something, have a look at Polkit, where someone brilliantly figured out that the best way to configure policies is to write basically undebugable JavaScript code in .rules files.