Hacker News new | past | comments | ask | show | jobs | submit login
A timesyncd failure and systemd's lack of debugability (utoronto.ca)
140 points by zdw 7 months ago | hide | past | web | favorite | 123 comments

Let's carefully step around the whole distraction that is systemd. Instead:

> The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it.

This seems to be really common practice in modern software development, and it sucks.

There's a rush to rewrite old software and to match some or most of that software's feature set, enough to get people to adopt it, but useful error messages and other diagnostics represent a lot of extra work and so nobody bothers.

Not enough programmers are spending time as systems administrators, trying to keep plates spinning, and so they aren't learning to appreciate the value of software that helps you troubleshoot it when something goes wrong. It's turning into the long-running conflict between mechanics and automotive engineers, where mechanics are tired of things like, "To replace [simple part that needs to be replaced every 5 years]: begin unbolting the entire front of the car. The front of the car uses 23 similar, but different, types of fasteners; keep track of them."

It needs to be a rule that if somebody's going to rewrite some software, they've got to match or improve the logging and error handling in it. Error handling needs to start getting the kind of attention and shaming for bad behavior that bottom-barrel security practices gets.

That's not quite right in this case. Systemd and gnome3 were cases where the new 'version' did not need full feature parity. Both those projects were written on leisure time, with no pressure for deadline or feature. and still adopted from top down. i guess some projects forget the benevolent part of benevolent dictator for life.

systemd arguments were just "boot faster" and "it's newer". Everyone knew it was going to be broken, buggy, lacking features. As those were never actual goals.

In fact, the goals had features never intended for init. So we could say they were rushing for feature anti-parity.


see the table at the end. All the features systemd have show "no" for init. And "verbose debug", something you would consider very essential on a piece of software responsible for making your machine something broken and unusable, just show "no" for systemd, and nobody ever cared, because: boot faster (or so they say)

systemd is a mistake. But i guess now it is a permanent mistake.

I want to expand on mikekchar's "You're getting down voted mainly because you are wrong."

systemd has been adopted by many different distros, however this is not because of "top down" pressure (From who, Red Hat? Most distros don't answer to Red Hat), and most of them don't have a (B)DFL who could have imposed that decision on them.

There are many arguments for systemd other than "boot faster" and "it's newer". In fact, the author of the article this is a thread for has written a pretty good list of them: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdRight

The article you linked... is bad for very many reasons. In particular, it makes no acknowledgement that what it refers to as "init" (http://www.nongnu.org/sysvinit/) is only really half of an "init system" as the term is commonly used today, and all distros would be pairing it with something else on top of it. The table at the bottom is particularly ridiculous, and I take issue with almost every row; a lot of that comes down to the silliness of comparing systemd to sysvinit-by-itself, rather than sysvinit+initscripts or sysvinit+OpenRC or sysvinit+LSB-init-scripts.

> systemd has been adopted by many different distros, however this is not because of "top down" pressure (From who, Red Hat? Most distros don't answer to Red Hat), and most of them don't have a (B)DFL who could have imposed that decision on them.

Often it was a "matter of fact" thing. If the ecosystem under the distro changes to use systemd distros will follow. There might be more advantages to it, but it being required to run Gnome 3 without additional effort, and in general other servies starting to rely on it, make the decision pro systemd the easy one for distro maintainers. You can see that point being made in the debian discussion on the issue.

Till they hit the showstopper bugs and can't solve them.

sysvinit is short enough to be trivially maintainable. If you're just using it as a shim to load a framework on top of it, you can ditch even more of it cutting it down to around 500LOC (see openrc-init). So that's not a large enough part of "the ecosystem under the distro".

Debian maintained their own init-scripts framework (and still does, for kFreeBSD and other kernels), so that certainly didn't move out from under them.

Arch Linux maintained their own initscripts framework, so that wasn't moving out from under them.

I suppose there is a case to be made that distros were pushed to systemd by Gnome 3's reliance on systemd-logind's D-Bus API.

You're getting down voted mainly because you are wrong. I'm actually not well versed in this, so hopefully somebody will correct my wrong version -- but at least I think it will steer you in the right direction ;-)

Systemd solves real problems that have to do with virtualisation and/or containerisation. Red Hat is the company that wrote Systemd and they did it for commercial reasons (i.e. they needed these facilities in order to fulfil contracts with paying customers). Red Hat contributes greatly to Gnome. I haven't checked, but I believe they are far and away the biggest contributor. They have financial reasons for doing so, but I don't really understand what they are.

The connection between Systemd and Gnome is real, but not quite what you think. Systemd does make some stuff in Gnome easier to do, but it isn't really necessary at all. Systemd was not developed for Gnome -- it was adopted by Gnome. It's a bit of a dog food problem. Most Gnome developers (i.e. Red Hat developers) have a vested interest in Systemd being successful. They want to make sure they are using it where they can. They saw no particular reason to make Gnome 3 compatible with any other init system because: 1. It would be more work and why should they pay for work that isn't in their interest 2. They want more people to adopt Systemd because that puts Red Hat in a good position to sell more contracts.

It's totally understandable and not really underhanded. I personally hate it with a passion, but things I hate are not always evil :-)

Systemd solves real problems that have to do with virtualisation and/or containerisation.

systemd predates the industry love affair with containerization and virtualization for workload management, although work on containerization within systemd may have helped increase awareness of containerization's value. I don't remember early reasons to use systemd mentioning either; the rationales trotted out in favor of systemd in its early days were reducing boot time and replacing shell scripts used during boot, shell scripts labeled things like "old" and "fragile", despite multiple decades of shell scripts being successfully used to reliably manage the boot process on open source and proprietary Unixes, alike.

systemd does not predate the industry's love affair.

According to wikipedia, the initial release was in March 2010. The industry started courting virtualization (as it is practiced today) with the appearance of EC2, in August 2006, and the love affair came shortly afterwards.

"Reducing boot time" is of great importance for short-lived containers - if it takes 60 seconds to boot a server, and you only run it for 120 seconds, then you have 50% overhead. If it takes 6 seconds, you're down to 5% overhead; If it takes 150ms (e.g. Intel Clear Container project), it's completely negligible.

Those scripts, are, in fact, old and fragile, and while they did manage the boot process, "reliable" is a stretch. I've personally filed 2 bugs against the horrible hodge-podge of isc-dhcp that, in rare occasions, randomly failed (which is a bit of a problem if your DHCPd box is unattended and a few hours drive).

Through the years, many systems much better than init scripts have been proposed and implemented - personally I like the daemontools family, which I think is inherently better thought out.

although it definitely helped to have red hat backing, systemd won on for technical reasons: it's much better than the init scripts; it's not inferior to any other system; and it comes with simple integration with network/resolvers/etc (by virtue of including those services) -- something that saves distributions a lot of work integrating various parts.

> "Reducing boot time" is of great importance for short-lived containers

Containers don't tend to run an init.

While true, i think the notion here is to treat containers like VMs in the cloud. Thus a container is spun up or down depending on the load on the service.

Then again, that notion brings to mind a certain comic about a self-ddos...


Systemd as a project has developed a history of moving goalposts. Their initial talk was about boot speed by using dependency trees and parallel execution.

But when containerization caught the zeitgeist, they were quick to trot out nspawn as an alternative to, say, Docker (never mind that Docker and RH devs had a bit of a spat around the same time).

Since then they have been shifting the goalposts towards whole system management, perhaps best likened to Windows Active Directory, by adding more and more sub-systems that frankly has crap all to do with booting.

On top of all this it seems the people involved regularly end up pulling grsec style "fixing" of systems that work in real life but are "broken" according to some spec or other (sometimes ignoring decades of hard earned experience in the process, as was the case with their DNS reimplementation).

Gnome 3 has a hard dependency at this point on logind. There are patches produced to allow it to work with consolekit but they got denied. What I hate systemd for is the fact that Redhat is making it a dependency on more and more of their projects. So much for making small tools that do one job.

This containerization and virtualization thing is repeated without evidence. There is nothing systemd specific to running containers. Containers and VMs work perfectly well on Alpine, Gentoo and any distribution.

Namespaces and cgroups are provided by the kernel. Any init system can use them. Cgroups are mounted at /sys/fs/cgroups and while there continues to be some friction on how these are mounted and used with systemd going its own way, other init systems like openrc manage them perfectly well.

The rest of it is standard networking, mounts and autostart services started by the container managers like LXC, Docker, libvirt for VMs and these can be done by any init system.

> > The real problem here is that it is impossible to diagnose or debug this situation. Simply to get this far I had to read the systemd source code (to find the code in timesyncd that printed this specific error message) and then search through 25,000 lines of strace output. And I still don't know what the problem is or how to fix it.

> This seems to be really common practice in modern software development, and it sucks

I have this discussion with the PM and CEO every now and then. It is impossible to give meaningful error messages for all errors. For known errors you can give a meaningful error message, but for errors not yet discovered it's more luck than anything if you give a meaningful error message. The best you can do it make the software robust so it'll work if a non critical subsystem/plugin fails.

The next part of the discussion is usually them suggesting to just allocate time to figure most errors and write meaningful error messages for them. But the problem is that this isn't a mechanical system, but a system several orders of magnitude more complex, and even if we did spend multiple man-years doing it, the cost in terms of added complexity would be enough to get any management directly responsible fired.

That said, I completely agree that developers should spend time supporting and running systems so they understand the impact of some of those decisions. Especially should it be expected that when you encounter errors that aren't bugs in the source code, you account for them.

If you structure your software simply and make it easy to reason about what is going on, then any half decent error message should give enough information to at least start down the right path.

But we don't get that in modern software. We get "Something went wrong :(".

> If you structure your software simply and make it easy to reason about what is going on

Complex problems are not easily made simple or easily reasoned about. Something like systemd (or any non trivial piece of linux) could never be made simple or, with your hands down in the guts of it, easily understood.

Breaking systemd's functionality into a million separate pieces wouldn't make the complexity go away, it would necessarily increase it.

I don't know what the solution is, but "making things simpler and easier to understand" is a bit hollow. Its problem space was never simple or easy to understand (unless you think systemd just runs some things at startup).

The trick is to have a environment where you can take the complex apart and examine each part in isolation, and then bring them together until you replicate the error.

With older _nix setups that was fully possible because daemons were pretty much (i am likely to get dogpiled on this) a user process without a shell to send output to.

With systemd on the other hand the state inside a systemd-managed unit, and the state outside of said "black box" may be wildly different. And to debug the person doing so have to attempt to maintain a massive mental image of state.

Frankly the kernel may well be the closest approximation, and it may well be better instrumented at this point. Never mind that building a second kernel in userspace is not a kudos point.

> It is impossible to give meaningful error messages for all errors.

This problem was, however, largely non-existent back when most of the init code was shell code. It was easy to trace, easy to debug.

(Edit: obviously, "easy" if you know sh and the idioms. They are, nonetheless, far easier than stracing stuff)

It certainly had a lot of other problems (hence systemd's popularity) but debugging was definitely not one of them.

> It is impossible to give meaningful error messages for all errors.

Not for end-users, no, because you're faced with a lot of future unknowns about who they are and what they were thinking of doing.

For developers, on the other hand... IMO this has similarities to the "exceptions or return-type checking" debate. Certain architectural choices will have a strong impact on whether the inevitable runtime/production failures are mysterious or not.

> This seems to be really common practice in modern software > development, and it sucks.

I recently deployed an upgrade of an "enterprise-type" jboss application, and it failed. The log file was literally 200kb of stack traces with some "INFO" entries sprinkled here and there.

I sent it to the vendor, and their response was "everything looks okay"

I disagree that nobody bothers. I've seen people bother about this.

Before systemd, a difficult bug was a difficult bug.

With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist.

I like systemd, it's a pity some vocal people aren't enthusiastic about it.

Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

I guess my question would be, is this really a general issue about systemd being difficult to debug, or some specific component subsystem of systemd. systemd is a big place.

Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

Why would someone speak-ill (attack) something they didn't have a bias against and were neutral on?

The question is not does a bias exist, it's if a bias is illegitimate. Many people who have issues with systemd have them because of their own experience, which they document. And often the very act of documenting their issues brands them as biased. When people say systemd is a regression and list their reasons, they are labeled as not having enough experience with systemd, and if they'd only give it a try, they'd see it's better. But that's the exact thing that's occurring, someone tried to do something that used to be obtainable, and their experience with systemd has been things being difficult, obscured, badly documented, undiscoverable.

If we're going to merely ask if attackers have a bias and use that as a reason to downplay or ignore their contrarian arguments, we should be asking if proponents have a bias and also use that to downplay and ignore their supportive arguments. "Are systemd developers in favor of systemd because Red Hat signs their paychecks?" is just as much an invalid assertion of bias, assuming the developers have made meaningful supportive cases that are independent of the bank account that funds them.

Before systemd the error were shell errors.

before systemd the bundled logger wrote to a file, didn't direct people to journald, that lags like a champ when a log file is > 1 meg.

before systemd errors in config files were on specific lines

I don't have bias against systemd I have bias against it's flaws.

I like its config file. I hate it's documentation.

I like that it has a cron like feature, I hate that its pretty impossible to debug.

_whats my bias_? I have >>5k machines to look after, I don't want to think about systemd. (and no, not all of them drain into splunk. that costs money.)

Worst of all, I hate the attitude: every problem is a deep and personal critism.

it's not, in the same way that when a bug brings down your entire estate at 4am isn't a personal attack on me. It's just terribly annoying, and I want it fixed.

I'll take the rare systemd bug over the buggy hell that most sysv init scripts were.

In fact, I run both distros with and without systemd and broken init scripts are a constant headache, while I've yet to encounter a single issue caused by systemd.

Your and zaarn's comments are the classic fallacious dichotomy between van Smoorenburg init+rc and systemd, called out by the Uselessd Guy years ago. One day, the world will have a systemd discussion without an instance of this fallacy cropping up.

* http://uselessd.darknedgy.net/ProSystemdAntiSystemd/

sysvinit has it's warts too and it can get quite ugly.

For example, atm I need to have 3 services. Two of these services need to run at the same time, they can run separately but then they are not useful. The third services can only run with both services online and is first run 15 minutes after boot (not earlier!) and then every hour but the hourly run MUST NOT run before the first run of the boot (ie the very first run may happen 15 minutes after boot, even if the hourly run would have happened in that time, it cannot)

This is not trivially achievable with cron and classical init other than resorting to complicated shell scripts in both.

With SystemD I can cut down on the complexity by declaring this dependency in the service files and the timers in systemd are sufficiently configurable to achieve what I need.

Sure, systemd isn't flawless, but it's a mile better than the average init script I'd have to hack together otherwise. Journald isn't pretty either but with the right configuration you can cut down on the lag.

And before systemd problems like the above wouldn't have errors in specific lines either, some shell script would be subtly wrong and the entire charade would collapse in an instant.

You don't have to think about systemd at all. Ansible and other modern tooling can abstract managing the machines for you (atm I managing about 200 VMs in Ansible playbooks and it works great)

I probably would have resorted to another process manager if I were on sysvinit such as supervisord, god, monit etc to avoid making those scripts.

I used supervisord for anything complex, which is where I think systemd get's it's syntax from.

Anisble abstracts some stuff (ensure running, running as use, etc) but for more complex stuff (process type, dependency management etc) you're on your own shipping system files.

My main beef is actually three things:

1) crap error messages,

2) crap validataion

3) journalctl being a crap VIM wrapper.

of all of them, journalctl is the most obnoxious. Instead of deffering to the system viewer it enforces their crap viewer. It truncates by default, and without restarting the viewer, or forcing it through less. (Why didn't they use less by default? less code to maintain..)

Journalctl uses less as a pager by default, you can set the env variable SYSTEMD_PAGER to specifiy what pager to use (or just PAGER) and SYSTEMD_LESS to specify parameters to pass to less.

journalctl has no own pager implementation and if no pager is installed then none is used.

I give you validation and error messages, systemd is a bit sparse on that front. Would be easier if there was a way for programs and services to signal back error messages to report.


You can actually enforce non-wrapping by setting SYSTEMD_LESS to FRXMK (the S option that is otherwise included causes truncating, if omitted you get wrapping)

This can all be configured in /etc/profile, .bashrc and friends

Systemd is the light that people don't want to look at and embrace, heh?

What I see is a an appalling amount of arrogance that comes with the "I'll solve all your current and future problems".

Systemd actually doesn't and in fact can't do what it pretends. The evidence is in the never ending list of posts like these that pop up all the time. And the arrogance of pretending to be the top of the init.d hierarchy of init scripts is not sustained by the competence of actually fixing the problems that people keep reporting.

There is a reason for the existence of the systemd opposition, and is similar to the opposition you find in any field where the top of the hierarchy is abusively taken by force instead of by competence. This in fact will be the reason for systemd downfall and I'm confident to say that it will take with it everything that sustained it in it's position.

> The evidence is in the never ending list of posts like these that pop up all the time.

So posts hating on something are proof that something doesn't work?

The open-source way used to be "come up with an alternative and win on merit". These days, since what systemd is trying to do is complex, it seems to be "complain about it online but keep using it" which leads to a super toxic user base, which might explain why bugfixing it (since you have to interact with said users) is so hard.

My understanding is that you are conveniently blind to several facts and this allows you to create a naive narative about what open source is supposed to be and what sustemd is leaving to the rest of us to do. Thus your downvotes.

With systemd, any really tricky low-level bug is systemd's fault, because systemd tries to do everything, in the most "advanced" way. And due gnome3 dependencies it's been adopted by all major distros. Before systemd, a difficult bug was not always the fault of one newfangled opinionated and unavoidable package.

I read the first blog post about systemd when it was written, and thought it was really interesting. Then I read the second blog post and thought "eh this is getting a bit complicated". And since then of course systemd has relentlessly taken over and added much more ... so of course practically any low-level system bug, randomly distributed, has a really high chance of being in systemd. And again, it does lots of stuff that was never needed before - in this case it's un-sharing a mount namespace for extra isolation of a small trusted service, which is overkill and complicated, and this struggle to figure out what the hell happened seems particularly unnecessary.

>And due gnome3 dependencies it's been adopted by all major distros.

That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files. That is very clear from nearly all mailing list discussions.

Also, people can and do use gnome3 without systemd. (https://wiki.gentoo.org/wiki/GNOME/GNOME_Without_systemd).

> That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files.

Or rather, because unit files work the same across all systemd-using distros, they can actually be upstreamed to the application developers and distros just package them, which is a huge time-saver compared to having to maintain init scripts for hundreds of daemons.

Makes sense.

timesyncd is a network connected daemon. Why would you not isolate it as much as possible?

The job of synchronizing the system clock is not a job that belongs to the init system. It is a job that belongs to people who can write and debug the very complex cases of NTP.

The init system simply has to fire it up, take it down, and report on whether it's still running. Any further elaborations are features of the init system.

All the problems mentioned here were failures of the init system in doing things that the init system should be doing, but were hard to diagnose because this particular init system does not and never has put "easy to debug" as a high priority item.

"as much as possible" would be to put it in a virtual machine, or actually separate physical hardware, maybe in a multi-million-dollar bank vault to prevent physical tampering.

Then there's a reasonable amount of isolation, which would be to run it under its own user, which can only write to one or two directories, and can't read user directories or non-world-readable system config files. This is already common practice. (Creating this user at package install time, or in the initial base distro config, is completely sufficient. And simpler.)

"as much as possible" is really not worth it (and sucks resources away from implementing reasonable security practices elsewhere).

It's disappointing to see the top comment here condoning ad hominem attacks.

But if you do believe that the identity of a speaker is more important than the arguments they make, note that the author of this blog post has a history of praising systemd (e.g. https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdRight) There is no anti-systemd bias motivating this post.

>>condoning ad hominem attacks

I'm not condoning ad-hominem attacks. In fact I am pointing out that often it is systemd itself that it the victim of ad-hominem attacks, which is to say that often people attack systemd, not its behaviors.

Systemd is not a hominem - it's a piece of software. It doesn't have feelings.

It's understandable that people don't like critique, especially delivered harshly as it so often is on The internet. But that only applies to those who worked on systemd. What I don't understand is why it seems to have fanboys.

Because every opinion can be used to build groups and people like to belong to groups, so you have a systemd fan and hate group.

But you wrote:

> I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

There is nothing wrong with examining the motive behind someone's argument. That's very different than an ad hominem attack.

That is literally an ad hominem attack: https://en.wikipedia.org/wiki/Ad_hominem

later parts of the page claim the ad-homs are legitimate. Most people would consider a claim of "ad-home" as implying illegitimacy though.

In this case it is literally the definition of the fallacy. The argument was not addressed at all. Not a single word of the post went to even acknowledge the content of the article.

Whenever I see a post that does nothing but dismiss an argument based on who made it, I do question the poster. There is a chance it is warranted, but in those cases evidence is at least presented. This post didn't even offer that.

As kemitche says, you have incorrectly said that I attacked the blog post author.

There are so many systemd haters out there that I immediately wonder about the motives when I read a new attack on systemd. Pointing this out is not attacking the author.

"I'm just asking the question" is hardly a valid defense against ad-hominem. If you're wondering about it and don't take the effort to research then that's your problem, not something you should share with the world. If on the other hand you have done the research, but can't be bothered to share the results the question is again why share the question with the world without sharing the answer?

> I like systemd, it's a pity some vocal people aren't enthusiastic about it.

To give some context as to why, for non-C++ developers systemd makes things very difficult to debug and understand in general.

Moreover the biggest issue /I/ personally have is the fact it was essentially forced on me by the mainstream distro maintainers.

When you discuss the merits and fallbacks of systemd it’s common for people to either try to discuss its merits only against sysvinit. Or they imply that issues that have affected me personally are not such a big deal and I should be quiet because: progress.

It’s a frustrating and exhausting position to be in. It’s like everyone shoving Windows registry on Linux down your throat, with all the opaque, obtuse and stifling qualities that would come with such a solution and telling you and your ilk to be quiet and suck it up.

Edit: I’m getting pounded by downvotes. But I guess offering any kind of other opinion about systemd for any reason isn’t met with kindness. However I can say this: neither google nor amazon use systemd to host AWS or GCP. At least consider why that is before you bury your head in the sand.

Distro maintainers didn't force you to do anything. They made a decision about how to allocate their limited resources. And we're past the days of Linux being a primarily volunteer OS, but a fair proportion of distro maintainers are still volunteers. They are not at your beck and call. If you want to have a Linux distro without systemd, there's several. If none of them meet your needs, pitch in with time or money to improve them. The purpose of free-as-in-libre software is that you can do the work yourself, not that you can expect others to do the work for you.

False dichotomy. I pay redhat a lot of money, short of paying people full time to maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything.

Some decisions cannot be unmade. Short of starting from scratch with gentoo (which is a Herculean effort at scale) I’m not sure what you’re saying. I cannot be disgruntled at choices because distro maintainers time is worth more than mine? Fair. But how many people have to be in my shoes for it not to be fair.

> maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything

Not RH-based, but have you checked out Devuan (systemd-free Debian)? I've always valued RHEL's (up until 6) stability, but I'm seeing Debian/Ubuntu becoming more and more used in enterprise roles where before these had already been strong in web roles. I'm also seeing a need for a systemd-free minimal O/S for container (not container host) roles.

> False dichotomy. I pay redhat a lot of money

Well why are you paying for a distro which doesn't meet your wishes then? Why not choose another one?

The company I work for runs Gentoo at a global scale...

You live in a better place!

Then you either make significantly more money than we do or are able to allocate resources to that project as it is key to your business strategy. The main focus of my company is to make Video games. Not maintain operating system distributions. That’s why we pay redhat so much.

I think Gentoo scales very easy. Especially in enterprises.

One thing I like about Gentoo that no other mainstream distro has is something like this https://packages.gentoo.org/packages/net-proxy/haproxy.

That page says that you can install any version of haproxy that suits you. You don't have to upgrade/downgrade the OS just because you need an older or newer piece of software.

But, as always, no distro is perfect. Gentoo has its issues too.

Systemd is straight C, not C++.

> C, not C++

The two wretched sisters.

> Uses UNIX-like OS

> C/C++ bad

> writes C/C++ every day.

> C is is stupid and breaks things

> C++ has serious mental health issues

spins finger in the air the difference is immaterial and irrelevant in this context.

A huge part of GNU/Linux is written in C. A small part is written in C++. I would accept the argument that C++ is less accessible than C to people maintaining Linux systems.

Based on this alone I would say that yes, the difference is material and relevant.

pish posh. It could be written in Egyptian and it would make no material difference to the ease of debugging for most people.

The number of people who are C++ programmers and can deal with strace vs. C programmers who can deal with strace is really that significant? I don't buy it. The two groups intersect significantly.

The point is that other people who are not C or C++ programmers have to deal with this kind of stuff from systemd.

This is just pedantry for the sake of it.

Systemd would be only marginally harder to debug if it was in C++ or heck, even rust, or go.

No one is opening the source code here.

> Moreover the biggest issue /I/ personally have is the fact it was essentially forced on me by the mainstream distro maintainers.

So pretty much the same as any other decision distro maintainers make, if you disagree with it then.

Most things can be replaced. Not systemd.

Prefer postfix or exim over sendmail? Sure, completely replaceable. Prefer another cron system, Logging system. Prefer another /anything/ and it used to be possible to swap.

Not any longer.

The author has gone into a lot of details on the issue so your question about whether 'this is a general issue' is answered quite clearly in the article and the follow up itself, including a link to Poettering's rationale and write up for dynamic users.

Characterizing a bug report with a detailed write up and follow up solution as 'attacks' and 'bias' is inexplicable.

Yeah, it's more like "Any time someone presents a problem with systemd, someone else will claim politics."

It's kind of fascinating to watch software become so polarizing. Like, it's technology. It doesn't inherently have feelings or bias. But we make it so.

You obviously didn't read the article before deciding this wasn't systemd's fault.

The bug is the lack of any error handling in the DynamicUser feature of systemd. The original failure is not logged anywhere - and launching timesyncd just goes ahead even though the setup for it fails completely. Naturally timesyncd then fails to start.

(Or perhaps the bug is the whole DynamicUser feature. From reading the bug report, it looks like it will fail any time anyone has a user-only FUSE mount point on their system...)

The other part necessary to trigger the bug is that uid 0 is not root and is not able to read the user-only mounts. root would still be able to read the user-only FUSE mount.

User 0 is root, and it is not able to read everything on other machines — as has been the case for decades.

Perhaps I misunderstood this line then?

> The machine that this happened on is an NFS client and (as is usual) its UID 0 is mapped to an unprivileged UID on our fileservers. On this machine there were some FUSE mounts in the home directories of users who have their $HOME not world readable (our default $HOME permissions are owner-only, to avoid accidents). When systemd was setting up the 'slightly modified mount name-space' it attempted to access these FUSE mounts as part of binding them into the namespace, but it failed because UID 0 had no permissions to look inside user home directories.


My reading is that (a) systemd can't read home directories on the file server, because clients understandably don't get root permissions there; and (b) the reason systemd was trying to look in those directories is that they contained (non-root-users') FUSE mount points.

The follow up article (https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...) explains the underlying problem.

Basically, to get around root-only readable directories, a special mount namespace is created when Dynamic Users are used. That fails when there are mount points that systemd can't access, which was due to FUSE filesystems mounted inside owner-only home directories, combined with an NFS mount.

This is the second time in a few days that I've heard about systemd-related user / ownership problems. The other was related to when a systemd-managed socket needs to be owned by a user that is created by a daemon that relies on networking to already be up (ldap / activedirectory / whatever). systemd wants to create the socket at the same time as networking is starting, so the user doesn't exist yet -- and there isn't a simple way to tell systemd to delay creating that socket.

Instead, the post-start script of the user daemon can chown the socket, and services relying on it will need to be started after that -- and there is no generic way to say to start a service after the auth provider is up, so you have to be specific, making the dependency fragile.

> and there isn't a simple way to tell systemd to delay creating that socket. … and there is no generic way to say to start a service after the auth provider is up

That's what

is for; just stick that in the [Unit] section of the .socket file. See the systemd.special(7) man page for more information.


Edit: However, there is an interesting problem related to that: when you use socket activation for the user-providing daemon itself, you'll get a boot deadlock. For instance, one of the common LDAP solutions, nss-pam-ldapd, uses NSS and PAM modules that use a local socket to talk to a daemon (nslcd) that handles actually speaking LDAP. If you adjust that daemon & its service file to use socket activation (a trivial patch), the system will deadlock! Systemd will start the nslcd socket, but then when systemd starts dbus.service as a special hack it will then call dbus_init() to register itself on the bus. This will cause dbus-daemon to use the NSS module and hit the nslcd socket. Since this happens very early, systemd probably hasn't started nslcd itself yet. Because systemd is single threaded, and it's waiting on dbus-daemon to reply, it won't be able to handle the request to start nslcd, and the boot process deadlocks. Eventually, something will time out, and it will hobble along in to a half-working system. You can hack around this by manually ordering nslcd.service before dbus.service.

That should be doable with a single additional line, IIRC


By itself that would only schedule it before dbus.service if they are both queued to be started; but since we'd like to be using socket activation to lazy-start it, you'll also need to make sure it's queued when dbus.service is queued:

That's what I meant by "manually ordering nslcd.service before dbus.service".

It might be config and security fragile, but ntpd is a safe bet.

When religions preferences come up, I prefer s6 because it’s scalable like daemon tools/runit. Old-school upstart was meh and ancient style rc scripts rarely did process supervision.

Debugging and simplicity are prime requirements for init’s.


PS: Logs aren’t files... they’re streams of structured messages often, that are unfortunately usually destructured into text lines. Log-rotation, logging to local files are horrible kludges that doesn’t scale for high-volume servers... log-structured log device, log sourcing or time-series structured db are more natural fits. Keeping log data in structured form instead of broken up in arbitrary line formats makes custom parsing unnecessary and log mining/transformation (ETL) much easier and usable. “Grepping” is simply looking at a text view of desired fields like a full-text db search. IOW, historical logs become random-access msgpack/protobufs streams while app logs are just always appending... and one string per log message for backwards compat, with a “logger”-type command for logfile-less 12-factor apps.

I'm repeating myself, but the best way to use systemd is as an init system only, and then staying away from those fancy features like dynamic users (wtf) or socket activation. Especially avoid all that bolted on crap like timesyncd, resolved, etc. Yes they integrate a bit better with systemd, but no you don't need this better integration, especially since these tools are usually less configurable than their bigger brothers and are often more buggy just because they are not that battle tested yet.

> systemd will tell me to reproduce on the latest version, Ubuntu will ignore it as always

Amen to that. Distro bug reporting/handling is a pretty sad story on its own, and so far dealing with Ubuntu has been the worst. The most insane instance was when I reported a bug in their ati x11 driver. It was about two weeks after a new non-LTS came out, I even tracked down the bug and found that it was already fixed upstream only 2 commits after the one they shipped. So they didn't need to do anything but apply that single patch. No response ever. 6 months later the next release just shipped newer drivers and the problem was gone, but hadn't I recompiled the fixed driver myself I wouldn't have been able to switch vt for half a year without crashing X. I really like open source and all and make an effort to produce usable bug reports to help improve things for everyone, but instances like this make me wonder what the f*ck is wrong with people maintaining such projects. <end unrelated rant>

> Especially avoid all that bolted on crap like timesyncd, resolved, etc.

You're pretty much stuck with systemd-resolved when you use systemd-networkd.

And systemd-networkd is a godsend. It's the first network setup tool that I've found that is:

- full-featured

- simple to set up for the 99% usecases like "just do DHCP whenever a physical interfaces goes UP"

- available and works the same on all major distros

- not a Lovecraftian monstrosity like network-manager

Agree, this is the only thing that's almost as well-designed as the init part of systemd. Unfortunately, you still have some weird edge-case every now and then with it that's just not possible to set up, and then you're screwed because you cannot just plug in in a little shell snippet somewhere.

In that case Ubuntu was a middleman whose interests were completely disaligned with their customers', which probably means they in reality felt obliged to someone else.

You tried to voice. Depending on your loyalty, you may think of exiting [0].

[0] https://en.wikipedia.org/wiki/Exit,_Voice,_and_Loyalty

Given things like this: "The default configuration is defined during compilation" [1] and the fact that you'd think 'timedatectl status' would tell you the NTP server you're supposedly trying to sync to (spoiler alert: it doesn't), I'm unsurprised of any further atrocities committed by systemd.

[1]: https://www.freedesktop.org/software/systemd/man/timesyncd.c...

> Given things like this: "The default configuration is defined during compilation [so a configuration file is only needed when it is necessary to deviate from those defaults....]"

I don't get it. What's so bad about that?

It means that a configuration file on two similar machines could have radically different behaviors.

That's always the case due to different software versions.

This implies they can be different with the same versions.

You might want to cast your mind back to https://news.ycombinator.com/item?id=9809427 .

The practical result is that on a machine without a configuration file, you can’t determine what NTP server it’s trying to use without having access to the source code (or some compile time include) or doing some other trickery.

That is a ludicrous design decision and is indicative of the developers having zero empathy towards their users.

Once you could just maintain some file from etc in a version control and setup a machine predictably with a script three lines long

Then it came puppet chef and a shitload other stuff because ‘devops’ which make the task easier once and then fuck up your environment at every single upgrade changing defaults, changing dependencies and changing the environment you run

I swear what could be done repeatedly the same way for a decade now it’s a frail mess that requires yearly fiddling to keep it running and current

No wonder companies need so much devops people/time these days, they need them to keep following the frail shit that’s dropped into stable distro.

Systemd is just one more thing that you have to reconfigure at every release because changes defaults subtly.

systemd's job is to start daemons. It can be reasonable to decide dynamically when to start them based on which others are healthy (maybe you care more about speed than determinism on a laptop). But there's no reason to choose a uid and mkdir dynamically. You want these errors at install time, not at the box's next (probably unattended) restart.

Everything that would have been "a weird problem with Foo Daemon's init script" we now call "a weird problem with systemd".

So you think that systemd-timesyncd's service file shouldn't choose the uid at service-start time.

That's not really the issue though. The issue is that for service isolation reasons (related to choosing the uid at service-start time), the service file wants to run systemd-timesyncd in a private mount namespace, but it is failing to properly set up that namespace. That's a more general issue, as there are a number of things that could trigger running the service in a private mount namespace; not just DynamicUser.

If SELinux doesn't cut it and you actually need to jail a badly-behaved service, "set up a jail in this specific way and exec this process in it" should be a tool to be invoked by an init script or a unit file or whatever. unshare(1) looks like it's halfway there.

That's exactly what's going on here. The systemd service launcher is doing the same thing as unshare(1); calling unshare(2). However, by "default" after you `unshare --mount`/`unshare(CLONE_NEWNS)`, changes made to existing mounts can propagate to the parent namespace, so then you need to do something like `mount --make-rslave /` to complete the isolation.

As I understand the bug, the problem is in systemd's version of the `mount --make-rslave /` step, which (for a reason unknown to me) tries to stat (as root) the mountpoint first, and if the mountpoint is on a remote FS that might fail, and systemd isn't correctly handling that.

Actually, wanting this stuff not at install/configure time is one of the primary driving forces behind the mechanism. Lennart Poettering notes that it moves the relevant tasks out of package pre/post-install/deinstall scripts and into service startup and shutdown.

Ironically, the alternative approach here is more descriptive and less imperative package management.

Witness manifest files on the old BSD pkg system. Rather than every package maintainer writing an install/deinstall script to invoke the relevant account database tools, packages place a @newuser or @newgroup directive in the file and the packaging system handles creating and deleting the account at appropriate times.

* https://man.openbsd.org/pkg_create.1

> systemd's job is to start daemons

But that's not all it does and the dependencies of systemd alongside making systemd a dependency, results in a very different observable reality.

Personally, I think even having to choose a UID at install time or dynamically is a pretty sad indictment of just how broken the user system on Linux (and pretty much every UNIX-like) is. It leads to a huge amount of incidental complexity where there's a much simpler way: User IDs should just be randomly chosen UUIDs -- either generated by upstreams or by the distros. Any metadata attached to the user UID could just be preinstalled for all possible daemons[0] -- and it should be separate from the mutable /etc/passwd stuff.

Obviously there's so much inertia in the existing system that it could realistically never happen at this point, but one can dream, right?

[0] Kind of like e.g. /etc/services but with UUIDs.

One can, and many people have, over the years.

Some of the more interesting ideas that people have had, in my view, have been:

* ID systems that introduce hierarchies, allowing (say) a user to create multiple sub-users (one for running the WWW browser, one for running the office suite, one for running the chat program, ...);

* proper nonce ID creation with segregation guarantees (c.f. nonce SIDs in the Windows NT world); and

* IDs that are reference counted, accessible via descriptors, passable from process to process via descriptor-passing mechanisms, and explicitly supplied in system calls for opening/creating things.

Except of course if you want multiple instances of the same daemon

Um, mix in a second randomly-chosen UUID?

My point wasn't so much that I had the perfect solution and that I'd thought of everything. It's that there are much better systems and that we should actually strive to get there.

And do the logging. My only gripe is that it logs in a binary format requiring special tools to read the logs. Otherwise systemd is a marvel.

I have switched between ntpd, openntpd, timesyncd and a couple of others I might have forgotten, and the only one that's worked reliably is ntpd, but I was trying to stay off it, because historically it had quite a few vulnerabilities.

I manage hundred of stand-along edge devices, mostly hardened PCs who have to bear up to 50 deg celsius environment from time to time; I've observered 5-10 times that RTCs were wrong by 5-10 hours in some direction after a power-loss, and openntpd fails miserably, timesyncd also not-so-good. ntp does ok. It's been much better since I started updating the rtc every 10 minutes from the system clock - most distros update from rtc on startup and to rtc on shutdown, but if the shutdown is after 6 months due to power loss, you're left with an rtc drift, which -- in harsh environment -- can be significant.

I am right now testing chrony, which seems to be a winner, but it's too early to tell.

You could always use htpd [0]. :-)

[0] http://rkeene.org/docs/oss/htp/htp.pdf

For various reasons, the powers that be mandated ntp; but thanks for the reference.

> this server drifted more than half a second out of synchronization before I found this issue entirely by coincidence

Did `systemctl status` not report "degraded" for this failure? Maybe the problem is that there was no monitoring set up for failing services?

I'd suggest that the time drift was tested directly. 0.5 seconds for most things is a reasonable threshold

I'm pretty sure, by the way the OP described the problem that both the service state _and_ the product of the service are being monitored

The more I use Systemd the more I actually like it. It has some weirdness to it and there are parts that aren't really useful.

Journald is pretty stupid when you think about it. Either you're setup is really small, and logging to text files by defaults would be simpler to use. Alternatively your setup is large enough that you have a log-host/log-analytics, and logging to text files would be a sane default, as you need to convert the binary log to text anyway to do log-shipping.

It hard to argue that writing startup script hasn't improved. I would much rather write a service file than an up-start script.

I'm always amazed at what systemd can do that I don't know about. I had no idea about timesyncd. How did anyone find out about it? Is there a super awesome book or video of systemd "hacks" that I can read or watch? I think I've read the official docs but I don't think I really learned anything, but that was a couple of years ago.

Edit: Wow, just realized that my debian laptop is running timesyncd. I run the same debian version on all my servers too but I installed ntpd on them. I guess I need to check how their conflicting, if any...

as a start, try "man -k systemd"

This blog is full of explanations to understand how systemd can cause issues (like https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...). This kind of information is often omited from official documentation but is so valuable when investigating bugs.

TL;DR: systemd has bugs.

Mountain out of a mole hill.

Lack of debuggability? It's not like this is some threaded golang program you'll lose your mind attempting to make sense of in strace. systemd is a single-threaded C program, if you consider this as not debuggable it's your failing not systemd's.

There are valid complaints to be made about systemd, but from what I've seen the crux of the problem is generally the haste with which its features are conceived and developed. For the most part it's a relatively conservative codebase, in a simple and easy to understand language without much of a runtime interfering with relating the code to the underlying system calls performed.

This project doesn't have very many core contributors. It's somewhat incredible how much has been achieved with so little. But as a result, it's not perfect.

The lack of contributors has a lot to do with Lennart's (apologies if I misspelled his name) attitude early on in systemd's development.

I have no idea if it has changed, as I gave up trying to get things fixed.

The one issue I still have with systemd is the way it handles the kernel's command line.

I work with Chromebooks a lot, using software that isn't ChromeOS. The cool thing is that they can use FIT images so you can load multile kernels and device tree bindings into one image. So I can use the same sdcard on the Acer tegra Chromebook, Samsung's Chromebook, as well as the ASUS Flip Chromebook. However, to boot from sdcard you have to enable developer mode. When you enable developer mode, it adds a flag to the kernel command line "debug" - systemd sees that and thinks you want systemd in debug mode (wat?) Unfortunately there is no way to override this. I've tried passing the log level as info, which sort of works, until journalctl starts, and IT parses the debug flag in the command line and ignores the flag that sets the log level to info.

No amount of explaining this helped and I (and many others who have had various issues) gave up on trying to help make systemd better.

The only workaround to this is to stop using 1 sdcard amongst all of my Chromebooks and use individual sdcards per machine that uses a u-boot built for each Chromebook that does not pass the debug flag.

I like systemd and many of it's features are helpful, but when you run into issues, it's almost always an uphill battle to fix them, even when you provide a patch.

> adds a flag to the kernel command line "debug" - systemd sees that and thinks you want systemd in debug mode

A reasonable assumption, given that systemd is taking care of the whole OS, and if you want the kernel in debug mode, you’d reasonably want the system in debug mode too. But it’s not obviously correct, either, and ultimately it boils down to “who owns the kernel command line flags”. There was some kerfuffle about exactly that between the kernel and systemd people, but they eventually worked it out – I think systemd no longer tries to assert ownership of the "debug" kernel command line parameter.

Ah yes, debug parsing. Didn't that attract Torvalds' ire some time ago when it was found that it made systemd flood the kernels log buffer with "junk" data?

Then again the other guy in the systemd team, Sievers, have a long history of pissing the kernel people off with out of the blue unilateral decisions.

I just worry when their buddy GKH is given the reins of the kernel when Torvalds steps down...

Lack of contributors? systemd had over 300 different contributors the last 12 months. That makes it in the top 2% of open source projects (tracked by OpenHub). Which is very impressive for being such a low level component. https://www.openhub.net/p/systemd

Core contributors. Not overall contributors. I should have specified but figured people could read the context based on the comment I was replying to.

Anyone who follows the systemd-devel mailing list and the systemd github repo knows that there's just around four maintainers regularly active to varying degrees, including Lennart.

Come on, it's not like shell scripts are debugable.

Err, they are very easy to debug. You can have them print out every line as they execute and follow along what was done. Prior to systemd, if there was ever anything critically broken, I could step through the entire startup from the initramfs and bring up the system line-by-line or script by script as I needed, and see exactly what was at fault.

With a script-based startup, the entire system was understandable by anyone with some rudimentary knowledge. It wasn't complicated, and it worked well. It did have its shortcomings, but the accessibility to modification and the ability to effectively debug were not amongst them. In comparison, systemd's behaviour is a black box with the unit files as input. The actual internal state is opaque, and it's inaccessible even to experts.

> this server drifted more than half a second out of synchronization before I found this issue entirely by coincidence

Sounds like time synchronization is not particularly important for you. We have alerts set up for high clock drift that trigger at abs(drift) >= 300ms.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact