Am I the only one who is getting fed up with everything under the sun having their own DNS resolver and DNS cache? It's border-lining madness right now.
Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache.
So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat.
But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance!
Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid.
And now systemd adds to this. What the flying fuck?
All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy.
Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again?
Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.
The short answer is: the glibc DNS interface sucks balls and systemd wanted extra features.
The long answer is:
Often, systems do not come feature-complete. There are many cases of a platform, or OS, or service, or framework, or some other piece of software being released without every possible imaginable feature being implemented. When this happens, some other software may need to re-implement some core piece of technology in order to add functionality to it. Usually care is taken to not interfere with other systems or cause additional burden to the user. In the case of DNS, our use of DNS and different networks simultaneously requires additional functionality in order to process requests independently and efficiently. As the DNS protocol itself has an inherent flaw allowing for cache poisoning, it's non-trivial for people to implement without running into this security hole. And as someone else mentioned, the kernel and glibc don't have their own DNS cache.
I didn't downvote you for the rant because you weren't being mean or insulting and you genuinely seem to want to know what's going on. But next time you feel yourself posting out of emotional reaction, I recommend typing it out in a draft and save it for a half hour, and see if you might want to reword it later to be less angry/ranty.
> And now systemd adds to this. What the flying fuck?
While I agree with you that these layers of caching can lead to headaches, Linux doesn't have a built-in cache (and distros often don't provide one by default). Systemd would be one way to provide this, instead of other daemons that have been used in the past, like nscd, dnsmasq or bind. So it's not adding yet another layer, just providing the layer at the local host.
As for technical reasons, I'm not sure what you want to hear, each layer implements a cache for performance and/or efficiency. No one throws in a DNS cache just because they feel like it.
I don't understand your comment. a DNS server is typically also a cache, and this is what systemd was attempting to provide, without the public server part. It's the same thing that distros were using e.g. dnsmasq for.
What does the kernel have to do with this? No one is proposing that the kernel should have a DNS cache.
GP says 'Linux doesn't have a built in cache'. I presume (hopefully correct) that he means 'the kernel', since Linux doesn't have anything 'built in' that you don't run as a user process. The kernel of course has plenty of stuff built in but (fortunately) it tries hard to stay away from things that can and should be done in userland.
Systemd is a user process and now tries to replicate functionality that we already had a perfectly good solution for.
Since a DNS server is typically also a cache we already had this functionality (and more), simply make it listen to a local port and nothing else. That way you don't need to resort to rebuilding things we already have with incompatible interfaces.
Linux hasn't been just a kernel for a while; it's a kernel with some associated userland utilities that are developed together and normally expected to be present. That's been the case in the filesystem part for a long time; the userland fs-related binaries are developed in sync with the kernel side. Systemd is doing something similar with e.g. cgroups.
Linux develops fewer things this way than the BSDs typically do, outsourcing a lot of userland to other projects like GNU, but it doesn't literally develop only kernelspace code. Imo it's generally an improvement; the situation with trying to get things like cgroups to work together with process management previously was terrible.
This is absolutely true, which is why linux is as large as it is and why you need certain stuff in order to bring up a functioning linux system. As for the filesystems they are modules plugged into the kernel (linked directly so you can boot our kernel straightaway or through a ram disk kludge as modules in linux, not the most elegant solution because technically that duplicates a buch of stuff but nobody seems to care too much) and a bunch of supporting programs, this has been the unix way since time immemorial, I think even the first unix I ever used had fsck.
The reason those have to be kept in sync is simply because you don't want to run fsck on a filesystem that was built with a newer kernel that possibly has datastructures that your fsck (or other filesystem aware utility) accesses directly without going through the kernel. Essentially such programs re-implement a bunch of kernel functionality because they access file systems that are not currently mounted.
Systemd on the other hand (in the case at hand) replicates the functionality of another user process that we already have, know how to use and have hardened over many millions of hours of use. That seems folly to me, it duplicates functionality, makes troubleshooting harder and increases the size of the codebase without actually improving anything.
What's next? The framebuffer? Keyboard interface? TCP stack?
There are no really hard reasons why those could not be user programs. It's fine with me if we make linux a micro kernel but let's do that in a way that we do not need to revisit the last decade or so of security issues and ill thought out interfaces. Once we start to need things like DBUS and X in order to bring up a server we've really lost the plot.
Not as much as you'd think, actually. The CONFIG_VT subsystem has been a bane on the Linux developers for a long time.
A userspace replacement entitled kmscon has already been around since 2012, by David Herrmann. It's a totally separate project.
consoled is a minimalistic fallback done by the same person. I'm not sure if it really needs to exist, but replacing CONFIG_VT in of itself isn't bad. Most people will probably still have kmscon primarily.
I had great success using that. Both installing it on my own or run it part of Ubuntu's base install.
(And the answer of "but it already provides everything + kitchen sink, might as well provide DNS caching too" would probably not be what most people would want to hear).
Exactly, so imnsho this is simply wrong and should be un-done. Systemd should do those things that can only be done at init time and/or those things that need to be there and no more and leave the rest of the lifting to those tools that we already know how to use and that have stood the test of time. Otherwise we end up with 2 sets of incompatible tools with one of them not only re-inventing the wheel but replaying a whole bunch of security issues from the past on top.
This is the worst case of feature creep in system land in a long time.
That is a good architectural design, but is in direct opposition to the projects mission statement of "embrace, extend, ..." you guess the third word.
If a group is actively trying to ruin something, noticing that they're trying to ruin things isn't going to stop them. That's a useful strategy with accidents and friendly groups, but not with actively aggressive oppositional destructive groups.
Mostly I'm just angry about having to waste the labor hours moving everything to Freebsd from Debian. Its interesting, but not exactly productive.
Funny, that was exactly my response to all this, if Linux insists on copying the worst of windows in the future then I'll be more than happy to go with FreeBSD. I'm a bit rusty but I'll survive.
Last I looked FreeBSD used rc.conf but it's years ago.
I don't see FreeBSD's init trying to absorb most of the userland system utilities.
Whether or not it is in one repository isn't really the issue at hand here. It is that a bunch of desktop related stuff is worming its way into the systems to the point where you'll be running a whole slew of new, buggy and sloppy code on your servers if you're not very careful just so that the systemd folks get to scratch their NIH itches.
Everything in one repository, yes. Tightly coupled, no.
And even when in one repository i think you can grab the BSD utils without getting the kernel and whatsnot in one big tar-ball.
If you try to grab say just the dns daemon from systemd you can't do so directly. you would have to grab the tar-ball for the whole of systemd, and then attempt to extract just the parts dealing with the dns daemon.
Damn it, LFS have found doing this for udev so demanding that they have switched to eudev for their non-systemd edition (yep, they now have their own systemd edition going alongside the "traditional" one).
I think we've reached the point where we can start being pedantic about what "Linux" means.
I don't think the kernel developers care that much about what systemd is doing (except for the notorious clash about systemd interpreting a debug option meant for the kernel and crashing the boot...), so you can't say the kernel is copying or doing anything.
The distros, however, are possibly the ones you want to blame for pushing and supporting systemd. Which is actually funny because the way it's going, systemd is laying foundations for a new kind of linux distributions to which distributions will certainly have to adapt if they want to keep it.
Right, because it drives people from linux to FreeBSD. Well, here is my take on that: the reason I run Linux is because my desktop runs linux, and I try to keep as little knowledge about systems in my head as I need to get through the day in one piece. Since FreeBSD lost more and more ground to linux when it came to supporting the kind of hardware that PCs (desktops) ship with I ended up standardizing on Linux. I know full well that if I switch back to FreeBSD I'll end up running it on my desktop as well and that's when I will probably regret my move. Let's hope the situation has improved over the years.
Systemd is a cancer, it will end up killing Linux if this goes on unchecked. At least Torvalds seems to (most of the time) know his stuff, the way they replaced the memory management left me with a bad taste but in the end it worked out. What's happening with Systemd is absolutely unpalatable and I hope that Redhat and other distros take note, there are a lot more Linux systems on servers than on Desktops and if they believe that expending all this effort on Linux on the desktop so they can better compete with Windows they might just lose their dominance on the server because of it.
> I hope that Redhat and other distros take note, there are a lot more Linux systems on servers than on Desktops and if they believe that expending all this effort on Linux on the desktop so they can better compete with Windows they might just lose their dominance on the server because of it.
Red Hat is developing systemd. It has nothing to do with making Linux palatable on the desktop, but a severe case of NIH, which to be honest is hardly new in Linux-land.
systemd won't kill Linux. Linux development is out of the hands of hobbyists and enthusiasts now and corporate users and developers don't care about the holy wars. Red Hat ships systemd, Red Hats users will use systemd and that will be that.
RH is pushing heavily into cloud computing, and appears to use systemd to do so (notice the amount of containerization talk there is in relation to systemd these days).
Their existing server customers are likely to be sitting on RHEL7 for quite some time. And i suspect that the workstation customers, mostly military as best i can tell, are in part pushing for systemd. Logind etc after all is heavily focused on multi-seats.
I'm one who you speak of. Systemd to me is just a scarily large attack surface. Working through my servers slowly replacing them with FreeBSD. ZFS raid root out of the box on install! That one really impressed me.
>Mostly I'm just angry about having to waste the labor hours moving everything to Freebsd from Debian. Its interesting, but not exactly productive.
It's scary for me, because everything I've done has been in a Linux, but it's also educational for me, because everything I've done has been in a Linux. If the systemd invasion results in a lot of attention going to the BSDs, it's not entirely a bad thing.
The biggest change will be that you might find it sometimes harder to get exotic hardware to work and that BSD tends to put stuff in different places. Most of your other unix knowledge (userland) will transfer just fine, the sysadmin stuff is a bit harder but learning another dialect of unix is along the lines of learning Spanish when you already know Italian.
FreeBSD always was targeted on running on servers. So if that's where you will run it then you should have minimal issues with drivers. If you plan to run it on desktop it might be more challenging.
As suggested reading "man hier" is a great way to understand the filesystem layout. I personally also really like clear separation between the system and applications. For example all ports/packages are installed in /usr/local including configuration files (in /usr/local/etc).
Read the hier man page of your chosen BSD, first. It is a great eye opener. From there OpenBSD and FreeBSD have some great documentation (not sure / no slight intended to NetBSD or Dragon Fly BSD as I have not used them much).
But that's not in line with what systemd intends to be. It's meant to be the standard suite that you sort of plug in as a middleware between GNU and Linux.
Your proposal would require a radical paradigm shift as to the fundamental project goals of systemd, in which case it'd probably become something like uselessd.
There are already excellent solutions out there that have been battle-tested, such as unbound or even dnsmasq. Best as I can tell, resolved has no good reason to exist.
Actually Unix out of the box (unlike for example windows) does not have system level dns cache, people often setup things like bind, unbound, dnsmasq, nscd etc.
Systemd here tries to be the official standard system level caching.
I agree though, about your extra caching sentiment to a degree. Actually as long the caches are RFC complaint (e.g. they obey TIL - looking at you nscd & java) or should be fine.
Every client operating system under the sun implements a DNS cache, its part of the intended design of DNS. This is just systemd's bikeshed of a function already available on Linux, via nscd.
Without caching, all DNS queries degrade to worst case behaviour, which is a recursive query involving 4-6 or more round trips, starting at the DNS roots (which, good luck finding them, since you don't have their addresses cached, which is exactly what the root hints file is).
As for invalidating the cache when network configuration changes arise, that sounds like a bug report for wherever you experienced it. Certainly I've never seen this kind of problem on OS X, nor any time in the past 10 years on Windows.
Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
Yes, that's the full commit message for what I think is the commit that adds the feature.
If true (that it is a 4 word commit message), that is a disgrace. Seriously, something as important as this should contain a lot more information, if not just a pointer to a place that provides it.
Lennart proclaimed therefore we all must follow. That appears to be how it works these days. No community, no discussion. Someone with commit access says bend over or fork it.
Actually fuck that. Which is why I'm sitting here on a FreeBSD machine.
I don't understand this attitude. Lennart has every right to write a DNS resolver if he feels like it. And you have every right not to use it. Nobody is being forced to do anything. (I'm not aware of any distribution currently using systemd-resolved by default, and you can disable it at build time and runtime.) So what's the problem exactly?
A couple of problems: (1) This resolver has security bugs which have been fixed in the existing daemons. That doesn't matter except that: (2) systemd utilities get an easier route into distros than competing utilities, even where the competition is much better or more secure. This really matters since Linux improves because of the Darwinian nature of the competition between the different components. You can't have one group of packages claiming they are the "one true standard", or this doesn't work.
2. Its another tentacle hanging off systemd. Some of us pay for our software (RHEL) so we'd in turn like to have some say in what we have to manage rather than prescriptive software development. We don't really get to choose to not use it because of the incestuous nature of the Linux ecosystem at the moment (I.e. Freedesktop/gnome/systemd).
3. Systemd is a buggy piece of crap. As are several of the "stable" bits of systemd I have encountered in RHEL7. I have quite a bit of experience with systemd and i really don't want it consuming any more things badly. Its been a complete pain in the arse to be honest. An init system has never been a problem until now. Its totally unreliable and incredibly difficult to debug.
4. Some of us get shit done, not use the ecosystem as a playground for egos and quote frankly incompetent twattery. This stuff can go on in the sidelines and mature independently, not knotted into the next "stable" drop because everything has suddenly been tangled with it.
Its a complete fuck up at all levels and I haven't even started on the technical marvels of introducing something with the complexity of COM into a process model that requires reliability.
"Nobody is forced to do anything" is a very lame argument, because it is quite obvious that systemd is effectively being forced on the Linux world, because it has become a dependency of Gnome. Other distributions besides Red Hat will make their life very difficult if they don't implement systemd as well, so yes, practically, systemd is being forced upon Linux users. You could of course continue the argument by saying that nobody is forced to use Linux, or even a computer, but it gets silly.
But the solution is not to complain, but to use /etc/hosts, run your own root and your own cache.
Any software that overrides /etc/hosts is not worth using, in my opinion. There are no "underlying technical reasons" for not letting the user control their own name resolution that benefit the user. Of course, there could be reasons that benefit someone else.
Sure, but your average application does not do a DNS query directly, it simply uses gethostbyname and depends on those local system libraries to figure out what to return.
I find the design of systemd-resolved to be very strange. It uses dbus to talk to glibc, and it seems to be a new, from-scratch implementation of a DNS resolver. To be clear, I don't really think it matters whether systemd-resolved is under the systemd umbrella, but I do think that the design has a lot of unnecessary NIH syndrome.
It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53.
Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation.
This is a perfect example of why the systemd approach of putting a bunch of disparate components under a single tightly-coupled umbrella is bad engineering. Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. A less-than-perfect attempt at implementing a caching DNS resolver will be insecure. For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
> This is a perfect example of why the systemd approach of reinventing a bunch of existing components under a single tightly-coupled umbrella is a bad approach.
Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver.
> Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well.
I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure...
Yet they're only attempting to do "one thing" so according to you they should have done it well.
> Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard.
This line oddly contradicts the line that directly proceeds it.
> For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
According to the previous line they aren't doing that one thing well...
The reason why a project with a bunch of tightly-coupled disparate components will have a hard time doing anything well is that an expert who could do a good job isn't going to spend their time working on an implementation that can only be used along with a bunch of unrelated baggage. They're going to work on an independent implementation that can be used by as many people as possible.
Take DJB. He's an expert and djbdns has a stellar security record. He's not going to spend his time trying to make systemd's resolver secure. Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.
> They're going to work on an independent implementation that can be used by as many people as possible.
So they would work on systemd then? Since it is set to have a far larger customer base in the near and long term than most independent cache resolvers (just see the list of distro's who have taken it up).
> Instead systemd is going to end up repeating all the same mistakes BIND has made. At least BIND has a substantial head start over systemd-resolved.
Potentially. But I don't think you've successfully shown that that is inherently due to systemd's design, complexity, or componentization. Instead it is likely due to systemd's relative immaturity of a codebase and that as discussed previously most cache resolvers have had a checkered history of security issues (and assuming systemd is "as bad" as everyone else, it will have security issues).
> So they would work on systemd then? Since it is set to have a far larger customer base in the near and long term than most independent cache resolvers (just see the list of distro's who have taken it up).
Is the systemd project passionate about DNS? Is the curl team passionate about protocols and files? Is the opensmtpd team passionate about e-mail?
I think two of the three are yes answers. The great strength of UNIX has always been people's ability to flock to places where their passion is shared and improve and learn (institutional knowledge) about their passions.
I don't think systemd will have passionate people for these extra services and it seems they don't have the institutional knowledge that others have gained since this is not a new kind of bug.
I think history favors projects, companies, and individuals that concentrate one something as opposed to big entities that try to be all things to everyone. This is how small companies are able to dislodge larger ones. Focus is important in anything. The argument is far from flawed, as limited counter-examples are hard to find.
> It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver.
Yes. And when it happened before, it was The Bug of the Century. Repeating that mistake is either stupidly incompetent or proof the systemd developers were literally born yesterday.
New development is expected to have some bugs, but it shouldn't sound like a tribute band playing the greatest hits of yesteryear. And they can't exactly fallback on the old standby of "oh, it's a legacy code base."
The init system is not vulnerable. The local dns caching daemon that you can optionally install and that shares the name and maybe some code with the init system is vulnerable.
It's like saying "package dnsmasq.deb is vulnerable, so sysv-init.deb is now vulnerable, they share an extension"
Pretty weird for Systemd to have its own cache and resolver. If you need a local resolver and a cache simply run a local DNS server rather than to re-invent the wheel (in a broken way at that). I'm sure they have their reasons but this is not the unix way.
Systemd is just not the Unix Way. It's some shit program that requires to be run as PID 1, and takes over the system in the forms of an octopus. And, you need some gods-awful binary program to even read their logfiles. Yes, it's like the Windows registry.
I've been thinking a lot about the Windows Registry (and /etc, etc) and I've come to the conclusion that the systems which get it the most "right" is the app model (e.g. Android, iOS, Windows 8).
You see systems like Docker containers, which try to isolate a complete application and its configuration files together, and really that is just a hacky solution to the lack of inherent silo-ing within the underlying design.
While silo-ed apps have their own limitations (e.g. two "apps" using the same configurations) most of them can be overcome with tinkering for the edge cases.
I'd never design a system in 2014 with either Windows Registry (too messy, impossible to manage), or /etc (too inconsistent with itself, also often results in orphan files).
The submitter doesn't state affected versions, but judging from systemd's NEWS file, the resolved component was first introduced in 213 and later dubbed as "a pretty complete caching DNS and LLMNR stub resolver" in 216: http://lwn.net/Articles/609740/
Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though.
Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.
> "a pretty complete caching DNS and LLMNR stub resolver"
That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive.
But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."
Of all the components of systemd that doesn't need to exist, resolved is definitely one of them. Why they're reinventing the wheel rather than allowing the user to set up something that works and is battle-tested, such as unbound, is beyond me.
I don't necessarily disagree with the heart of your statement, but it's worth mentioning that systemd does not require one to use resolved. The user is still "allowed" to run unbound, dnsmasq, etc., and the preferred solution will be a question for distribution maintainers and users to answer.
I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
Small comfort. When a project has the incredibly poor judgment (and, frankly, hubris) to write a caching DNS resolver without even the most basic cache hardening protections, and then declare it "a pretty complete caching DNS and LLMNR stub resolver"[1] instead of just leaving it to people who know what they're doing, it calls into question the rest of the project. This is a major reason why so many of us are wary of systemd.
Personally, I'd prefer that they concentrate on stuff that's properly within the scope of the project. This is way, way outside what the reasonable scope of the project ought to be.
Systemd... a very large mass of c code being written in a short amount of time with huge functionality creep. Yeah, this is going to end/begin really well. I swear all of the people commenting about it being a good thing are not seasoned developers.
Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache.
So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat.
But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance!
Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid.
And now systemd adds to this. What the flying fuck?
All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy.
Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again?
Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.