
Systemd: Don’t fallback to Google NTP and DNS - throw_faar_away
https://github.com/systemd/systemd/issues/12499
======
zaarn
This discussion is the worst to have to repeat every time.

This fallback will only be used when:

* You have no DNS and NTP servers assigned via DHCP (your router does by default in 99.99% of cases) * You have no DNS configured in /etc/resolve.conf * Your distro does not change the defaults to nothing or something else (almost every distro does) * You still have internet connectivity (somehow)

In most cases, these conditions would simply not be met because everyone has
DNS configured somehow. This isn't a fallback for when your DNS is
unreachable, it's for when no DNS has been configured at all anywhere
(including by distro upstream) but you still want to use the internet.

So in that rare circumstance, where your distro didn't change upstream
defaults and you don't have any DNS configuration, it will fall back to Google
DNS.

~~~
manjalyc
> /etc/resolve.conf

It’s /etc/resolv.conf, and systemd overwrites it, Network Manager overwrites
it (like every 10 minutes), dhcp overwrites it, etc. What do they overwrite it
to? Most just get it from the router even if the router has no assigned
servers. These days manually managing /etc/resolv.conf by hand requires
hooking into every service that overwrites it on most systemd distributions.
So really the people this change would affect are those with misconfigured
routers.

My issue is why they didn’t go with a more private DNS such as Open DNS or
even Cloudfare DNS. Google’s privacy policy is worse than either of the two.
It seems the questioner asked in bad faith but I would’ve hoped Poettering
would respond to the validity and dismiss the malintent instead of dismissing
the whole issue.

~~~
znpy

        It’s /etc/resolv.conf, and systemd overwrites it, 
        Network Manager overwrites it (like every 10 minutes),
        dhcp overwrites it, etc. What do they overwrite it to?
    

I still haven't understood where is the definitive place to set dns nowadays.

/etc/resolv.conf is routinely touched by stuff like NetworkManager and other
stuff. Systemd-resolved config seems like the correct place, but I'm not sure
how that plays with network-manager.

~~~
manjalyc
Universal practice is editing the resolv.conf file and then write-protecting
it with.

    
    
      chattr +i /etc/resolv.conf
    

This always works, but certain applications NetworkManager have their own
preferred way of doing things that I quite frankly do not care to learn.

~~~
kevin_nisbet
I think it get's a bit harder than this, with newer systemd supporting name
resolution over dbus. It's conceivable that some programs may move away from
libc resolvers using resolv.conf and start querying the service directly over
dbus.

I'm not sure I'm personally a fan, just saying it's something that might
become more common based on my read of the systemd docs awhile back.

~~~
vetinari
> with newer systemd supporting name resolution over dbus.

Not only that, with suitably configured nsswitch.conf, (e.g. with nss-
resolve), you might find out that your /etc/resolv.conf is used only by apps
that have their own resolver and ignore the glibc one.

------
jancsika
At flare-up #237 you'd think one would have figured out a professional way to
handle questions like this.

Buried in the pointless flamewar is the maintainer's own reply which is the
way forward:

> Anyway, if there's a well respected network of community DNS servers that
> can roughly match Cloudflare/Google we can add, sure.

That is the correct and _initial_ response the maintainer should have given.
It puts the onus on OP to show how their preferred DNS stacks up against
Cloudfare/Google.

Instead, the maintainer plays dumb wrt the importance of defaults, essentially
guaranteeing that OP will escalate the rhetoric. Worse, the maintainer
responds in kind.

If the maintainer had gone the correct (and easy!) route, we'd all benefited
by having learned how this non-profit DNS stacks up against the others from a
technical standpoint. (How does it, anyway?)

And to cover my bases-- when spam gets through to the inbox, no serious person
asks, "Why doesn't anyone ever talk about the spammer's responsibility to
prevent spam? What about the spammer?" Similar logic applies for why I'm
talking about maintainer and not OP.

~~~
psadauskas
_Buried_? As the 6th post in an 8-post thread? And I don't see any flaming at
all. Your comment here is far more aggressive than any in the Github issue.

~~~
eqvinox
It's after another maintainer post, and then the post begins by ... weird name
calling about the funding question (which, albeit maybe a bit excessive, I'd
still consider valid to ask.)

I think "buried" is an OK description of that.

------
psanford
> I believe Google's DNS service is used for monitoring DNS traffic from IPs.
> This is then most likely used for personalisation, which aids Google's
> mission to take freedom and privacy away from nontechnical and technical
> users.

The Privacy Policy for Google's public DNS service explicitly says that they
don't do that. If you really think Google is doing it then you should take it
up with the FTC who would really enjoy making an example out of Google.

~~~
Nextgrid
> you should take it up with the FTC who would really enjoy making an example
> out of Google.

How do you prove it though? Google will not just give you the source code of
their DNS infrastructure and even if they did there is no way to prove that
it's indeed that code that is running and not a modified version that _does_
collect data?

~~~
jeffbee
Actually google is one of the few organizations out there with real
reproducible builds. If they gave you the sources and the program image, you
could prove the latter was derived from the former.

~~~
Nextgrid
This has nothing to do with what software they're running internally though,
and again as I said even if you can audit and build your own version of the
Google DNS infrastructure there's nothing guaranteeing that it's exactly
what's running internally.

The problem is that ultimately there is a conflict of interest - Google is an
advertising company that benefits from knowing as much as possible about
people for ad targeting purposes, and as such some people (like me) might not
be willing to trust them.

~~~
jeffbee
My personal bias is that I've worked at Google and I've worked elsewhere and I
know that Google's control over what code runs in prod is years ahead of the
other places I've worked. They have strong controls that audit that binaries
running as sensitive roles (roles with access to real user data and logs) were
produced from reviewed and submitted code and built in the official hermetic
build farm. For very sensitive roles (gmail etc) they audit the command line
and everything. The controls everywhere else I've worked are a complete joke
by comparison.

This quote is from Google's security infrastructure whitepaper:

"""Google’s source code is stored in a central repository where both current
and past versions of the service are auditable. The infrastructure can
additionally be configured to require that a service’s binaries be built from
specific reviewed, checked in, and tested source code. Such code reviews
require inspection and approval from at least one engineer other than the
author, and the system enforces that code modifications to any system must be
approved by the owners of that system. These requirements limit the ability of
an insider or adversary to make malicious modifications to source code and
also provide a forensic trail from a service back to its source. """

I don't think the conflict you mention exists. Google benefits when people use
the open web. They run DNS because DNS is critical to web user experience and
ISP DNS is garbage. Also, by the way, ISP DNS privacy story is a complete
disaster.

~~~
Nextgrid
I am not talking about malicious code contrary to Google's intention being ran
on their infrastructure. I am talking about code that Google _wants_ to run.
Code that harvests DNS queries for ad targeting might be within Google's
objectives and wouldn't be considered as malicious, but it would be malicious
when looking at it from the user's point of view.

~~~
jeffbee
You're saying that Google has a plan to intentionally subvert their published
privacy policy, and act which if discovered would end the company's existence,
and that some engineer on the project wrote and another reviewed this change,
that none of the dozens of privacy zealots in their internal privacy org[1]
have managed to notice, and that the people who operate 8.8.8.8, some of whom
are just as privacy-deranged as anyone you've ever met, and who collectively
own a disturbing number of fedoras, kilts, and unicycles, who are the biggest
nerds you've ever seen, happily run this service 24x7 without blowing the
whistle?

Seems unlikely.

1: [https://gizmodo.com/meet-the-woman-who-leads-nightwatch-
goog...](https://gizmodo.com/meet-the-woman-who-leads-nightwatch-google-s-
internal-1825227132)

~~~
Nextgrid
Facebook has been caught doing that where phone numbers for 2FA purposes that
were promised not to be used for ad targeting started being used for exactly
that purpose. Facebook is of comparable size and operates within the same
regulatory environment as Google, so if they can do it and get away with it
there's no reason to believe it would be different for Google.

~~~
jeffbee
That’s basically ridiculous. Facebook is a corrupt organization. It was
established to sexually harass young women. It was irredeemable from the
beginning.

------
lazyjones
From a technical standpoint, there seems to be absolutely no reason to put
cruft like this in systemd. If the network isn't configured (or configured
correctly), there's no point in making sure DNS "always works". It has more
potential to leak information at the wrong moment than to fix anything the
user/administrator did wrong.

~~~
diegocg
This is probably part of networkd, which is a different daemon in charge of
managing the network

~~~
creeble
Or is it systemd-resolvd? Who can tell these days?

~~~
takeda
if it is systemd-resolvd, then privacy oriented solution would be to not use
google, cloudflare or even the non profits. It would be to just have list of
13 DNS root servers and do recursive name resolutions (i.e. what caching
resolvers were meant to do).

But the absolute correct behavior would be a hard failure, because if you have
IP properly assigned and routing setup but no DNS, then you either
purposefully set it up that way and don't want DNS to work, or you have
misconfigured network setup which you should be aware about it, without having
to troubleshoot why some machines don't have working Internet while others
seem perfectly fine.

------
different_sort
What are the 'obvious privacy reasons' to not use google DNS/NTP?

I am being 100% serious when I ask: What bad does someone expect to happen as
a result of this.

~~~
livre
Google logs and stores forever every DNS request (and other info related to it
like approximate geolocation), they also temporary store even more information
that can identify you (like your IP address).

[https://developers.google.com/speed/public-
dns/privacy](https://developers.google.com/speed/public-dns/privacy)

Edit: let's suppose for a moment that Google is not evil, does not cooperate
with the NSA and doesn't make bad use of that information. It's still a
liability and any data leak will be a problem for anyone using Google's DNS.
Some things are better not being logged at all or at least not permanently.

~~~
WillPostForFood
_stores forever every DNS request_

That’s not what your link says.

 _The permanent logs are a sampling of the temporary logs_

~~~
livre
I may have misinterpreted this part:

> The permanent logs are a sampling of the temporary logs where your IP
> address is removed and replaced by a city or region-level location.

What I understood from it is that they replace your IP address with an
approximate location but keep the rest of the data. Can you explain me how you
interpreted it?

------
kd913
Why is this repeatedly raised? This is only done like this for development
purposes. It has been discussed for years at this point.

Distros/users are encouraged to change these defaults and many do.

Seems pretty clear cut and simple to me.

~~~
fartcannon
Probably because it shows the character of the systemd devs.

~~~
kd913
I can imagine it getting a little frustrating to have to constantly reiterate
the same point over and over again.

Especially when people are refusing to understand the reasoning and logic
explained from last time. It's already been explained, a quick duckduckgo
search can show quickly what the reasons are and why the whole privacy
argument is irrelevant here.

~~~
fartcannon
Easy fix! Just change it or dont set it at all.

Instead, he chooses drama.

~~~
kd913
He chose the option that was the fastest for development purposes/testing. On
average Google/Cloudflare worldwide has the fastest and most reliable DNS
services.

Why would he change it?

He says it clearly that it's the responsibility of distro maintainers to
modify these defaults which they are encouraged to do.

It's everyone else who is choosing drama. This is for development purposes,
the entire privacy argument is completely irrelevant.

~~~
fartcannon
If it's so irrelevant, then why _not_ change it? That's what good leaders do.
Small price to pay to get good will and a large group of bystanders on your
side.

His way generates conspiracy and alienation.

~~~
kd913
Because all of the alternatives are simply less reliable, less performant,
less available and offer no benefits for development/testing. That is the only
factor that matters here.

>That's what good leaders do. Small price to pay to get good will and a large
group of bystanders on your side.

I don't think he cares about people who are determined to stay ignorant.
Especially of those who refuse to comprehend the needs/use cases for different
parties.

Frankly considering he has received death threats about systemd, I am
surprised he even put a response at this point. It would make sense to me if
he just closed the ticket immediately with a link to the previous discussion.

~~~
fartcannon
Sure, he doesn't care, and so then this happens. We happen. Drama.

~~~
kd913
The only factor here that matters is development/testing.

Drama is irrelevant and wouldn't change anything. At worst, he gets his issue
tracker filled. If it bothers you, then the only thing that you can do is
change to Devuan or don't use systemd-networkd/systemd-resolved. Distros are
going to continue using systemd irrespective of this nonsense drama.

------
wrkronmiller
Could someone explain why systemd needs DNS/NTP fallbacks to begin with? Why
aren’t these both runtime configs?

EDIT: I assumed there were also runtime configs based on the name "fallback."

My question is primarily why you need compiled-in fallbacks, and secondarily
why the fallback configs aren't pulled from config files.

~~~
peterwwillis
More to the point: does _any_ Linux software at all use a default DNS or NTP
server? NSS resolver library doesn't default to using a specific DNS provider,
and it's a DNS resolver. NTPd _might_ default to using ntp.org pools, but that
might just be my distro's default configs.

To me this is more evidence that Systemd is really just "Poetteringd":
whatever he feels like it should have, it has. Really disappointed that
distros continue to use this software rather than start a community
replacement that addresses its shortcomings.

If I re-did systemd, it would retain the same functionality, but not come
enabled by default, and it'd be broken up into independent packages to be
installed as needed. It would also use standard interfaces and do away with
the binary formats, defer to existing system software rather than default to
its own custom versions, de-complicate its awkward filesystem tree and
simplify management of service configs, make the command-line interface more
sane, and begin integrating more cloud-specific features to replace more
complex cloud software (such as for service discovery, cluster management, and
HA).

All these fixes would allow the "anti-systemd" crowd (of which I am a card-
carrying member) to run a simple init system if they want, or extend it to the
full range of systemd functionality. It would remove glaring pain points and
user friction, embrace the existing system's default functionality (and
backwards compatibility), and extend the software to be more useful in cluster
environments. In addition, if this were maintained by the community, all
distros could adopt it like they did systemd, and we don't have a half dozen
different service systems like before.

Now please somebody reply and tell me why this is a terrible idea...

~~~
hedora
It more-or-less worked that way for decades (and it worked better), so it’s
not a terrible idea.

My theory is that the big distros are intentionally destroying forwards and
backwards compatibility on the desktop as much as possible.

The more often window managers, desktop software, etc., stop working, the less
their own teams have to compete with existing software. Wayland’s X11-as-a-
second-class-citizen and gtk’s elimination of API stability are two other good
examples of this.

Also, tearing down established standards is a great way to sabotage the
various BSD’s out there.

What they don’t get is that sabotaging the rest of the community is
undermining their own market share within Linux, even as it shrinks Linux’s
market share.

~~~
cycloptic
> My theory is that the big distros are intentionally destroying forwards and
> backwards compatibility on the desktop as much as possible. The more often
> window managers, desktop software, etc., stop working, the less their own
> teams have to compete with existing software. Wayland’s X11-as-a-second-
> class-citizen and gtk’s elimination of API stability are two other good
> examples of this. Also, tearing down established standards is a great way to
> sabotage the various BSD’s out there.

I see this sentiment a lot and I don't get it. The developers I've talked to
have been doing a _lot_ of work to get XWayland working and to fix all the
edge cases. And GTK has been API stable for years, the only thing that really
seems to have broken was themes. And actually now GTK3 is feature complete and
will not be receiving any more changes except bug fixes.

As far as init systems go, there seems to have never been any real standard
and most distros always seem to have done their own thing. It does not seem
like anything could be done there especially when a low level piece of
software like this needs to use Linux- or BSD-specific APIs. It simply is not
possible to have a portable standard there right now.

I have a humble request, please don't continue by promoting these unfounded
theories and accusing people of sabotage. I personally would welcome a
friendly fork of systemd to make some positive changes but it won't happen if
people are doing it out of hostility. Many of the issues in the GP post have
been brought up over the years. Upstream and the distros all have a stance on
them and there is a reason why things are the way they are. It would be unwise
to proceed without having substantial discussions with them to put resources
where they are actually needed most. There was already a hostile fork that
died because of not paying attention to this.

------
falcolas
> I will block discussions here now, since I don't think we need the input
> from the script kiddie peanut gallery here.

How _not_ to interact with your community 101.

Defaults matter, because even if they are changeable via a build-time flag,
most of the time they won't (how many people knew this was a default before
now?)

~~~
theevilsharpie
> How _not_ to interact with your community 101.

On the contrary, I think he handled it perfectly.

I've become exhausted with most discussions on social media, because
conversations frequently get hijacked by loud people with strong opinions that
don't express them in good faith. Without strict moderation, it can become
impossible to have a rational discussion on certain topics. For example,
pretty much anything on HN involving Google devolves into worthless noise.

In the case discussed in the OP, Poettering clearly has no intention of
changing the default configuration (at least based on the reasons given in the
issue), and has provided a way of modifying the configuration for those that
care enough to change it. There's no value in further debate, and closing the
issue to further discussion keeps Poettering and other contributors from
getting distracted/burned out by unproductive conversation.

If people don't like it, they're free to fork Systemd or start up their own
competing software project, and then they can handle community interaction for
their own project in whatever way suits them.

Frankly, I wish more discussion were handled this efficiently.

~~~
KaiserPro
Its not really efficiently.

Something like: "we've discussed this a number of times, see #12345" closing
now

would have closed the ticket and not made the front page.

By being a top draw dick about it (when the originator was at least being
earnest) the debate rolls on.

He had given a full answer, he could have left it there, but chose not to.

its this kind of "rock star" and cheerleaders that give IT a bad smell.

~~~
bad_user
Are you talking about the comment that closed the ticket [1]? Are we not
seeing the same thing, because that's not a " _top draw dick_ " I'm seeing.

He was perfectly reasonable. And I don't know if you've ever managed any
popular open source projects, however it's best to close tickets fast instead
of letting them linger on, especially when you have over 1000 opened issues
and especially if, as a maintainer, you know that you won't change your mind
about that issue.

Then he got accused of being funded by Google. He actually showed restraint.

[1]
[https://github.com/systemd/systemd/issues/12499#issuecomment...](https://github.com/systemd/systemd/issues/12499#issuecomment-490057558)

~~~
wtallis
> Are we not seeing the same thing, because that's not a "top draw dick" I'm
> seeing.

> He was perfectly reasonable.

The original complaint had the form "this default should be changed for reason
X". Poettering's response was "our defaults are Y, and you can change them".
That's not perfectly reasonable. It's a non sequitur. Poettering's response
passively-aggressively ignores what the original complaint was all about,
instead of simply issuing a direct rejection.

~~~
bad_user
It isn't a non-sequitur, he offered a solution for users that would like
different settings, while choosing not to engage in pointless arguments.

While professional courtesy is recommended, maintainers don't really owe an
explanation for their work when provided for free. Put that man on a payroll
and you might deserve a more detailed answer.

~~~
wtallis
It is absolutely a non-sequitur to tell someone what the defaults are when
they are complaining that they do not like what the defaults are. "Choosing
not to engage in pointless arguments" would be _not responding at all_ ,
rather than responding in a manner that is implicitly insulting. Poettering
phrased his response as if he was answering a different issue than the one he
was actually responding to. Deliberately misinterpreting someone's question
instead of directly saying he didn't want to get into that matter is poor
behavior.

~~~
growse
If they don't like the defaults, they could just fork the project.

------
alibert
> I believe Google's DNS service is used for monitoring DNS traffic from IPs.
> This is then most likely used for personalisation, which aids Google's
> mission to take freedom and privacy away from nontechnical and technical
> users.

Not questioning if Google does it or not but I was made to believe that IP is
not really useful for ads targeting because it just doesn't work great with
multiples users behind a single IP. Now with CGNAT on both type of networks
(fixed and mobile), using IP for targeting does not seem pertinent to me.

~~~
hedora
Timestamps are usually enough to infer end users, even on much larger data
sets. (Researchers used the Twitter firehose to deanonymize a large dataset a
few years ago).

------
dbergamin
So what colour should we paint the bikeshed?

Fair response IMO, this is the type of issue that can attract a lot of noisy
opinions so very reasonable to lock comments to contributors.

------
nimbius
FYSA: you can override Potterings delights in
/etc/systemd/resolved.conf.d/fallback_dns.conf by specifying an empty string
for the fallbacks.

------
dec0dedab0de
I'm kind of behind on systemd, can someone explain why it has DNS and NTP at
all? Shouldn't it just use whatever the rest of the computer uses?

~~~
AlexandrB
Systemd is kind of like the Borg. It slowly assimilates daemons until it
eventually replaces all system level functionality. Its next target is home
directories[1], then who knows what else.

[1] [https://systemd.io/HOME_DIRECTORY/](https://systemd.io/HOME_DIRECTORY/)

------
paulcarroty
How to delete the fallback Google DNS and setup Cloudflare for example:

/etc/systemd/resolved.conf.d/fallback_dns.conf

    
    
      [Resolve]
    
      FallbackDNS=1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001
    

The same with NTP:

/etc/systemd/timesyncd.conf

    
    
      [Time]
    
      FallbackNTP=time.cloudflare.com

~~~
hedora
How do you completely disable the fallback, though?

------
cbsks
What fallbacks do the major downstream distributions use?

~~~
megous
Arch Linux for example:

#FallbackNTP=0.arch.pool.ntp.org 1.arch.pool.ntp.org 2.arch.pool.ntp.org
3.arch.pool.ntp.org

------
marcinzm
>Christ, what's next? You accuse us of controlling people's minds with
vaccinations we get directly from Bill Gates? And that systemd uses 5G to
spread CoV-2?

That seems a very aggressive response to a perfectly valid question in modern
times. Google pays Mozilla tens of millions to be their default search
provider. Companies use money to influence things to their benefit all the
time.

edit: Another example was all the extensions that Kite bought out and had
their service offering added to (which sent all your code back to Kite
servers).

~~~
jasondclinton
Asking someone if they are a shill is never "a perfectly valid question".

~~~
wtallis
> Asking someone if they are a shill is never "a perfectly valid question".

It was more of a question whether there is a conflict of interest; "shill" is
a more loaded term than what was used. And the answer was actually a qualified
_yes_ , so there was at least a sliver of validity to the question.

------
znpy
the more i grow up the more i emphatize with Poettering.

------
sam_lowry_
The original title was much more explicit: "Systemd Fallbacks to Google NTP
and DNS".

I guess @dang changed it with good intentions, but he accidentally destroyed
the meaning. "We wanted the best, you know the rest."

~~~
threatofrain
> Don't fallback to Google NTP and DNS. #12499

------
microcolonel
The singular party who is entitled to complain about this default
configuration is Google.

------
2trill2spill
> Christ, what's next? You accuse us of controlling people's minds with
> vaccinations we get directly from Bill Gates? And that systemd uses 5G to
> spread CoV-2?

> I will block discussions here now, since I don't think we need the input
> from the script kiddie peanut gallery here.

Ohh my what a toxic open source community. I can't imagine working on a
software project where the lead dev/maintainer says things like that to
people.

Edit: I suppose with all the downvotes apparently lot's of people here think
such responses from the leader of a open source project is acceptable
behavior.

~~~
grandinj
More like - that community leader takes an unbelievable amount of flack and is
still generately polite and helpful in most of his interactions.

How about we cut them a little slack for occassionally being human?

~~~
hedora
I don’t care if Poettering's an a—hole or not. I enjoy reading Torvald’s
rants, in fact. They’re usually informative.

I’m upset that he has done more than any other developer to break my machines,
and he keeps getting more authority so screw things up.

PulseAudio was bad enough, but at least it only broke one subsystem. SystemD
has unapologetically broken: background process management, time keeping,
network resolution, pam authentication, screen saver session management, x11
startup, logging, and probably more.

~~~
manuelabeledo
> I’m upset that he has done more than any other developer to break my
> machines, and he keeps getting more authority so screw things up.

I may be wrong, but you should be blaming the ones who made the _actual_
decision of replacing whichever init system, with systemd.

systemd was developed under Red Hat's umbrella. Nowadays, the vast majority of
mainstream Linux distributions use it, and I doubt it was because Red
Hat/Poettering pressure them to do so.

------
AlexandrB
So to summarize:

When discussing privacy and Google a common refrain is "just don't use Google
products then". But Google's DNS is used as the default for a key system
daemon in the majority of Linux systems?

I'm going to file this under "voting with your wallet is bullshit".

I also hope that the systemd team is watching the Google DNS ToS like a hawk
because: "We may modify the Terms or any portion to, for example, reflect
changes to the law or changes to our APIs. You should look at the Terms
regularly."

