I get why people are paranoid about ISPs blocking content and net neutrality, but let's not cry wolf prematurely. The technical details here strongly suggest a bug rather than intentional blocking of 18.104.22.168 DNS traffic.
CF CEO tweets that 22.214.171.124 is also blocked.
Others have confirmed that the ipv6 address belonging to CF appears to be blocked.
> For IPv6, we have chosen 2606:4700:4700::1111 and 2606:4700:4700::1001 for our service. It’s not as easy to get cool IPv6 addresses; however, we’ve picked an address that only uses digits.
For me up in Canada, ping 126.96.36.199 works. But
We also need to confirm IPV6 works outside AT&T's network.
Edit: Just tried Google's DNS. 188.8.131.52 works, but their IPv6 doesn't, so I guess this was a bad test.
Edit2: Learned about nslookup, but it does not seem to work with either Google or CloudFlare's DNS.
nslookup reddit.com # Works
nslookup reddit.com 184.108.40.206 # Works
nslookup reddit.com 220.127.116.11 # Works
nslookup reddit.com 2606:4700:4700::1111 # Does not
nslookup reddit.com 18.104.22.168 # Works
nslookup reddit.com 2001:4860:4860::8888 # Does not
nslookup reddit.com 2001:4860:4860:0:0:0:0:8888 # Does not
Okay, guess my PC/LAN/ISP doesn't support IPv6 yet.
I'll ask them about it when they ring me up next time asking for more money.
I recall on Teksavvy I had to pay extra for a "static IP" to get IPv6. Not sure if you're with Bell directly, though.
Connecting to WiFi (Time Warner), I got a 403 from cloudflare (presumably there just isn't a web server set up on that address).
Using mobile data (AT&T), I got ERR_ADDRESS_UNREACHABLE. However, 22.214.171.124 actually works on AT&T cellular, so I'm not sure what to think.
(US based) frontier, vz, and spectrum all can ping that ipv6 address (though all have way over 10ms latency)
the nslookup reddit.com 126.96.36.199 does not return for me, if I connect to work via VPN it does. 188.8.131.52 and 184.108.40.206 do work without VPN. while the AT&T modem shows IPV6 I did not test.
Manufacturer Pace Plc
I'm pretty sure the other end has to be running `pingd` to get a response from ping. Some do, some don't.
I might be wrong but that's always been my understanding.
What is unfortunately common though is people blocking ICMP at their firewall, either at the host level itself or further upstream. Sometimes they just block echo requests, but often they block ICMP entirely which breaks things in very weird ways from time to time.
Blocking ICMP in any way is generally to be considered harmful. It's not 1997 anymore, the "ping of death" is not a thing on any OS you should actually be connecting to the internet.
""With the recent launch of Cloudflare's 220.127.116.11 DNS service, we have discovered an unintentional gateway IP address conflict with 1 of their 4 usable IPs and are working to resolve the issue,"
A few of you will be disappointed to know its not a evil attempt to block you from using it. Same way they have literally never blocked the ability to use any other DNS service before.It's simply a bug caused by the way the BGW-210, and Pace 5268AC operate and make use of 18.104.22.168 internally in some way and it will be fixed with a firmware update.
Maybe the firmware update has a bug, but it's very suspiciously timed. Notice that the OP is dated April 2, while 22.214.171.124 was released April 1.
I have ATT U-verse internet service and use their Arris BGW210-700 gateway
One interesting thing is that if I go to the gateway management page, and use their diagnostic tools, I'm able to ping / traceroute the address - but I can't from any devices connected to the gateway
From gateway diag page:
PING 126.96.36.199 (188.8.131.52): 56 data bytes
64 bytes from 184.108.40.206: seq=0 ttl=64 time=0.568 ms
64 bytes from 220.127.116.11: seq=1 ttl=64 time=0.156 ms
64 bytes from 18.104.22.168: seq=2 ttl=64 time=0.164 ms
64 bytes from 22.214.171.124: seq=3 ttl=64 time=0.144 ms
--- 126.96.36.199 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.144/0.258/0.568 ms
traceroute to 188.8.131.52 (184.108.40.206), 30 hops max, 38 byte packets
1 1dot1dot1dot1.cloudflare-dns.com (220.127.116.11) 0.285 ms 0.177 ms 0.090 ms
The times on the pings make it look like its hitting a loopback address instead. Pings to 18.104.22.168 from the diagnostics page take about 23 ms. No way 22.214.171.124 is completing in under 1ms haha
They had the choice of "fix the whole backend" or "block 1.x on the user end".
Guess we know which one was easier. If all this wild speculation is true, maybe they're working on a fix to the root cause and will roll back the patch when complete.
This would make the situation both due to incompetence and intentional.
I ask because I don't know. I figure any traffic headed that direction would go anyway it just wouldn't get routed very far with no valid destination.
Maybe stupid question, but why would AT&T block it?
According to the announcement post, part of the reason that Cloudflare was allocated the 126.96.36.199 address is that they were ready and willing to handle the expected inundation of all kinds of bizarre traffic.
It seems that one of those "off-label" uses of 188.8.131.52 is an internal / network control interface on [some?] AT&T networks. I'm just speculating, but it's definitely possible that 184.108.40.206 suddenly becoming publicly routable and pointed to a real thing caused some problems. "Patch it out" may be an acceptable emergency response depending on the breakages, but not really acceptable long-term.
Back in 2010 there were problems that came up when IANA started allocating out of 220.127.116.11/8 (e.g. ). Things that were once assumed to be unused started being used, leading to strange issues.
Also, why on earth would AT&T block 18.104.22.168 and not Google DNS and OpenDNS?
Also- If this was intentional- I'm betting they'd filter it for the mobile network as well. This has got to be a fuck-up.
"Bugs are worth more points if, once discovered, they are plausibly deniable as an innocent programming error."
Just like how only the true messiah denies his divinity, it doesn't give innocent bugs much of a chance.
In fact, now we can show that all bugs are suspicious, with apologies to the interesting number paradox:
The least intentional looking bug is the most effectively hidden, and therefore should probably be suspected of being intentional. Since it's now suspect, it's longer the least intentional looking bug, so the next least suspicious bug suddenly deserves a bit more scrutiny, and so on.
Blocking 22.214.171.124 and 126.96.36.199 -> what are the odds here?
Remember to scrub EXIF data!
- If the action was malicious, the people involved in writing this code are likely okay with it and not likely to leak details of it.
- If the issue is a bug, the people involved in writing this code are probably working to fix it, and not likely to leak details of it.
- People not involved with making it would likely leave an internal access trail (independent of EXIF data) when they access that code.
Which is to say, expecting an Ed Snowden every time a company does something unethical is kinda silly, otherwise we'd have Google's search algorithm by now.
3. Unlikely; it's likely in a big repo that's synched all at once.
Alternatively, we can just obtain the firmware from a device and diff it against the last-known-working version, to see how the routing is failing.
Lots of cisco example config use 188.8.131.52 for router internal identifier / DHCP server / OSPF dummy network .
Not suprised if it break anything.
AT&T routers also don't let you use a 10.x address at home (possibly to prepare for carrier grade NAT, although there is an official 100.x address reserved for that; so fuck you ATT).
I'm so sick of my AT&T router/modem for various other reasons. I hate how you are required to use it for many of their offerings (including Fiber to the home).
There are a number of tools out there for putting their router behind your Linux box. Most of them configure ebtables or use scripts to forward the 802.1q authentication packets to/from the router.
Hanlon's razor: https://en.wikipedia.org/wiki/Hanlon%27s_razor
It's a shitty hack and it adds a weird layer of indirection that's kinda buggy and doesn't always flow traffic the way you think it's being flowed. The IPv6 stuff gets confusing as well because the modem is still dishing out public IPv6 address, so if you want to advertise them as well, you've got to start slicing up your prefix.
However, I still can't ping 184.108.40.206.
But hey, if you have advanced needs, no problem, let me refer you too our Gaming Provider and Streaming Provider subsidiaries.
Oh you need actual technical access to the internet because you write your own software? Tricky, but I'm sure our Business Technology Services Provider subsidiary will have the service you need. (You do have a business, right?)
They'd also become unreliable and untrustworthy.
"Mom, I'm going over to Timmy's house tonight. They have _good_ Internet"
(Meanwhile this whole exchange is probably already obsolete because who visits their people's houses when you have phones?)
Taking a moral stand is honorable, but using your customers to do it isn't.
Certainly that's the first step.
There's options for the second step. But advertising seems like it would be the most powerful.
"Why use us over AT&T? Because you're not getting the Internet. You're getting what AT&T decides you should look at."
"We don't block Netflix or Hulu or a whole host of other streaming services, unlike AT&T"
And in that case, the town just lost it's internet. What makes you think the residents won't remember this come election day?
In most areas there is effectively a government imposed monopoly on who can provide you access. So there is no "market" to normalise things. You simply cannot vote with your feet.
In Europe, where the regulatory framework is different, people would just switch ISPs if one started acting in bad faith.
And that government is elected by the people, right? Which means they could make this an election issue and vote candidates that don't support monopolies, right?
I don't understand what part of my statement you're arguing with.
Most people don't have the grasp on the technicalities to even be able to make the decision to vote for a specific candidates because their internet access is sub-par
Not to mention if you vote for someone you also get all the other things that candidate aligns with, not just better internet.
(not super sure how voting on city/state level works in the us, but it should be accurate enough)
How do we provide a kiddie day care service level for people who won't or don't want to care, and a full service level for the rest of us?
Or do I owe the Internet an apology?
I really love your idea.
"Last year, a number of industry groups lobbied for a change to the FDA’s definition of chocolate — a change that would have allowed cocoa butter to be replaced with vegetable oil. At the time, Hershey’s spokesman Kirk Saville told the Harrisburg Patriot-News that “there are high-quality oils available which are equal to or better than cocoa butter in taste, nutrition, texture and function, and are preferred by consumers.”"
Also see the UK as well for an example of how previously unregulated speech has become regulated because the authorities have pushed over and over again, backing off every time there's a loud enough protest, but trying again after a short time.
So exactly what parent said, happened.
They do because it's true and that's exactly what the law says.
Digital Economy Act 2017 14 (1):
>A person contravenes this subsection if the person makes pornographic material available on the internet to persons in the United Kingdom on a commercial basis other than in a way that secures that, at any given time, the material is not normally accessible by persons under the age of 18.
Section 23: Regulator’s power to require internet service providers to block access to material
(1) Where the age-verification regulator considers that a person (“the non-complying person”) is—
(a)contravening section 14(1), or
Go read the "commencement" section - it's actually eye-opening to do this for other laws you've heard are supposed to have drastic effects.
(note that its not an article but a debate post)
It's likely incompetence, not malice. If they didn't want people using other DNS, and were willing to fuck with ip addresses they don't own to accomplish that, they'd be blackholing google's and opendns's public caching nameservers too.
It might even have been a conscious decision. Even though it's horrible and the people involved in developing the firmware need re-education. The decision probably went like this: we need an internal address to do something. We can't use 10, 172.16, or 192.168 ranges because those might conflict with internal LANs. 1.x is safe because we all know nobody uses them. The correct decision obviously would have been to get at&t corporate to commit to never using some tiny corner of their address space, and use that. Or 127.a.b.c if that works on the OS. Those options are only needed if they really need an extra IP address. They might not need one after all if they designed their firmware better.
I'm still not entirely sure what the best option is there. Maybe some clever use of network namespaces, with a named pipe to bridge between the "internal" and "external" universes? Just typing up that idea makes me cringe though.
One workaround to this is to have the container host also put in an SNAT rule, so that anything that it forwards to a container would have the source IP address re-written to appear to come from the container host's IP, or the docker0 bridge IP (172.17.0.1/16)
This is also the correct solution for your cluster.
This was an organization that sustained five mines of uptime for decades.
Crazy to see a fallen (or broken up) titan struggle with basic stuff. I mean, basic compared to their heyday.
It's allocated to "DLA Systems Automation Center," a branch of the US military. The addresses are probably used on NIPRNet/SIPRNet, but not publically routed. (Much like 220.127.116.11/8.)
Don't use Kaspersky!
There is not enough data to attribute this to malice yet, but it does not look good (see CloudFlare's tweet).
I think they're blocking 18.104.22.168 because customers are now using DNS that isn't them, which deprives them of valuable data on which domain names their customers go to, which they can sell to advertisers. Yes, there's other ways to get that information but the DNS server is an easy one.
On what basis? Google started Google Public DNS in 2009 and, as far as I know, it was never intentionally blocked by any ISPs. The issue with 22.214.171.124 is a lot of hardware treats it as though it was reserved for private networks. For instance, I can't access 126.96.36.199 right now since I'm connected to a Cisco router. So this could very well be a technical issue.
But even if 188.8.131.52 is taking off more than 184.108.40.206 did, your assuming the DNS queries people are sending are secure anyway. I'll admit I'm not completely up-to-date on the whole "DNS over TLS" thing but I haven't noticed any support for it on my fully-updated Windows machine or Android phone. I'd love for someone to correct me, but I don't believe any major electronics ship with secure DNS by default. If people are sending DNS queries unencrypted the ISPs can just sniff them.
Net Neutrality wasn't considered much of an issue back then, it was just taken for granted (and the administration at the time was attempting to enforce it as vigorously as possible).
Forcing independent internet technical infrastructure off the internet and through their own proprietary infrastructure would be the opening shot you would expect if they wanted to open that battle. After all, you gotta boil the frog slowly, and nobody but a tiny minority of technical users would really care about not being able to use third-party DNS servers.
I've never seen or heard of a Cisco router doing anything that would interfere with access to 220.127.116.11.
Their wireless LAN controllers on the other hand, use 18.104.22.168 as the default (but entirely configurable) Virtual IP to use as an anchor for the captive portal.
If you can't access 22.214.171.124 behind a Cisco router it's likely because someone set it up incorrectly.
I have news for you...
"After very little research we quickly came across Cisco mis-using 126.96.36.199, a quick search for “cisco 188.8.131.52” brought up numerous articles where Cisco are squatting on 184.108.40.206 for their Wireless LAN Controllers (WLC). It’s unclear if Cisco officially regards 220.127.116.11/8 as bogon space, but there are lots of examples that can be found on their community websites giving example bogon lists that include the /8. It mostly seems to be used for captive portal when authenticating to the wireless access point, often found in hotels, cafés and other public WiFi hotspot locations."
Well, now you have.
> If you can't access 18.104.22.168 behind a Cisco router it's likely because someone set it up incorrectly.
That’s kinda the point.
Allow me to rephrase, I've never heard of a Cisco router doing that from a reliable source.
> That’s kinda the point.
Then it has nothing to do with Cisco and everything to do with the person who configured it.
My Spanish ISP (Vodafone ES) doesn't block external DNS at the ISP level. However, the router they give you:
1) Blocks outgoing DNS requests from the internal network by default. This can be disabled.
2) Doesn't let you specify any other than Vodafone's DNS servers on the DHCP Server configuration. This cannot be changed.
I'll let you decide whether this is blocking or not...
What's the theory exactly? What would be the benefit for AT&T to block a new 3rd party DNS? Did they do similar things in the past for other 3rd party DNSs such as OpenDNS, Quad9 or Google's? Seems odd to target this one service in particular.
The ship may have sailed on blocking 22.214.171.124 at this point; some things _hard-code_ it.
Definitely. So if this truly was their strategy, why are they blocking 126.96.36.199 instead of pointing it at their own DNS? It would be less immediately obvious what’s happening versus outright blockage. I really think people are prematurely attributing this to nefariousness.
I would have expected 188.8.131.52 to already be blocked if anyone filters on bogon-space (or has dealt with i
Is there a database of who blocks what? I searched but didn't find a collection anywhere.
Unless we are looking at port 25 and whatnot. Yes, it is not allowing you to use a (not technically)-arbitrary port, but most would agree that the internet is better off for that.
Using unallocated IPs for "internal" or bogus purposes is sketchy, continuing to use them after they are allocated is something else. Especially so nearly a decade on.
Not upgrading equipment and configs for 10 years is nothing in the ISP world.
I had my stint at an ISP that worked with around 40 state level and national orgs. I saw the underbelly of how things work, and its frankly scary.
I had to end up touching one of them, because of things breaking with that subsystem and the new ticketing system that was being implemented. It had the wonderful line
database_user = root
database_password = [current mysql root password]
This team provide a great side service - you can setup BGP with them using an internal AS. It's one of the few ways you can get practical experience setting up BGP in the home with a third party. I'm running it right now.
> A bogon prefix is a route that should never appear in the Internet routing table. A packet routed over the public Internet (not including over VPNs or other tunnels) should never have a source address in a bogon range. These are commonly found as the source addresses of DDoS attacks.
I know there was an amount of collateral damage, but if you think about it, it's been many years since malware would get in user desktops and just send spam, largely due to this.
The right response is to contact the owners of the servers/services they're running and tell them to configure them correctly - if they continue to abuse them or don't show the technical skills, then that's another matter.
even diagnosing the issue and finding someone on the other side that understand the topic is hard. I'm no network engineer and definitely neither are the support guys.
it's just a roulette. you have to change until you find one that works. and it sucks.
Me: hi can you open up some ports on my router?
CSR: sure which port?
Me: all of them
 According to google, it's defined as:
"the principle that Internet service providers should enable access to all content and applications regardless of the source, and without favoring or blocking particular products or websites."
Well, since the formal repeal of net neutrality has been delayed, I think it technically is a violation of the no-blocking rule.
OTOH, it's not like the FCC is enforcing net neutrality while delaying the official effect of its repeal.
Clearly, they’re discriminating against certain client devices, and were under the Obama administration too.
However, the documentation says it should work, and AT&T won’t provide support.
They’ve been getting away with this for years, so I guess plausible deniability (it is “just a bug”) can work wonders in this space.
See: Monopoly/duopoly of telecom in the USA, net neutrality protections, and the speed limit signs on 280.
until they are selectively enforced.
Still, it looks more like malice since there are other addresses besides 184.108.40.206 that are also blocked.
By their own admission CF receives a ridiculous amount of garbage traffic at this IP, it was not absurd for AT&T engineers in the past to thing "well, we need an IP that we can be reasonably sure nobody is going to use and is never going to conflict with anything on any network, 220.127.116.11 seems reasonable". Seeing everybody in this thread jumping into conspiracy theories instead of the much more likely configuration issue is a bit disappointing for a community that's supposed to understand technology.
That is an utterly unreasonable conclusion.
The same logic resulted in Y2K, which was generally a huge waste of time, money, and resources.
The same logic has resulted in the anemic adoption of IPv6, which is NOT a correct solution, because it doesn't work properly for large swaths of the public.
The correct answer always was, and will continue to be, to use internal routes for internal routing, and external routes for external. Clashes with your LAN? Too fucking bad.
This sort of pushing out of externalizes onto the customer results in the same exact outcome anyway: customer gets screwed.
The customer always gets screwed. Don't rationalize the incompetence of engineers who should know better, and corporate execs who don't give a fuck.
Do that and the FCC will come down on you like a tonne of bricks, and I feel that is absolutely justified.
That said, we agree on the probable cause of this particular issue.
Unless you're actually trying to say that AT&T has a grudge against Cloudflare and are only doing this to harm their company.
This is something more like negligence or gross negligence.
That's not wholly unthinkable as far as AT&T is concerned. They're historically a bad player.
This caused issues where my phone would try to get on wifi, but the DHCPACK would be sent along on the existing interface rather than the one coming up. So the wifi icon was continually bouncing back and forth. My only solution was to go into airplane mode and bring the cellular down before bringing the wifi up. I don't think Android ever addressed this issue, and I had to switch around the entire subnet to avoid the conflicts.
If I knew enough about how Android worked, I'd write a patch to have all android interfaces in their own linux netns, with the dhcp client exec'd in that netns, that way you'd never have to worry about this sort of conflict.
One requirement of CGNAT, for instance, requires that the carrier's router be able to handle "address crashes". 
> In particular, Shared Address Space can only be used in Service Provider networks OR on routing equipment that is able to do address translation across router interfaces when the addresses are identical on two different interfaces.
(edit: no, very clearly, "Devices MUST be capable of performing address translation when identical Shared Address Space ranges are used on two different interfaces." )
Also, is it wrong to assume that cellular networks are able to handle "address crashes" due to the inherent centralization that comes as a result of having clients maintain the same IP (and same connections) as the device hops from tower to tower? Maybe I don't understand the topics at play here...
The 18.104.22.168/8 range was owned by IANA from _1981_ up until 2010, when it was transfered to APNIC. (The 22.214.171.124/8 range was also owned by IANA until 2010, thentransfered to RIPE NCC).
If you want to get technical, use of the space could be construed as theft.
As for the continuity issue, it was stated that it was an old device, so they have no responsibility to continue supporting it, and considering the age of the device in question, it may not be able to connect to the existing network.
Reversing that would likely require AT&T to make a firewall change to literally every piece of equipment they operate, and that's assuming that they don't use the blocks internally. That, and I can guarantee that some customer, somewhere, would be using whatever IP they chose.
I'm eagerly awaiting your evidence for this statement, which to my knowledge is entirely incorrect.
So far's I know, existing uses of 126.96.36.199 have almost universally been illegitimate (e.g. captive portals, internal-ish services, …)
Otherwise, you can purchase a different device, preferably a Nexus/Pixel, or at least one that's unlocked. If that's impossible for you then, yes, you're stuck with AT&T's "best efforts."
stupid to think using the IP was a good idea
malice to break my device in order to paper over their stupidity.
And from a telecom no less - of all people, they should know better.
Technological websites noted that by using 188.8.131.52 as the IP address for their service, Cloudflare created problems with existing setups. While 184.108.40.206 was not a reserved IP address, it was and is used by many existing routers (mostly those sold by Cisco Systems) and companies for hosting login pages to private networks, exit pages or other purposes, rendering the use of 220.127.116.11 as a manually configured DNS server impossible on those systems. Additionally, 18.104.22.168 is blocked on many networks and by multiple ISPs because the simplicity of the address means that it was previously often used for testing purposes and not legitimate use. These previous uses has lead to a huge influx of "garbage" data to Cloudflare's servers.
A wake-up call for all those (ab)users of public address space is also desperately needed. All IPv4 addresses will soon be allocated. Failure to use only private address spaces will cause problems, very soon.
CloudFlare likely did this on purpose, because so many people can't get their heads out of their own asses and follow spec. Now there's a big spotlight on the people purposefully breaking the network. And it will be fixed, eventually, whereas previously, AT&T would have just said "take a hike".