
AT&T explains why it blocked Cloudflare DNS: It was just an accident - jgrahamc
https://arstechnica.com/information-technology/2018/05/att-is-blocking-cloudflares-privacy-focused-dns-calls-it-an-accident/
======
jamiek88
Well that thread of conspiracy theories, hot air and cries of censorship
yesterday was instructive.

These corporations have zero benefit of the doubt level of trust.

A few voices to be fair pointed out 1.1.1.1 is often incorrectly used and
issues with it becoming ‘live’ mostly likely caused the outage.

Not judging this response btw, I can clearly see why ATT and other big telco
wouldn’t get much leeway.

~~~
amaccuish
I wasn't one of the conspiracy theorists but I do expect ISPs to get these
things right. It's literally their job, they should be experts.

~~~
supertrope
AT&T has long provided a sub-par Internet connection service.

Their wireless router and DSL modem "U-Verse" gateway device lacks a true
bridge mode, has a small NAT table, and you must use it because it contains a
802.1x certificate. You can use your own router with a rather involved hack:
[http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-
NA...](http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NAT-table-
limits) They instructed customers (including small businesses) to get off
10.X.X.X, presumably because they were considering CG-NAT, although they have
since done some IPv6 deployment.

The original AT&T faded away with declining long distance revenue which they
milked to the end until cellphones and Internet access became the main market.
It took tax credits meant for "video dial tone" broadband deployment and used
them to build their cellular network.

Its ADSL deployment use PPPoE and ATM which can cause MTU mismatch. ATM itself
is a telco mistake. European access oligopoly members wanted 32 bytes because
that suited voice frame transport and they thought data transport would never
sell. American counterparts wanted 64 bytes for better data delivery. They
compromised on 53: 48 for payload and 5 for header. With newer "U-Verse" ASDL
they've finally eliminated this layer of overhead. (But no more 3rd party
modems).

It decided to deploy VDSL instead of make the true upgrade to fiber. But they
also cut costs by not building enough remote terminals so many customers have
slow and unreliable links. Fast VDSL speeds (e.g. 100Mbps) aren't possible if
customers are a mile away, or more. How much money have they spent on truck
rolls diagnosing issues? Is it less than just building a more robust network
to begin with?

Since 2005 they've been complaining that customers are using too much Internet
and they need to slice and dice the Internet with data caps, application
specific throttling, etc. And get more tax credits and deregulation which they
promise will be used for broadband investment.

~~~
tptacek
I've had better service from U-Verse and, more recently, AT&T Fiber than I
have from Comcast.

But really, the point this thread is making isn't about whether you approve of
all of AT&T's business decisions, but rather whether their backbone engineers
can properly manage a network.

I'm interested in the opinions of people who have managed default-free BGP
peering before on whether AT&T does a good job, but I'm not so much interested
in reading another dslreports thread about whether AT&T is a "good ISP".

~~~
supertrope
This technical glitch is in their CPE boxes not the backbone network.

~~~
tptacek
Good to know! Add to the set of people who's AT&T network engineering opinions
are interesting "anyone who has managed a very large scale distribution layer
network". :)

------
alex_young
OK, I'm not sure I buy this explanation. If they want to revert whatever
happened to 1.0.0.1, and fix 1.1.1.1, I'm fine with that.

Here's what I do know:

* I am an AT&T fiber customer.

* 1.1.1.1 did not resolve DNS for me any time I tested it.

* 1.0.0.1 did resolve DNS when I first tested it.

* Neither 1.1.1.1 or 1.0.0.1 work from the external interface of my AT&T router now.

* Strangely, 1.1.1.1 responds to ICMP requests from the diagnostic interface of the router, but not from the external interface.

* 1.0.0.1 does not respond to ICMP requests from the diagnostic interface of the router or from the external interface.

It sure looks like something upstream from my router is blocking requests to
1.0.0.1, and that this is new behavior.

Like I said earlier, I don't especially care what the reason is, as long as
they do something to fix it.

~~~
vermilingua
The reason you are getting an ICMP respose from 1.1.1.1 is that _your router_
is responding. This is consistent with the explanation. If you still want to
doubt it, clear all DNS and ARP tables, ping 1.1.1.1, and check the ARP entry
for it. It will match the MAC of your routers inward facing interface.

~~~
alex_young
From the same diagnostic interface:

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 38 byte packets 1
1dot1dot1dot1.cloudflare-dns.com (1.1.1.1) 0.327 ms 0.156 ms 0.120 ms

~~~
nitrogen
The fact that the reverse DNS is correct doesn't mean that it's reaching the
real deal. The fact that the ping time is fast enough that light could only
have traveled 5 or 6 miles one way means it's almost certainly not the real
deal.

~~~
ajross
FWIW: light can go ~23 miles in the 120us measured.

~~~
nitrogen
Hmm, not sure how I managed that; must have divided by 2 twice or something to
find one-way length.

------
_bxg1
I could believe they tried to sneak it through for malicious purposes and
wrote it off as a mistake when they were caught. I could also believe it was a
genuine mistake. Either way, I'm glad there are communities like Hacker News
to make a stink when stuff like this happens and get it into the spotlight.

------
abarringer
Assigning a 1.1.1.1 to a local modem interface? Cox: that ain't noth'n here
hold my beer" and watch a 10.x traceroute go seven hops out the door and
resolve somewhere on their network.

~~~
icedchai
There is a big difference: 10.x.x.x is for private use. Cox is using it
privately for device management. You can also use it privately and there will
be no conflict (assuming you are not attempting to manage those same devices,
which is probably a safe assumption 99.99% of the time.)

~~~
pacificmint
To be fair, there is a network range specifically allocated for carrier grade
nat (100.64/10), sounds like that might have been better to use here.

~~~
MertsA
Those IP addresses are just being assigned to their own networking equipment.
Cox using the CGNAT network for that would be just as wrong as you using it.
There's absolutely nothing wrong with Cox using private address space
internally. If anything, there's something wrong with routing requests to
private address space out your WAN in the first place.

What Cox is doing is completely appropriate and what AT&T and Cisco are doing
is squatting on someone else's IP assignments.

------
johnvega
They need to explain why it is an accident in detail, otherwise they are just
being incompetent for disrupting large number of their valued customer's
internet connections, and likely wasting their valuable time. They can't just
say it is unintentional, they need to convince me that it is. Technical word
salad will not cut it. It has to be clear to most people.

------
magoon
I believe this. I’ve had at&t business connections that black hole certain IPs
and ranges because they (incorrectly) use them internally. What’s worse is
their own techs spend a large amount of time troubleshooting because it’s not
clear to them.

------
half-kh-hacker
IIRC, 1.0.0.1 was also blocked by the firmware update.

------
louky
I've got AT&T fiber using a BGW210-700 with firmware 1.5.11 and 1.1.1.1 and
1.0.0.1 have been and still are blocked for me. I'll be keeping an eye on it
to see if it changes.

------
drawkbox
Large ISPs wanting to get into CDN revenues and DNS, just a little hiccup to
start testing the waters... ISPs are mad most people use Google or Cloudflare
DNS now instead of their own which facilitate their ad networks and future CDN
properties.

Large ISPs want that DNS endpoint and are aiming for CDNs as well, they are
gonna eventually sell it as other CDNs/DNS can't be workable due to issues
like this.

Large ISPs stopped innovating for decades and are mad others kept innovating
on top of the network. ISPs think they own the network and are taking back
this property not by innovation, but by bribes to 'representatives' and by
force.

Now that net neutrality is demolished and ISPs can sell your private data,
they have a reason to ruin DNS and CDNs to eventually own that for tracking
and revenue streams. We made IMMENSE mistakes in 2017 with allowing ISPs to
bribe their way into removing privacy protections and removing net neutrality.
We gave them reasons to dismantle systems built ON TOP of the network so they
can own those areas by controlling the network. 2017 was regressive for the
internet markets built on our internet utilities.

Rather than ISPs building their own companies to compete on top of the
network, they want to use their network lever to barge their way in with
bribes like the Kool-aid man through walls that shouldn't be breached.

ISPs are in all out war mode fighting being a utility or commodity, rather
than building competitive products or innovating on top of the network, making
the network better/faster/fast they want to win by force and milking it, not
by building products and improvements people want.

ISPs are also gunning to help build our government firewall/filter that is for
censorship and IP protection [1]. AT&T in particular has been a privileged
insider to the surveillance state and filtering[2]. It is also an extension of
FOSTA/SESTA censorship. Turns out, building the government filter for
surveillance also gets you a super efficient ad network by peering into all
private data. No way blocking Cloudflare was a mistake, it was ISPs kicking
the tires of their 'innovations'.

In 2017, we allowed emboldened network provider monopolies to get more
emboldened and now that they won, they want to run all ad networks and will do
so in unison with the surveillance state and filtering/censorship.

[1] [https://www.wired.com/2017/04/internet-censorship-is-
advanci...](https://www.wired.com/2017/04/internet-censorship-is-advancing-
under-trump/)

[2]
[https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)

~~~
PMan74
> most people use Google or Cloudflare DNS now

Do they? I'd say most people would be troubled to find their DNS settings let
alone have the confidence to change them. I'm open to correction on that, is
there any data on Google /CF DNS usage?

~~~
drawkbox
Most people that know anything about the internet have switched away from
their ISPs DNS largely due to tracking and slowness/peering.

These tools will spread and eventually everyone will, a big fear for the ISPs
who want to be ad networks and profiling systems.

> _is there any data on Google /CF DNS usage?_

Only ISPs would know that information, Google and Cloudflare know how many
people use it but not how many are connected that choose it. ISPs know who
uses what DNS, who uses VPNs and common clients and other competitive
advantages that should be protected. Pretty soon they will have some 'identity
protection' app that provides VPN and DNS which is used in their ad
network/tracking systems. Even better if large DNS providers are flaky due to
their own interference. This was definitely AT&T kicking the tires to see what
would happen, either that or they are incompetent and need competition. They
should be broken up either way.

ISPs now that they are able to sell your private data and build ad networks by
removing privacy protections, actively do not want others using
alternative/competitive DNS. Along with building the government filter and
surveillance state, ISPs are not in the business of providing network quality
and speeds, they are milking it with data caps and bundles noone wants, they
want to be the biggest/baddest ad network/profiling system ever invented, that
they of course control 100% over competitors.

From the OP article:

> _The blocking is affecting AT &T home Internet customers who use an AT&T
> gateway. Cloudflare unveiled its DNS service on April 1, and users in
> DSLReports forum threads almost immediately started complaining that they
> couldn't access it. One thread began on April 1, within hours of
> Cloudflare's announcement._

> _Cloudflare pitches 1.1.1.1 as a privacy tool that can help deter ISPs from
> monitoring one 's Internet usage. AT&T lobbied against broadband privacy
> rules last year, and the company used to charge fiber Internet customers
> extra for privacy. AT&T fiber customers who did not opt in to a traffic
> scanning system that analyzed Internet usage in order to deliver
> personalized ads had to pay at least $29 more per month than customers who
> consented to the scanning._

> _AT &T ended the controversial traffic scanning program in September 2016,
> but it says that it still wants the "flexibility" to expand advertising-
> focused business models to compete against Facebook, Amazon, and Google._

By allowing ISPs to remove net neutrality and privacy protections, we've
emboldened network provider monopolies to be even bigger and ultimately it
will harm internet freedom and competitive business on top of the network.
Events like this will happen more and more as they are empowered more and more
as they pull away from being privacy protected utilities, which they are but
don't want to be.

------
bradknowles
AT&T: Oh, gee. It sure would be a shame if something happened to that nice new
DNS service of yours....

------
dsl
It is obvious Cloudflare has no interest in operating a real recursive DNS
service. If they did, they would have provided additional backup IPs for users
they knew would be impacted by these issues.

Building a production service on what even APNIC refers to as "research space"
was foolish. If this was Cloudflares only line of business, they never would
have taken such a risk.

~~~
eastdakota
We (Cloudflare and APNIC) knew exactly what we were getting in to. We knew
this was the only way that we'd be able to reclaim the space to be usable.
And, in about a month, we've gone from 1.1.1.1 being routable by only 92% of
networks to 98.7% of networks — and climbing. Don't accept that you can't
change the status quo.

~~~
dogecoinbase
The space is allocated for research. To what end do you imagine it will be
"usable", and for whom?

~~~
jhall1468
No it was not.

1.0.0.0/8 was unassigned space until 2010 when it was assigned to APNIC. It
was never "allocated for research" and the reason nobody wanted it is all the
cross-talk with internal devices using something they shouldn't have.

~~~
icedchai
Whois literally says 1.1.1.0/24 is a research prefix.

    
    
      netnum:        1.1.1.0 - 1.1.1.255
      netname:        APNIC-LABS
      descr:          APNIC and Cloudflare DNS Resolver project
      descr:          Routed globally by AS13335/Cloudflare
      descr:          Research prefix for APNIC Labs

~~~
eli
APNIC designated 1.0.0.0/24 and 1.1.1.0/24 out of the whole /8 to APNIC Labs
for "research" _because_ they were getting so much bogus traffic. That just
happened a few years ago.

~~~
icedchai
I know.

