
AT&T stealing/misrouting 1.1.1.0/29 for some of their residential customers - harshreality
https://www.dslreports.com/forum/r31901379-AT-T-gateway-5268ac-maybe-others-misrouting-1-1-1-0-24
======
MertsA
Sigh, those AT&T gateways are such a dumpster fire. What's sad is that it's
not likely that there will be an easy way to fix this as those gateways do not
have any kind of bridge mode which is a crime in and of itself. The best
they've got is DMZ+ mode which fakes giving the LAN device the public IP
address but it's still doing NAT behind the scenes and routing everything as
normal so it doesn't work for any IP protocol other than TCP, UDP, and ICMP so
no IPSEC tunnels or GRE with those modems.

~~~
toast0
The biggest crime is that AT&T uses these on their FTTH to turn ethernet from
the GPON transceiver into ethernet for the home. Thankfully, in that
application, you can bypass it, but it's still a tragedy. My equipment is
perfectly capable of handling Ethernet, thanks (it's also perfectly capable of
bridging 802.1x packets to the gateway, and keeping all the rest of the
packets :)

~~~
benjaminl
Do you have a write-up of this anywhere? When I installed AT&T fiber I looked
into this briefly, but I didn't find any success stories other than manual
VLAN reassignment.

~~~
boshaus
[https://bzsparks.com/2016/10/05/using-an-ubiquiti-
edgerouter...](https://bzsparks.com/2016/10/05/using-an-ubiquiti-edgerouter-
with-att-gigapower-fiber/) here's an example for a ubiquiti edgerouter to not
have to do the vlan garbage. I'm going to try getting it to work on pfsense
later this week.

------
CaliforniaKarl
I think Icomera’s bus WiFi systems also use 1.1.1.1 somewhere in the hotspot
process.

TBH, I’m not surprised, and this definitely isn’t the first time something
like this has happened. For example, Hamachi (before being bought by LogMeIn)
squatted on 5.0.0.0/8, and then (when 5.0.0.0/8 got allocated) on 25.0.0.0/8
(which was already allocated to the British MoD).

This is really nothing new, and now that I think of it, not really surprising!

~~~
buzer
It's probably running Cisco WLC.
[https://www.reddit.com/r/networking/comments/88qt5k/announci...](https://www.reddit.com/r/networking/comments/88qt5k/announcing_1111_the_fastest_privacyfirst_consumer/dwmlcst/)

------
DominoTree
Semi-related: once some hardware arrives, I should be able to finish finding a
way to rip the 802.1x certs off of the 5286AC so we can use our own routers,
or at least put these things into a proper bridged mode.

[https://spun.io/2018/03/18/getting-into-the-pace-5268ac-
rout...](https://spun.io/2018/03/18/getting-into-the-pace-5268ac-router-
part-1/)

~~~
rhexs
Please fully disclose your work! I was looking into this earlier and found the
other post you mentioned. Unfortunately, they stopped following up with RE
details on the 5286AC after they got a bunch of CVEs assigned. Was really
disappointing.

Either way, thank you! Would you be willing to send me the URL for the
firmware? I don't have time to desolder anything at the moment but would love
to look at the image.

~~~
DominoTree
Update: private keys acquired. Need to figure out the password for these now.
[https://twitter.com/DominoTree/status/984272671549677568](https://twitter.com/DominoTree/status/984272671549677568)

Here are links to current firmware images:

[http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.5300...](http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.530094-PROD/5268.install.pkgstream)
[http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.5300...](http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.530094-PROD/att_config.pkgstream)
[http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.5300...](http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.530094-PROD/att_eapol-
certs.pkgstream)
[http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.5300...](http://gateway.c01.sbcglobal.net/firmware/00D09E/10.6.0.530094-PROD/att_cms-
certs.pkgstream)

------
jlgaddis
Yep, I noticed this yesterday and mentioned it [0] when trying out
Cloudflare's new public DNS service.

A Cloudflare employee (marty) also mentioned on NANOG that they were aware of
several other devices with the same issue.

[0]:
[https://news.ycombinator.com/item?id=16729845](https://news.ycombinator.com/item?id=16729845)

------
chx
What is truly disturbing here... who wrote these firmwares? You'd think
something as widely distributed as an Internet gateway for a major ISP would
be written by someone who at least know some Internet basics (like which IP
ranges are public and which are private). And, how do you test if this can
happen. And if they made this mistake then what else is lurking in that
particular piece of software?

~~~
zentiggr
Wait, you weren't aware that most equipment manufacturers' software teams
leave every edge case and design decision to a coin toss between 'fastest to
code' and 'least effort to code' and let the rest of us bitch and scream about
the stupidest of egregious bs for years until that piece of gear isn't sold
anymore, then do it again with the next model?

~~~
arghwhat
"Weren't you aware that most equipment manufacturers' _managers_
downprioritize quality and demand that the developers to produce code that can
handle the managers' incomplete solution requirements given the smallest
possible budget?"

FTFY

------
RKearney
Comcast / xfinity was null routing 1.1.1.1/32 for residential customers up
until Monday morning/afternoon. They removed the null route fairly quickly. I
assume it was done to reduce the amount of garbage traffic going to 1.1.1.1
from hitting Comcast's core.

~~~
raverbashing
Which garbage traffic? Where does it come from?

~~~
matthew-wegner
Part of CloudFlare’s arrangement with APNIC is to study this traffic.

See “Enter 1.1.1.1” section on this post:
[https://blog.cloudflare.com/announcing-1111/](https://blog.cloudflare.com/announcing-1111/)

~~~
cpeterso
So is all this 1.1.1.1 garbage traffic now directed to Cloudflare's servers?
Or will ISPs port filter everything sent to 1.1.1.1 except DNS (and DNS-over-
HTTPS) requests headed for Cloudflare?

~~~
Arnt
Can't really filter out such things efficiently, so Cloudflare gets it, yes.

[https://www.nanog.org/sites/default/files/wed.general.traffi...](https://www.nanog.org/sites/default/files/wed.general.trafficdiversion.serodio.10.pdf)

~~~
PhantomGremlin
Is there still some sort of traffic "balance" requirement for settlement-fee
peering?

Cloudflare sends so much data out that they probably want to get more sent to
them so things balance out. The more garbage traffic, the better! :)

I'm totally out of my bailiwick here, it can't be that simple, can it?

------
joeseeder
RIPE Labs ran some tests back in a day, on this subnet, with interrsting
conclusions

[https://labs.ripe.net/Members/franz/content-
pollution-18](https://labs.ripe.net/Members/franz/content-pollution-18)

~~~
taurath
Super interesting! It’s the IP address space equivalent of a no mans land - a
dead zone where stray packet radiation will cook your servers. One can only
wonder what goes on under the surface..

------
nodesocket
I just got bite by this. If your ISP is AT&T (I have AT&T fiber) don't use
1.1.1.1, only use the CloudFlare secondary DNS resolver 1.0.0.1.

    
    
         <<>> DiG 9.10.6 <<>> google.com @1.1.1.1
        ;; global options: +cmd
        ;; connection timed out; no servers could be reached

------
eli
T-Mobile US did (and maybe still does?) hijack 4.2.2.2 and 8.8.8.8 to silently
redirect them to their own crappy DNS server.

[http://esd.io/blog/t-mobile-dns-hijack.html](http://esd.io/blog/t-mobile-dns-
hijack.html)

~~~
taborj
From that link you provided, there's an update dated 11/5/2014 stating they
don't do that anymore.

~~~
eli
I wrote the link :) Someone sent me a message a while back that it was
happening again but I haven't investigated myself.

------
swiley
Never ever ever use the hardware firewall that your ISP tries to sell/rent to
you.

Better still is to use an old Linux desktop with IPtables doing masquarade,
that's what these cheap things are doing anyway but now you have control.

~~~
Klathmon
Sadly you can't with AT&T, they have a cert on their routers and won't give
out any IPs to anything else other than their hardware.

------
Coincoin
Are people really that surprised 1.1.1.1 won't work everywhere?

I suck at networking, but I worked that job just long enough to know there is
some pretty horrible stuff such as 1.1.1.1 being hard coded all over the
place, even in high-end hardware.

AT&T's crime isn't really that they are stealing stuff, but rather that they
aren't doing anything to fix the millions of legacy stuff with broken 1.1.1.1

Granted, the most plausible explanation is they thought: "To hell with it,
it's broken, we are going to use it."

------
gerardnll
Few ISPs (Movistar, Vodafone) here in Spain also use 1.1.1.1 for router
internal stuff... So we can only use 1.0.0.1 at the moment.

------
aarongolliver
My CenturyLink provided gateway gobbles up 1.1.1.1 as well, thankfully 1.0.0.1
works.

------
wemdyjreichert
Can confirm. Have that same router; same issue. 1.0.0.1 works fine, though.

------
robertcope
I didn't have any problem getting off the ATT Fiber network, but it was
getting dropped down the line towards the datacenter.

------
saas_co_de
does the term "stealing" really apply here?

~~~
tialaramex
The usual non-technical word that would get picked here is "hijacking". If I
get on a bus home, and some nutjob with a gun to the driver's head insists it
goes to Springfield instead, the bus wasn't stolen, but it was hijacked and
now I don't get where I was headed.

Hijacking can occur in several places in Internet infrastructure, but
hijacking IP addresses is arguably the worst since there's almost nothing we
can do about it. Cloudflare might have (perhaps even purposefully) struck the
one thing you can do if you want to, which is put a popular but optional
service on the address, and then let ordinary users scream blue murder because
it doesn't work.

[ Other examples of hijacking: Web browsers will fetch the path /favicon.ico
from your site. Why? Because Internet Explorer did that to add "favourite
icons" for web sites, so now that's all you can use it for;
administrator@example.com can't just be the email address for somebody who
fancied the handle "administrator". Why? Because Certificate Authorities
decided that if somebody receives email sent to administrator@example.com that
person must be authorised to have certificates for any name in example.com. No
existing rules told them this was safe, but they did it anyway, so now you
have to allow for that ]

~~~
tialaramex
Oh, I should point out that "The Internet" (to the extent it's any distinct
thing) also hijacks things. All ISO/ITU object identifiers used in Internet
standards are under the OID 1.3.6.1, but, er, 1.3.6 belongs to the US
Department of Defense, so how did the Internet get the nice compact 1.3.6.1?

Turns out there's just an RFC from 1988 which says "This memo assumes that DoD
will allocate a node to the Internet community", it says it assumes this will
be 1.3.6.1, and of course thirty years later it would be pointless to say "No,
the DoD does not allocate this node to you".

~~~
arghwhat
To be more specific, RFC 1065 states its use of the subtree that was initially
held by the U.S. National Bureau of Standards. The NBS transferred the subtree
to DoD, which had not stated how it intended to manage it. Thus, the DoD might
have chosen a different value for the last digit of the internet subtree, but
that's all.

This isn't really hijacking, although it does seem a bit haphazard.
Considering the "official" handling of the subtree transfer, I suspect that
the DoD did accept the decision after the RFC was submitted as a draft, with
the RFC just not updated to reflect this.

30 years later, I don't think anyone cares about OIDs.

~~~
tialaramex
"I suspect that the DoD did accept the decision after the RFC was submitted"

You suspect that an enormous Federal bureaucracy "accepted" something but it
isn't written down anywhere? Does that sound right to you?

If you mean "accept" in the tacit sense that they can't do anything about it
now, well, of course, that's why I called it "hijacking". I can't do anything
about the fact I'll be home late when my bus is hijacked, but let's not
pretend I've "accepted" the new destination and somehow now the nutjob with a
gun has my permission to take it there.

Funny thing about OIDs, everything that needs to enumerate things and touches
any of the ISO/ITU X series standards uses OIDs, yes, still in 2018 and
presumably forever. For example you might have noticed that TLS 1.3 is now
(probably) finished. Grep through that and you'll find it mentions all the
OIDs you need to use to make it work, and introduces OID filters so that you
can write key-value matches for client certificates like "I only want client
certs which have the following policy OID".

OIDs are fine, there is no reason to create a new parallel system that would
work the same way and presumably either duplicate the OID hierarchy or try to
displace it.

------
notafxn
When 1.1.1.1 launched here in Spain, it was inaccessible from several major
carriers. Some had network-wide routing problems to that IP address, and some
had installed CPEs that included static routes to 1.1.1.0/24 and stuff like
that, probably for internal purposes.

Nothing of this strikes me as odd, let alone malicious. It's such a weird IP
range, I even remember having my LAN configured as 1.0.0.0/24 at some point,
because who would ever use those IP addresses?

Also reminds me of when Spanish ISPs were given IP ranges by RIPE for their
customers beginning with 37.* -- those had never been used, so many network
administrators had them added to their bogon list, which meant for those
customers lots of web pages were inaccessible. The solution was to reboot
their CPEs until they got a good ol' IP address from the ol' ranges :D

~~~
darklajid
Nothing like that is an excuse. You have 10.0.0.0 for that - it's huge and you
can use it for whatever you want without stepping on anyone's toes.

There are absolutely zero reasons I've seen so far (I'd be interested to hear
abstruse ones? "I thought who cares" isn't one) to avoid using one of the
private ranges.

10.0.0.0/8 isn't even noticeably harder to remember or type, really.

~~~
tssva
Using the 10/8 or any of the other RFC1918 had great potential to step on
their customers toes. That is exactly why rightly or wrongly they used the
1.1.1.0/24 range. Hardware manufacturers generally used the range for
interfaces that were local to the device and often only used on interfaces
internal to the device. They knew this equipment would be deployed into
environments where RFC1918 addressing would be used but they had no idea what
RFC1918 address ranges, so using addressing from the RFC1918 networks meant
potentially impacting their customer's data. They chose to instead use
addressing which at the time they believed would not impact their customers.

APnic is not blameless here. They knew the issues with this space when it was
assigned to them as a research network. For quite awhile they allowed Google
to advertise the space and collect data on it's usage. I assume Google no
longer was providing the infrastructure to do so and APnic saw an opportunity
to have someone collect data for them for free.

Collecting data on traffic sent to this ip range is one thing but approving
its use for a service available to the public knowing the accompanying issues
much of the public would have accesssing it is in my opinion not responsible
use of a research network.

~~~
Dylan16807
> Using the 10/8 or any of the other RFC1918 had great potential to step on
> their customers toes.

There are better options though. Why not the class E reserved addresses, or
just using some address space you actually own?

Though apparently an RFC from... 2012... has a solution. 100.64.0.0/10 is for
internal ISP use.

