
Possible BGP hijack of 1.1.1.1 - pstadler
https://bgpstream.com/event/138295
======
akw28888
I'm using AnchNet's services. And We've asked AnchNet when I recieved a e-mail
from our BGPMon. They said their staff was configured a wrong config on
router. Also they don't know 1.1.1.0/24 is used by CloudFlare&APNIC. So they
used this prefix to test.

~~~
sdfgdfhjdgj
Why let people access BGP that don't even know that 1.0.0.0/8 or 1.1.1.0/24
are part of the public internet or that decide they can use random prefixes to
"test" things? :-/

~~~
charonn0
To be fair 1.1.1.1 had been unassigned/non-routable up until April.

~~~
RKearney
To be fair, unallocated or unassigned IP space isn't fair game to use for
testing outside of an air gapped lab. I've never in my career thought it would
be a good idea to "test" unallocated public unicast address space on my edge
routers.

~~~
Fnoord

      On an airgapped lab it is bad practice. Same -though DNS related- with using *.local as a LAN TLD
    

We _just_ should not.

~~~
a1xndr
.local is perfectly fine according to the IETF
[https://tools.ietf.org/html/rfc6762](https://tools.ietf.org/html/rfc6762)

~~~
slrz
For mDNS, yes. There are resolvers that don't even try DNS when looking up a
name in .local.

Some of them were made more lenient because just as Microsoft finally stopped
encouraging this nonsense, the Kubernetes people picked it up from ancient
Windows Server folklore and apparently decided to make it web-scale.

------
amaccuish
Does anyone else find it sort of beautiful watching replays of events like
this? It's amazing to watch how the routers organise themselves, making and
breaking connections when needed.

~~~
kchr
There are at least two of us, dear friend!

~~~
davidkuhta
Three's company.

------
zimbatm
ASN 58879 belongs to Shanghai Anchang Network Security Technology Co.,Ltd
(China) according to [https://ipinfo.io/AS58879](https://ipinfo.io/AS58879)

website: [https://www.anchnet.com/](https://www.anchnet.com/)

------
jacquesm
Ah! That may have been the reason why my site wasn't resolving earlier today.
It was the weirdest situation with people from all over the planet complaining
without any apparent pattern, a RIPE check of the site from 10 different
locations showed no issues in connectivity.

Thanks for posting this.

~~~
minxomat
No, the issue persists. While I can access your site from mobile and
residential connection, any static business connection fails. No 1.1.1.1
involved. I tested this with two different Fibre connections (Berlin).

~~~
jacquesm
Is the BGP hijack over?

~~~
ColinWright
As a data point - your site still isn't loading for me.

IP : 79.69.113.214

Time: Tue May 29 15:28:30 BST 2018

~~~
jacquesm
I wonder if it is a resolution issue or an access issue.

What happens when you go to: [http://62.129.133.242/](http://62.129.133.242/)
? That should come up with a 'domain for sale' page, that's the same server.

~~~
carbocation
Not loading for me from MA, USA.

    
    
        $ httpstat http://62.129.133.242/
        2018/05/29 10:54:04 unable to connect to host 62.129.133.242:80: dial tcp 62.129.133.242:80: connect: connection timed out

~~~
Diederich
+1 for httpstat; I'd not seen this utility before, quite nice!

------
solotronics
Large companies misuse "unassigned" space all the time. I have heard engineers
at my work propose using the non public routed DOD /8 before. Not on my watch!

~~~
TheAceOfHearts
If you happen to know why, could you explain their reasoning? In what ways are
10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16 insufficient?

~~~
solotronics
I help design one of big 3 cloud providers and we're about to run out of
private space for customer IPv4. We are addressing this is in a number of ways
but I think others have run into this same issue.

------
walrus01
Network engineer here: I'm going to guess that this is a mistaken effort on
the part of a Chinese ISP or the GFW to hijack traffic to 1.1.1.1 internally
within China, but probably not intended to propagate beyond the major Chinese
international-transit-ISP's connections to the global Internet. BCP38 is your
friend.

~~~
unethical_ban
It's been a while so I should re-read it, but I thought BCP38 was best applied
at the terminal ISP/client network level. If you're a transit network, you
can't do that kind of fitering because you have a legitimate chance of
forwarding traffic to and from any network address.

~~~
walrus01
More about the principles of not just ingress filtering, but the same as
BCP38, but as an ISP with its own ASN, applied to your own egress:

a) don't announce shit you don't own

b) know how to set up ACLs and prefix-list filters on your own egress IP space
announcements which face towards your peers and IP transit upstreams.

Conversely, as a big ISP which has many small ASNs downstream of it, be
responsible and set up filters on your own ingress which prevent your
customers from announcing mistaken shit to you.

Using an example of a clueful and attentive major ISP: For example if you are
a small to medium sized regional ISP and buy IP transit from NTT, one of the
world's top-ten global commercial transit providers, they actually do take the
time to verify each and every prefix you announce to them and will require an
interaction with their NOC if you want to announce a new /22.

------
fiber
I doubt that this is a genuine hijacking attempt. All it takes is a Cisco
router and some IT admin making up an address.

~~~
bodyfour
Agreed. As many pointed out when the 1.1.1.1 DNS service was introduced, it's
an address that is often used (incorrectly) as an internal or temporary IP.
Then all it takes is a slight mistake in your route redistribution and
suddenly you can find yourself accidentally announcing the prefix to eBGP.

I wouldn't be surprised if this becomes a semi-regular occurrence.

~~~
walrus01
Hanlon's razor applies to a great degree. I have no doubt that there are a
great many enterprise-type organizations that have been using 1.0.0.0/8
internally and cluelessly for a very long time.

------
ancarda
How effective is this? Looking at
[https://bgp.he.net/ip/1.1.1.1](https://bgp.he.net/ip/1.1.1.1), 1.1.1.0/24 is
apparently "ROA Signed and Valid". I don't know a lot about BGP. Does this
mean hijacking this subnet is a bit harder than unsigned ones because some or
all ISPs verify this announcement? Or is it faster/easier to detect?

Maybe a wider question: is there some way to prevent BGP hijacking?

~~~
walrus01
Basically, the bigger Chinese ISPs that are upstream of this small one which
is making the false 1.1.1.0/24 announcement are not actually verifying that
this small ISP is allowed to announce the space.

As for prevention, the only thing that will work is proper use of IRR/route
registries and RPKI validation of peer announcements. Which a great many ISPs
do not currently do.

[https://www.noction.com/blog/bgp-hijacking](https://www.noction.com/blog/bgp-
hijacking)

The other method is more blunt, and can be more effective if the people with
'enable' on various ASNs' core and edge routers actually have a spine. ISPs
which repeatedly announce space that they're not allocated (as per RIPE, ARIN,
APNIC, AFRINIC records) should be depeered by their local peers, and their
owners/operators publicly shamed. It's a reputation thing. As a neighbor of
other, more clueful ISPs, it's basically the same thing as being a bad
neighbor by leaving garbage all over your front lawn and causing a public
nuisance with loud parties and trashy behavior.

------
n1c
Interesting!

My ping to that address went terrible for a brief window today -
[https://i.imgur.com/KjCcBeT.png](https://i.imgur.com/KjCcBeT.png)

Wonder if this was the cause.

*edit: I'm in Cape Town and the ping looks what was routing to a DC down the road decided to go to Europe instead.

~~~
ancarda
Hey, what software are you using to collect the data and display that graph? I
wouldn't mind running something on my server that could notify or log if
1.1.1.1 (and other services I rely on) are slow or down.

~~~
benjojo12
That looks like smokeping

~~~
walrus01
more specifically looks like smokeping, rra file storage and rrdtool to draw
the graph. Compared to modern things like influxdb+grafana it is a very
oldschool setup but still works perfectly fine.

------
highace
What does this mean for those unfamiliar?

~~~
jpollock
If true it means that Cloudflare's DNS server can't be trusted.

~~~
Latteland
That's not correct, anyone could 'accidentally' do this to every other
provider. It's not a special weakness of cloudflare.

~~~
jpollock
I'm sorry, I went for brevity because someone was asking for possible impacts.
If I were using 1.1.1.1 as a DNS server and saw this news story, I would
change to a different DNS server until the problem was resolved.

My goal was to provide actionable information quickly.

I never said it was specific to Cloudflare. :) It is specifically a mistake
which would break an assumption - that putting 1.1.1.1 into your resolver
results in an answer from Cloudflare. DNS doesn't necessarily have any
protections (not current, so maybe they were added?), so the only level of
protection is that the IP address routes UDP traffic where we expect it to.

It also isn't a long-term problem, it only remains for the length of time the
route is wrong.

It could also be argued that we're already trusting every router between the
device and 1.1.1.1 anyways, so there's not much difference. Except that
there's already a trust relationship between those groups, and the new route
subverts them.

It's the same level of risk if someone had done a BGP hijack of any backbone
router.

~~~
Latteland
Okay, thanks for your polite explanation.

~~~
Lennie
If you use your providers resolvers you would be very unlikely to have this
problem (it's not likely someone will re-route the traffic on the network of
the provider themselves).

------
floatingatoll
Would this affect certificate-validating clients doing DNS-over-HTTPS to
1.1.1.1 — doesn’t it have an ipAddress certificate and demand HTTPS resolution
only?

~~~
moviuro
Well, if you control the host behind the IP, you could have any CA issue a
challenge, and successfully pass it (e.g. if Let's encrypt uses the erroneous
routes).

So no. The only thing protecting you would be to have the expected hash of the
certificate you expect to see (TOFU - Trust on First use, though you're
screwed if you didn't contact 1.1.1.1 before the incident!).

~~~
floatingatoll
Is there a CAA equivalent for ARIN assignments?

~~~
devicenull
RPKI, but it's barely used

~~~
vbernat
This is currently used to sign ROA. A rogue actor can easily work around that
by including the original AS in the AS path of the announce.

------
ChuckMcM
So who is going to tell the 13 peers that they should not accept BGP path
advertisements for 1.1.1.0 from anyone but Cloudflare?

------
throw9991999
I use 1.1.1.1 Do I need to do anything? Can I just continue using it or do I
need to clear some cache etc?

------
spacenick88
That awkward moment when you read an IP and the first thought is "But that
belongs to Cloudflare I read about this"

------
vesche
Are people here really using 1.1.1.1 as a DNS server...? Do people here
_really_ think that Cloudflare isn't giving your data away to _someone_? I
have been using DNS servers from OpenNIC for sometime now, and I will continue
to.

------
amaccuish
And that is why I'm using dns over tls :)

~~~
moviuro
Not enough. You have to check the certificate's fingerprint, along with its
validity.

TLS is not a silver bullet. If an attacker controls the host behind what
everyone believes to be 1.1.1.1, nothing is to prevent them from applying for
a legit certificate.

~~~
tialaramex
Nothing prevents them from applying but...

* They need to do that, and get the resulting certificate, and install it, during the attack. The weirder the product (and certificates for IP addresses are relatively weird) the more humans end up involved in your order, and humans are slow.

* This leaves a smoking gun in the Certificate Transparency logs. So we all get to know (in maximum 24 hours but usually the reality will be minutes) about this extra certificate.

~~~
peterwwillis
1) would take about 20 seconds, thanks to Let's Encrypt, but probably only
marginally more time for some other CA with an API.

2) Who exactly is staring at CT logs and going "oh, I don't remember this
domain using this CA, maybe I should investigate this" ? Sure there's a record
of it. Doesn't really matter _during_ an attack, because public attacks like
this aren't intended to last long.

All you need is a half hour or less to steal a couple hundred million from a
bank, or cryptocurrency wallet, using this attack. That's more than enough
incentive for most unscrupulous 3rd world hackers.

If you're an authoritarian government, you could require CAs in your country
to selectively quiet CT logs by certain users, and just issue certs willy-
nilly for your private government org for MITM purposes. Google Chrome would
detect them for Google-owned properties, but smaller sites would never know.
And spy agencies can use this at their leisure and basically never be held
accountable, because world politics.

Let's face it. The CA system is a joke and BGP is the butt of it.

~~~
tialaramex
"would take about 20 seconds, thanks to Let's Encrypt"

Let's Encrypt does not offer certificates for IP addresses. They choose only
to offer DV certificates using methods 3.2.2.4.6 and 3.2.2.4.7

How much of your hypothetical half hour will you spend trying to figure out
why your chosen ACME client reports "Policy forbids issuance" when asked for
an IP address?

As to who is staring at CT logs, well there's the fun thing about the design
of CT, the _logs_ aren't where you would be staring, you would be looking at a
_monitor_ and a monitor can be configured to do whatever it so happens you
think is important. We know that commercial CAs already sell monitoring as
part of "Enterprise security" type offerings.

Facebook took... I want to say minutes here, but I can't find an exact
timeline, to spot that a certificate had been issued by Let's Encrypt for a
name in their DNS hierarchy and begin investigating what went wrong.

Certificate Transparency isn't finished. As it stands today your authoritarian
government "only" needs to ensure that nobody notices these shenanigans as
unavoidably they create a "smoking gun" which would lead to their pet CA being
distrusted. That's a tall order, but certainly not impossible in the short
term, or for attacks in which the bogus certs are shown to a small number of
individual targets rather than a broad population that will invariably notice
by accident.

But longer term CT is intended for use with a gossip protocol so that it's
impossible for the pretence to be kept up. Sooner or later a node somewhere
will end up realising that there's an inconsistency, either it has seen SCTs
that weren't logged, or it has seen logs that aren't consistent with the logs
other nodes see, either of which is a matter for distrust.

BGP has no relationship to the Web PKI, which is what I presume you're
referring to by "the CA system". The relatively small number of parties
interested in BGP have developed their own PKI to fit their needs.

~~~
peterwwillis
BGP allows anyone to issue fake certs, so it does have a relationship to web
PKI.

And sure, if you have a couple billion dollars, you can set up fancy
infrastructure and response teams to monitor the whole web (that you are aware
of, that participate in CT) for a strangely-issued cert.

Or you could just get a cert issued by the CA you normally use. In which case,
now we have to track some kind of "customer number" per CA. If you use more
than one CA (Google does) now you're tracking different customer numbers on
different CAs. And all that has to be standardized.

Most of these "fixes" for web PKI's glaring holes are intended for large
multinational corporations, or are optional, or specific to a particular
browser, and don't address the main concern: _do not trust an IP address to be
who it claims to be_.

~~~
tialaramex
No, you wouldn't use a "customer number". The usual approach taken for these
genuinely concerned outfits goes like this:

1\. Agree contractual terms with particular Certificate Authorities in which
all certs for names in your hierarchy need explicit approval from your
security people.

2\. Set DNSSEC-secured CAA records for your names forbidding issuance by other
Certificate Authorities.

This funnels requests from hypothetical bad guys into your security people,
which is exactly what they don't want. It loses you the shiny capability to do
"spur of the moment" issuance, but presumably if you want these sort of terms
the phrase "spur of the moment" causes you to start writhing and clutching
your throat anyway. Insider threats will usually be much _worse_ than
outsiders.

As to things being "specific to a particular browser" we can't and don't want
to be able to force, say, Microsoft and Apple to do things just because
everybody else decided they're a good idea.

The same with the trust store programmes. I can't make Microsoft take this
seriously, but Microsoft can't make me use their SChannel and associated trust
store. Maybe you have a lot more pull with Microsoft than I do.

If we're in a world where the "easiest" way to get a bogus certificate is now
to do global BGP hijack of an entire /24 then I think we're on the right
track.

------
jacksmith21006
Curious how is this different than the similar? issue with Amazon route 53
getting hijacked not too long ago?

