
Route leak incident on October 2, 2014 - jgrahamc
https://blog.cloudflare.com/route-leak-incident-on-october-2-2014/
======
jtchang
BGP is one of those things that make the internet fragile yet resilient at the
same time.

Fragile in that any person who controls an Autonomous System can advertise
routes and if your neighboring AS'es are not configured to filter out stuff it
will just keep propagating. Imagine a government deciding to just reroute
traffic to their data center maliciously. Obviously a good chunk of the
internet will go WTF and blacklist the route but not before you get a deluge
of data.

In the same way it is easy to route around traffic. Congestion in the midwest?
No problem. Send the traffic down to Texas. All this is built on network
admins trusting each other for the most part.

~~~
feld
BGP route hijacking is not uncommon, and it's used to capture traffic of your
victim without them knowing.

BGPMon can tell you if this is happening to your AS

[http://www.bgpmon.net/](http://www.bgpmon.net/)

------
lorddoig
I'm a little hazy on the technical details of BGP routing - am I correct in
thinking that if you're a first-class BGP citizen, you can just advertise
yourself as handling others' traffic and that's it, it just comes?

~~~
pyre
That's pretty much my understanding of it, though "it just comes" is a bit of
a simplification.

Edit (clarification):

\- BGP has no authentication so anyone can advertise 'anything' (still has to
be a valid address).

\- "it just comes" isn't entirely accurate as the changes propagate outwards
through your peers (in p2p fashion). I'm not sure what happens technically
when two networks are advertising themselves as serving a particular IP though
(I have a fuzzy idea, but don't know how the edge-cases would be resolved).

~~~
lorddoig
Well I imagine it would only be a portion of traffic. I wouldn't think a
router in the UK would capture any US to US traffic through this method, for
example. Is that what you mean?

~~~
nknighthb
BGP hops are actually AS (autonomous system) numbers, which are not closely
correlated with routers or distance.

If an ISP in Los Angeles transits five autonomous systems to reach an ISP in
New York, but four to reach an ISP in London, and the London ISP suddenly
starts advertising the New York ISP's IP addresses, oops, your packets are now
headed to the UK.

Great care is taken to avoid bizarre physical routes for packets, but it still
happens constantly. One DSL connection I used in Santa Clara had a propensity
for routing packets through Seattle to get to Dallas for a while.

~~~
tacticus
Also the smallest advertisement wins.

if an isp in india advertises all of amazons AWS ranges as /24 or even smaller
the routes will go there even if the bigger ranges are closer.

------
biot

      > For the time being, we have quarantined the Medellín data center
      > and disabled connectivity with Internexa.
    

Does this imply that it was CloudFlare's trust of Internexa's announced BGP
routing which caused or contributed to the outage? Did CloudFlare redirect
traffic that it should have handled internally to Internexa? If so, isn't it
more appropriate for CloudFlare to prioritize its own routes over those
claimed by external parties?

I would have anticipated that Internexa's routes would have affected routing
by third party networks (eg: Internexa claims it handles traffic for
CloudFlare's IP range, and Level3 redirects all that traffic to them) which
isn't something that CloudFlare can do much about other than notify Internexa
and those third parties of the problem and hope they resolve it themselves.
Having not worked directly with BGP, I'm sure I'm misunderstanding something
and would appreciate any additional clarification.

edit: Incidentally, this is the kind of scenario which prevented me from using
CloudFlare recently. I wanted to only CNAME our production web site to CF's
systems which is something they only offer with the $200/month Business plan
and not with the $20/month Pro plan. Otherwise, you have to delegate ALL of
your DNS for the entire domain to CloudFlare. As one user in the comments says
in response to how someone could have worked around the outage:

    
    
      "That [bypassing CF] would be a good idea, except cloudflare.com
       and control panel was inaccessible during this period too, so not
       sure how this could have been done..."
    

I really hope they revisit their policy of not allowing Pro customers to CNAME
individual sites to CloudFlare. Putting all your eggs into CloudFlare's basket
limits the ability to mitigate around these kinds of issues.

~~~
BuildTheRobots
>>Does this imply that it was CloudFlare's trust of Internexa's announced BGP
routing which caused or contributed to the outage? Did CloudFlare redirect
traffic that it should have handled internally to Internexa? If so, isn't it
more appropriate for CloudFlare to prioritize its own routes over those
claimed by external parties?

"This downtime was the result of a BGP route leak by Internexa, an ISP in
Latin America. Internexa accidentally directed large amounts of traffic
destined for CloudFlare data centers around the world to a single data center
in Medellín, Colombia. __This was the result of Internexa announcing via BGP
that their network, instead of ours, handled traffic for CloudFlare. __This
miscommunication caused a flood of traffic to quickly overwhelm the data
center in Medellín. The incident lasted 49 minutes, from 15:08UTC to 15:57UTC.
"

The problem wasn't that cloudflair thought their IPs were somewhere else, the
problem was that the _rest of the internet_ [1] thought cloudflair lived in
Medellin (somewhere else) and not with cloudflair.

[1] Complete hyperbole, but "huge swathes"[2] of the internet had the wrong
idea.

[2] "The exact impact of the route leak to our customers’ visitors depended on
the geography of the Internet. Traffic to CloudFlare’s customers sites dropped
by 50% in North America and 12% in Europe. The impact on our network in Asia
was isolated to China. Traffic from South America was also affected as data
centers there had to cope with an influx of traffic normally handled
elsewhere."

~~~
biot
I understand what you quoted (refer to what I wrote in the second paragraph).
If the problem is with the rest of the internet, why did CloudFlare quarantine
the Medellín datacenter? To use an analogy, if your company's upstream phone
provider redirected calls to your phone number to Siberia, what good does it
do for _you_ to quarantine Siberia?

~~~
jgrahamc
We quarantined it because it is connected directly to Internexa and we were
unhappy with their accidental route leak.

~~~
dsl
Can you explain "quarantined" a bit better? You make it sound like they are a
transit provider. Did they "leak routes" you were passing to them?

------
nknighthb
There is an obvious solution, but it's an enormous undertaking.

Each level of delegation must be cryptographically validated. Route
announcements not signed using a certificate to which authority for that block
has been delegated must be rejected.

~~~
bcoates
How would that work?

You could sign that you trust your next-to-last hops to deliver traffic to you
but transitive delivery isn't guaranteed and not every chain of valid hops is
actually workable as a route capacity-wise. A completely signature-valid route
with near 100% packet loss isn't much better than an undeliverable leak.

~~~
noselasd
There's an architecture for doing orignation validation in BGP, lot's of info
if you want to listen to [http://packetpushers.net/show-105-bgp-origin-
validation-with...](http://packetpushers.net/show-105-bgp-origin-validation-
with-resource-public-key-infrastructure-rpki/)

However, it pretty much requires everyone to do it.

------
Goopplesoft
I spent a good bit of time trying to figure out why GAuthify had brief
downtime and do a post-mortem. Luckily I'm read most CF blog posts here on HN
and saw this. Is Cloudflare planning on having any real-time notification
system for things like this? I'm sure it'll save a lot of headache trying to
figure out what happened especially if you don't read the cloudflare blog.

Its very likely that I and especially the other non-hn folks could have missed
this all together (checking cloudflare for an issue is likely my last check
list item due to its amazing record).

~~~
jgrahamc
Yes. We are actually in the process of redoing our status system and web site
to communicate problems better.

However, we are also working to ensure that we don't need to use it.

~~~
nodesocket
I'm sure you guys know, but recommend
[https://www.statuspage.io](https://www.statuspage.io). They allow end users
to subscribe and get e-mails and SMS notifications.

------
jey
How does something like this get fixed? Is the only option to get on the phone
and yell at the guys in the other NOC to fix their announcements?

~~~
jgrahamc
Yes. Without going very deep into what exactly happened here, the other NOC is
the only network with control over the route announcements and so you get on
the phone.

------
nnx
Would IPv6 help reducing the impact or even completely avoid this type of
incident?

iirc IPv4's "bloated" routing tables are getting harder to maintain.

~~~
jbert
No, aiui, the trust model is the same. If a peer announces routes to you, all
you can do is:

\- throw away "martians" (routes to known-reserved/bad parts of IP space)

\- throw away stuff you _know_ they shouldn't be announcing (your stuff
basically)

other than that, I think you have to trust them when they say that they have a
magic route to IP 16.0.0.1 which is shorter than any other you know about.
(Since it _might_ be true).

------
wtbob
Why don't we use cryptography to guard against false routes? If each IP range
were signed-for, then I think a relatively simple protocol could be used to
document which networks a network could route for.

