
Widespread impact caused by Level 3 BGP route leak - pdcerb
https://dyn.com/blog/widespread-impact-caused-by-level-3-bgp-route-leak/
======
jlgaddis
From elsewhere:

> Root Cause: A configuration issue impacted IP services in various markets
> across the United States.

> Fix Action: The IP NOC reverted a policy change to restore services to a
> stable state.

> Summary: The IP NOC was informed of a significant client impact which seemed
> to originate on the east coast. The IP NOC began investigating, and soon
> discovered that the service impact was occurring in various markets across
> the United States. The issue was isolated to a policy change that was
> implemented to a single router in error while trying to configure an
> individual customer BGP. This policy change affected a major public peering
> session. The IP NOC reverted the policy change to restore services to a
> stable state.

> Corrective Actions: An extensive post analysis review will be conducted to
> evaluate preventative measures and corrective actions that can be
> implemented to prevent network impact of this magnitude. The individual
> responsible for this policy change has been identified.

[snip]

Sounds like "the individual responsible" forgot to set some communities on the
peering session. Oops.

~~~
irl_zebra
Do you have a link for this? Interested in the corrective action.

Also, any suggestions on reading to learn about bgp generally to the
specificity one might learn IP/TCP from a networking book?

~~~
knorrie
I wrote a routing protocols tutorial a while ago, that tries not to be overly
complex and not "dry".

[https://github.com/knorrie/network-
examples/blob/master/READ...](https://github.com/knorrie/network-
examples/blob/master/README.md)

It uses the bird routing daemon on linux to build some networks on the go and
see OSPF and BGP happening.

Maybe it can help you a bit. :-)

~~~
irl_zebra
I know this is late, but I’ve also been going through this, learned a ton.
Thanks a lot!

------
NDizzle
Every few months/years I'm reminded that a dozens to a few hundred people are
responsible for routing packets across the entire internet. One typo could
affect millions.

------
jwilk
It took me a while to realize that "Level 3" is a company name.

~~~
rnhmjoj
I know it's a company because regularly they seem to mess something up and
there is a post about it on /r/networking:
[https://www.reddit.com/r/networking/search?q=level+3&restric...](https://www.reddit.com/r/networking/search?q=level+3&restrict_sr=on&sort=relevance&t=all)

It's so common that someone even made a website about it:
[http://fuckinglevel3.com/](http://fuckinglevel3.com/)

~~~
DarronWyke
Remember when they nearly broke the internet back in IIRC '06, when they had
that peering dispute spat with Cogent?

~~~
scurvy
Telia de-peering with Cogent was bigger. And as others have pointed out,
Cogent is frequently a bad partner in these disputes.

------
rbanffy
Early in the history of commercial internet in Brazil we had a couple issues
like this one with the recently privatized telco that operated the big
backbone that connected us to other countries. At that time we more or less
seriously mused about whether all commercial ISPs should pool their resources
and pay a top-tier consultancy to properly configure everything for the telco,
provided they never ever touches those routers again.

------
pc2g4d
"Machine learning classifier predicts which route announcements are legitimate
and which ones are erroneous" <\- headline I'd like to see

------
AlphaWeaver
I don't get it... I thought the Internet was supposed to be decentralized...
How is it possible that one or two companies can cause such widespread issues?

~~~
toast0
BGP is such a great foot gun precisely because the internet is decentralized.

There's no central repository of how to route traffic for an IP [1]. If there
was, it would probably mess things up from time to time, but not to such a
large extent.

Instead, we just have to kind of trust BGP announcements -- especially if they
come from ISPs that credibly could route anything (Level 3, other "teir 1"
isps).

[1] Actually there are some efforts to develop this. After all, IP allocations
are essentially centralized under the five regional internet registries. There
are some registries of routing information (RADB is the most well known, I
think); but not all ASNs participate, and filtering routes from large transit
ISPs is still a major problem.

