
BGP super-blunder: How Verizon sparked a ‘cascading catastrophic failure’ - totaldude87
https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconfiguration_cloudflare/
======
Isamu
I was interested to find out which steel mill, since they are almost non-
existent in Pittsburgh these days, but this is about Allegheny Technologies,
headquartered downtown, which owns a number of mills and specialty metals
businesses scattered around.

There is some steel produced around here still but it's nothing like the old
days. The gigantic J&L plant was still in operation when we drove my brother
to Carnegie Mellon.

~~~
steveklabnik
I was also interested to find out which one, given that I'm from Pittsburgh.
Turns out that that this is what Allegheny Ludlum is called these days; they
pay my grandpa's pension, and several other relatives worked for them for a
very long time as well. And today, I work for Cloudflare, though not on any of
the stuff that was involved here. It's a small world.

------
mtmail
Related HN discussion about the incident from June
[https://news.ycombinator.com/item?id=20267790](https://news.ycombinator.com/item?id=20267790)

------
Domenic_S
Tangentially related to Verizon: I lost Verizon cell and cell data service
(along with my cable internet through a different provider) this weekend when
a construction crew accidentally cut a fiber line.

I had not realized that literally all modern communications were flowing
through a single point of failure. It was a surreal experience.

~~~
WrtCdEvrydy
Yeah, redundant lines are expensive.

Even multiple lines to the same DSLAM were considered expensive back in the
day... so if you sliced a single line, you're SOL.

------
ToFab123
If you were the sys admin at that stell mill and you come into the office on
that day. What would be different?

~~~
justusthane
You'd essentially be DDoSed. The internet wouldn't work, and your firewall
logs would show that you're being flooded with inbound traffic. The firewall
would probably be dropping the unsolicited inbound traffic, so internally you
shouldn't see any impact.

Unless as part of the misconfiguration that caused this, the firewall also
thinks it knows how to route traffic to the affected prefixes, in which case
it would be accepting the traffic and routing it, in which case segments of
the internal LANs could be flooded too.

~~~
icedchai
More than likely this wouldn't even hit their firewall. BGP misconfigurations
generally occur on the router(s), in front of their firewalls, that connect to
the upstream providers. Packets come in one connection and go out the other
because you're now the shortest path from A to B.

------
Causality1
Odd how often Verizon's name comes up in these periodic stories about mass
mis-routing of internet traffic.

~~~
toast0
Part of this is Verizon's poor practices, and part of it is Verizon is a major
transit provider; if a routing leak affects a large amount of the internet,
it's almost certainly because a major transit provider accepted the
announcement and propagated it.

------
neuronflux
BGP filters fix this. Only accept routes from customers for IP space that they
control.

I imagine Verizon has some sort of webUI with bad defaults (no filter) so that
their helpdesk can setup new customers - Hanlon's razor being what it is.

~~~
yusyusyus
easier explanation is that they were doing troubleshooting and had removed the
filter to do that.

~~~
icedchai
In the early days, some upstream ISPs did _no_ filtering. You could announce
anything.

------
appleshore
Thanks, this is why I’ll never know if I actually fixed the WiFi. My
girlfriend says I haven’t.

------
salawat
Steeling internet. Neat.

~~~
mc32
Forging the internet?

~~~
davidkuhta
Metaling with the internet!

------
donalhunt
not the first time and won't be the last time...

------
OrgNet
so, who's going to get fined?

