Hacker News new | past | comments | ask | show | jobs | submit login
BGP super-blunder: How Verizon sparked a ‘cascading catastrophic failure’ (theregister.co.uk)
134 points by totaldude87 7 days ago | hide | past | web | favorite | 26 comments

I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days, but this is about Allegheny Technologies, headquartered downtown, which owns a number of mills and specialty metals businesses scattered around.

There is some steel produced around here still but it's nothing like the old days. The gigantic J&L plant was still in operation when we drove my brother to Carnegie Mellon.

I was also interested to find out which one, given that I'm from Pittsburgh. Turns out that that this is what Allegheny Ludlum is called these days; they pay my grandpa's pension, and several other relatives worked for them for a very long time as well. And today, I work for Cloudflare, though not on any of the stuff that was involved here. It's a small world.

> I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days

As evidenced by the smell of sulphur yesterday morning the coke plants down in Clairton are still alive and well!

Related HN discussion about the incident from June https://news.ycombinator.com/item?id=20267790

Tangentially related to Verizon: I lost Verizon cell and cell data service (along with my cable internet through a different provider) this weekend when a construction crew accidentally cut a fiber line.

I had not realized that literally all modern communications were flowing through a single point of failure. It was a surreal experience.

Yeah, redundant lines are expensive.

Even multiple lines to the same DSLAM were considered expensive back in the day... so if you sliced a single line, you're SOL.

If you were the sys admin at that stell mill and you come into the office on that day. What would be different?

You'd essentially be DDoSed. The internet wouldn't work, and your firewall logs would show that you're being flooded with inbound traffic. The firewall would probably be dropping the unsolicited inbound traffic, so internally you shouldn't see any impact.

Unless as part of the misconfiguration that caused this, the firewall also thinks it knows how to route traffic to the affected prefixes, in which case it would be accepting the traffic and routing it, in which case segments of the internal LANs could be flooded too.

More than likely this wouldn't even hit their firewall. BGP misconfigurations generally occur on the router(s), in front of their firewalls, that connect to the upstream providers. Packets come in one connection and go out the other because you're now the shortest path from A to B.

Odd how often Verizon's name comes up in these periodic stories about mass mis-routing of internet traffic.

Part of this is Verizon's poor practices, and part of it is Verizon is a major transit provider; if a routing leak affects a large amount of the internet, it's almost certainly because a major transit provider accepted the announcement and propagated it.

Since they fail to filter these out, perhaps some enterprising person could route the entire internet through someone's closet ISP. Maybe, just maybe, they will then care.

Among the many ASes of things they've acquired, Verizon is AS701 / UUNet, which is historically one of the largest and most widespread ASes on the planet. I still think of it as uunet.

BGP filters fix this. Only accept routes from customers for IP space that they control.

I imagine Verizon has some sort of webUI with bad defaults (no filter) so that their helpdesk can setup new customers - Hanlon's razor being what it is.

I remember setting up a T1 in the mid 90's. Our upstream was InternetMCI. There was no Web UI. There was no documentation of any sort. The guy (MCI tech) just had me read him routes over the phone and he announced them. Fun times!

easier explanation is that they were doing troubleshooting and had removed the filter to do that.

In the early days, some upstream ISPs did no filtering. You could announce anything.

Thanks, this is why I’ll never know if I actually fixed the WiFi. My girlfriend says I haven’t.

Steeling internet. Neat.

At least they tried to prepare for this situation. The software they were using was very close to the metal.

Forging the internet?

Metaling with the internet!

Oh the irony.

not the first time and won't be the last time...

so, who's going to get fined?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact