There seems to be an issue with Level 3 at the moment which is causing all sorts of sites to be unreachable for some people but not others. Rackspace (mail, hosting, cloud, etc), Facebook, Hotmail, Github, Digital Ocean, etc.
For those experiencing connectivity issues, try a different ISP/network. For example, I can't connect to any of the above networks via my regular Time Warner cable connection or via my T-Mobile hotspot. But I can connect via my Verizon hotspot.
Too funny -- I diagnosed this to L3 on my home net this morning and I found myself saying "I wonder if anyone else has this issue?". When it comes to the backbone providers there is no Twitter account or status.level3.net or status.theinternet. Hard to do when everything is decentralized. I usually go hit NANOG when stuff like this happens but you're not guaranteed to find something real time there.
It's funny, because when you think about it, the Internet is remarkably centralized despite its potential to be almost completely decentralized.
A coworker of mine was telling me just this week that the building we work in, in New York City, is one of two single points of failure for the Northeast (and both points of failure are in New York).
This is an incredibly chilling thought when you consider just how little control humans still have over natural disasters (we saw this firsthand last year in NYC).
By "single point of failure" I don't mean that every single person would be unable to access every single website, but that large chunks of the Internet (particularly anything coming from Europe) would be unavailable for most of those users.
I guess it's a question for NANOG and not HN (if there are any TWC netops here, chime in!), but I'd love to hear a PM why the very expensive routers they buy are not able to automatically re-route to a different peer AS when this happens.
Confirm. Woke up this morning to limited connectivity, realised I was getting to some sites and not others. While on hold with TWC (where they said there was an 'outage affecting service in your area'), loaded HN to see if, well, HN would load, and this was the #3 story. Thanks HN, GFY TWC!
You just described my morning. Opened HN right after this:
Tracing route to debian.org [126.96.36.199]
over a maximum of 30 hops:
1 2 ms 1 ms 1 ms 192.168.1.1
2 42 ms 28 ms 28 ms cpe-74-73-128-1.nyc.res.rr.com [188.8.131.52]
3 14 ms 13 ms 12 ms gig-0-2-0-3-nycmnya-rtr1.nyc.rr.com [184.108.40.206]
4 12 ms 15 ms 15 ms 220.127.116.11
5 17 ms 15 ms 15 ms ae-3-0.cr0.nyc20.tbone.rr.com [18.104.22.168]
6 13 ms 12 ms 15 ms 22.214.171.124
7 15 ms 12 ms 13 ms xe-4-2-0.edge4.frankfurt1.level3.net [126.96.36.199]
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
Level3 is in the midst of grooming/updating/changing some of their fiber connections.
One Level3 DC I have servers in, was served out of Baltimore, MD ; but in the last few weeks they changed it to be served out of NYC. By "served" I mean that the actual huge-Cisco that the fiber terminated to, was in Baltimore.
Very glad to have moved out of Level3, actually :)
I thought this is only associated with my routers "DNS Amplified DDoS vulnerability" on Asus RT-N56U router when using Level3 DNS "188.8.131.52 and 184.108.40.206" but after upgrading my firmware the issue continued. So I've changed my DNS to "220.127.116.11 and 18.104.22.168" google dns and everything started working normal for me. Good to know that this is not just me ^_^
These issues have been around for at least a couple of weeks now. I too have been monitoring Internet Pulse and what I see is not only Level 3 puking on a regular basis but several of the other backbones also. Quite honestly, it's either a gross lack of competence or the feds are beginning to condition the net as they are in the military.