

Internet Issues in the US - Possible Backbone Issue - JohnTHaller

There seems to be an issue with Level 3 at the moment which is causing all sorts of sites to be unreachable for some people but not others.  Rackspace (mail, hosting, cloud, etc), Facebook, Hotmail, Github, Digital Ocean, etc.<p>For those experiencing connectivity issues, try a different ISP&#x2F;network. For example, I can&#x27;t connect to any of the above networks via my regular Time Warner cable connection or via my T-Mobile hotspot. But I can connect via my Verizon hotspot.
======
raiyu
We can confirm that there are large issues on Level3's network we've had to
pull them from the routing mix until they get everything resolved and
stabilized.

Thanks, Moisey cofounder DigitalOcean

------
apaprocki
Too funny -- I diagnosed this to L3 on my home net this morning and I found
myself saying "I wonder if anyone else has this issue?". When it comes to the
backbone providers there is no Twitter account or status.level3.net or
status.theinternet. Hard to do when everything is decentralized. I usually go
hit NANOG when stuff like this happens but you're not guaranteed to find
something real time there.

~~~
malandrew
Out of curiosity, what do you do to diagnose something like this? What tools
and commands?

~~~
bcl
[http://www.internetpulse.net/](http://www.internetpulse.net/) and
[http://traceroute.org/](http://traceroute.org/) are two sites I used in
addition to the usual unix tools like traceroute.

------
JohnTHaller
According to Internet Pulse, the Level 3 issues seem to be solved as of
11:03am. Still showing a slow connection between AT&T to Cogent and AT&T to
SBC. [http://www.internetpulse.net/](http://www.internetpulse.net/)

I am still unable to reach any of the above sites/networks via my Time Warner
Cable connection, though.

------
korphol
Usually the looking glass for each tier 1 provider will give you a good idea I
can see [http://lookingglass.level3.net/](http://lookingglass.level3.net/) is
down now.

[http://www.bgp4.as/looking-glasses](http://www.bgp4.as/looking-glasses)

------
elliottcarlson
While most of these services have now been rerouted via Above Net etc, some
ISPs such as Time Warner (in NY at least) still seem to be routing via Level3
and unable to get to a lot of destinations.

~~~
JohnTHaller
Time Warner always seems slow to re-route. T-Mobile is having the same issue.

~~~
nayefc
Because Time Warner sucks.

~~~
ajpiano
Confirm. Woke up this morning to limited connectivity, realised I was getting
to some sites and not others. While on hold with TWC (where they said there
was an 'outage affecting service in your area'), loaded HN to see if, well, HN
would load, and this was the #3 story. Thanks HN, GFY TWC!

~~~
schandur
You just described my morning. Opened HN right after this:

    
    
      Tracing route to debian.org [128.31.0.51]
      over a maximum of 30 hops:
    
      1     2 ms     1 ms     1 ms  192.168.1.1
      2    42 ms    28 ms    28 ms  cpe-74-73-128-1.nyc.res.rr.com [74.73.128.1]
      3    14 ms    13 ms    12 ms  gig-0-2-0-3-nycmnya-rtr1.nyc.rr.com [24.29.98.1]
      4    12 ms    15 ms    15 ms  184.152.112.105
      5    17 ms    15 ms    15 ms  ae-3-0.cr0.nyc20.tbone.rr.com [66.109.6.76]
      6    13 ms    12 ms    15 ms  107.14.17.169
      7    15 ms    12 ms    13 ms  xe-4-2-0.edge4.frankfurt1.level3.net [4.68.63.121]
      8     *        *        *     Request timed out.
      9     *        *        *     Request timed out.
     10     *        *        *     Request timed out.

~~~
nayefc
Time Warner asked me to give them the milliseconds of a working website
traceroute, instead of the broken link to a non-working website! It's
hilarious (and quite sad) how incompetent Time Warner is.

------
tux
I thought this is only associated with my routers "DNS Amplified DDoS
vulnerability" on Asus RT-N56U router when using Level3 DNS "209.244.0.3 and
209.244.0.4" but after upgrading my firmware the issue continued. So I've
changed my DNS to "8.8.8.8 and 8.8.4.4" google dns and everything started
working normal for me. Good to know that this is not just me ^_^

------
pinkmist
These issues have been around for at least a couple of weeks now. I too have
been monitoring Internet Pulse and what I see is not only Level 3 puking on a
regular basis but several of the other backbones also. Quite honestly, it's
either a gross lack of competence or the feds are beginning to condition the
net as they are in the military. ~Ginn

~~~
joezydeco
We lost telecom for almost all of Thursday and our provider told us there was
a major metro fiber cut somewhere. Related?

------
dec0dedab0de
I have to remember to check HN before driving in to work when a bunch of
people call me about outages.

------
superuser2
Isn't there a website that shows a giant table of all the Tier 1 providers and
the current ping latency and packet loss between them? I've been unable to
find this again... does anyone have a link?

~~~
agnokapathetic
[http://www.internetpulse.net/](http://www.internetpulse.net/)

------
patrickg_zill
Level3 is in the midst of grooming/updating/changing some of their fiber
connections.

One Level3 DC I have servers in, was served out of Baltimore, MD ; but in the
last few weeks they changed it to be served out of NYC. By "served" I mean
that the actual huge-Cisco that the fiber terminated to, was in Baltimore.

Very glad to have moved out of Level3, actually :)

------
wglb
This [http://www.internetpulse.net/](http://www.internetpulse.net/) is always
a handy tool to get an internet-wide picture.

------
gesman
My dedicated server went on to serving data as dialup modem speed and provider
said it was optic fiber issues on their end. That seemed to be fixed later on.

------
neals
what is a level 3?

~~~
johnchristopher
[http://en.wikipedia.org/wiki/Level_3_Communications](http://en.wikipedia.org/wiki/Level_3_Communications)

------
JohnTHaller
As of 1pm EDT, I can now reach Rackspace, Facebook, Hotmail, Github, Digital
Ocean, etc.

------
TimHordern
We're beginning to see sites come back up on our TWC connection.

------
zebra
Maybe NSA filter overload?

~~~
motoboi
NSA updating firmwares.

------
JohnTHaller
11 out of 37 routers down as of 11am EDT.
[http://www.internettrafficreport.com/namerica.htm](http://www.internettrafficreport.com/namerica.htm)

~~~
jessaustin
Are they typically all up?

~~~
JohnTHaller
Nope. But that seemed like a few more than normal to me. I might be wrong
about that, though.

