
Ask HN: Is Comcast having internet problems? - collinmanderson
I believe others are having problems too: https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15637299<p>Update: The issue seems to be around 50% packet loss.<p>I called Business Support and the person was unaware of the issue, so please also call in to report the issue so Comcast starts to realize how much of a problem this is.<p>I had the support person ping facebook.com from my modem which showed packet loss. I asked them to send me to the next tier of support which they did. (This ends up being a ticket where someone will call me back.)<p>The issue started for me around 12:50pm Eastern time.<p>Update 2:10pm eastern: There&#x27;s another thread on the homepage: https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=15637607 (update: 2:26pm the other thread was removed from the home page)<p>Update: I contacted Comcast Network Operations Center. They confirmed they&#x27;re aware of the issue and working on fixing it, but didn&#x27;t provide any more details.<p>Update 2:30pm eastern: The issue appears to be resolved.
======
funwithjustin
Serious question - what could cause this? I don't know much about network
configuration or DNS. Could it be a coordinated attack on Comcast's
infrastructure somewhere? Seems a little crazy that they'd be experiencing
problem nation-wide with no redundancy baked into their system.

I tried manually flipping over to Google's DNS servers with the same result.

~~~
Shelnutt2
Since we are resolving dns, we know its not a dns issue. This appears to be
pack loss / routing issue. The most likely cause of this is someone has leaked
internal routes, or leaked bgp.

I don't see anything from bgpmon
([https://twitter.com/bgpmon](https://twitter.com/bgpmon)) or bgpstream
([https://twitter.com/bgpstream](https://twitter.com/bgpstream)) for North
America though. There have been multiple bgp leaks today but nothing that I
can see that effects comcast/rnc.

It does appear at least some site I'm seeing are being routed through Europe.

    
    
      traceroute kayak.com
      traceroute to kayak.com (185.6.169.69), 64 hops max, 52 byte packets
       2  96.120.68.17 (96.120.68.17)  7.840 ms  8.995 ms  8.354 ms
       3  96.108.154.229 (96.108.154.229)  8.136 ms  9.174 ms  8.325 ms
       4  be-23-sur03.westroxbury.ma.boston.comcast.net (69.139.221.226)  9.879 ms  7.478 ms  7.667 ms
       5  be-20-ar01.needham.ma.boston.comcast.net (68.85.106.21)  9.527 ms  12.000 ms  8.524 ms
       6  * * *
       7  ae-1-16.bar1.zurich1.level3.net (4.69.142.129)  118.578 ms  118.947 ms  118.229 ms
       8  ae-1-16.bar1.zurich1.level3.net (4.69.142.129)  127.931 ms  116.424 ms  117.852 ms
       9  nexellent-a.bar1.zurich1.level3.net (213.242.82.110)  118.340 ms  117.555 ms  115.950 ms

~~~
funwithjustin
Thanks for the info. How on earth could Comcast have a single point of failure
for their entire network nation-wide?

Would this suggest that there's possibly something like a MITM attack going on
that's hijacking their traffic somewhere?

~~~
tclancy
>Thanks for the info. How on earth could Comcast have a single point of
failure for their entire network nation-wide?

I guess I've been a Comcast customer too long given I don't even understand
your question. It would seem more shocking if it were not something like this.

------
ozfive
This was caused by a misconfiguration at L3 NETWORKS which btw i should add is
now owned by centurylink. Go figure.

------
angrydev
Comcast user in SE Michigan here, can't reach certain websites (reddit, cnn,
basecamp)

~~~
rbritton
Same here in Washington.

~~~
ashwn
Same in Chicago

~~~
vimpire
Houston, checking in

------
jetrink
RCN is also having issues[1]. I'm seeing 50%+ packet loss.

1\.
[https://twitter.com/RCNconnects/with_replies](https://twitter.com/RCNconnects/with_replies)

------
funwithjustin
As of 11:30 PST in Oakland on Xfinity, things appear to be coming back to
normal. Anybody else?

~~~
Twisol
Same, shortly before you in Sacramento on (I believe) Comcast.

------
kreisys
Bell Canada business customer in Ottawa. Getting about 50% packet loss to
about 50% of the internet.

------
uncled1023
I'm in Northern California and also having issues.

One thing I noticed is that it is only Downloads having issues. My uploads are
as fast as ever.

I then connected to my VPN, and everything is working great. I think it is
their routing/DNS that is having the issues, not the connection itself.

------
amorphid
Mildly off topic...

For Chromecast users, I discovered that there's a wired Ethernet dongle you
can use to eliminate the need for wifi...

[https://store.google.com/us/product/ethernet_adapter_for_chr...](https://store.google.com/us/product/ethernet_adapter_for_chromecast)

This past weekend my wifi was super slow, to the point streaming music would
have buffering issues on my Chromecast. A quick speedtest.net assessment
showed wifi was only getting 2mbs (2 meters from the router) and wired was
getting 50mbs. I googled for options, and found the dongle.

------
senthilrajndren
Yes it is having issues across USA CHeck this
[https://www.downstatus.com/comcast-xfinity-
outages](https://www.downstatus.com/comcast-xfinity-outages)

------
dashwin
It may be packet loss on certain routes.

I resorted to using VPN and that resolved the problem at-least while over VPN.
Without VPN its a CRAWL for many sites.

------
caseyohara
Comcast residential customer in Denver, CO. Lots of sites intermittently
slow/unreachable right now.

------
aaronmiler
Comcast home customer in Portland OR

Netflix servers responding and streaming fine, but almost everything else is
slow

------
funwithjustin
Oakland checking in. Can't reach about half of the internet, dropping a lot of
packets.

~~~
toomim
Berkeley too.

~~~
funwithjustin
Guess we should just grab a pint and wait for this whole thing to blow over.

------
brianwawok
South Bend, IN is no bueno here.. both from comcast business and comcast
residential

------
tclancy
Dover, NH Comcast, same here.

~~~
donutdan4114
Manchester, NH... Sllooowowww....... Droppin' packets like it's hot.

------
trebor
Same trouble in Frederick, MD. Packet loss appears to be high too.

------
raincom
Same in San Jose, CA, Berryessa area: can't access reddit; however speedtest
shows 210M upload, 10M download. I use Google DNS for my apartment network.

------
kevin2r
South Florida as well, can't reach some websites.

~~~
dfcab
Same here, been crawling since around 1pm EST

------
chipuni
Same problem in Hayward (i.e. Bay Area), CA, USA.

------
justinlink
Lot of packet loss. Pennsylvania Comcast Business Class.

Twitter has a lot of people complaining all over the country when searching
the tag comcast.

------
maccam94
I was also seeing problems with Comcast in San Francisco, but they resolved
just before I found this thread (~11:30 PST)

------
jipiboily
Getting a very similar issue with Bell Canada. So similar I'm wondering if
they are Comcast customers themselves.

------
noncoml
Comcast has been so unreliable for me the last year, that I pay for a second
backup DSL connection from AT&T.

------
bguillet
Millbrae, CA, same problem with Comcast

------
aacook
Same issues in Plymouth, MA. Noticed it first w/ Stripe but now other sites
like Reddit too.

------
Todd
Comcast Business is bad near Seattle

------
oregontechninja
Same in Albany, OR. Any theories or answers as to why we're seeing such a
performance loss?

------
wiremine
West Michigan is seeing similar: 40% to 50% packet loss for most websites.

------
Pieman103021
I currently can't reach doodle, something is definitely up.

------
8000800008
Secaucus, Edison NJ also having issues

------
stormcode
Boston, Comcast Biz. Same issues.

------
dhess
Yes. Comcast Business, SF.

------
methodover
Same in San Francisco

------
booleanbetrayal
Boulder, CO = issues

------
deadcast
same in Modesto, CA

------
cjbenedikt
same in Baltimore

------
gregimba
some sites are timing out for me in WA

------
exikyut
Wow. I'm reading through what's left of this thread and quite amazed at the
impact.

There should be a service sysadmins can run that sends controlled pings,
download/upload requests, etc, to specific hosts. The idea being that the
results are shared publicly.

\--

Copypasting [https://www.wired.com/story/how-a-tiny-error-shut-off-the-
in...](https://www.wired.com/story/how-a-tiny-error-shut-off-the-internet-for-
parts-of-the-us/) (dated today):

\--------------

A YEAR AGO, a DDoS attack caused internet outages around the US by targeting
the internet-infrastructure company Dyn, which provides Domain Name System
services to look up web servers. Monday saw a nationwide series of outages as
well, but with a more pedestrian cause: a misconfiguration at Level 3, an
internet backbone company—and enterprise ISP—that underpins other big
networks. Network analysts say that the misconfiguration was a routing issue
that created a ripple effect, causing problems for companies like Comcast,
Spectrum, Verizon, Cox, and RCN across the country. Level 3, whose acquisition
by CenturyLink closed recently, said in a statement to WIRED that it resolved
the issue in about 90 minutes. "Our network experienced a service disruption
affecting some customers with IP-based services," the company said. "The
disruption was caused by a configuration error." Comcast users started
reporting internet outages around the time of the Level 3 outages on Monday,
but the company said that it was monitoring "an external network issue" and
not a problem with its own infrastructure. RCN confirmed that it had some
network problems on Monday because of Level 3. The company said it had
restored RCN service by rerouting traffic to a different backbone.

The misconfiguration was a "route leak," according to Roland Dobbins, a
principal engineer at the DDoS and network-security firm Arbor Networks, which
monitors global internet operations. ISPs use "Autonomous Systems," also known
as ASes, to keep track of what IP addresses are on which networks, and route
packets of data between them. They use the Border Gateway Protocol (BGP) to
establish and communicate routes. For example, packets can route between
networks A and B, but network A can also route packets to network C through
network B, and so on. This is how internet service providers interoperate to
let you browse the whole internet, not just the IP addresses on their own
networks.

In a "route leak," an AS, or multiple ASes, issue incorrect information about
the IP addresses on their network, which causes inefficient routing and
failures for both the originating ISP and other ISPs trying to route traffic
through. Think of it like a series of street signs that help keep traffic
flowing in the right directions. If some of them are mislabeled or point the
wrong way, assorted chaos can ensue.

Route leaks can be malicious, sometimes called "route hijacks" or "BGP
hijacks," but Monday's incident seems to have been caused by a simple mistake
that ballooned to have national impact. Large outages caused by accidental
route leaks have cropped up before. "Folks are looking to tweak routing
policies, and make mistakes," Arbor Networks' Dobbins says. The problem could
have come as CenturyLink works to integrate the Level 3 network, or could have
stemmed from typical traffic engineering and efficiency work.

Internet outages of all sizes caused by route leaks have occurred
occasionally, but consistently, for decades. ISPs attempt to minimize them
using "route filters" that check the IP routes their peers and customers
intend to use to send and receive packets, and attempt to catch any
problematic plans. But these filters are difficult to maintain on the scale of
the modern internet, and can have their own mistakes. Monday's outages
reinforce how precarious connectivity really is, and how certain aspects of
the internet's architecture—offering flexibility and ease-of-use—can introduce
instability into what has become a vital service.

