
Massive Comcast Outage Hits the US - crazysane
http://www.newsweek.com/comcast-xfinity-down-outage-not-working-internet-my-area-1002278
======
JonathonW
Comcast's confirmed the fiber cut:
[https://twitter.com/comcastcares/status/1012767042158510080](https://twitter.com/comcastcares/status/1012767042158510080)

> One of Comcast’s large backbone network partners had a fiber cut that we
> believe is also impacting other providers. It is currently affecting our
> business and residential internet, video and voice customers. We apologize &
> are working to get services restored as soon as possible

They seem to be at least starting to route around it now; I was having
intermittent connectivity issues to various sites here all morning, but as of
~45 minutes ago, I'm not having any issues reaching the sites I couldn't reach
earlier.

~~~
onewhonknocks
Sounds like it was 2 cuts, which likely means it was intentional IMHO:

NY| Manhattan| _Communication Failure_ | 30 Rockefeller Plaza (Comast)|
Comcast is experiencing a National outage due to 2 fiber line cuts. One
between NYC & Chicago, the second between Ashburn & South Carolina. customers
urged NOT TO CALL 911, no ETA on a fix, police monitor

Edit: source below. Thank you dfee.

~~~
thebluehawk
> customers urged NOT TO CALL 911

Do people really do this? I can't even comprehend this.

~~~
wilde
Yes. [http://komonews.com/news/offbeat/bothell-police-dont-
call-91...](http://komonews.com/news/offbeat/bothell-police-dont-call-911-to-
tell-us-facebook-is-down)

------
LeoPanthera
On Comcast here in the bay area. Some sites are fine. Some are just gone.
Luckily, my corp VPN endpoint is accessible, and if I VPN out, I can get to
everything again.

~~~
gm-conspiracy
This leads me to believe they are testing to see what they can get away with
for fastlane access.

~~~
abhiminator
If what you're saying turns out to be true, it would be very disturbing.

Disrupting your paying customer's access with unannounced 'tests' is a sure-
fire way to torpedo your brand value and rating.

~~~
jedmeyers
“Comcast” and “brand value” do not go together very well as it is.

~~~
reaperducer
This is true.

When your company name is so hated by the public that you have to re-brand
your flagship product "Xfinity" to distance it from yourself, you know you're
running a crappy company.

------
smaili
> Customers were also taking to Twitter to reach out to the company about the
> possible outage. The company recommended that some users try checking the
> status center online, which would require an internet connection of some
> kind.

Got to love you need internet to check the status of your internet :)

~~~
nadavami
To be fair those customers were reaching out online via Twitter.

~~~
Moral_
People have phones.

~~~
varjag
Phones have browsers.

------
tlynchpin
What's that NANOG .sig about 'always carry a piece of fiber, that way if you
get lost just bury it and when the backhoe comes by to dig it up you can get
directions back to civilization.'

"I always bring a length of fiber with me while backpacking. If I find myself
lost, I just bury the fiber and a backhoe should be along shortly to dig it
up." \--Sean Lally

Similar, not the one I'm thinking of though.

ah-HA! We'll string it up on poles, no way the backhoes can get it up there!

------
mcculley
This is why it would be better to pay by bit consumed, like most do for
electricity. This would encourage carriers to build more infrastructure. It
would also create an economic incentive for the ever increasing number of
Internet connected devices to be more conscientious of data usage. Of course,
we can't have that in the oligopoly situation in the United States.

~~~
charleslmunger
I'm not sure that makes sense - bit consumed neither reflects the value of the
service to the customer, nor the unit economics of the provider.

You want a service level guarantee (99.9% available, with some extra wiggle
room for peak hours and degradation. With electricity, you're paying for
energy generation, plus grid maintenance. With internet, you're only paying
for grid maintenance.

~~~
mcculley
It reflects value to me. I want my connection to be as fast as possible. If I
pay a fixed monthly fee, the incentive to the ISP is to throttle my connection
at some point. If they have no unit economics, why do some ISPs throttle and
cap after hitting a maximum?

~~~
learc83
The OP didn't say they have no unit economics--just that bits consumed doesn't
reflect their unit economics.

------
AlphaWeaver
Allegedly the outage due to a utility truck in North Carolina knocking down a
few power lines. When I heard this, I realized it could've been what happened
in my hometown!

There was a video going around Facebook of a truck taking down several utility
lines in Wilmington, NC.

[https://www.facebook.com/110795228128/posts/1015572616317812...](https://www.facebook.com/110795228128/posts/10155726163178129/)

~~~
thecosas
In case you don't want to go to Facebook:

[http://www.wect.com/story/38541106/video-accident-with-
crane...](http://www.wect.com/story/38541106/video-accident-with-crane-snarls-
power-lines-shuts-down-nc-road)

~~~
AlphaWeaver
Thanks! Wasn't sure where to find the non-facebook link!

------
zucked
Been ongoing for us for almost an hour - was surprised to see it not posted
here. We cannot reach any of our US-East AWS resources at the moment. Getting
about a 50% success rate on other sites - Facebook (& Co.) pages load but no
media displays.

~~~
gm-conspiracy
You have to pay extra for fastlane access if you want to go from Comcast ->
AWS.

~~~
GilbertErik
Thanks Ajit!

~~~
gm-conspiracy
Serious question, anybody try VPN, and then connect to AWS?

~~~
GilbertErik
I didn't notice for a while because my primary Amazon Workspace wasn't
affected, but when various requests weren't working on wifi but were working
from my cell phone, that's when I figured out something was up.

------
hvoiiita
I work for an ISP, seems the issue was localized to Comcast's AS. Probably a
network engineer made a change that went bad :)

~~~
jonknee
A backbone fiber was cut, lots of ISPs are going to be making network changes!

------
SpaceManNabs
Seems to be caused by third party fiber cut

~~~
std_throwaway
Good thing that the internet was built for redundancy so single cuts don't
affect the connectivity.

~~~
dev_dull
It actually looks like two simultaneous cuts in this instance.

------
drewr
I use them at work and home (no choice). From a technical perspective, Comcast
is usually rock-solid for me. Sure it's expensive, and an absolute joke that I
have to call every year to keep the discounts and a TV plan that I don't use,
but I'll give credit where they have actually done a good job. I'd much rather
a stable service that's a bit of a pain on the billing side than a terrible
service that has a nice website.

~~~
bithive123
Yes, everyone loves to complain about the cable company but in my experience
Comcast's network, technicians, and support people are the best in the
business. You could not pay me to use CenturyLink, for instance.

~~~
decebalus1
> in my experience Comcast's network, technicians, and support people are the
> best in the business.

in the context of US based ISP 'businesses', that's like saying 'the smartest
kid on the short bus'

------
rietta
Heroku is currently very, very slow loading. This is the traceroute:

    
    
      traceroute dashboard.heroku.com
      traceroute to dashboard.heroku.com (151.101.2.49), 30 hops max, 60 byte packets
       1  router.home (192.168.xxx.xxx)  0.355 ms  1.103 ms  1.074 ms
       2  96.120.4.xxx (96.120.4.xxx)  9.575 ms  14.987 ms  15.859 ms
       3  ae113-rur01.d9chamblee.ga.atlanta.comcast.net (96.108.191.229)  18.802 ms  18.846 ms  18.882 ms
       4  ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (162.151.88.217)  15.913 ms  15.978 ms  15.733 ms
       5  be-7725-cr02.56marietta.ga.ibone.comcast.net (68.86.93.125)  19.719 ms  19.659 ms  19.528 ms
       6  be-11486-pe03.56marietta.ga.ibone.comcast.net (68.86.85.254)  16.524 ms  17.280 ms  16.821 ms
       7  * * *
       8  * * *
       9  * * *
      10  * * *
      11  * * *
      12  * * *
      13  * * *
      14  * * *
      15  * * *
      16  * * *
      17  * * *
      18  * * *

~~~
rietta
I think this is consistent with backbone / fiber cuts. Local networks in the
Atlanta area are pretty consistently okay. Other stuff is just down or slow,
very slow.

------
rietta
At least this outage is not local. It took me months to get them to replace
the coaxial aerial run from the pole to my house that had been compromised and
had liquid water inside the cable and enough pressure to force said water
around the cable loops, inside my house, and into the splitter. Water-caused
signal attenuation was making for some bad service quality.

I need to consider my plans for obtaining backup Internet. The only real
choice is 4G and the only way to pull that off is to somehow run a hoist up
into one of the very old, mature oak trees on my property to get an antenna
way up above the house (thinking like we do with ham radio antenna runs). I
have no usable cell signal in the house except through the personal cell that
uses the comcast connect to get back to the carrier.

~~~
jessaustin
Before letting one's walls fill with water, one might have made a small hole
in the jacket just before the cable entered the house, to let it run out where
it couldn't do any damage...

------
rietta
Good to know its just not me. I finally had to call it quits and head to
Starbucks with the laptop. My home office is in a no-cell signal black hole so
when this outage hit, my cell phone goes out too until I drive up the street
at least a few houses.

------
gnicholas
I just realized the huge downside to having moved my cell phone from Verizon
to xfinity mobile. Although neither my wired Comcast or my cell service is out
right now, I would be screwed if these both went out.

~~~
onwchristian
Xfinity Mobile is just an MVNO (Mobile Virtual Network Operator) of Verizon.
So it is Verizon's towers/networks that would be providing your connectivity,
not Comcast. Comcast basically just handles your billing and customer service.

~~~
gnicholas
Good point about using the Verizon network. They also use their own access
points for internet service, which could have an impact.

I’d be more worried if they take Verizon traffic and route it across their own
network, which most MVNOs don’t — but Comcast might for data collection
purposes. Hopefully not!

------
abhiminator
I sincerely hope this is a technical malfunction and not a co-ordinated to
attempt to take down Comcast's infrastructure by malicious actors, domestic or
foreign.

Non-localized outages (especially a coast-to-coast one) are signs of deep
faults with the network architecture -- not surprising given the way Comcast
is run.[0]

[0] [https://www.pcmag.com/news/350979/comcast-is-americas-
most-h...](https://www.pcmag.com/news/350979/comcast-is-americas-most-hated-
company)

------
bkmartin
I'm on Comcast and cannot get to Twitter and a few other places... But Netflix
and YouTube work just fine. I wonder why I can get to some stuff but not
others...

~~~
jasoncartwright
Netflix and YouTube have caches inside Comcast's network. That'll probably be
why they are still serving?

[https://openconnect.netflix.com/en_gb/](https://openconnect.netflix.com/en_gb/)

[https://peering.google.com/#/options/google-global-
cache](https://peering.google.com/#/options/google-global-cache)

~~~
bkmartin
I'm on HN...seeing your update...I an connect to my employer remotely... Feels
like Comcast is not the whole story...

~~~
jasoncartwright
HackerNews is served from Cloudflare, which employs caches close to you as
well. Seems like Comcast has a problem routing over distance.

~~~
bkmartin
Seeing talks about 2 separate fiber cuts... Lots of issues all over the US.
Chicago to NY and also another cut that goes south to SC. r/sysadmin has good
info

------
drxyzzy
Discussion here reminds me of Rule #3 in Richard Cook's classic "How Complex
Systems Fail": Catastrophe requires multiple failures – single point failures
are not enough.
[http://web.mit.edu/2.75/resources/random/How%20Complex%20Sys...](http://web.mit.edu/2.75/resources/random/How%20Complex%20Systems%20Fail.pdf)

------
duxup
My Comcast internet has been flakey for a good week.

Sadly the alternate provider offers "up to 100Mbps" ....and in my area that
means 5Mbps... maybe.

I hate a lack of competition.

------
0xcafecafe
Here in Atlanta I had a comcast outage on this Sunday and Monday evenings last
for 4-5 hours each. I wonder if the current outage is related.

~~~
rietta
About two months ago, they took the Internet down for the whole neighborhood 3
days in a row like clockwork to "split the node" (because I went and talked
with the crews out on the road to see what was up). No notification, not even
for business class customers. Completely bonkers!

------
aprao
Switching my DNS to 1.1.1.1 alleviated some of the problems (access to
Google/YouTube)

------
GistNoesis
I'm picturing plenty of bundled optical fiber cut in two. How big are these
bundles ? How can they repair the optical fiber (i.e. how to reconnect each
single fiber to its previous half ?) Do they have to reinstall a section of
cable ?

~~~
dunham
My dad used to do this for AT&T (mainly for train derailments if I remember
right). As I understood it, he would cut/polish the ends of the glass and glue
together. The process may have changed since then, it’s been about a decade.

~~~
GistNoesis
What I don't get is how they match each individual fiber on the left with its
right one, as I imagine there are probably thousands and individual fibers
inside a bundle. When they installed the fiber at my home they had some laser
to check the fiber correspondence. Is it just that, pick left illuminated by
laser fiber, pick right illuminated by laser fiber, put in a glueing/fusing
machine, and doing it n times (where n is huge)? Where do they get some slack
to access the fiber bundled inside ? (Do they need to add a patch section and
do the above work two times, or is there some neat trick ?)

~~~
windowsworkstoo
Fibre buffer is standard colours, you just match and splice.

Service loops in the run provides slack to take up, else you just cut and
splice in a short segment.

------
jdorfman
Coworker in Colorado can connect to slack but certain websites fail. Weird.

~~~
GilbertErik
Confirming from S.Florida. I can get to some/most of my AWS workspaces and GCE
instances, but not to wikipedia.org... unless I turn on my VPN in which cases
they block some traffic or make me go through JS verification, but then at
least wikipedia works. :-(

~~~
gm-conspiracy
Is your VPN IPSec or SSL or other?

~~~
GilbertErik
I'm using PIA

------
dsfyu404ed
Well, good thing we had our scheduled maintenance last night.

------
spike021
I'm not having any issues with Comcast in San Jose. Been using every normal
website for approximately the past hour and a half. Odd.

------
ajoy
try changing your DNS. I got mine working after switching to the Level3 DNS
server IPs : 209.244.0.3 & 209.244.0.4

Listed here : [https://www.lifewire.com/free-and-public-dns-
servers-2626062](https://www.lifewire.com/free-and-public-dns-servers-2626062)

Seems like the list is legit but do verify.

------
wybiral
downdetector.com isn't even working for me:
[https://twitter.com/davywtf/status/1012778410714812416](https://twitter.com/davywtf/status/1012778410714812416)

------
drewr
Most stuff has worked today. GitHub hasn't. IPv6 has been hit-or-miss as well.

~~~
rietta
First thing I noticed failing was Github. Then nothing for a while. I'm not
home, but was able to VPN back into my home and run traceroute and it seems to
have healed with the route to Github at least.

    
    
      traceroute github.com
      traceroute to github.com (192.30.253.112), 30 hops max, 60 byte packets
       1  router.home (192.168.xxx.xxx)  0.455 ms  1.454 ms  1.431 ms
       2  96.120.4.xxx (96.120.4.xxx)  10.247 ms  16.604 ms  15.646 ms
       3  ae113-rur01.d9chamblee.ga.atlanta.comcast.net (96.108.191.229)  17.851 ms  17.839 ms  17.810 ms
       4  ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (162.151.88.217)  16.559 ms  16.673 ms  16.660 ms
       5  lag-5.ear2.b0atlanta2.Level3.net (4.68.71.45)  17.321 ms  17.411 ms  17.353 ms
       6  * * *
       7  GITHUB-INC.bear2.Washington111.Level3.net (4.14.98.158)  128.508 ms  122.411 ms  120.312 ms
       8  * * *
       9  * * *
      10  * * *
      11  * * *
      12  * * *
      13  * * *

~~~
Chlorus
> 5 lag-5.ear2.b0atlanta2.Level3.net (4.68.71.45) 17.321 ms 17.411 ms 17.353
> ms

> 6 * * *

Obviously Comcast is trying out a nefarious scheme to give their own routers
Level3 hostnames, to throw us off their trail, and not, say, a fiber cut. /s

------
barneygumble742
It started last night. Luckily I had two other modems and kept switching them
out. Each modem would work for roughly 30 mins and then the service would go
down. I gave up and went to sleep.

------
tachang
Some of you conspiracy theorists are insane. No Comcast didn't purposely do
this. I can guarantee it was just a backhoe digging around. Happens more often
than anyone would care to admit.

------
bandrami
The fiber-seeking backhoe strikes again

~~~
zipwitch
Backhoes. The internet's natural predator.

------
makecheck
Well, we all know they’ll learn to do better because of this: they’ll take all
that money from monthly fees and make sure to improve quality of service!
Right?

Nope. They’ll probably raise prices if anything, and receive no ill effects as
a company for basically failing to provide what they happily accept money for.
(Buried somewhere in an “update” to Terms of Service, they probably pardon
themselves from exactly this type of outage.)

There ought to be strict laws around long service outages, resulting in
automatic multiple months of free service, etc. as a deterrent for allowing
things to just collapse for hours at a time. This isn’t just Internet anymore;
entire businesses can be affected for example.

~~~
ajmurmann
Of course nothing will happen. Comcast is too good at lobbying and Internet
still doesn't get treated with the same legal importance as other
infrastructure like road, running water or even snail mail.

~~~
tomcam
Snail mail is handled by one designated quasi-private firm, the USPS. Roads
are often handled by contractors on a per-job basis. How would handling it
like roads and USPS help?

~~~
nolliebs180
The USPS (at least in my neighborhood) does a far better job than
Amazon+OnTrac?

~~~
H1Supreme
I prefer USPS now. I order a lot from California, and it's to me in Ohio in
two business days generally. That's with Priority Mail, which is similar to
UPS and FedEx Ground. Both of those are 4-5 business days.

~~~
emodendroket
Beyond that it's way more economical to use, in my experience.

------
Simulacra
How could they tell?

------
txru
I know that other commenters are being flippant, but could this legitimately
be them testing their throttling ability for specific types of traffic?

~~~
SpaethCo
No.

