
What happened with your Internet last night, and what we are doing about it - hw
http://corporate.comcast.com/comcast-voices/what-happened-with-your-internet-last-night-and-what-we-are-doing-about-it
======
thaumaturgy
The trouble description seems a bit off. They had a backbone switch in San
Jose fail (according to mtr), but it only affected DNS depending on your
routing. In my case, 8.8.8.8 and 8.8.4.4 were both on the working side of the
switch, but a lot of other sites and systems weren't -- even when accessing by
IP. So it wasn't really a DNS issue.

I couldn't ping Hurricane Electric servers, but Slashdot worked OK for
example.

The frustrating thing was that I was going through a Sacramento hop first but
still couldn't reach systems in Reno -- I thought there was fiber all the way
up the 80 corridor, but apparently most of my Comcast traffic has to go to San
Jose before it can proceed East.

~~~
LukeShu

        but a lot of other sites and systems weren't -- even when
        accessing by IP. So it wasn't really a DNS issue.
    

That's just an intermittent issue that Comcast customers deal with--I
experienced that in Indiana a couple of months ago.

I believe that it relates to them re-jiggering routers such that the
appropriate default gateway for your modem changes, but because your modem is
already up, it doesn't get notified of the change. So the modem doesn't send
the message to the correct router, and the router it sends it to may or may
not know how to get it to the destination.

~~~
thaumaturgy
I typically have logs streaming through terminal windows from several servers
and multitask online -- I notice even brief transient network issues. FWIW
what you're describing hasn't been a problem for me (though I don't doubt it
is for lots of customers), and the San Jose outage was definitely out of the
ordinary.

Anyway, I wish Sacramento was a bigger network hub, is all.

------
taspeotis
What a generous window of time...

    
    
        This $5 credit offer is being extended to our customers with active XFINITY Internet service in
        California, Washington State, and Tucson, Arizona who were affected by the issue with our DNS
        servers that was first reported on 6/1/15. The form above will be available until 6/11/15

~~~
reagency
Comcast exists to prove that government isn't always less efficient at
providing service than the private sector.

~~~
aggie
Comcast doesn't face typical market competition; it is uncommon for it's
customers to have good alternatives. That's not a good representation of the
efficiency and quality the private sector produces.

~~~
sdenton4
Surely no true scotsman....

~~~
aggie
I think it's fair to say that taking any single company and holding it as
evidence of what the private sector produces is flawed, but Comcast is a
particularly flawed example. Likewise with any single government entity.

------
anigbrowl
That's nice. I figured out what was going on (though I was very frustrated by
that point) and switched my DNS servers to Google's public DNS servers
(8.8.8.8 & 8.8.4.4) which worked fine. I figured that calling would be a waste
of both my time and Comcast's - the people on the other end of the phone have
no idea how long a fix is going to take other than 'as soon as possible'.

The straightforward explanation and humble acceptance of responsibility were a
refreshing surprise, however. A $5 credit is not that big a deal but it is
good to see a firm putting its money where its mouth is instead of trying to
deflect or minimize the disruption to customers.

~~~
yalogin
So Google runs a public DNS service too? First time hearing this. I was
annoyed with the constant DNS outages but did not do anything.

~~~
cmdrfred
8.8.8.8 man, what else do you ping to see if the internet is working?

~~~
AnkhMorporkian
127.0.0.1, but I do get quite a few false positives.

------
sjg007
It would be nice if these regulated monopolies would actually charge a decent
flat price instead of promo rates that then jack up 6, 12 or 24 months later.

This $5 is a different percentage of about 6 different price levels for
Internet.

------
georgemcbay
How generous of them to offer this for anyone who happens to learn about it in
the limited time window they are offering it for... (/s)

Also: $5 is half of the $10/month modem rental fee (that I assume is commonly
paid by a lot of their customers who don't know any better) on a ~$70 modem
(probably much cheaper at the scale they buy them at) that has probably been
paid for in rental fees 5 times over already.

------
stox
If my recent experiences with them in regards to DNS is any indicator, their
DNS infrastructure is a colossal joke. 4 days to get a reverse DNS entry done?
Really?

I wont even mention the typos in some of their SOA records.

------
hw
I'm not sure if $5 adequately compensates for the productivity (and in some
cases entertainment) loss encountered. I guess that equates to roughly a day's
credit depending on what your plan is like. Plus I don't see why they require
us to file a compensation claim, when they could just credit every account
that was in the affected areas.

I wonder if they would be compensating more if the Internet and TV went down
during a big event like the NBA Finals yesterday.

~~~
pbreit
Step outside. Do some jumping jacks. Talk to your kids.

You pay $150 for Comcast Internet?

Obviously requiring an opt in cuts the payout by 99%.

~~~
FireBeyond
Technically I pay $224/mo for Comcast Internet, but apparently (home) business
customers aren't eligible...

------
dba7dba
NOW it makes sense.

I was writing up a tutorial which involves working with subdomains. I was
QAing the tutorial by following the tutorial along but the DNS changes
wouldn't work. I know it worked before but not this time. And meanwhile I was
able to visit the websites that I usually visit.

After seeing the Comcast letter, I'm concluding that the DNS changes weren't
working because of Comcast's issue. I was still able to visit websites because
my home DNS server had cached DNS names.

Oh well.

------
ntakasaki
>We are also building a Web site that impacted customers can visit to receive
their credit. We will update this post with a link to that site as soon as it
is available and will share the link on Twitter through our customer support
handle @comcastcares.

If you need to use a social media account name that says "company name cares"
then its obvious that most customers think you don't care.

------
kzahel
My home comcast connection in SF started being extremely flaky last week and I
switched over to 4.2.2.2 etc and haven't looked back.

~~~
LukeShu
Level 3 isn't cool with non-customers using their DNS server.

Google operates 8.8.8.8 and 8.8.4.4 that are for public use.

~~~
userbinator
Level3's VP disagrees:

[http://blog.level3.com/level-3-network/a-flawed-study-of-
cdn...](http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-
dns/)

"Open DNS is provided by companies like OpenDNS, Google _and Level 3_. You can
use it wherever you are on the Internet with no restrictions or authentication
required."

~~~
LukeShu
Huh, that's interesting. As an organization, there's no public info on the
availability of those servers.

Last year they showed ads for failed DNS requests for non-customers.
[http://james.bertelson.me/blog/2014/01/level-3-are-now-
hijac...](http://james.bertelson.me/blog/2014/01/level-3-are-now-hijacking-
failed-dns-requests-for-ad-revenue-on-4-2-2-x/)

------
Aloha
So either business class customers were not effected, or are not eligible -
personally. I didn't notice a DNS outage, I run my own DNS locally.

