> One of Comcast’s large backbone network partners had a fiber cut that we believe is also impacting other providers. It is currently affecting our business and residential internet, video and voice customers. We apologize & are working to get services restored as soon as possible
They seem to be at least starting to route around it now; I was having intermittent connectivity issues to various sites here all morning, but as of ~45 minutes ago, I'm not having any issues reaching the sites I couldn't reach earlier.
There is a fiber cut somewhere between NYC and Chicago.
At the same time there is also a fiber cut between Ashburn, VA, and locations in South Carolina. The cut is somewhere in NC.
Copy and paste from Level3 notification system:
"* CASCADED EXTERNAL NOTES 29-Jun-2018 15:13:37 GMT From CASE: 14815828 - Event Field Operations dispatched to the estimated failure location and upon arrival was advised by local law enforcement there was a truck that struck a utility pole, which then fell across the street and struck another pole, which then tore down the aerial fiber lines. In addition to the fiber lines being down, live power lines are down as well. OSP will be dispatching a fiber repair and construction crew to the site; however, until the local utility company has completed their repairs and deemed the area safe, fiber repairs cannot begin.
* CASCADED EXTERNAL NOTES 29-Jun-2018 19:21:05 GMT From CASE: 14815828 - Event The OSP has contacted the personnel transporting the replacement fiber, and they advised they remain en route, and on schedule for their ETA of 20:15 GMT. The next update will be provided upon their arrival, or as new information becomes available.
There are also a lot of long-distance links carrying DWDM traffic that are >95% aerial, not on poles that can get hit by trucks, with singlemode fiber inside high voltage AC transmission lines (https://3ee2108sdg.files.wordpress.com/2014/07/cropped-20140...).
I guess the plus side is that utility pole lines are probably the easiest to fix when cut.
A lot of the fiber built during the dotcom 1.0 boom with VC money was trenched along railways or pipeline right of ways.
Intentional, targeted damage is really rare, even when multiple paths experience concurrent outages. The fact that 2 paths were affected isn’t enough to assume it was intentional.
NY| Manhattan| Communication Failure| 30 Rockefeller Plaza (Comast)| Comcast is experiencing a National outage due to 2 fiber line cuts. One between NYC & Chicago, the second between Ashburn & South Carolina. customers urged NOT TO CALL 911, no ETA on a fix, police monitor
Edit: source below. Thank you dfee.
Or when traffic re-routes, a major ISP will temporarily congest its own intra-AS city-to-city links on a few key paths, for their own router-to-router interfaces.
So no, two cuts like this are not something out of ordinary. The fact Comcast is cheap and don't have redundant pipes/contracts to operate secondary backup lines is simply being cheap and saving money on their part.
the more cynical side of me says that an intentionally crashed truck looks the same to outsiders as one that was crashed to cut the fiber line.
but i really don't think it's the case that there is foul play here today. this is where calculating p values comes in handy though.
Do people really do this? I can't even comprehend this.
Wow. It's sad that has to be said. Do people not get fined for abusing 911 like that?
Isn't IP supposed to do this automatically?
Maybe they were overloaded from all of the additional traffic they got sent?
Disrupting your paying customer's access with unannounced 'tests' is a sure-fire way to torpedo your brand value and rating.
When your company name is so hated by the public that you have to re-brand your flagship product "Xfinity" to distance it from yourself, you know you're running a crappy company.
America has states and even counties larger than many countries. And even where there are lots of people, population density is typically less than most other nations.
Distance is a killer in last-mile applications like internet.
See also: Australia.
I know what you mean when you say this, but there are strong differences between American ideals and American realities. When there's a conflict of interest between a large company and consumers, I think it's a safe bet to assume the company in America comes out ahead.
Contrast Los Angeles. There, Stokab’s business model would be illegal. Builiding out based on demand might mean that wealthier neighborhoods would get fiber a decade before poorer neighborhoods. Disproportionately, white residents would get fiber before hispanic residents. That would be politically untenable (and would be completely impossible to put the government’s imprimpteur behind such an effoet as with Stokab). For that reason, most US cities make Stokab’s business model illegal.
Los Angeles had a fiber proposal: https://www.wired.com/2013/11/la-fiber. It tried to get companies to build a fiber network. By contrast with Stockholm’s approach, neighborhood income and population density could not be a “factor” in the rollout. That means that any ISP would have to build to new neighborhoods that were not economically justifiable. The ISP moreover would have to provide a minimum level of free access to all residents. It would thus have to recover the cost of that from other residents, driving the price higher and decreasing the competitiveness of the product. Unsurprisingly, nobody took up Los Angeles on that proposal. Nobody would build it, it made no business sense.
In Stockholm, fiber was a simple business proposition, built with private capital. In California, it was a social justice initiative, unattractive to private capital. Which is not necessarily itself a problem, but if you want to do that you need to be willing to build it with public money.
On average, houses in Sweden and closer together. Apartments are smaller than the U.S.
Perhaps instead of "last mile," the term should be "last meter."
Edit: I know the <reasons>, but the debate about that is not particularly relevant here.
Except you can't actually do that. There's a huge list of companies you simply can not choose to ignore. That's sort of why monopoly/oligopoly is so evil, and why strong independent antitrust action is so critical for preserving freedom.
Yes, you can escape all companies if you live like a hermit in a prepper dungeon on your property (that you somehow purchased while 'ignoring all companies' with money you got ahold of by 'ignoring all companies'), That is technically one form of freedom. But most people define Freedom a bit more broadly than that.
I agree that I probably understand "freedom" differently than most, but my definition demands nothing from my fellow citizens (which is what "companies" are made of). :)
Also, could you please not post unsubstantive comments or flamewar comments of any kind? We're trying for better than that here, and need everyone's help.
A variation of the bong native to SouthWest England, where a straight tube is used instead of a conventional "gauze type" bong tube.
The end of the shottie tube (also known as a downie or downpipe/tube) is plugged with a small amount of rolling tobacco, onto which a thin layer of crushed or chopped cannabis is applied. The user then lights the cannabis/tobacco plug until such time as all of the cannabis glows red and the plug can be seen to slide down the tube. At this point, the user gives a sharp tug to pull the cannabis/tobacco into the bong water, the smoke is then inhaled, providing the user with a more intense hit than that provided by conventional smoking paraphenalia, this means shotties provide a both economical and enjoyable smoking experience
Got to love you need internet to check the status of your internet :)
"I always bring a length of fiber with me while backpacking. If I find myself lost, I just bury the fiber and a backhoe should be along shortly to dig it up."
Similar, not the one I'm thinking of though.
ah-HA! We'll string it up on poles, no way the backhoes can get it up there!
You want a service level guarantee (99.9% available, with some extra wiggle room for peak hours and degradation. With electricity, you're paying for energy generation, plus grid maintenance. With internet, you're only paying for grid maintenance.
The econ 101 solution to this is variable pricing based on hours of the day, to shift consumption to off peak hours. However, that's not attractive to ISPs for a few reasons:
1. Customers don't have a good understanding of how much data various activities use.
2. Just like airline tickets, you may be paying substantially more or less than your neighbor for the same service. ISPs do not want to add additional pricing transparency or control - they would much rather adjust the rate as high as they can regardless of usage, then lower it for individuals newly signing up for service or when existing customers threaten to cancel.
3. Many large ISPs also own media and cable TV businesses that are in direct competition with high bandwidth services like Netflix. Putting caps on that apply regardless of time of day directs users to their own offerings, which do not count towards the cap. It's also a hedge to gain revenue as cable dies, and extract rents from it's replacement.
4. If you run a gym, your best customers are the ones who show up least often - the same is true for the current ISP model. A cap is a good way to shed heavy users and provides a chilling effect on existing users who don't have a good grasp of computer networking.
I think it's dangerous to let those with deep pockets prioritize their traffic over mine.
If only a finite amount of bandwidth is available to the world, then yes, the rich will be able to spend more to take bandwidth from us.
However more cable can always be added, so what would actually happen is that the pie would grow.
This assumption is the same one behind energy grids, and assumes an ideal world with no bad actors.
Instead, what happens is that incumbent players with the deepest pockets buy up all the (relevant) competition and then corner the market on capacity.
Let me check. IPv6? Nope. Public IPv4? Nope. HTTP interception proxy? Check! Peak bandwidth? Only a handful of mbps over LTE. And this is in markets like the UK and France where there is supposedly “competition”.
This model doesn’t work, the carriers will still do the bare minimum that they can get away with.
There was a video going around Facebook of a truck taking down several utility lines in Wilmington, NC.
in the context of US based ISP 'businesses', that's like saying 'the smartest kid on the short bus'
traceroute to dashboard.heroku.com (126.96.36.199), 30 hops max, 60 byte packets
1 router.home (192.168.xxx.xxx) 0.355 ms 1.103 ms 1.074 ms
2 96.120.4.xxx (96.120.4.xxx) 9.575 ms 14.987 ms 15.859 ms
3 ae113-rur01.d9chamblee.ga.atlanta.comcast.net (188.8.131.52) 18.802 ms 18.846 ms 18.882 ms
4 ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (184.108.40.206) 15.913 ms 15.978 ms 15.733 ms
5 be-7725-cr02.56marietta.ga.ibone.comcast.net (220.127.116.11) 19.719 ms 19.659 ms 19.528 ms
6 be-11486-pe03.56marietta.ga.ibone.comcast.net (18.104.22.168) 16.524 ms 17.280 ms 16.821 ms
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
I need to consider my plans for obtaining backup Internet. The only real choice is 4G and the only way to pull that off is to somehow run a hoist up into one of the very old, mature oak trees on my property to get an antenna way up above the house (thinking like we do with ham radio antenna runs). I have no usable cell signal in the house except through the personal cell that uses the comcast connect to get back to the carrier.
I’d be more worried if they take Verizon traffic and route it across their own network, which most MVNOs don’t — but Comcast might for data collection purposes. Hopefully not!
Non-localized outages (especially a coast-to-coast one) are signs of deep faults with the network architecture -- not surprising given the way Comcast is run.
Sadly the alternate provider offers "up to 100Mbps" ....and in my area that means 5Mbps... maybe.
I hate a lack of competition.
Repair depends on the type of cut, too. Aerial run? Buried? Did the fiber get damaged as a result of an underground gas explosion in a nearby vault that caused several meters to fuse together? :)
Often, a splicing is necessary - that involves using special equipment to re-align and fuse the damaged pairs. Slack in the run can be used to bring the damaged ends flush, or a segment can be added to couple them.
Service loops in the run provides slack to take up, else you just cut and splice in a short segment.
Listed here : https://www.lifewire.com/free-and-public-dns-servers-2626062
Seems like the list is legit but do verify.
traceroute to github.com (22.214.171.124), 30 hops max, 60 byte packets
1 router.home (192.168.xxx.xxx) 0.455 ms 1.454 ms 1.431 ms
2 96.120.4.xxx (96.120.4.xxx) 10.247 ms 16.604 ms 15.646 ms
3 ae113-rur01.d9chamblee.ga.atlanta.comcast.net (126.96.36.199) 17.851 ms 17.839 ms 17.810 ms
4 ae-29-ar01.b0atlanta.ga.atlanta.comcast.net (188.8.131.52) 16.559 ms 16.673 ms 16.660 ms
5 lag-5.ear2.b0atlanta2.Level3.net (184.108.40.206) 17.321 ms 17.411 ms 17.353 ms
6 * * *
7 GITHUB-INC.bear2.Washington111.Level3.net (220.127.116.11) 128.508 ms 122.411 ms 120.312 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
> 6 * * *
Obviously Comcast is trying out a nefarious scheme to give their own routers Level3 hostnames, to throw us off their trail, and not, say, a fiber cut. /s
Nope. They’ll probably raise prices if anything, and receive no ill effects as a company for basically failing to provide what they happily accept money for. (Buried somewhere in an “update” to Terms of Service, they probably pardon themselves from exactly this type of outage.)
There ought to be strict laws around long service outages, resulting in automatic multiple months of free service, etc. as a deterrent for allowing things to just collapse for hours at a time. This isn’t just Internet anymore; entire businesses can be affected for example.
Anyway, Flint isn't the only place in the US with water problems. https://impact.vice.com/en_us/article/wj4qvx/these-us-cities...
I've seen a lot of "Flint still doesn't have clean water!" talking points in the past month or so. Without passing moral judgment, the talking point is not reflective of current reality.
For various reasons this narrative has subsided. It probably wouldn't even occur to younger people to perceive in the post office reflections of a wider debate about government. Indeed, for some the post office is seen as a pragmatic answer to providing banking services to poor communities, only distantly related to or even entirely divorced from the big government vs small government debate. Actually, I think in general the debate on both the right and left has shifted away from bickering over the size of government, per se, and instead emphasizes individual rights and injuries.
It's why you'll hear the echos of demands on the right to completely privatize the USPS, almost entirely from the older generation because newer conservative generations couldn't care less unless you're talking about Jeff Bezos.
 IMO I think that conservatives won the small government debate in the 1990s, for better or worse. There was a confluence of events in both America and Europe that effectively resolved the debate. Rather, policy proposals these days either rely on some public-private model, or simply attempt to exact concessions from private industry. The government directly doing something themselves is completely off the table. See, e.g., rayiner's comparison of Stockholm's and LA's fiber initiatives.
It was easy to pick on the USPS when people could only compare them to UPS or FedEx. But these days complaints about the USPS fall flat as its more obvious how much value they provide. I'll happily keep shoveling junkmail into the recycle bin if that's the cost of Priority Mail.
Business internet accounts can and often do have SLAs. But, should consumers pay more for SLAs like businesses do? It’s a question of cost.
In France, I had a four hour outage on a business connection once; no refund, not lawsuits from the government. I have also had at least five or six outages that lasted 15 minutes or so. Expecting 5 nines uptime is just unrealistic unless prices rise. My gigabit Comcast connection costs me $101 per month, including taxes. In France, I paid €79 for a “business” connection that had 10mbs down and 0.8mbs up — with an 8 hour SLAS. I am getting vastly better service from a consumer Comcast connection than I did from a business Orange France connection. And France is heavily regulated. My point is to be careful what you wish for.
Instead, we see many issues of their own making (monopolies and refusal to invest, mainly) that make it hard to sympathize.
Verizon? Same shit different name.
Even where there's competition, they don't try to compete. I presume it's mainly a ruse to avoid any further regulations, etc.
It does bring back fond memories of trying to explain to WoW guildmates that we needed to hurry up and finish the fight, because my power just went out. (Still online thanks to telco central-office based DSL and a good UPS.)
> There ought to be strict laws around long service outages, resulting in automatic multiple months of free service, etc. as a deterrent for allowing things to just collapse for hours at a time.
FYI, that means even higher prices...
When your ISP goes down, you can't just switch to another one on the fly. Many people can't switch to another one at all in the first place. When there's a road closure, you just go around.
Their rate limiting tools like Sandvine do not causing these kinds of outages. They cause hassle for specific protocols and destinations. (rate limits, tcp resets, blackhole) but not massive outages.