Hacker News new | past | comments | ask | show | jobs | submit login
Rogers Canada, please fix the damn internet in Canada
93 points by gabugsat on July 10, 2022 | hide | past | favorite | 91 comments
Its been 24+ hours, have mercy on us.



I just can't believe how much relies on just one company Rogers seemingly every debit machine in Canada, anything to do with passports, many doctor's offices.

A similar situation occurred just last April (2021). https://mobilesyrup.com/2021/04/20/nationwide-rogers-outage-...


Canadian ISPs definitely have a stronghold on the infrastructure. Can't call it a "monopoly" because there are three of them, but a common name given is "Robelus" which is portmanteau of Rogers/Bell/Telus.

I don't have the numbers on how much they spend to lobby the government every year, but I imagine its a staggering sum.


A 'Canada' should be another word for an oligopolistic (maybe that's a word?) state.

Mobile carriers - 4 control 98+ % of the market

Internet - 4 control likely 90+ % of the internet market

Banks - 5 control 90+ % of retail banking. The rest? Credit Unions who have themselves mostly merged into https://www.central1.com/

Insurance - 6 control 89%

Media - 4 control 70%

Grocer - 5 control 60%

Movie theaters - 2 control 99%

Links:

[0] https://financialpost.com/fp-finance/megadeals-fuelling-unpr...

[1] https://financialpost.com/investing/how-canadas-oligopolies-...


Revolut neo bank (GB) tried to enter the canadian market in November 2019. It was in beta mode for one and half year, then it has quit the canadian market!

Why? because they failed obtaining a bank licence!

It is very hard for a new bank to come to Canada. They have to "partner" with one of the 5 existing oligarch banks! Imagine.. as a business, you have to "partner" with a competing bank in order to operate!

Meawhile, Revolut has successfully opened in 10 other countries at the same time..

Tourists and strangers don't really catch how the canadian market is ruled by oligarch / oligopoly so few players/businesses.

There is NO COMPETITION here! Welcome to CanaRussia!

Sources: https://portfolioplus.com/what-happened-to-revolut-canada-wh... https://www.fintechfutures.com/2021/03/revolut-exits-canada-...


After a banking charter is given, the Canadian PM gets put on their board (after the PM leaves politics).

Definitely no quid pro quo there.


What percentage of mobile phones in the us are not Apple or Samsung? How much wailing and gnashing of teeth would occur if Apple screws up an iOS upgrade that can’t easily be undone? Or just repeats what rogers and Facebook have done re dns?

Glass houses, eh?


The word you are looking for is “oligopoly”. This is when a small set of firms has a monopoly on something.


That's a good word, but if I were to be honest, I think a better word might be "cartel" ....I looked up oligopoly and it seems like it implies no collusion..... However, when I think about the large telecoms, I feel like they are colluding with (each other, surely, but also) the government.


Investopedia were pretty sure they were:

https://web.archive.org/web/20171230012928/https://www.inves...


Canada is a country of oligopolies - everywhere from the products you can buy to the Federal Liberal Party's Trudeau Foundation being the receiving end for paid political favours.

It is hard to think of areas where there is actual competition in Canada, because there are so few.

Media? None, the Federal Liberals fund the approved media and leave the unapproved to fend for itself or die. The approved media is then highly encouraged (unofficially) to tow the Party line and back up anything Trudeau says - especially when it's a fabrication.

Banks? Highly controlled via charters, although actual regulation would make a 1970s Cayman's banker blush.

Food? There is a literal milk cartel. There are three major grocers - all highly vertically integrated. One was just on the news a few years ago for price-fixing bread, and also received untold millions for a refrigeration overhaul from the Federal Government.

Telecom? See the outage.

Law enforcement? The RCMP (Canadian CIA) was recently caught editorializing a report to suit a gun bill Trudeau's Liberals had been waiting for a reason to pass. The Federal Government was also crucial in telling the RCMP what to report and when. In any first world country this would be scandalous and met with a wave of resignations. In Canada, it's a Trudeau Tuesday.


The media is not at the whim of the Trudeau govt. There’s the National Post and Rebel media. CBC itself has been highly critical of the govt if you care to watch it. Then there’s CTV and Global News. And I recall a number of newspapers and radio stations are owned by conservative interests.


They don’t like to bring it up, but they all owe a lot of their existence to government regulation (can content requirements), media production funds ( https://en.m.wikipedia.org/wiki/Canada_Media_Fund ), other grants and tax credits: https://www.canadaland.com/qcjo-news-media-bailout-explainer...


Wow the hatred people have for Trudeau. I'm pretty sure whatever is wrong with Canada's telecommunications industry started long before he was involved in politics.


This is a very common problem with politics in canada these days.

Whatever "current event" has a negative outcome, attribute it to the politician you currently don't like.

Post office in the high-arctic is closing, must be "politician i don't like's" fault.

Never mind basic logic as you posted, these are not new problems. JT hasn't done anything to address the issues, nor did his predecessors.

Our CRTC is effectively useless so little will change until CRTC is overhauled.


I think it's closer to:

CIA -> CSIS (intelligence)

FBI -> RCMP (federal police)

HN, please correct me if I'm wrong!


Close, except for an important nuance: CSIS is primarily for domestic intelligence and security and doesn’t do much foreign work. From what I understand, the CIA isn’t supposed to work domestically; for CSIS, working domestically is in their charter.

From what I recall, CSIS was established after the FLQ Crisis in the 1970s, when the government realized that having a single organization responsible for clandestine domestic intelligence and law enforcement was an idea that was rife with abuse.


CSIS does foreign work. They are responsible for anything related to national security. Internal and external.

From Wiki:

“There is no restriction in the Canadian Security Intelligence Service Act on where CSIS may collect "security intelligence" or information relating to threats to the security of Canada.[31] The agency may collect information on threats to Canada or Canadians from anywhere in the world. While CSIS is often viewed as a defensive security intelligence agency, it is not a strictly domestic agency. CSIS officers work partly domestically and often internationally in their efforts to monitor and counter threats to Canadian security”


Ahhh, I was out of date:

> Previous law stated that CSIS was only allowed to collect this intelligence within Canada but due to an updated law in 2016 they are now allowed to collect that intelligence abroad as well.


I am Canadian and you are correct. OP lost me at "CIA" as well. Canada has a "spy agency" and it is CSIS as you state.

I don't think many Canadians would think of the RCMP as "federal police" but more "national police" because their have odd responsibilities:

" The Royal Canadian Mounted Police (RCMP) is Canada's national police force. The RCMP is the only police force in the world that acts as a federal, provincial and municipal force. "


As I understand it your correct. Parent comment is wrong on a lot of things.


Respectfully, this just sounds like the screed of somebody who doesn’t like Trudeau/liberals for political reasons not like an unbiased, cogent analysis of Canadian policy. Though I guess not always a lot of blue sky between those two. But I have a feeling you didn’t like liberals before Trudeau, and nothing was going to change your mind.


The majority of Canadian media is right leaning.

And not sure why you seem so set on blaming the liberals and JT because basically nothing changed when JT took power except a different set/type of scandals and legal weed.


The problem is bundling and integration. The customer is best served by dumb-pipe vendors. Imagine if your devices conducted a reverse auction periodically, and could switch providers without significant cost.

My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that happened a lot, I could buy a dedicated adapter but it would cost another "line" of service. Which is a monthly fee, regardless of use. It's not practical for most people to pay redundant contracts - but there's no technical reason for contracts. And it's not even a matter of fixed costs or physical interfaces - the cost of an alternate modem isn't a big problem.

Breaking the contract norm would benefit uses in other conditions, too. For instance, roaming. Heck, if I could negotiate service minute-by-minute, I might routinely spread some traffic - based on latency, routing, even bandwidth bursting. Heck, even privacy.


>My phone service is Rogers and my internet is Cogeco, so this outage isn't a huge problem for me. If it were the other way around, I could route through my phone, but it would be a serious impairment. If that happened a lot, I could buy a dedicated adapter but it would cost another "line" of service. Which is a monthly fee, regardless of use. It's not practical for most people to pay redundant contracts - but there's no technical reason for contracts. And it's not even a matter of fixed costs or physical interfaces - the cost of an alternate modem isn't a big problem.

AFAIK there are IOT/esim vendors that provide this sort of service. They give you a sim card that can "roam" between whatever providers you want around the world, and you only pay per MB of data transferred. The only downside is that such services carry a huge markup compared to plans from the providers themselves.


Canadian mobile is so expensive that it actually isn't a huge markup if you only use mobile data lightly.

I am curtusing a Bell "tablet" pay as you go plan and IIRC if I use under a GiB a month an internal roaming SIM would be cheaper. Plus it was measured by MiB instead of huge jumps like 1/2GiB, 1GiB, 2GiB, 5GiB, 10GiB. For these days where I am still almost always at home with WiFi I would have saved a decent bit of money.

The main reason I didn't take this option is because almost all of the providers tunnel their traffic yo some other country where their headquarters are. So the performance isn't necessary great. And even if they don't you usually get a foreign IP which results in blocks and wrong languages for many sites.


If you were in an area also served by Rogers coaxial plant, the Bell/Telus wireless network bogged down in many areas as it couldn't handle the traffic of those that had simultaneous Rogers mobile and Rogers DOCSIS outages.

Also experienced some jittery/failed calls to USA on Bell/Telus around noon-time EST once the west woke up.

In many countries, the mobile networks can handle heavy bandwidth because they compete against wireline providers for data and have virtually unlimited plans. Not so in Canada.


All of the profit in telecom is selling more service than can be provided at once. People paying for service they never use is their entire business.


Again, I see this is light grey. HN downvoting. What, specifically, about this, is incorrect:

> All of the profit in telecom is selling more service than can be provided at once. People paying for service they never use is their entire business.


Wireless internet not working for us except google.com and youtube.com site, its been 2 days i am unable to check my work emails, so much anger and frustration. btw i am in GTA


I had to change the apn to get data to work on my rogers phone. when they brought it up the apn had defaulted to the petro-can one so going into settings and changing it to the other proper rogers apn had it all working again for me.


Change your DNS to 1.1.1.1 and 4.4.4.4 for secondary that should get you going for now.


Is this actually true?

If the entire internet is routable but dns is screwing anything that is not major player, that raises a whole host of other questions and is inconsistent with everything i have heard about this outage so far


The original outage wouldn't be circumvented by this however in this person's case they local Rogers resolvers or third party were probably just getting online so they were serving their local resources which is why the bigger entities were loading. So this just piggy backed into their DNS. That's was my thinking anyways.


AFAIK, on iPhone, there is no way of overriding carrier provided DNS.


I think you can get around this by installing VPN app (eg. dnscloak or 1.1.1.1)


Either install an app like the sibling comment suggested, or use one of these profiles: https://encrypted-dns.party/


Thanks, this might be life-changing. Going with MullVad ad-blocking.


Carrier-provided, not without third-party stuff, as others have explained. But the vast majority of users are on WiFi most of the time they are using their phones. When on WiFi, you can control the DNS quite easily.


Any (technical) information on the cause of the outage?


CloudFlare thinks it had to do with bad BGP routes:

https://blog.cloudflare.com/cloudflares-view-of-the-rogers-c...

Not much technical details from the Rogers CEO:

> "We now believe we've narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning,"

https://www.cbc.ca/news/business/rogers-outage-interac-debit...


Bad BGP routes are an externally visible symptom of the outage -- they're how Rogers was telling other ISPs that its own networks were unreachable, but just knowing that they were doesn't explain why.


Read the Cloudflare post:

=======

Cloudflare Radar shows a near complete loss of traffic from Rogers ASN, AS812, that started around 08:45 UTC (all times in this blog are UTC).

What happened? Cloudflare data shows that there was a clear spike in BGP (Border Gateway Protocol) updates after 08:15, reaching its peak at 08:45.

=======

BGP storm first, outage second.


If an internal route goes down, the information is propagated to adjacent networks via BGP. So the BGP storm could have been caused by routes becoming unreachable (possibly intermittently).


>BGP storm first, outage second.

Not really. If you overlay the two graphs you see that the outage happened almost in lockstep with the outage. While the two events are correlated, that doesn't imply causation. There could be a common cause factor that was behind both of them, for instance (which is what the parent poster was saying).

https://i.imgur.com/OnP4Kcz.png


Also … bad routes can cause outages for sure but this long lasting?!?

That raises other questions about their ability to recover from what isn’t an entirely unique situation (bad routes).


It’s [time to live](https://en.m.wikipedia.org/wiki/Time_to_live). Once a bad route is published and propagates, TTL determines how long it remains cached.

The real issue is, it took Rogers engineers so long to diagnose and address the problem, allowing incorrect route maps to propagate to deep and far corners of the internet.

Anecdotally, a friend in Greece roaming with a Canadian Rogers SIM card was unable to use her phone. Perhaps other Canadian Rogers subscribers, traveling on far continents can confirm?

My guess (pure speculation) is Roger’s TTL values have been dramatically decreased by now, which will increase the bandwidth requirements for network state management. This is also not great news for Rogers customers and peer networks.


Generally speaking, BGP route propagation does not include a TTL cache invalidation mechanism (i.e. routes are never invalidated and rechecked -- the BGP session's ongoing existence validates already learned routes and conducts the addition or deletion in-band -- if heard, "TTL" in the context of BGP means that routes should not be propagated onward, like TTL for IP packets).


> a friend in Greece roaming with a Canadian Rogers SIM card was unable to use her phone

I've heard this from several individuals. It makes sense: Rogers, being totally dark, couldn't authenticate any roaming sessions.

I wonder how fast a roaming provider boots you off their network though. Roaming usually tunnels your traffic to your home provider, so I could see data halting immediately. Maybe local calls work for a while.

This effect sunk in when they said they couldn't receive any SMS 2FA. Even people with a local SIM SIM setup (highly recommended!) still need to slide in their home SIM to log into some things.


I don’t know if they were from Canada but I photographed a family on vacation here in Minnesota and they said their “phones don’t work in this area.” They seemed surprised.

It was an extended family of 14 people and there are plenty of cell phone towers of various sorts that reach where we were.


India High Speed Internet 4G - 2GB per day - Rs. 450 per month ($ 6.50) + plus unlimited nation-wide calling.Everything in Canada is an oligopoly.


One can say that India just has a low per capita income. Look at Sweden and Finland then; it's not about (just) per capita income.


What speeds do you get in India for those prices.


On 4G have seen 5Mbps consistently, and reaches 25Mbps if you are closer to the cell tower.This 4G connection was my backup for remote working over a VPN. Primary was a 50Mbps symmetric fibre to my home, which costed about $10 per month, unlimited with peering for youtube, netflix, amazon, disney and other streaming platforms. Fun fact, I don't have a municipal supply of water at my place, but oddly enough get Fibre to home easily.


Few notes, ad an EU mind: why a private company should fix? Who own roads? The public isn't it? Then they one of the most used road by the public, named internet, is not run equally by the public whose target is not profit but offering a service to the public?

Aside: do we really want to make a cash-less society? No ones who of course have e-cash BUT ALSO physical money payable in person no extra network needed?

Do we really want connected stuff, cars included (see last "blackout" of SF autonomous taxi fleet in the USA) instead of having optional connectivity under user control and a local-first approach by default?


Wired or wireless?

Where are you? Most of GTA was back up around midnight (or about 22 hours ago).


For me, it was both the wired home internet which is TekSavvy as well as mobile phone which is on Fido. Both TekSavvy and Fido rely on Rogers behind the scenes.

Ours came back at around 2am EST.


For me, teksavy (wired home internet) never went down at all. Cell phone (fido) came back up sometime when i was sleeping last night/early morning. I am in vancouver.


that's because your teksavvy service is over Shaw or Telus lines. This outage was out of Teksavvy's hands: if you were in their Rogers-based territory on DOCSIS, you were down with the ship.


You left out Bell... does Rogers basically never lease lines from Bell? Or it depends a great deal on location? (everything you said is correct afaict btw)


they do (well, Teksavvy leases Bell lines), just not in vancouver!

AFAIK, Rogers doesn't lease Bell lines ever, not for home internet anything anyway.


I have Teksavvy via Cogeco and had no issues (obviously). I just discovered Rogers owns 41% of their shares though :|


don't worry, as with Shaw and Rogers, Audet family controls the votes through supervoting shares. This isn't a democracy.

Rogers only has 13% control of Cogeco itself: https://crtc.gc.ca/ownership/eng/cht043.pdf


I want more info, please email me. I put address in my HN proflie


What more info do you want?


*EDT


Why the downvotes, HN?

edit: 1 hour later: I was originally asking why HN was downvoting parent comment, but the masses have since reversed their decision and have been upvoting the parent to this comment. Unfortunately, they have begun to downvote me. Ah well, I suppose it comes with the territory. But hey, no question = no answer...


it's best to ignore the noise. Votes here don't pay the bills so they can't hurt me or anyone.


You should read into how the son of the founder is suing his sister and family for control of the company. Total trust fund bullshit and incompetence.


And that current mayor of Toronto and former Rogers executive John Tory was called in to mediate the family dispute.


Better yet, listen to the analyst calls from when the son was running the company. His answers were totally nonsensical. It was abundantly clear that if his last name had not been Rogers, he wouldn’t have been in a senior position.


[edit] Fixed reference to Rogers.

Not to mention… Rogers is actively working to secure further control over the Canadian market through its acquisition of Shaw.

Related to this outage though, I found it weird that my cell had no bars during the outage. Why would the “last mile”be out when it seems like the problem was their core backbone network that was at issue?

I am very curious to read the post mortem on this one if it ever reaches the light of day.


” Shaw is actively working to secure further control over the Canadian market through its acquisition of Shaw.”

Wow, modern corporate finance really keeps surprising me


I thought letting businesses do business without government involvement was peoples ideal state? Allowing private companies to handle debit transactions for a country turns out to be something that should be well regulated. Who’d have thought.


It is highly regulated. Most of the problem with Canada (and many other places including aspects of American business) is that businesses are granted oligopolies through highly customized/lobbied for regulation/laws and then they get shitty because they have no incentive to control costs or invest in improving their offering. The role of government in business should be making sure business isn't obviously harming individuals and breaking up business when they get too big and start exercising market power to win rather than innovating. This doesn't happen in Canada and it's why we suck. If you go look at our stock index the S&P 60 basically everything on there that isn't base resource extraction should be broken up but our government is too bought and paid for to do it.


I am in Canada and I do not use Rogers (nor Bell/Telus), although I often communicate with people who do, so it still affects me, although not as much as others.


Badly regulated monopolies suck.


Internet in Canada isn't a monopoly, it's a government-protected oligopoly.

And yes, it sucks. And no, nobody's going to fix it, because Canada's telecoms are far more motivated to lobby to protect themselves than Canadians are to lobby for the government to fix them. It's a classic collective action problem. It's why we can't have nice things.


> Internet in Canada isn't a monopoly

Depends where you are, and if you can consider Teksavvy et al as Rogers competitors. Where I am, Bell will only sell DSL1 at 6mbps. A newer subdivision that took a payoff from the local (now Rogers-owned) cableco. Before that acquisition forced Rogers' hand, that cableco didn't have anyone else selling on its lines.


I'm sure they suck for plenty of reasons (eg. price gouging from lack of competition), but I'm not sure how those reasons relate to this outage. For instance I can imagine that a country with a more diverse telecommunications sector would be less vulnerable to a single provider going down, but this argument doesn't really pan out. I looked up how the market share is broken up in european countries[1][2] and they're roughly split 3 ways between 3 major competitors, which is similar to how the market share is split in canada. If there was a major outage with a major provider in germany, the impact would be roughly the same as with canada.

Overall comments like this feels like a case of "maligned entity is in the news for some reason, so let's bring up all the other reasons we hate them" to me.

[1] https://www.statista.com/statistics/469074/mobile-telephony-...

[2] https://www.statista.com/statistics/375986/market-share-held...


In Canada, mobile doesn't compete against wireline. With 3G Carrier Aggregation, LTE and now 5G, that's doable technically, but won't happen if one eats the revenue from the other.

Personally, I blissfully slept through support tickets because my work phone is on Rogers, and my home wifi (for app pings) is on eBox.ca, which runs on Rogers lines (and is now owned by Bhell), so my work phone was totally dark.


Canadian telecom is joke. I am on Bell, but neither are great.


Did they state what the root cause was


I posted a bit of info here:

https://news.ycombinator.com/item?id=32041473


Not in any technical sense, but the latest official statement is "We now believe we've narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction early Friday morning."

I'm wondering if this is a software update gone awry.


Asking the ISP will do nothing. The government needs to step in and fix this.


Yes, governments are great at solving technical problems.


I have heard almost all good things about municipal internet service.


The concept of municipal internet is widely liked (ie. "let's put the government in charge of internet rather than greedy corporations"), so that accounts for a lot of the positive reception. Meanwhile municipal internet services are local by their very nature, so when there's some sort of outage it's not widespread enough to get nation-wide coverage.


The time for the government to fix it was however many years ago they did whatever it was they did that led to not having a second source for telecon services.


How? You want them to deploy the national guard to rogers HQ to assist them in recovering their networks? All the people who know how their network works are probably helping in some capacity.


> You want them to deploy the national guard to rogers HQ to assist them in recovering their networks

I also don't see how this would help, but yes I do want to see that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: