Hacker News new | comments | show | ask | jobs | submit login
Comcast demands Level 3 pay fee to deliver online movies to their subscribers (businesswire.com)
269 points by jsm386 on Nov 29, 2010 | hide | past | web | favorite | 82 comments

Read the press release carefully:

"“On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast’s customers who request such content."

...and other content.

I'd like to know the entire story here. Is Comcast demanding that Level3 Pay a fee to deliver online movies, or is Comcast demanding peering fees. One of those is content neutral, the other is not.

Note - as I read it, Comcast is demanding peering fees from Level 3. Big Whoop.

I'm fine with Comcast demanding money from inbound-feeds to transit their network, and shutting them off if they don't pay it. That's why any sane content provider multi-homes . Think about the position Netflix put L3 in by announcing an "exclusive" CDN deal with L3. Comcasts negotiating position with L3 got a whole lot better.

This is not unusual on the internet - think about how often cogent gets de-peered because it's unwilling to pay peering fees. I've had a ton of colleagues get blackholed because they made the mistake of single-homing through cogent. Ironically, L3 depeered from Cogent a few years ago... So, what's good for the goose...

For a taste of how this type of negotiating constantly is going on in the background, see: http://www.renesys.com/tech/presentations/pdf/nanog43-peerin...

Net-Net: Always Multi-home through at least a couple Tier 1 providers, so when one of them gets depeered, your customers can still communicate with you. Be wary of exclusive transit deals.

Anybody with some actual insight care to comment on what's really going on here?

While peering disputes have happened throughout Internet history, it is unusual to see an eyeball ISP trying to charge a tier 1. Usually the money flows the opposite direction.

How many internet backbone providers are also content delivery networks (other than Google)?

Now, almost all of them: http://www.renesys.com/blog/2010/10/internet-transit-sales-2...

I wouldn't really consider Google a "backbone provider" since they don't sell transit; they're more of an "extremely ginormous end site".

You hit the nail on the head. It makes it incredibly difficult to compete as a CDN when your upstream backbones start selling their "free" bandwidth as a CDN service that competes with yours.

I think your comment is right on the money. The devil is undoubtedly in the details here. We know netflix just switched away from akamai for video cdn to l3. Aside from scale, it's easy to guess one of the major reasons was price - akamai is one of the highest cost providers and L3 has been aggressively bidding on work. We also know that akamai has way more POPs or at least caching locations (2000+ is the pr) and is very well positioned to locally deliver on comcast. L3 video CDN in contrast is more in the dozen or two POP range, and several of those may not yet be up to the weight of netflix traffic. If comcast is going to have to handle a ton more national traffic themselves because netflix is saving money by switching to L3 it only makes sense for comcast to try to offset those costs by charging for transit.

What if I started CheapoStreamCo and started undercutting everyone in the business, but the only place I was on net was NYIIX. Think I'd be getting settlement free deals? Hell no.

If we can set aside the "Comcast is inherently evil" mindset for a moment, this to me reads simply that Comcast notified Level 3 that they were terminating a previously agreed upon settlement-free peering [1] agreement where Comcast and L3 find exchanging traffic without charging each other mutually beneficial.

This kind of thing is not an unusual event in the history of the Internet and can happen for any number of reasons -- including arguably justified criteria such as unbalanced traffic ratios, as well as business strategy (or greed, depending on which side you are on).

Level 3 has elected to frame this as a net neutrality issue and take the dispute to the press and the court of public opinion, but it's probably about a different question: who will pay, and how much will each party pay, for the infrastructure required to deliver content from L3's customers to Comcast's customers?

If viewed in this light, this peering agreement renegotiation is Comcast pushing back some of the costs on L3 and/or it's customers (i.e. streaming video providers). The other alternative would be to raise it's own customer rates.

It's possible that Comcast is being greedy and leveraging their position to rent-seek, but it's also possible that the traffic mix that resulted in the previous peering agreement is now significantly out of whack, justifying this renegotiation. It's definitely not black and white, and this is only one side of the story.

[1] http://en.wikipedia.org/wiki/Peering

Pretty damn good framing if that's what it is. However, if they really are charging L3 just to deliver competitor's video content online that's pretty much pisses everyone off. I write this using Google's DNS because last night comcast's default DNS was broken. What a terrible company that only stays in business because of near monopoly powers :/

"On November 22, after being informed by Comcast that its demand for payment was 'take it or leave it,' Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions."

Leave it. Level3 set a terrible precedent here by paying the shakedown money. Even with their subsequent complaints, they've now put Comcast in a much better position by implicitly acknowledging (whether true or not) that Comcast brings more value to the table by providing access to its subscriber base than content-providers bring to the table by supplying content to subscribers.

Like the old Churchill story goes, "now we're just negotiating the price."

(See PS #4 below for a more benign interpretation.)


PS #1: Yes, of course it might have been expensive for them to play hardball here, with respect to the short-term harm it might cause their customers. That's exactly the point. By acknowledging that the harm to Comcast would be lower (which I actually doubt), they've significantly strengthened Comcast's position against other carriers and perhaps even with regulators.

PS #2: Game theory would suggest that it's critical for Netflix to repel this assault now, even at the cost of short term pain. Once this precedence is established, Comcast will be able to extract nearly all of the profit from Netflix sales to Comcast customers. Each time Comcast raises the toll, it would be in Netflix's rational self interest to go along so long as they are still making some net profit on the channel (modulo opportunity costs and whatnot).

PS #3: It's not uncommon for carriers to play hardball. The big Tier 1's have been trying to prevent the ascension of Cogent into their club for years, which (along with Cogent's aggressiveness) resulted in some nasty peering dispute stalemates in 2005 (with Level3) and 2008 (with Sprint), each time "breaking the internet."

PS #4: It's also possible that Level3 is representing this as a net neutrality issue when in reality it's just a peering dispute. Traditionally local ISPs have had to pay for transit across the networks of the big carriers. When a local ISP becomes a large regional or national one (Comcast), they have enough leverage to extract no-charge ("settlement-free") peering agreements with most of the major carriers. This is probably the first case, though, of a cable-company ISP becoming large enough to extract transit fees from a big international carrier. That's noteworthy, but if it is content-neutral, it's still the way the internet has always worked.

I don't think I've ever seen anyone cherry-pick various points from other posts without attribution before. Anything wrong with responding in the the thread instead of constantly updating your post with other people's (including mine) comments?

Editing a post is typically for the purpose of fixing typos, making your (own) thoughts more concise or clear.

It's not for the wholesale amalgamation of other people's posts, as useful a function as you think you might be serving.

Let your original post stand on its own merit, or don't post it.

This was simply a case of simultaneous editing. I saw your comments only after writing the substance of my updates (and in the case of wmf's comments, only because of this meta-point). I don't want anyone to think that your comments were copied from mine, though, so I'm sorry if it might otherwise appear that way.

I've operated an ISP, and I'm deeply familiar with the issues involved here. I posted my main comment quickly and updated it as I thought more deeply about the press release. To my mind, a legitimate use of the editing feature is to improve a comment upon reflection.

Again, I want it to be clear on the thread that I'm comfortable that your (and wmf's) comments weren't copied from mine, but neither is your accusation fair. I expect that we've all simply worked in or around the same industry and have seen similar things.

He's at the top of the thread, so those other comments will appear to be duplicates of his post. :-/

Agreed. I recognized " This is probably the first case, though, of a cable-company ISP becoming large enough to extract transit fees from a big international carrier. " as being _your_ comment. The thing is - I learned that from _you_. So I'd kind of like to remember who I can trust to provide useful insight into these topics.

Unless Level 3 had commits to their customers (e.g. NetFlix), in which case it could be very expensive to drop Comcast users.

It's pretty easy to change internet providers when you suddenly lose netflix. They should have called Comcast's bluff.

Oh really?

The Comcast alternatives in Seattle are Qwest's elderly DSL, and Clearwire Wimax which can struggle to get over 1mbps. Compared to a lot of US cities, even those options look good. High-speed internet service in the US does not resemble a functioning market.

Only AT&T provided non-dialup access where I live, until (apparently) very recently when RoadRunner finally ran wires.

Swapping's not easy. I do work and research online - dialup isn't an option. I'm a student - pricier cell-based solutions aren't an option. I do a bit of gaming - ping time matters.

There are also a couple of very small providers who provide really cheap fast connections. If you have access to one of them you can get either 30Mbps symmetric for $40/month or 100Mbps symmetric for $60/month. Hopefully more of these will spring up if the bigger players don't change their ways.

Those smaller providers will be paying for transit with someone larger, no?

Yes, but I think that since the wholesale bandwidth market is more competitive it won't be as much of a problem.

But Quest is an option, as is Verizon and Earthlink.

The fact that Comcast may be the best option and the market state aren't the points I'm arguing.

Would you stay Comcast if it meant losing Netflix? I bet many times the answer is no.

I'm not arguing that it's a wonderful thing to you, I'm arguing that by pissing you off, you'll be much more likely to act.

Would you stay Comcast if it meant losing Netflix? I bet many times the answer is no.

Anyone in my area code switching to Qwest for Netflix will be disappointed. Friends a few blocks away with DSL can't finish a streamed program without dropouts and rebuffering. AT&T delivers better service to my phone. The "pretty easy" option you alluded to entails moving out of the city to one of the suburbs with FiOS.

How crap, I haven't heard anyone say Earthlink in years. Next thing you know someone busts out Sprynet.

I watch 1 hr of Netflix every evening in Seattle with Qwest and it is very reliable.

He's referring to NetFlix being a customer of Level3, not the end user being a customer of Comcast. NetFlix uses enough bandwidth that they'd have direct deals with Tier 1 providers like Level3.

I get that, but Netflix and Level3 not only just signed a major agreement, they have similar interests.

Level3 should have attempted to sign Netflix up for this fight and point the issue squarely at the consumer.. A big "You can't get Netflix because you are on Comcast" or "It now costs you $20 a movie because you are on Comcast" frames the battle back into Level3's favor.

All they did by backing down was show Comcast that they will back down when challenged again. You beat a bully by fighting him.

That creates a real net neutrality issue, wheras the simple issue of rather imbalanced peers doesn't.

Free peering only works, and is only reasonably fair as long as traffis is, on average, about equal. If one network is really genreating 5x the traffic of another, why would the smaller one take the hit fo free?

Not for everyone. My options are Comcast, or dial-up.

very limited in choices in "Small Town" USA when it comes to ISP

On the peering front, interesting case study is MWeb, an ISP in South Africa. They have the opposite problem in that they recently refused to continue to pay other ISP's in the country for the right to peer with them. So all local traffic to and from their network is routed internationally since they have their own international links.


Framing the argument as a peering dispute is an interesting strategy. But I think the differentiator is that if peering is not allowed, there is no alternative route to the customer. Whereas if it were a traditional peering dispute, the quality of the service would degrade with longer routes rather than be terminated altogether.

"Whereas if it were a traditional peering dispute, the quality of the service would degrade with longer routes rather than be terminated altogether. "

Nope, if a pair of Tier-1 Providers De-peer, their customers can't communicate to each other unless the new intermediary receives some form of settlement. They won't do it for free.

See: http://www.renesys.com/tech/presentations/pdf/nanog43-peerin... for an example of this.

I'm sure it had nothing to do with this announcement from two weeks ago:

"Netflix (NASDAQ:NFLX) announced Thursday (Nov 11th) that it has signed a multi-year deal with Level 3 (NASDAQ: LVLT) for Level 3 to become a primary content delivery network (CDN) provider for the online movie rental company."

This is why we need to use encryption for everything, not just shopping, SSH, and banking. The less distinguishable packets are from noise, the more open the Internet can be.

Sadly, I'm beginning to envision the future of the Internet as a bunch of point-to-point VPN connections to trusted sites, almost exactly like UUCP or the BBSes of days past. VPNs are already essential if you live in certain countries (China, the US if COICA had passed, etc.) or want to BitTorrent your TV shows without getting sued for 30 million dollars. Someday, it might be required just to watch Netflix or search with Google. Sad.

In what world do 65536 octets representing a video cost more to deliver than 65536 representing an email? A world where greedy ISPs can tell the difference.

The ISP can still tell where the data is coming from/to and how much of it there is. Encryption is not a solution to the threats against net neutrality.

Entirely valid point, but there are ways to handle that as well (tor networks come to mind), but it does devolve into an arm's race. Where each side is spending more time/energy protecting or cracking a packet to charge it at a specific rate.

"there are ways to handle that as well (tor networks come to mind)"

That's not a viable solution. Unless the TorProject has made MASSIVE gains since last I checked, it's simply not viable for low-latency or high-bandwidth use. It's not like everyone on Comcast could feasibly switch to getting their Netflix through Tor.

If you think a party the size of netflix is going to use the likes of tor to stream their video I suggest you use tor right now and try to view a netflix movie through it.

tor just doesn't cut it. I have tried to use or similar services like yourfreedom.net with VPN (german tor) but there is no way to stream even half-way decent video through these... They usually top around 350 kb and you need at least twice that to start to stream well.

It's not clear that encryption would make any difference in this case, because presumably Comcast can easily measure the traffic going through their direct links to Level 3.

And they can easily measure the outrage from their customers when they stop routing any traffic to or from Level 3.

You're doing that whole "charge by cost" thing that everyone here seems to frown upon, preferring instead to recommend "charge by value".

In what world can you extract more cash from 65536 octets representing a video than 65536 octets representing an email? A world where end users value them differently.

However in a truly competitive commodity market (and surely bits moving from A to B are a perfect example of a commodity), "charge by cost" and "charge by value" should converge, and when they don't we should be alarmed. If ISP X is making 1% profit on regular bits and then decides to make 10% on premium bits, why couldn't ISP Y come along and undercut them? Because the market is being manipulated through monopolies on infrastructure and lobbying for special treatment.

This truly is the clearest articulation of the economic danger of net neutrality I've ever heard.

Good argument, but you've got the main point backwards. Byte-for-byte people will value the email more. It's an incredibly economical use of bandwidth. A given 64KB chunk of my video may not even arrive and it's just a momentary annoyance in the stream, if not entirely error-corrected away.

Proof in point: When IT blocks YouTube, the slackers groan. but if the exchange server goes down...

Nobody would mind Exchange being down if Youtube was available :)

After further consideration of your argument, I stand corrected.

Everyone prefers to sell on value. Everyone prefers to buy on cost. Which end it falls on reflects who has the stronger negotiating position. More money can be extracted from the video bandwidth in a world where the person buying the bandwidth doesn't have another option.

"On November 22, after being informed by Comcast that its demand for payment was ‘take it or leave it,’ Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions"

I do not understand how Comcast had leverage in that situation. Had Level 3 told them no then Comcast customers would have been affected. Why is Comcast even negotiating with Level 3 and not the content providers?

"Customers" in this situation mean the big streaming players like Netflix and Hulu. Comcast would surely point fingers at the streaming services (Netflix, Hulu, etc) who would then have to move off Level3. A pretty ballsy move and exactly why Net Neutrality is important.

I think they went for Level3 because average users have no idea who they are. Attacking Netflix gets you in the NY Times. Attacking Level3 gets you on a BusinessWire press release.

Any content provider who's single-homed to Level 3 (which may include L3 CDN customers like Netflix) would become unavailable to Comcast customers if they disconnected.

Traffic would go through another tier one to reach LVLT.

Update on this from: http://mediadecoder.blogs.nytimes.com/2010/11/29/netflix-par...

Peering Dispute.

"Comcast on Monday rebuffed the notion that the new fees were related to Netflix by saying that the type of traffic distributed by Level 3 was irrelevant. Joe Waz, a senior vice president at Comcast, says it has had a peering agreement with Level 3 to swap traffic fairly evenly. Now Level 3 is sharply increasing its traffic, he said, while resisting a commercial agreement to pay for that.

Comcast is “already carrying huge amounts of video to our high-speed Internet customers every day through commercial arrangements, and it seems to be working for everybody else,” Mr. Waz said. “Level 3 is trying to change the rules of the game.”"

Comcast responds, seeming to claim that Level 3 wants CDN peering with a tier-1 peering agreement:


This makes the most sense, especially considering Level 3's very careful wording in their press release.

This kind of thing happens all the time, relatively speaking. There's really nothing to care about here.

If you get upset over this, you're just playing into Level 3's hand. They're exploiting the "net neutrality" concept to try to get a better deal out of Comcast, even when this has nothing to do with it.

How extremely disappointing of them.

I'm glad this is happening now. The concept of Net Neutrality has been too abstract for most consumers and legislators thus far, so it's been easy for telcom lobbyists to manipulate the narrative.

Something concrete and egregious needed to happen before the average consumer would understand what it really means to allow companies to filter the type of traffic that people are allowed to get through their pipes.

Comcast is going about it in a way that consumers won't understand. I've seen a lot of people expecting things like tiered pricing (think the "YouTube package"), but this is much more dangerous.

Comcast is going to isolate the content upstream so that consumers won't ever know that there was other content accessible, just like they don't know about the TV channels that Comcast doesn't carry.

Comcast isn't being overt about it, but it's easy enough to explain that Comcast is making things more expensive for Netflix. It doesn't take much for consumers to understand the conflict of interest between those two entities.

Of course, you can frame this as either the correct operation of the free market, or as an example of why net neutrality is critical to the free market.

If we had proper amounts of competition in ISPs the entire net neutrality problem would simply disappear. It's only because we're forced to negotiate with monopolies that net neutrality is an issue at all.

If I was Level 3 I would have said "leave it" to Comcast and let Comcast block the service for all their customers. It would have caused a bigger media sensation and outrage by Comcast customers against their ISP. By giving in Level 3 has just encouraged this "tollbooth" attitude that ISPs are developing.

Netflix's software traffic balances among Akamai, Limelight, and Level(3).

If Comcast blocked Level(3)'s Netflix hosting, customers would still get streams, just pulled from Akamai and Limelight.

I'd guess that Level3 is counting on "Tier 1 backbone" peering agreements to make delivering this bandwidth affordable while avoiding having to negotiate with each cable co separately, while Akamai and Limelight have direct agreements with the cable companies already.

If this is what's going on, that would give Comcast leverage.

Which suggests Comcast's real aim may be to force Level 3 to peer.

Could be. Comcast is easy to hate for whatever reasons as so many people actually have contact with them. Level 3 in my experience is run by complete incompetents and perhaps just as aggressive with lock-in tactics.

Anyone know anything about L3's current strength? Their stock doesn't look good but I have no idea what's going on inside. I wouldn't at all be surprised if this is because L3 wouldn't peer to comcast or wanted to give them some sort of premium tier pricing for no added value or some other mess happened in the past between the two...

Comcast and Level 3 were already free peering before this dispute; now they're paid peering.

Bad news for the Internet, especially considering how central Level 3 is to the current landscape. It was inevitable that they'd shake down the CDNs.

If enough Netflix customers live in Comcast-only areas, Netflix would have had little choice but to lean on Level 3 to capitulate. Even if the customers are willing to switch ISPs just to get Netflix, they wouldn't have had the option. That translates to less money for Netflix, more pissed off customers for Comcast, and a true win for nobody. With Netflix and Level 3 falling on their swords, their customers at least come out ahead in the short term.

In the long term, it's obviously disastrous. At some point, companies will have to refuse en masse to pay the protection money. In the areas where it has a monopoly, Comcast could stomach losing Netflix, but not another 10 or 20 big sites on top of it. There are antitrust issues with collective refusal to pay, but there's got to be a way around that.

It seems to me that Netflix management is a little more forward-thinking than that. Hastings, et al. know that streaming is going to be their primary market and if they're having to pay off every possible intermediary in the future, their margins are going to disappear completely.

While certainly not an ideal situation, I could easily see Netflix going after Comcast with highly-visible notices to their Comcast customers rather than pay them off and set a very, very bad precedent.

There is a Comcast employee who occasionally posts on HN, he was active in the OpenDNS thread over the weekend. I would like to get his input on this.

Ideal solution would be for Netflix to offer a discount to users on ISPs that don't do this. Importantly, send an email with "this is what you would pay on the different ISPs serving your area".

How can Comcast argue they incur so much costs? when other providers are able to offer triple-play (isp/cable/phone) offerings for a third of the price or less. Example: in France Proxad (http://www.free.fr) offers 179 cable channels, on demand movies and shows, 18 Mbits DSL, free unlimited VOIP calls to the whole country + 40 foreign countries - all for $40 flat (no catch - I have used them happily for 5 years and I am still a customer in France where I own a home).

Comparing France to the US is nonsensical. US is obviously way bigger and thus the network infrastructure required is much more complex, which greatly increases the cost to provide connection to each household.

This is BS. If Level 3 is paying Comcast why am I paying Comcast?

I am so looking forward to the day when the Comcast monopoly is brought to it's knees! Bring it on GOOGLE!

Comcast pulled quite a bluff, but I don't see how they can win in the long term. As a Comcast subscriber, I'd more than happy to switch to a different ISP if Comcast starts to limit access to Netflix or any other content provider.

As a Comcast subscriber, I'd more than happy to switch to a different ISP if Comcast starts to limit access to Netflix or any other content provider.

How many broadband ISPs are there in your area? In many areas, there are no more than two. If they both adopt this policy, without net neutrality you're out of luck.

As Comcast subscriber, I'm more than happy to switch the minute I have a viable alternative. Whether its a constant barrage of billing issues, poor connections, terrible customer service, abusive monopoly actions, the list goes on...

They win in the long-term if no one calls their bluff (or only minor players call their bluff).

Yep. This is just your local television monopoly protecting its interests. We all knew it was coming sooner or later.

I wonder if this has anything to do with Comcast being down last night.

Absolutely chilling. Glad I abandoned comcast a few years ago.

Well, there goes the (net neutral) neighborhood.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact