That's Netflix paying Comcast directly, not Cogent giving Comcast the few grand to upgrade the link that they were purposefully not upgrading to force Netflix into paying.
EDIT: Also worth noting that Netflix has to pay every month (which is what Comcast wants) rather than Cogent just giving them $10k once for new hardware (not as delicious as lots more money!)
It's not just the hardware upgrade, the peering agreement is only free when each side sends the same amount of data. Netflix is sending a lot more than it's receiving, in which case their ISP needs to pay for peering.
Edit: I'm just explaining why a hardware upgrade was besides the points that were being made. These are the facts regardless of who you think is in the right. The argument was over bandwidth usage, not hardware upgrades. Is anyone disagreeing with that?
> Netflix is sending a lot more than it's receiving, in which case their ISP needs to pay for peering.
All Internet traffic has more download than upload (for the end-consumer). This is literally why ISPs offer faster download speeds in end-consumer packages than upload speeds - they know that consumers will need to download more than they upload.
In vector calculus, this is known as divergence. Traffic on the highway has divergence of 0, meaning that if you draw a closed loop of any shape, over the course of a day/week, the number of cars crossing that loop in both directions (in/out) will be the same.[0][1] Internet traffic does not have zero divergence, because bits are interchangeable and easy to create, destroy, and copy (whereas cars and humans are not).
Comcast isn't dumb. They know this is the way the Internet works. They just also know that it's more profitable to pretend they don't and extort money based on a false definition of "equality".
>All Internet traffic has more download than upload (for the end-consumer). This is literally why ISPs offer faster download speeds in end-consumer packages than upload speeds - they know that consumers will need to download more than they upload.
Yes, but they also have commercial customers that send out lots of data. Netflix is an outlier all by itself, outweighing the connection.
That's not for businesses running websites, that's for using the internet in a business. http://business.comcast.com/internet/business-internet/web-h... is relevant, although it doesn't give speeds. But presumably anyone who buys that is sending more data out of comcast than they are receiving.
And as I mentioned elsewhere in this thread, I'd like to see a chart of data in and out excluding Netflix and large companies. If the problem only affects them or other large companies (like Youtube, Apple, Microsoft, etc, which all pay for direct connections), but not small websites, then I think that would support my argument.
You must realize that these are absolutely NOT true commercial offerings. They don't even list what your commit bandwidth is!
Also you can't run your own applications there! Do you really think Netflix runs off of a shared server like this? They're a HUGE AWS customer running many thousands of VMs. How do you access control things, how do you deploy your code, how do you do basically ANYTHING with this offering from Comcast? I suspect that you don't.
If you notice, they have a page dedicated to their network. What familiar names do we find on this page? Cogent and Level3! http://www.hostgator.com/network
Notice that there is no mention ANYWHERE of ANY COMPANY that might even a LITTLE BIT be considered a consumer ISP. Comcast, AT&T, Verizon, Cox, etc, none of them are mentioned.
>Notice that there is no mention ANYWHERE of ANY COMPANY that might even a LITTLE BIT be considered a consumer ISP. Comcast, AT&T, Verizon, Cox, etc, none of them are mentioned.
From the page:
>We utilize a wide array of network carriers including Level 3, nLayer, Comcast, Cogent Communications, and Hurricane Electric.
You've got me dead to rights on that one, of course, so I have no problems taking it back. I'm not going to change the above comment where I misspoke because I did write that, and I did think it was correct at the time.
I think it is telling, however, that Comcast's logo isn't anywhere on the bottom of the page with all their partner logos. And I think it is also telling that it's only Comcast and not all the consumer ISPs.
Netflix is only sending the data at the request of Comcast's customers. Comcast is the company which created an asymmetric network such that their customers can request far, far more data than they are able to send. They are reaping the rewards of that decision. It hardly seems reasonable to expect that all the companies that Comcast peers with to have data flowing in equal amounts both ways since the design of Comcast's network explicitly prohibits this.
Trying to boil this down to "Netflix is sending a lot more than it's receiving" is to completely ignore WHY this is happening and really makes the debate useless. Symmetric (or nearly symmetric) peering agreements made sense when everyone had symmetric connections like modems, ISDN, T1, T3, etc.
When the majority of connections are not symmetric, it makes no sense at all.
Again, remember that Netflix isn't DOSing Comcast. Any packets that Netflix sends to Comcast's network was at the request of one of Comcast's (and Netflix's!) customers.
Why aren't the customers that send less data than they receive balanced out by those that send more than they receive? For every byte being sent, there's someone on both sides, so an ISP serving a large enough area should balance out. Except with huge outliers like Netflix.
Netflix is not an outlier when it comes to asymmetrical traffic. The typical Comcast customer receives from Google far more data than they send to Google. They receive far more data from Amazon than they send to Amazon. Same for their interaction with Facebook, Twitter, Hacker News, Reddit, the bank, their email provider, their gaming guild forum, and so on. About the only time they might be sending as much as they receive is when they are using the internet for a voice or video chat.
But some of those sites are going to be Comcast customers, who serve customers from other places.
I'm not saying that an average consumer is sending as much as they receive, but Comcast doesn't only have consumers, they also serve websites. That should balance out for small websites in and out of their network.
Uh no. Comcast's customers are requesting more traffic than they send. That is how residential asymmetric ISP services are specifically designed to be used. Netflix isn't the bogey man just because they offer a service consumers want and have already paid their ISP to deliver. If Netflix was split into 1000 smaller VOD services streaming the same traffic volume who would Comcast blame for having to shoulder the same burden?
Three years ago if you were willing to buy in bulk, you could get bandwidth for as low as $0.65/meg as this HN user points out. https://news.ycombinator.com/item?id=3892854 Based on his comments I suspect that you could get it for less than $0.50/meg today, maybe less than $0.25/meg.
Sure looks like Comcast wanting to have its cake and eat it too.
Let me ask you something else. How large should a company need to be to get free connections to last mile ISPs? If any company could do it, then instead of paying for hosting or internet, I'd just pay for one computer and hook it up directly to the ISP. That isn't possible for every website.
Why should Netflix be able to do that and not smaller sites?
I think you've misunderstood the scenario at play. Netflix did not want to connect to Comcast for free. Netflix did not want to connect directly to Comcast at all. They wanted to connect to their own ISPs, and they were paying to do so. Comcast didn't want to upgrade the peering links to Netflix's ISPs, though, because they'd then lose their leverage over Netflix. Comcast's refusal to upgrade peering links effectively forced Netflix to connect directly to Comcast.
Comcast sells asymmetric connectivity that's skewed as far as 10:1 download vs upload. And then they claim that peering should be symmetric as if they aren't a huge contributor to the asymmetry.
While we're asking each other silly questions, let me ask you this: if my ISP says that they're providing me with "internet access" but actually only provides me with "most of the internet minus a few people we'd like to shake-down access", this is OK?
And my question was serious. If you think that what Netflix was asking for should be given as a matter of principle, then it shouldn't matter how large they are, and absolutely every website should never have to pay for bandwidth again. What exactly am I missing?
Netflix does pay for their bandwidth. They pay their ISP for it. What you are missing is that Comcast is not Netflix's ISP, and Netflix does not use any of Comcast's bandwidth.
When a Comcast customer uses their Comcast internet connection to use Netflix, or Google, or HN, or any other site, it is that customer who is using Comcast's bandwidth. It is not Netflix, Google, or HN using it.
Why should Comcast care whether Netflix pays their ISP or not? Either way, someone's trying to give them data for free.
And Netflix has its own CDN, which wants to connect directly to last-mile ISPs. In that setup, they aren't paying anyone other than the end ISP.
>When a Comcast customer uses their Comcast internet connection to use Netflix, or Google, or HN, or any other site, it is that customer who is using Comcast's bandwidth. It is not Netflix, Google, or HN using it.
This is pointless semantics. If I send you something, we can either call it me using the bandwidth, or you using it. Doesn't change my point here.
Not really. If I mail something to you, I pay the postage. That's how mail works.
When I connect to the internet, if I request something from you, we both pay our respective ISPs to make it happen. That's how the internet works. And the limit to connection speed has historically always been the last mile on one side or the other.
So what Comcast did is pushed the congestion from the last mile to a peering point by purposefully not upgrading the peering point. And predictably, what happened is that the last mile wasn't the limiting factor anymore. According to 20-30 years of internet history, that's breaking the internet. Maybe not Comcast's contract with their customers (since they can unilaterally change that at any time) but definitely in terms of expectations.
That would be like gas stations suddenly advertising the pre-tax price on their signs only once you're done pumping $30 worth of gas they charge your card for $40 because of the taxes. Might not be explicitly illegal since so many other products are advertised on the pre-tax price. But it would be breaking with decades of convention. People might get really pissed as a result.
Sadly few of us have options to get internet aside from Comcast, where as in the gas station scenario we could easily stop patronizing BP or Texaco or Shell or whoever. That also gets people riled up. And rightly so!
> Why should Comcast care whether Netflix pays their ISP or not? Either way, someone's trying to give them data for free
The data was paid for by Comcast's customers. I pay Comcast for the service of transport of packets from my network to the internet, and from the internet to my network. That's the whole point of an ISP.
> Either way, someone's trying to give them data for free.
And why shouldn't it be free? I, the Comcast customer, requested this data. Why should Netflix have to pay for the last mile when that's what I pay for already?
Why does Comcast get to charge for both ends of the pipe when they only own one side?
The principled way to solve this is to give Comcast a choice of receiving traffic over transit or free peering, but either way they have to deliver customer traffic without congestion. https://news.ycombinator.com/item?id=9193591
> And, while it is also worth considering whether the impending merger between Comcast and Time Warner cable might exacerbate the current situation, I predict that this is unlikely. In general, as one access ISP gains more market share, a content provider such as Netflix depends on connectivity to that ISP even more, potentially extracting increasingly higher prices for interconnection. In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation.
I'm having trouble with this logic. Increased leverage over content providers seems like it would exactly exacerbate this situation.
This is why the Net Neutrality ruling was not a big step forward.
The right way to solve the internet problem is "unbundling"; that is, the people who own the last mile lease it out to companies that sell you internet, video and other services. If Comcast customers were getting a bad netflix experience they could switch to an ISP that gives a good one.
>> The right way to solve the internet problem is "unbundling"; that is, the people who own the last mile lease it out to companies that sell you internet, video and other services.
That's terrible. The ISP should give me "internet access" which includes everything out there. They should not get to pick and chose what I can get.
>> If Comcast customers were getting a bad netflix experience they could switch to an ISP that gives a good one.
This is called competition. It's what you get with net neutrality. Since every ISP has to give you everything equally, the only way they can compete is by building better infrastructure and compete on ping time and bandwidth. This is how it should be. Allowing them to prioritize certain services for a fee (regardless of who pays) means allowing them to stagnate - in fact paying them to stagnate - which is bad for everyone.
> This is called competition. It's what you get with net neutrality.
You are stating that "Net Neutrality" will magically create competing ISPs in places where only a duopoly of <Cable Company> and <DSL Company> currently exist. Please explain to me how net neutrality will somehow affect things like local municipalities granting ISPs local monopolies. Or how "net neutrality" will allow new-comers to overcome the initial regulatory and capital investment hurdles to setting up a new ISP.
What you're missing is the "unbundling" is talking about forcing (e.g.) Comcast to lease it's "last-mile" lines out to competing ISPs. This means that competing ISPs don't need to invest in "last-mile" infrastructure. (If you don't understand what "last-mile" means, it would behoove you to look it up).
Net neutrality does play a part here in that if Comcast partners with Netflix, they could provide a better Netflix experience than a smaller ISP without bargaining power, but the lack of ISP competition is arguably a bigger problem and calling out "Net Neutrality" as the solution to all Internet connectivity woes doesn't help.
>> Please explain to me how net neutrality will somehow affect things like local municipalities granting ISPs local monopolies. Or how "net neutrality" will allow new-comers to overcome the initial regulatory and capital investment hurdles to setting up a new ISP.
It doesn't. Local monopoly support from the government is a big problem, but it's also not the only problem. I actually have 3 options at my home and it doesn't seem to be causing an increase in service or a reduction in prices.
That said, as video moves to the internet and as the ISPs lose differentiation, we can hope for actual competition on access to what FCC regulation will make a commodity service. But I'm not entirely convinced that will happen either.
> This is called competition. It's what you get with net neutrality. Since every ISP has to give you everything equally, the only way they can compete is by building better infrastructure and compete on ping time and bandwidth. This is how it should be. Allowing them to prioritize certain services for a fee (regardless of who pays) means allowing them to stagnate - in fact paying them to stagnate - which is bad for everyone.
Access to everything equally is not physically possible; if we pretend it is physically possible, it's not logistically possible: no ISP is going to peer with all networks at all peering points; and no regulator is going to compel an ISP to peer with all networks at all peering points.
Unbundling is easy to define, and easy to regulate, and offers a way to get competition where it's most effective. Unfortunately, it seems to be not easy to put in place. The basic idea with unbundling is that last mile delivery is cost intensive, and has high barriers to entry, etc: a market solution isn't going to work: instead require that the last mile providers offer wholesale access to get packets to a centralized point (or points) where other providers can deliver the connectivity to rest of the networks. This actually makes sense: once you're at a carrier hotel, you have a large number of options to interconnect; getting from a central last-mile provider building to a carrier hotel is managable, but getting from each home to a specific building is capital intensive.
Growth in internet speeds was way faster from 2004-present than from 1996 to 2004. In 1996, my parents had 28.8 kbps dial-up. In 2001, they got 256 kbps DSL. That's 10x over 5 years. In 2006, they got 50 mbps fiber. That's 200x over 5 years. See: http://www.ieee802.org/3/ad_hoc/bwa/public/sep11/cloonan_01a....
For various reasons, ISPs tend not to build competing infrastructure. Many municipalities grant time limited monopolies to Telecoms to get them to accelerate infrastructure construction. Also, after an ISP lays its lines and pays for them, it can make all future pricing decisions on the margin, meaning, it can price its service just above the cost of maintaining it. Another company will find it very difficult to incur the cost of construction and keep its price competitive.
I don't know enough about unbundling to say if I support it or not but I believe it has been pretty popular in countries that have adopted it.
> Since every ISP has to give you everything equally, the only way they can compete is by building better infrastructure and compete on ping time and bandwidth.
How would this solve the problem though? Having competition at the last mile doesn't solve the problem of congestion in miles 2-100 where there is still one company.
The point is that if 'last-mile' services were open to competition, then there would (could) be many companies in what you call the 2-100 mile arena. Building an ISP infrastructure is merely difficult and expensive. On the other hand, thanks to municipal involvement, lobbying, time-based monopolies, etc. building last-mile everywhere is basically impossible.
Cable built an internet infrastructure across America on top of existing outdoor wiring plant (including access to poles) built up over many years.
Cable built up the outdoor wiring plant by offering a compelling new service (great reception for local TV and exclusive national programming) without significant competition (you could get a giant satellite dish, but compact dishes were not available until after cable was mostly entrenched).
It's not easy to overlay additional infrastructure wiring to compete with existing services (most areas already have good enough internet and video services from one or two providers; especially when a new entrant prompts upgrades from the incumbents). On the other hand, it's very easy to use wholesale access (if available) to provide middle mile competition. You would just need a couple routers (one at last mile provider building; one in a local carrier hotel/colo), a leased line between them, and a transit account.
In theory there wouldn't be "one company" anywhere other than the wires. If you have a choice between ISP A who is deliberately congesting Netflix, ISP B who receives Netflix over uncongested transit, ISP C who was uncongested extortionate peering with Netflix, or ISP D who has uncongested free peering with Netflix then you just choose whichever ISP provides the price and performance that you prefer. And if all the customers choose ISP D, maybe the other ISPs will change their policies to try to win customers.
Unfortunately there are other problems (like "your DSL being down is not our fault") with unbundling.
Ok, this makes more sense now. I had interpreted "the last mile" to be closer to the home than the level of the network that is hashing out peering deals with transits.
Unbundling requires sharing of last mile infrastructure to allow for competition in the middle mile. Last mile wiring has huge barriers to entry, middle mile wiring barriers are significantly lower.
The last mile is where competition becomes difficult due to physical buildout restrictions. There are quite a lot of "mile 2-100" companies; it's not like Cogent is the only game in town.
Is that what "unbundling" actually means in this case? I thought it was just referring to letting other companies use your cables to resell the service at better prices/speeds.
> the people who own the last mile lease it out to companies that sell you internet
The only way that would happen nationwide is via federal law. What chances do you think the Tea Party congress would have of passing this?
The FCC policy is pretty much our only shot here and makes a lot of sense. The industry will pivot itself around it now that they can't fight it. They are now forced to play nicely with each other. Yes, there is going to be an ugly interim period, but I think we're going to remember Wheeler as the man who saved the internet as we know it.
>Thus, we will certainly see ISPs continue to use prioritization to manage congestion—prioritization is, in fact, a necessary tool for managing limited resources such as bandwidth.
Bandwidth may very well be a limited resource in some cases but the dispute between Comcast and Netflix is not one of them. The Comcast network has the bandwidth available to serve the customers (since not all sites are congested) and Netflix is willing to peer for free wherever they want them to, so the slowdown is completely artificial and Comcast generated.
I have always felt that this issue has been overcomplicated. It seems what this really stems from is the classic ISP business model. From a pure network design standpoint, nearly every network has an uplink that is not large enough to sustain simultaneous high usage from most, or all, of its devices. Your classic endpoint data closet example would be a 48-port switch with 1Gbps access ports, and a 10-20Gbps uplink.
This problem seems much worse in the ISP world. Many years ago, ISPs could easily oversell their connection. Realistically only worrying about the small percentage of extreme users downloading high volumes of data via Usenet, BitTorrent, or other file sharing services. Life for an ISP was probably pretty great.
Now, with the sheer number of households streaming video, it seems to be apparent that ISPs are drastically overselling their connections. It does not even make logical sense to blame a single service if an ISPs upstream connection is saturated. If the ISP did not oversubscribe their connections at such ridiculous rates, this would not be an issue. Why does it matter that a single service is causing the largest portion of the traffic congestion?
Netflix recommends a 5Mbps connection to stream HD video. Most all devices today would stream at HD quality, if possible. How is it that an ISP such as Comcast doesn’t have the uplink capacity to sustain many users at a mere 5Mbps? I find it absurd that multiple ISPs are overselling their connection to the point where a speed of 1/7 of the average connection in the USA is too much of a strain. It really seems that the level of oversubscription is equivalent to having a data closet with a 48-port 100Mbps switch uplinked at 2Gbps . The whole situation just stinks of pure greed.
Just as few as 5 years ago 50-100x over-subscription rates were common for ISPs. For example 100 homes with 10mb connections would be fed by 10-20mbps. The reality was much closer to 48 port 100mbps switches having 100mb uplinks. Netflix (and other streaming services) have been a huge shock to their business models.
>In the specific case of Comcast and Time Warner, it is difficult to predict exactly how much additional leverage this will create, but as it turns out, these two ISPs are already in markets where they are not competing with one another, so this specific merger is unlikely to dramatically change the current situation.
This does not follow. A larger ISP with more market share, regardless of the state of the competition, can extract larger peering agreements. So this merge would absolutely make the current situation for Netflix worse and other peering agreements more expensive.
It's refreshing to see more writing popping up from people that 'get' it. It's a shame that most of it is after people spent a decade campaigning for the wrong thing.
Wrong, preserving Net Neutrality was very important but it's just not the whole solution and no one ever said it was... It's called taking the first step.
Comcast claims that Netflix was sending traffic at such
high volumes as to intentionally congest the links
between different transit ISPs and Comcast, essentially
taking a page from Norton’s “peering playbook” and
forcing Comcast and its peers (i.e., the transit
providers, Cogent, Level 3, Tata, and others) to upgrade
capacity one-by-one, before sending traffic down a
different path, congesting that, and forcing an upgrade.
>Resolving the dispute: Paid peering (March 2014). Both sides of this argument are reasonable and plausible—this is a classic “peering dispute”
Both sides of this argument are reasonable and plausible?
Are you kidding me? If Netflix was intentionally shaping their traffic to disrupt Comcast, how is that not a denial of service attack? That sounds like a criminal accusation more than a bargaining position.
If Netflix actually was doing that, shouldn't comcast be sueing rather than asking for money for a peering arrangement?
also, and this is more to my own ignorance than it is to a counterpoint. I thought peering arrangements were things that happened between ISPs. Netflix is a customer of an ISP.
It's a pretty poor response. They're using the fact that Level3 had a dispute with Cogent over unbalanced traffic to say the traffic between Level3 and Verizon should also be balanced. Verizon is not in that condition, it's not a transit provider.
Basically there are 4 different situations:
1) ContentProvider<->Level3<->Cogent<->ISP<->User
2) ContentProvider<->Cogent<->Level3<->ISP<->User
3) ContentProvider<->Level3->ISP<->User
4) ContentProvider<->Cogent->ISP<->User
In all cases the first network gets a payment by the content provider and the last network a payment by the user. In cases 1 and 2 the middle network is not getting a payment because neither side is their direct client. So the Cogent/Level3 dispute is about there being the same number of 1 and 2 cases so that it all evens out and no compensating payment is necessary. Cases 3 and 4 require no additional payments no matter the upload/download ratio because both network links have been paid for, one by the user the other by the content provider.
This whole thing is about the ISPs noticing that the content providers are making money offering content over the internet (shocking I know) and trying to reach across the network and grab some of that revenue, even though everyone has already been compensated for the actual transfer of bits.
Verizon acquired the Tier 1 transit network of UUNet/Worldcom. It has a huge transit/datacenter business. ATT(SBC+BellSouth) has Tier 1 transit network of the old ATT long distance. Centurylink has the old Qwest Tier 1 network. One of the things overlooked in the peering disputes is that many of the ISPs ended up merging/buying Tier 1 transit networks.
Netflix is one of the first ContentProviders to be pumping so much traffic that they will imbalance the peering traffic ratios of any almost any Tier 1 they buy transit/CDN from, except for probably Google, which ended up doing what Netflix is doing now: paid peering.
> Netflix is one of the first ContentProviders to be pumping so much traffic that they will imbalance the peering traffic ratios of any almost any Tier 1 they buy transit/CDN from
That's only relevant if the transit/CDN is actually peering with another Tier 1 provider to deliver the Netflix traffic (cases 1/2). What's been in the news has been cases 3/4 where there's a single transit provider in the picture and there's no reason for the ISP not to accept the content with no compensation for the asymmetry as it has a client paying him for that service.
> except for probably Google, which ended up doing what Netflix is doing now: paid peering.
Do you mean Google is also paying ISPs for access to users like Netflix has had to? Because if you mean Google is having to do paid peering with transit providers because their traffic is asymmetric that would just be normal payment for transit, as if Level3/Cogent/etc where providing transit to them.
Verizon (and ATT, and CenturyLink) are Tier 1 transit networks for peering operations. If Level3s peering traffic ratio are off balanced by netflix's traffic, I see no reason why Verizon shouldn't do the exact thing Level3 did to Cogent back in the day. Also, keep in mind Netflix is using Level3's CDN service, so you have a scenario on the flipside where Level3 has become a transit network and ContentProvider also.
The issue is really that the idea of transit providers in general are becoming obsolete. So much fiber was laid, with DWDM and further subdivision abilities, it costs comparatively nothing to get a nationwide network these days. That was Cogent's entire business model, buy cheap fiber, sell transit for almost nothing to content providers, practically (in some cases, actually) give away transit to ISPs/businesses to keep their traffic ratios as balanced as possible.
Now, we're in a situation where it's not technically or financially hard to have a presence in every regional peering point. The major content providers don't need transit providers (or in the case of Level3's CDN, are already provider) except out of convenience and their access to settlement free peering. The major ISPs have no need for transit providers(or are already providers), as all the content providers wants to peer with them, and most of them have nationwide networks to do so. (see comcast: http://business.comcast.com/images/default-source/about-us/t... )
Unregulated peering has lead to the cost of transit plummeting to almost nothing. ( http://drpeering.net/white-papers/Internet-Transit-Pricing-H... ) We've just hit an interesting point where 2 contentproviders (Netflix and Youtube) are half of all internet traffic, and the eyeball networks are all basically tier 1 transit providers. There's basically unlimited choice for transit providers for ContentProviders, but each eyeball only has 1-3 ISPs to choose from.
My thoughts are rather than attempting to regulate transit/peering, regulate that last mile network and let people choose their own transit provider. Back in the DSL days, I used a local ISP that provided cheaper/better service than our telco. If I could pay a flat $x/mo for 1gb connection to a transit provider of my choice, I'm sure Google or Level3 would offer cheap uncongested access.
They are huge companies so they can probably also provide transit service but that's not the case in this dispute. Netflix is willing to deliver the content anywhere Verizon/Comcast/etc require it either directly or by hiring a transit provider. So Netflix doesn't want transit service from these ISPs, it's already paying or building that by itself. The user is also already paying for the ISP network to work, so all segments of the network are already properly paid for once, no need to pay again.
Should anyone be able to connect to an ISP for free? Why can't I host my website for free, just paying an upfront cost for a server, and get it hosted in an ISP's facility?
Is there a relevant difference between that and what Netflix wanted?
> Should anyone be able to connect to an ISP for free?
Yes if they are willing to interconnect with that ISP at every exchange point. In practice that's expensive and what transit providers are for. Here's my proposal for a simple set of rules that would solve all these disputes:
This doesn't allow you to host your website for free because it only adds connectivity to the users of that specific ISP, not the whole internet. For that you have to buy transit.
> How many times would they need to interconnect with a particular ISP? Once?
It's right there in the link. You'd have to interconnect with Verizon/Comcast at every one of their regional interchange points. It would become very expensive very fast for a single small website, but it would be reasonable for a transit provider or a very large content provider that already has servers everywhere (like Google). There would be no free riding, everyone would need to pay for their own way, either by directly physically connecting to the interchange points or by paying a transit provider that already does.
With Comcast, it's almost obvious that the net neutrality won't change this because Comcast was already operating under net neutrality requirements. To win approval of its NBC merger, Comcast agreed to be bound by net neutrality requirements until 2018. When Verizon won its lawsuit to strike down the FCC's 2010 rules that did not change this. Comcast remained bound by its agreement.
If your ISP wanted to give you good service they could also add a few Netflix OCAs to their rack[1]. Opening up the last mile could allow the competition to use this technique to improve service while using Comcast's (or Verizon's...) network for all of the data.
They want someone to blame for not being able to watch Game of Thrones in HD or for fear of having to pay extra to continue doing so. Providers who fell in between the eyeballs and the content got in the crosshairs.
This is really "Game of Thrones" vs. "The Internet" and GoT won.
Yet again, someone fails to understand that this is Comcast abusing their (effective) monopoly to extract additional money via double-paying.
First I pay for internet access. 50Mbps "theoretical max" but it's fairly reasonable to expect to get say 10% of that most of the time.
Then Netflix has to pay again, in order for me to get the internet traffic that I requested. If Netflix doesn't pay, then Comcast schedules upgrades more slowly than they normally would such that my Netflix experience is degraded. They're not reducing capacity, they're just growing capacity in some areas much, much more slowly than in the rest. To get extra cash.
I think Prof. Feamster does understand that (he says "Comcast arguably (and, empirically speaking, as well) has more market leverage: They can afford to ask Netflix to pay for that direct link—a common Internet business relationship called paid peering—because they have more market power.") but he is taking a faux-objective, "teach the controversy" approach. I've seen the same philosophy in papers from his students.
You pay for internet. Netflix has to pay for internet too. To deliver packets to Comcast's network, Netflix can chose to pay any number of transit companies, or they can cut out the middle man and pay Comcast. Why is it ok for Netflix to pay a transit company, but not Comcast?
So Comcast has historically been a consumer ISP, not a transit provider. Consumer ISPs have traditionally had to buy service from transit providers in order to give their customers access to the internet. So in this case your monthly service fee went partially for the last mile and partially for the actual internet connection from the transit provider. Bandwidth is expensive.
As some ISPs got bigger, they were able to negotiate peering agreements instead of paid service since it was in both party's best interest. A could bill B $10k/mo for service and B could bill A $10k/mo for service and in the end it was a wash. This makes sense too. Once this happened your bill largely reflected the cost of the last mile, since the transit was now free. Bandwidth has gotten cheaper.
What we're seeing now is that some ISPs have gotten so big that instead of paying transit providers for access to the greater internet, or simply getting free peering, they're charging others for peering to them. How are they able to do this? They have an effective monopoly on their customers and they're "leveraging" this to get even more money. Now my monthly fee goes to pay for the last mile, and someone else also has to pay for the last mile. Am I getting a discount? No. Bandwidth isn't getting cheaper, but the last mile is getting paid for twice. Is the last mile getting faster? Also no.
So this feels very much like a monopoly abusing their position of power to get extra money.
Isn't the last mile getting used much more than previously, without prices going up? Is it valid to look at this as paying for that, and if CDNs can't pay, then end user prices will have to go up?
Where on earth are the prices not going up? I used to get cable internet for $40/mo and now it's more like $60/mo.
I had to buy the DOCSIS 3 modem for my house, so they're not paying for that. I suspect that they had to buy new modems on their end, but my modem was only about $100 so I can't imagine that theirs was more than $100 especially if they're buying millions of them.
They haven't upgraded the coax that runs to my house, or the pulls that run along the telephone poles. Those are all the same. So it's an equipment upgrade? $100 once so that they can charge $20/mo more forever? They made their money back on the last mile well inside of a year.
And they haven't had to put more fiber into the ground either. Fiber transceiver keep getting faster and cheaper every year. As slower ones break you can replace them with faster ones for less money. Even if you upgrade early, it's not as though it's hugely expensive.
I pay for 50/5 but I suspect that Comcast only allocates 5/0.5 or so for me, since it can't manage to keep up with Netflix all that well. So that means that Comcast can do the last mile for $100 or less of new investment and the last 25 miles for $4 (2000 people per pair, $8k per pair) of incremental investment. This is not a lot of money relative to the $60/mo that I pay.
> Is it valid to look at this as paying for that, and if CDNs can't pay, then end user prices will have to go up?
So first I disagree that the CDN should have to pay anything, again since I'm paying Comcast to connect me to the CDN!!
But even if I admit your point (which again, I emphatically DO NOT!) the amount of money to make an upgrade happen is TRIVIAL relative to yearly revenues. $104 in one-time hardware versus $720 in yearly, recurring revenue.
I agree with msandford that Moore's Law is supposed to take care of this. Assuming that it doesn't (e.g. the "exaflood"), we come to the trickiest part of net neutrality.
If extra money has to come from somewhere, I think it's better for the Internet ecosystem for all those costs to be passed on to the customer as price increases, but obviously a lot of customers are more concerned about their own bill than the health of innovation in the Internet.
I think Netflix customers should be paying for Netflix data. If an ISP can't charge netflix and instead raises the price to end users, they're basically charging everyone for Netflix users. If I don't use netflix, why should I subsidise those who do?
> I think Netflix customers should be paying for Netflix data.
They are. They're buying 50/5 or 50/10 internet and then asking their ISP to (horror of horrors!) make good on their claim. If the ISP isn't DRAMATICALLY overselling their capacity, things are totally fine. If they can actually deliver on 10% of what they're "promising" (but have cleverly worded the contract to avoid having to actually make good on those claims, a shady practice also), there are no problems with Netflix whatsoever.
Level3 offered to pay for the direct cost of the upgrades, namely a few grand for the fiber transceivers on Comcast's side of the peering point in the carrier hotel. That would mean it's a "free" upgrade, and no end-users are paying for it. But Comcast didn't bite.
>They are. They're buying 50/5 or 50/10 internet and then asking their ISP to (horror of horrors!) make good on their claim. If the ISP isn't DRAMATICALLY overselling their capacity, things are totally fine.
The question is whether all consumers' prices should go up, or Netflix prices should go up. In the first case, consumers are subsidising netflix for those that use it.
>Level3 offered to pay for the direct cost of the upgrades, namely a few grand for the fiber transceivers on Comcast's side of the peering point in the carrier hotel. That would mean it's a "free" upgrade, and no end-users are paying for it. But Comcast didn't bite.
This is meaningless. As I said elsewhere, they are not complaining about the cost of the upgrade. They're complaining about the cost of taking the extra data. That cost is what would either be passed on to users, or to Netflix. Talking about the direct cost is missing the point.
ISP makes a promise to deliver data at a certain rate for a certain amount per month.
ISP can't actually deliver said data, and so, can't deliver on their promise.
I can't see how this is Netflix's fault. ISP makes a promise, ISP can't deliver.
Yes it does happen to be coming from Netflix, but Netflix didn't FORCE the ISP's customers to become Netflix customers, did they? Or does Netflix not only have some magical ability to FORCE data upon people, but also mind control to MAKE certain people sign up?
You're arguing that the ISP should be allowed to overstate their capacity as much as they want, and if it causes any problems anywhere, that the counterparty is at fault no matter what. That's completely nonsensical.
> The question is whether all consumers' prices should go up, or Netflix prices should go up.
> They're complaining about the cost of taking the extra data.
Basically they're complaining that they have to make good on the promises that they've made to customers. They were hoping that they could way, way, way oversell their capacity (and charge more for a theoretical peak capacity that you can never use) and get more money from their customers as a result.
Now what's happening is that many of their customers are asking them to make good on 10% of their promises all at the same time. This causes problems for the ISP, so they flail about trying to blame other entities than themselves.
Nobody is asking the ISP to give them a HIGHER data rate than they've already paid for.
They're asking the ISP to give them a mere 10% of what they've already paid for.
> Whether ISPs should need to build out enough network to satisfy peak usage completely is a separate issue, but irrelevant here.
No, that's exactly the issue. If the ISP doesn't massively overstate their capacity, there is no congestion. If there is no congestion, there are no complaints. If there are no complaints, there is no problem!
Remember, customers aren't asking the ISP to allow them to utilize their connection 100%. They're asking for 10%. Asking an ISP to not overstate their capacity by more than 10x is a pretty fucking reasonable thing to do, Net Neutrality or not.
So you can see that they recommend 3Mbps for SD and 5Mbps for HD.
The problem is that the ISP has promised 50Mbps per customer to said customer, but then they are not allocating enough resources at various points throughout their network to actually deliver on 5Mbps per customer to Cogent or Level3. Ostensibly Cogent and Level3 can handle the upgrade and increased bandwidth, and ostensibly the ISPs internal network can handle the increased bandwidth, the congestion was at a number of peering points only. Those were the bottlenecks.
This doesn't JUST affect Netflix customers, but in fact ANY traffic that runs from Cogent or Level3 to the ISPs customer or vice versa. It just so happens that A LOT of that traffic is Netflix, but it's not EXCLUSIVELY Netflix. There are a great many people that buy transit from Cogent and Level3, meaning that ISPs are harming more than simply Netflix customers but ALL their customers.
Please understand that if the ISP isn't making an effort to ensure that their peering arrangements are sufficient to satisfy the demands of their customers, that is the ISPs fault, and not the counterparty's fault. Especially when the counterparty is willing to make the necessary upgrades to relieve the congestion, and the ISP is not because the ISP wants to get paid for it.
Remember that the ISP is the one that's offered highly asymmetric internet access to their customers. They should not be surprised that their customers utilize said service in an asymmetric fashion. Failing to upgrade the peering connection isn't "standard operating procedure" for a consumer-facing ISP. It's that ISP doing something highly unusual because they think they can get away with it.
> Failing to upgrade the peering connection isn't "standard operating procedure" for a consumer-facing ISP.
This seems to be the main point of departure for us. If you could show that large consumer-facing ISPs used to give free peering even when the other side was sending much more data than they were, then I'd agree with you. The ISPs claim it is standard for someone sending more data to pay. Can you show otherwise?
Consumer-facing ISPs have generally arranged their business around a highly-asymmetric, consumer-pays model. Right? If I want more download, I have to pay more. I don't get free, unlimited "as much as I want" downloads, and only pay for the uploads, right? So for a consumer ISP, the principle is "receiver pays"
For a commercial, content producers, the model is "sender pays" which makes sense. That's why hosting companies buy transit from Level3, Cogent, etc. They're paying to send.
What's happened is that Netflix has already paid to send, and ISP customers have already paid to receive, and ISPs have said "yeah that's fine, but if you REALLY want to send, you're going to have to pay AGAIN"
That's the sticking point.
Nobody's arguing that anyone should get a free ability to send, or a free ability to receive. All traffic is generally double billed, once for the sender's connection and once for the receiver's connection.
The "innovation" here is that some ISPs have decided that SOME traffic should be TRIPLE billed. Once at the sender, once at the receiver, and a third time when it crosses from the sender's network to the receiver's network.
Why, if triple billing is OK, should only the consumer ISPs be allowed to do it? Why doesn't Cogent or Level3 charge consumer ISPs extra for receiving more than they're sending? Under the consumer ISPs model of "receiver pays" it would be entirely reasonable for Cogent or Level3 to charge extra, wouldn't it?
You're basically arguing that the way consumer ISPs handle this is acceptable. OK. Why isn't the exact opposite also acceptable? It's not as though most consumer ISPs have a global transit network! They're the ones that desperately need more download than upload to satisfy their highly asymmetrically provisioned customer connections, right?
Ultimately the problem here is that there are several analytical frameworks that one can use to decide who is the asshole here and who is the saint.
Given that I'm held captive by my consumer ISP and have a fair amount of choice over what content I subscribe to, I'm more inclined to believe that the consumer ISPs are abusing their power over the content providers. I don't have pleasant interactions with them basically ever so I can't see why I would extend them the benefit of the doubt.
>Under the consumer ISPs model of "receiver pays" it would be entirely reasonable for Cogent or Level3 to charge extra, wouldn't it?
Right now, they want to send to the ISP more than the ISP wants the data. So they have to pay. If they wanted to charge, the ISPs would say "sorry".
And you sort of ignored my question. You claimed above that paid peering for asymmetrical connections was not standard practice. I want you to source that, because the ISPs claim it always was standard practice.
>Ultimately the problem here is that there are several analytical frameworks that one can use to decide who is the asshole here and who is the saint.
That's why it's better to let everyone bargain it out amongst themselves. If it's important to netflix to have a faster connection, they'll pay for it. If it's important for the ISP to have a faster connection, they'll pay. In a free market, the resources should flow to whoever needs them most. (I believe that's the Coase theorem.)
> That's why it's better to let everyone bargain it out amongst themselves. ... In a free market, the resources should flow to whoever needs them most.
If I had the opportunity to ditch my ISP who wants to throttle Netflix through underhanded, shady tactics I WOULD ABSOLUTELY DO SO!
But the various entities were granted regional monopolies on certain wire plants like coax and utp and those huge, huge, huge advantages have yet to be leveled by the same entities that enacted the laws which caused them.
If I had the option to choose from at least say 5-10 different ISPs and some of them did what Comcast, Verizon and AT&T choose to do and other did not, I would vote with my wallet and choose a provider who does not engage in such tactics. This would cause those providers to lose customers and eventually, those providers to reconsider their "paid peering" stance. Or not! But I wouldn't be subject to the whims of a few business folk at a company that I have no practical way to influence.
As it stands, I do not have that choice. So I'm not terribly opposed to various ISPs being told through various means to play nice or have their toys taken away.
But given that I don't have any kind of real choice (duopoly at best and I'm in a major metro area!) it's kind of disingenuous to say "the free market knows best!" when there is in fact not a free market. Free the market up and let people duke it out. But don't pretend that the un-free-ness of the market somehow doesn't exist just because there are multiple long-haul transit providers.
I'm not talking about the market between you and your ISP, which I agree is not very free. I'm talking about the market between the ISP and the other ISP, or between an ISP and a CDN. I think every participant should be allowed to make their own decision whether to pay, and the other side shouldn't be forced to carry traffic it isn't paid for from that side.
I've argued elsewhere that throttling based on content shouldn't be allowed for reasons similar to what you're saying, but I don't want to extend that to paid peering or prioritization.
> I'm not talking about the market between you and your ISP, which I agree is not very free.
Right, but those two issues are for me, inseparable. If I don't have the freedom to choose my ISP and my ISP doesn't take my opinion into account when making business decisions, then I have no recourse.
If there were other ISPs for me to choose from, then I'm all for absolutely ZERO laws about net neutrality because then actual market forces can actually go to work.
Market forces on the consumer side of consumer ISPs can directly influence the happenings on the peering side of consumer ISPs.
> I think every participant should be allowed to make their own decision whether to pay, and the other side shouldn't be forced to carry traffic it isn't paid for from that side.
Sure, so now Cogent and Level3 can demand $100/Mbps/mo from Comcast right? Because without buying global transit, it's not really INTERNET, it's just Comcast. What if all the transit providers decided that those were the new rules? Would Comcast have any alternative?
What if these transit providers all got together and agreed that none of them would offer Comcast global transit for less than $100/Mbps/mo? Would that be OK? I mean, Comcast has highly asymmetric flows right? They should be allowed to do this, shouldn't they?
Because as it stands right now, I've got a million Comcasts I can choose from, but they've all agreed with one another that the pricing is the same no matter which one I pick. If Comcast can hold me hostage, why can't the transit providers all agree to hold Comcast hostage?
> Why is it ok for Netflix to pay a transit company, but not Comcast?
Because transit providers are actually providing a service to Netflix. They allow them to dump all the traffic in one location and then the transit provider will build out a network to take care of moving the traffic to anywhere in the globe where there's an ISP requesting it. The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user. When Netflix connects directly to Comcast that's not what's happening, Netflix has to get the content to Comcast's network by itself and the Comcast network is already paid for by the user. So if Netflix pays Comcast then Comcast is getting paid to create the network by both the user and Netflix.
> The transit provider will then not pay the ISP to take the traffic and deliver it to their requesting user.
The "not pay" part is, I assume, based on the prevalent use of settlement-free peering in the industry. These agreements were based on a understanding that, over time, the traffic would wash out even between the two, so let's just not keep track and call it even from the start. This is not necessarily a given in all situations. Comcast asserted that the quantity of Netflix traffic was breaking this understanding.
>These agreements were based on a understanding that, over time, the traffic would wash out even between the two, so let's just not keep track and call it even from the start.
I don't think that's the case. Transit providers will always tend to deliver more traffic to the ISP than the ISP delivers to them, because users download more than they upload. That's still fine because the transit provider's network is paid for by the content provider and the ISPs network is paid for by the user so the settlement free peering always makes sense. Without it the ISP wouldn't have content so no users and the transit provider wouldn't be able to deliver traffic so would have no clients. The fact that Netflix is now a significant single content provider in the transit network doesn't change this dynamic at all, it just makes it easier for ISPs to squeeze them for money. Compensation for unbalanced peering only makes sense between transit providers since for each connection only one of them is being compensated (the one whose client sent the traffic) so the other one will only accept that peering if it is balanced.
I've explained this more clearly elsewhere in the thread:
I'm having significant difficulties with this paragraph:
"Comcast claims that Netflix was sending traffic at such high volumes as to intentionally congest the links between different transit ISPs and Comcast, essentially taking a page from Norton’s “peering playbook” and forcing Comcast and its peers (i.e., the transit providers, Cogent, Level 3, Tata, and others) to upgrade capacity one-by-one, before sending traffic down a different path, congesting that, and forcing an upgrade. Their position was that Netflix was sending more traffic through these transit providers than the transit providers could handle, and thus that Netflix or their transit providers should pay to connect to Comcast directly. Comcast also implies that certain transit providers such as Cogent are likely the source of congested paths, a claim that has been explored but not yet conclusively proved one way or the other, owing to the difficulty of locating these points of congestion (more on that in a future post)."
Now, peering negotiations are an ugly, unpleasant activity that I'm quite happy never to have been a part of. But I don't find this particular claim to be "reasonable and plausible", in the author's words. Where is the congestion?
* Inside Netflix? Nope.
* Between Netflix and their transit providers? Seems unlikely; Netflix is paying them, after all. Such congestion would effect many of Netflix's customers and be against Netflix's interests.
* In the transit providers? Possibly, but again, Netflix is paying them. Further, Netflix connecting directly to Comcast is the first thing Netflix would do, if the transit providers were trying to squeeze more money out of them.
* Between the transit providers and Comcast? That seems to me to be the most likely interpretation of that paragraph. But if Netflix is doing this intentionally; say, by directing 90% of their traffic to Comcast customers through Tata, those customers are most likely to be unhappy with Netflix, not Comcast---any traffic from a customer out that doesn't go through Tata is fine, so they wouldn't likely perceive any problems with anything but Netflix. Further, that seems to contradict the statement that "[customers] routinely experienced high latency to many Internet destinations every evening during 'prime time'". And if the congestion is not hitting one specific transit provider, then the idea that it is intentional on Netflix's part is silly and Comcast's peering relationships are simply inadequate for the traffic its customers are using.
* Inside Comcast? Well, I could see almost anyone wanting to have an unrelated third-party pay for their infrastructure upgrades, but....
Then, there's this: "The best technical solution (and what ultimately happened) is that Netflix and Comcast should interconnect directly." The best technical solution? The best technical solution? (Here's a hint: a fully-connected network is not a good technical solution.)
But the ultimate problem here is: "This is where market leverage comes into play: Because most consumers do not have choice in broadband Internet providers, Comcast arguably (and, empirically speaking, as well) has more market leverage: They can afford to ask Netflix to pay for that direct link—a common Internet business relationship called paid peering—because they have more market power."
Comcast has more market power because it is a monopoly and the problem with monopolies is precisely that they use their market power to squeeze money out of counter-parties.
As an aside, I find Feamster's comment reply, "What is false in Oliver’s presentation (and many of the claims in the popular press) is his claim about what causes* that slowdown. The cause of the slowdown is congestion, not intentional throttling,*" to be simply disingenuous. What exactly is the difference between intentional throttling and intentionally introducing congestion by not upgrading inadequate peering relationships?
>As an aside, I find Feamster's comment reply, "What is false in Oliver’s presentation (and many of the claims in the popular press) is his claim about what causes* that slowdown. The cause of the slowdown is congestion, not intentional throttling,*" to be simply disingenuous. What exactly is the difference between intentional throttling and intentionally introducing congestion by not upgrading inadequate peering relationships?
He discusses that in a later comment. The point I think he's making is that it wouldn't qualify as "paid prioritization", and shouldn't be characterised as such.
I thought that was from his comment. Did I miss a later one?
It was paid; it succeeded in getting Netflix to write them a check. And it may not be strictly "prioritization", but to me the intent means more than the letter.
http://www.cogentco.com/en/news/press-releases/631-cogent-of...
Pretty sure Comcast didn't take them up on the offer. Doing so would have fixed the issue and Netflix wouldn't be writing checks to Comcast now.