
The Case Against ISP Tolls - acgourley
http://blog.netflix.com/2014/04/the-case-against-isp-tolls.html
======
nostromo
Netflix and Google and Amazon and Apple should join forces to let Comcast
customers know when they're being throttled.

"Your video is buffering because Comcast is slowing your connection down.
Click here to contact Comcast customer support. Click here to find Netflix
Certified providers in your area."

Netflix and Google and Amazon and Apple all have great brands with high
customer satisfaction. They should utilize all those eyeballs to funnel even
more hate toward Comcast.

~~~
brandon272
> "Click here to find Netflix Certified providers in your area."

Followed by:

"I'm sorry, there are no Netflix Certified providers in your area."

When Comcast is rated as one of the worst companies for customer satisfaction,
customers aren't staying with them because they are ignorant when it comes to
the alternatives. They are staying with them because there are no
alternatives.

~~~
CoolGuySteve
"I'm sorry, there are no Netflix Certified providers in your area."

Seems like a natural place to put "click here to contact your state/federal
representative"

While I grant that cynisim is definitely warranted, at the end of the day,
these companies have monopoly jurisdiction because we the people allow them
to.

~~~
logicchains
Because there are laws on the books explicitly preventing anyone else from
competing with Comcast.

~~~
anologwintermut
[http://arstechnica.com/tech-policy/2014/02/isp-lobby-has-
alr...](http://arstechnica.com/tech-policy/2014/02/isp-lobby-has-already-won-
limits-on-public-broadband-in-20-states/)

~~~
logicchains
Can anyone tell me what would be the problem with just having a law "no law is
allowed to be passed preventing any individual or entity from providing
broadband service to any other individual or entity"? And how said problem
would be worse than the current dismal arrangements?

~~~
arg01
"Look I'm sorry mate but I'm just running this wire through your house to give
your neighbor internet, if you don't let me you're in the wrong"

"Sorry but we need to tear up this highway and disrupt traffic for the next
couple of weeks to lay some fibre. I know someone else did the same thing last
week but they were another company and it's illegal for the local government
body to stop any of us laying stuff on public land when we want"

Obviously that extreme is untenable. You may be after something a bit more
useful like unbundling, but in the US you have to explain how forcing a
private company to let competitors use it's cable is not outright stalinistic
communism.

~~~
logicchains
There's a difference between not allowing ISPs access to public lands and
outright prohibiting them from offering their services.

Obviously if they wanted to run wire through a house they'd have to get the
owner's permission, or the local government's permission to dig up a road. The
thing is, as the law currently stands they're forbidden from doing so; even if
every household in they city signed a contract saying "sure, let company X run
cable through our yard", company X still wouldn't be allowed to sell them
internet over that cable.

Maybe a company could innovate and develop some form of fast wireless mesh
network. But even if they had the technology and finances to do so, local laws
stating "the only company allowed to sell fast internet here is Comcast" or
equivalent prevent them from doing so.

------
notdonspaulding
Comcast can do what it does because Democrats portray the company as a problem
with capitalism, while Republicans portray it as a problem with regulation.

In truth, it's both. It's a problem where a company has co-opted well-meaning
regulation to snuff out competition, which is a thing capitalist companies do.
Meanwhile, citizens point the finger at each other instead of removing
Comcast's ability to monopolize.

Either the regulations that make competing as an ISP difficult need to be
stripped out (my personal favorite), or the last mile needs to be public
property. Either way, Comcast getting to be the only one sitting on that last-
mile connection is what has given rise to its stranglehold on the internet.

We need to support fixing this on a local/state level. Support smaller ISPs in
their attempts to disrupt the incumbents, and support legislation that
actually encourages innovation in the marketplace (even if that means striking
policies from the law books).

~~~
maxsilver
As someone who's trying to bootstrap a small, local, last-mile ISP right now,
my only wish is that there were _many_ more people that held your opinion on
this matter, and were willing to back that opinion with their dollars.

~~~
antimatter
Best of luck. May I ask for a high level overview of what is involved with
bootstrapping a small ISP?

~~~
maxsilver
Sure, you can ask me anything :)

The process so far has involved :

\- Figure out how to get access to right of ways [Completed] \- Figure out how
to get internet access to sell [Completed] \- Figure out how much it will cost
to start a CO [Mostly Completed] \- Acquire a place to put a CO [Completed] \-
Acquire high quality, redundant backhaul [Under Construction Now]

\- Figure out who's willing to pay to switch away from Comcast [I have some
names, but this is definitely not completed] \- Beg them for money [Some have
subscribed, but definitely not enough yet]

There's a couple of big things that work against me:

City + State taxes for right of ways are pricy. We're talking multiple
thousands of dollars per mile per year, on every single mile, just in taxes.
(And I'm lucky, my city still owns a lot of their own right of ways. A lot of
cities just wholesale gave them away to AT&T/Comcast).

Backhaul is expensive. Most ISP's are monopolies, so they can safely buy in
scale. I have zero scale, so I pay full retail for everything. This is
obviously very pricy. (I'm guessing I pay at least 3 times more, per Mbps,
than Charter pays for backhaul, and probably 4 - 6+ times more than Comcast
pays)

Fiber lines are dirt cheap. Install is expensive. Easily 95% of the cost of
the network is that initial installation, and it can run into past the million
dollar mark even in the smallest of neighborhoods.

If you want more info, let me know. I'd happily talk more about this, and
provide solid numbers.

~~~
gaadd33
Is backhaul more expensive than getting a big Cogent connection for like
$1/Mbps? Or do places like Charter/etc pay even less than that for their paid
peering connections?

~~~
maxsilver
Yes. I've never seen a quote at $1/Mbps regardless of scale.

It wouldn't surprise me if speed/pricing like that existed around major
network hubs (like Chicago).

I'm paying $8/Mbps. If I were to afford five times more backhaul than I have
now, that price eventually drops to about $3.8/Mbps.

Places like Charter and Comcast pay less for their backhaul, in part because
they _are_ their backhaul, in part because they are large enough to negotiate
their own peering agreements.

Recently, that's swung even more in their favour. Netflix, for instance, pays
Comcast money for the "privilege" of paying the costs for all of the backhaul
Netflix uses over Comcasts lines.

People sometimes call this "double dipping". But it's actually "triple
dipping". Comcast charges subscribers money, Comcast makes Netflix buy the
backhaul to Comcast, and Comcast charges Netflix again, to allow them to
connect the backhaul they already bought, to Comcast's network.

~~~
gaadd33
Yeah I'm not sure the cost for the infrastructure but I've seen quotes of
$1-$2/Mbps for Cogent in various datacenters (assuming a 1-2Gbps commit I
think).

Makes sense though, I'm sure if you had to get the fiber to your location that
would easily push the price up a ton.

------
6cxs2hd6
When you think about it, Comcast NBC Universal is doing the same stuff with
Aereo.

Broadcasters get a government monopoly on a finite resource. Broadcasters
don't provide a good signal to all customers. Some customers want to use Aereo
(that's not the _only_ reason, but a _real_ reason) instead of paying for
cable.

But no, the broadcasters want to be paid for not-providing a signal to those
customers. Instead of investing to improve their infrastructure, they want to
be paid for _not_ improving it. With a straight face.

Of course it's especially ridiculous in the case where it's Comcast NBC, and
they are also an ISP. If Aereo wins at SCOTUS, and grows big enough, Comcast
NBC can get their pound of flesh as an ISP like they do with Netflix.

It is all well and truly fscked.

~~~
lessnonymous
> But no, the broadcasters want to be paid for not-providing a signal to those
> customers. Instead of investing to improve their infrastructure, they want
> to be paid for not improving it.

This is the best one paragraph summary of the Aereo case I've heard. Certainly
makes it absolutely plain why it needs to be allowed to continue.

------
fizzbar
This image [http://www.willchatham.com/wp-
content/uploads/2014/01/net.pn...](http://www.willchatham.com/wp-
content/uploads/2014/01/net.png) has been making the rounds on reddit,
surprised it hasn't appeared here yet. This is the most visceral/succinct
explanation of the consequences of allowing the ISPs to have a "fast lane"...

~~~
wmf
The clever thing about Comcast's intentional congestion extortion loophole is
that the customer never sees it, so it _doesn 't_ look like that image.

------
higherpurpose
This could be the right time for Netflix to move to P2P, so then most of the
bandwidth isn't consumed by them, but by their users.

[http://finance.yahoo.com/news/netflix-switched-p2p-video-
str...](http://finance.yahoo.com/news/netflix-switched-p2p-video-
streaming-180229987.html)

It's going to take a lot to fight the likes of Comcast in this one, which
means Netflix will also need support from the community - so Netflix, please
stop it with the boneheaded DRM push on the web. Guess what, if you _are_
going to use P2P, you're going to need a plugin or native app _anyway_. So
just stop trying to corrupt W3C already in a pointless move that can't and
won't stop piracy anyway (which I'm sure you already know).

~~~
wmf
P2P would shift the congestion from transit links to shared upstream last mile
links which will probably make ISPs even more unhappy.

~~~
kiallmacinnes
it would also shift the cost to consumers.. at least once you use up your data
cap.

~~~
shakethemonkey
"Though Comcast doesn't disclose it specifically, analysts have estimated that
Comcast's profit margins on broadband service are 80 percent or higher. In
2008, Sanford C. Bernstein & Co. analyst Craig Moffett estimated Comcast's
data margins at 80 percent, and Credit Suisse reported in fall 2010 that
Comcast's gross margins on high-speed data were 93 percent."

[https://www.freepress.net/sites/default/files/resources/Comc...](https://www.freepress.net/sites/default/files/resources/Comcast%20New%20Caps%20Factsheet_FINAL.pdf)

------
debian3
You only need two or three major website who blackout their services to the
Comcast user and redirect them to a page explaining that situation. Imagine if
their subscriber can't access to Google and Facebook, and on that redirection
page they target ads for alternative provider in your area. Would be nice to
see how long Comcast would take to change their mind.

~~~
kevinchen
That is incredibly user-hostile. Google and Facebook would end up looking like
the bad guys, and nothing would change.

~~~
falsecelaona
No it is not. ISP throttling their paying customers because of their internal
nonsense is user-hostile. Informing consumers that they're being undermined by
their ISP is quite possibly the most pro-consumer act you could do. If it gets
us one step closer to treating the bandwidth like electricity or water, it's a
good thing.

~~~
kevinchen
Two possibilities from the user's point of view:

"Facebook is a little slow because my ISP might be throttling, but I'm not
sure, it could also be my computer"

"I cannot access Facebook at all because Facebook blocked my ISP"

Getting users pissed off at web services is not a good way to effect change.
We need users on our side, so we can't have them think of us as crybabies.

------
jebblue
This:

>> "In this way, Comcast is double dipping by getting both its subscribers and
Internet content providers to pay for access to each other."

..and this:

>> "Put simply, there is one and only one way to reach Comcast’s subscribers
at the last mile: Comcast."

...should be illegal. This really torques me. This smacks of how AT&T acted in
the 80's when I was in my 20's and starting out on my own and how Microsoft
acted in the 90's before they got busted, which is what lead me to Linux.

~~~
rayiner
> ...should be illegal.

And Facebook is the only way to reach Facebook's customers. Should that be
illegal?

~~~
pritambaral
Except Facebook's customers use Facebook to access Facebook's data. Comcast's
customers use Comcast to access the internet's data, not Comcast's data. At
least until the definition of Internet Service Provider is changed.

~~~
rayiner
Not to the extent that Facebook is a platform for other services (Zynga, etc).

------
oelmekki
Same problem exists in France with the "Free" (actual business name) ISP.

It's a large ISP here, and it throttles access to youtube (and probably other
content providers). It has been so for years. For some reason, there is no
public awareness about that, people just say "my internet is not working
well".

It's an act of aggression against their customers, to me, since all ISPs are
marketing their offers around connection speed. And it's violating net
neutrality, too. I don't get how it does not become a major PR problem for
ISPs doing that.

~~~
liotier
Not just Iliad's Free - I know first hand that others admit internally that
they are doing it too... But they choose to let Free be publicly vocal about
it while they are just as violent behind closed doors.

~~~
oelmekki
Yes, I've heard some vague reports from others ISP users too, but not anything
I have measured of even just experienced first hand.

That's the reason why I've switched to the new ISP services from ovh, though,
as they made net neutrality their first marketing point (but for the same
price, you don't have tv, if that matters).

------
andrew93101
Has there been discussion of passing this fee on to customers who access from
Comcast in the form of a surcharge? If Netflix hiked prices for customers who
were connecting from Comcast networks, it would encourage news of this
business practice to spread.

------
zaroth
Netflix presumably was paying Cogent a lot of money for all those 10Gbit
uplinks. Now a few cross-connects and BGP tweaks later, a traceroute shows
Netflix packets leaving a Netflix server and going more directly onto
Comcast's network. I can only speculate that they turned on some dark fiber to
build cross-connects between their cages and Comcast cages. Maybe they were
already in the same building...

How much did Netflix's Cogent bill go down? How much of the "work" that Cogent
was doing is Netflix now doing themselves? How many hops of the traceroute are
owned by Netflix, and how many hops by Comcast?

When I rent a rack in One Wilshire in LA, CA, it comes with power, cooling,
and a network link. When I pay let's say $1000/mo for that 1Gbps link, it's
not just the link from my rack to their router that I'm paying for. And when I
send 1Gbps of packets up through that link, it's not just CoreSite at One
Wilshire carrying the cost of delivering those packets.

At some point, yes, I do expect Comcast to get their "fair share" of that
$1k/mo. If Comcast is PAID zero dollars for delivering that 1Gbps, then I want
this free bandwidth gravy train to propagate all the way back to me. Comcast,
please sign me up for 10 of those free 10Gbe links you're giving out, ok? Just
tell me where to rent space where I can get those free 10Gbe cross connects to
your network.

But then, you can also flip the perspective and ask the question, why doesn't
Comcast have to pay for all their bandwidth? Shouldn't they be paying
$1,000/mo for their Gbe's?

So why does bandwidth cost money, or more specifically _when_ should bandwidth
cost money? Price, as always, is just a means of efficiently allocating scarce
resources. Price can _only possibly be zero_ if there is no scarcity. Is it
possible there's no scarcity of bandwidth on Comcast's network? Now that would
be comical!

I think the answer lies partially in another development. Ask Netflix to chart
the cost in terms of Watts and Rack Units to deliver 1Gbps of video streams.
My theory would be the cost in terms of space and electricity to generate
10Gbe of video streams has dropped orders of magnitude. This means bandwidth
will be an increasing portion of your bill no matter what. It also puts more
pressure on scarcity of bandwidth when the ease of which you can generate
10Gbps of useful traffic is down to just a few servers.

~~~
aidenn0
> If comcast is PAID zero dollars...

Comcast gets paid $50/month per subscriber for that bandwidth. It costs
Comcast less money to peer with Netflix than to get it from cogent. Comcast
wants Netflix to pay to solve a problem that Comcast's customers have.

> Is it possible there's no scarcity of bandwidth on Comcast's network?

Absolutely! In fact, that's part of what the chart shows; when Netflix peered
with Comcast, the video quality (which is a proxy for bitrate to the consumer)
went up significantly, which indicates that there was no bottleneck on the
Comcast network.

> Price ... is just a means of efficiently allocating scarce resources.

Not in the case of a monopoly. Comcast has a monopoly on a large fraction of
the broadband market. They also compete directly with Netflix, so they will
leverage that monopoly to make it as expensive as possible for Netflix to
continue to compete with them. This is Comcast creating an artificial scarcity
of paths from Netflix to Comcast's customers.

On another note, I have referred to Comcast subscribers as "customers" above;
if agreements like the above continue, then Comcast subscribers are no longer
customers, but rather the product that Comcast charges for efficient access
to.

~~~
zaroth
Lets assume Comcast honestly thinks that Netflix owes them money for the
bandwidth. I know it's highly generous, but it's a fair starting point.

How does it costs Comcast less money to peer with Netflix than to get it from
Cogent? Does Comcast pay Cogent too?

Of course the whole confrontation was manufactured, I don't think Netflix was
ever arguing that Comcast needed to boost its entire buildout just to support
their video streams, and by the way they shouldn't pay a cent. Comcast doesn't
have to wait till its pipe are full before controlling traffic flows.

------
phrasz
They should use classic Cable TV vs. Local Broadcasters tactics: Block all IPs
from Comcast and force them to pay access to your content...which happens to
be the exact amount they want for the Tolls.

If we consumers have to pay to access to content why shouldn't ISPs?

~~~
SourPatch
That might work if Netflix had compelling content. Maybe they do, I don't
know. But it is not helping Netflix that Comcast is ranked near the top for
ISP market share and also owns a ton of its own content (NBC Universal).

------
dec0dedab0de
This makes me so mad. I kind of wish all of the Big Internet providers like
Level3, and Sprint would team up and block all traffic in and out of Comcast.
Maybe redirect all http traffic to a page listing alternative ISPs.

~~~
johnpowell
There aren't really alternate ISP's. It is Comcast or Centurylink here. And I
live in a relatively big city of 200K people.

------
mikekij
Reading this actually makes me hope that Facebook gets their wifi-beaming
balloons up and running quickly. I'd like to support Comcast as little as
humanly possible.

~~~
wmf
Neither balloons or drones are going to be fast enough to watch Netflix.

~~~
beamatronic
I wish there was a service that would just let me rent discs through the mail.

------
naviwins
I see all these people saying Comcast was feuding with cogent... that is just
not the case. For anyone in the hosting business we all know cogent is the
cheapest option, and a pretty spotty service as a whole. What do we do when
our customers complain and complain? Well, we seek out more reliable options
and spend the money needed better serve our customers.

So why does Netflix get to complain? They have already spent the money for the
ridiculously one sided peering(Netflix sending tons of traffic and Comcast
sending close to none). You would think Netflix would be happy to spend this
money to better their service! Right? Or is it so detrimental to their cash
flow that they would rather complain? They should have just told their
customers "hey, we heard you were having some problems with our service, look
what we did for you!"

In the end this is just a smear campaign to distract Netflix customers from
the price hike that is coming. It is obvious Netflix was going to raise its
subscription prices eventually. So why not point the blame to a company who is
already disliked. The majority of HN seems to be pretty smart people, it would
be a shame if you fell for this rather underhanded ploy.

~~~
josephlord
Who sends traffic is just easier to measure, the person receiving it may want
it more and in the Netflix case they are requesting the traffic.

In any case all Comcast's traffic is for their customers, when peering needs
to be balanced to be free is when to parties are offering transit to other
networks beyond themselves.

------
nkcmr
I really hope Netflix has what it takes to do what is necessary if it comes to
that point. And what it is going to take is for Netflix to simply blackout
Comcast.

But actions like that can be easily spun out in the media (media of which has
very close ties to Comcast) to make Netflix look bad.

What would make it better is for all of the major players (Google, Facebook,
Twitter, Apple) to collectively blackout Comcast, whilst educating the public
on how they are being used and screwed at the same time.

------
josephlord
If Netflix are right an Comcast are providing insufficient connectivity
through to all of the major transit networks (rather than just Cogent) it
feels to me as if Comcast are not providing the internet connection speeds
that they are selling. It would be interesting to see what would happen if
Netflix bought from all the many transit providers and spread the load across
them, would all of Comcasts general service grind to a halt.

------
brianmcdonough
The problem is sufficiently complex and boring to favor Comcast. What I would
like to know is this: It is already a regional monopoly in high speed internet
because no on can touch their speeds, why would anyone consider extending
their monopoly nation-wide?

------
clubhi
I dropped Netflix about a month ago when I first heard they made this
agreement with Comcast. They are part of the problem.

------
pixelcort
I think the real problem is unlimited internet connections.

If ISP customers paid per-GB and/or per-Mbps, then the ISP shouldn't care if
it's Netflix or Hulu or YouTube being consumed.

The issue is that today, it's difficult to get a good feel for how much
bandwidth is being used at any given moment. Unlike a water faucet, our
devices don't really have a knob we can adjust the speed with.

~~~
kalleboo
I think the real problem is poor infrastructure.

None of this has been a problem here in Sweden, where the infrastructure is
solid and easily supports end users with 100 Mbit home connections without me
hearing any complaints.

~~~
dsjoerg
The quality of infrastructure is the result of political and cultural factors,
which are therefore the more underlying/real problem.

Do you have a choice of multiple ISPs? If yes, then _that 's_ the critical
difference vs the U.S., not the infrastructure. If no, then _why_ has your
monopoly ISP built such good infrastructure?

~~~
nolok
France's situation: national public telecom company (France Telecom, now
Orange) built the network, private competitors appeared, Orange got
privatized, competitors didn't have the network to be everywhere and allow
proper competition so the law forced Orange to allow other ISPs to use
Orange's network for a fee (so I could be a client of, say, Bouygue Telecom,
but using Orange network to join Bouygue's).

A huge and easy justification for that is that Orange's network had mostly
been built using public funds since it was a public company at the time.

Since then, several competitors have built their own network, but most still
use Orange's sharing for remote place, small villages etc where it's not worth
it for them to deploy their own.

So your question is hard to answer. Why has the monopoly ISP built the
infrastructure ? Because it was a public company, and if you wanted your house
to be connected to the network they had to do it. Why have the others built
their own ? Because it was much cheaper for them long term, especially since
they also use it for their cellphone network.

And from what I see, it seems to be the same in a lot of Europe, the national
companies made the original network, and competitors piggy backed on it until
they had their own.

(I've oversimplified of course, but that's the gist of it)

------
skkbits
Is that "tata" from India ? How do they play role here ?

~~~
wmf
Yes, Tata Group from India owns an Internet backbone (formerly VSNL &
Teleglobe). Netflix traffic used to go through Tata to get to Comcast until
Comcast deliberately congested it.

~~~
cheald
Could you provide a link for that story? I hadn't heard of it til now.

~~~
wmf
[http://www.merit.edu/mail.archives/nanog/msg15911.html](http://www.merit.edu/mail.archives/nanog/msg15911.html)

[http://www.internap.com/2010/12/02/peering-disputes-
comcast-...](http://www.internap.com/2010/12/02/peering-disputes-comcast-
level-3-and-you/)

------
exelius
This is a very public battle between two companies about who should shoulder
the cost of delivering video to your home.

Comcast's argument is that they have built a network at great expense for
carrying content that others make a profit from, and they want to capture at
least some of that value.

Netflix's argument is that they have built a CDN to deliver that content to
Comcast's doorstep, and that Comcast won't accept it.

In my mind, Netflix charges money for their service and is not entitled to a
free ride. The fact remains that running an ISP is not free, and you can't
scale out a network faster than the cash flows (of which, yes, profits are a
part) allow. But think about this: Netflix has a 36% gross profit margin, and
Comcast's cable/Internet business has an 11% gross profit margin. Netflix's
cost structure is lower in large part because Comcast does a lot of the
expensive work for them (i.e. last-mile network maintenance and distribution.)

Netflix drives significantly increased Internet traffic usage, and Comcast's
investors should not have to accept lower profit margins so that Netflix's
investors can enjoy larger margins because of investments that Comcast's
investors have made.

I would have a lot more sympathy for Netflix if their margins weren't so high.
It just comes off as greedy when they have a 36% gross margin.

~~~
dangrossman
Comcast's customers are paying Comcast to deliver Netflix's bits to them. Why
does Comcast need to be paid for those bits twice?

If we're going to throw out some numbers, Netflix's net income was just $112M
on $4.3B in revenue last year (2.6%). Comcast's net income was $6.82B on
$64.66B in revenue (10.5%). Netflix's spend on content acquisition and
production is increasing, which means that 2.6% is shrinking; they're already
cash flow negative IIRC. Not much room to start doubling up on bandwidth fees
they already pay.

~~~
exelius
Because Comcast does not incur cost per-bit, they incur cost based on the
maximum required capacity of their network. Netflix significantly increases
those costs on a per-customer basis.

~~~
pdonis
No, _customers who want to watch Netflix_ increase those costs. Which really
means, _customers who want HD video streaming bandwidth_ increase those costs.
To which the right answer is, _build more bandwidth and charge the customers
for it_. Customers who want to watch Netflix, or any other HD video streaming
service, can pay Comcast for "HD streaming" bandwidth; customers who don't can
pay for the bandwidth they need. None of this should have anything to do with
Netflix.

I understand that Comcast doesn't like this because it makes them a simple
seller of bandwidth instead of a "provider of premium services", or whatever
marketing mantra they're using this week. But if customers do not _want_ their
"premium services", but just want bandwidth, the solution to that is to _build
better premium services_ , not to try to get paid twice for one stream of
bits.

