
Comcast is definitely throttling Netflix, and it’s infuriating - ghayes
http://mattvukas.com/2014/02/10/comcast-definitely-throttling-netflix-infuriating/
======
pilif
In the last two years, there were periods where YouTube was unusable for me in
the early to mid evenings. My provider freely admitted that their current
peering arrangement just wasn't good enough for all the traffic that their
customers were creating by watching youtube videos.

Over time, this was fixed, only to return later, at which point it was fixed
again.

Huge kudos to my provider (Cablecom in Switzerland) for actually fixing this
and even in a somewhat timely manner.

But.

As more and more of the internet users traffic gets centralized to a few big
services (Netflix, YouTube, to some extent Facebook), the usual unbureaucratic
peering agreements start to become less and less feasible as traffic starts to
be centralized with a few very major content providers.

Not only that: some players might start to produce so much traffic that it
starts to become useful to go as far as to start doing special proxy solutions
for specific content providers (another provider here in Switzerland has a
special solution in place for youtube after talking with Google).

At which point does all that effort in order to have your end-users be able to
access some of these huge content providers stop being worth it in the name of
net neutrality?

Providing network neutrality is easy when everything just works, but when you
have to make new peering agreements, build new pipes or create special
infrastructure just for one content provider that doesn't pay you a cent, then
that whole neutrality thing becomes very inconvenient and, I have to say,
unfair to the providers.

Yes. I really, really want network neutrality, but I see that aside of all the
political and business-model-preservation issues, we are running into
technical issues too and those need solving too.

~~~
izacus
Last I worked at a ISP, Google actually DOES provide YouTube caching servers
to ISPs that want it - they bring you a black box machine which you plug into
your backbone switch that does transparent caching for users. Getting one has
significantly reduced load on internatinal peering lines (I'm from a small EU
country) and made quite a differece for our customers as well (suddenly
YouTube started working smoothly for most of them while on other ISPs it
sometime still stutter).

I guess same thing can be provided by Netflix as well.

~~~
pilif
True. According to RKearney in this thread, Netflix does that too. But isn't
that, too, a break of network neutrality? The moment you start accepting
content-provider specific equipment, aren't you preferring them over those
that don't provide equipment?

Wouldn't accepting equipment from one provider compel you to accept from
others too in order to stay neutral.

~~~
soneil
I don't see anything that says they wouldn't accept equipment?

"If your endpoint drives enough traffic, we're open to solutions" doesn't seem
silly.

What about ISPs that provide mirrors? An example I've used before was
Australian ISPs providing steam, linux, etc mirrors to save their transoceanic
costs. Is that still neutral?

------
wmf
This kind of test has been discussed at Ars Technica in the last few days.
[http://arstechnica.com/information-
technology/2014/02/verizo...](http://arstechnica.com/information-
technology/2014/02/verizon-could-be-throttling-netflix-and-amazon-but-theres-
no-actual-evidence-of-it/) It doesn't conclusively prove anything since the
Netflix and VPN traffic take different paths, only one of which may be
congested. In particular, I suspect the encryption has nothing to do with it.
Ultimately I think this is the same old problem:
[http://www.internap.com/2010/12/02/peering-disputes-
comcast-...](http://www.internap.com/2010/12/02/peering-disputes-comcast-
level-3-and-you/) but as Netflix increases bitrate and number of subscribers
it gets worse.

------
RKearney
An alarming number of people don't seem to understand the different between
throttling and congestion.

Comcast does not participate in Netflix's OpenConnect project. If Comcast
wanted to, they could receive a 4U server from Netflix (multiple ones, in
fact, for each densely populated area they service) completely free of cost.
Obvious costs for power and rack space apply, although the appliance itself is
free.

This would then allow your Netflix viewing to go to a 4U server at a Comcast
datacenter or colocation in the area instead of having the entire South East
US try to fit through a small pipe at Marietta, GA.

Comcast's stubbornness to participate in the OpenConnect project is the reason
for slowness, not throttling. Of course it's going to be faster over a VPN
because whatever endpoint you're connecting to most likely takes a different
AS path to Netflix.

~~~
Dylan16807
What do you mean 'of course it's going to be faster over a VPN'? The VPN is
taking a non-optimal route, it should in almost all cases be of similar or
slower speed. If the VPN path is fastest then comcast should be sending data
in that direction.

Call it congestion if you want but it's intentional congestion caused by
intentionally wrong routing.

~~~
RKearney
BGP path selection does not route based on link utilization.

What you define as a "non optimal route" isn't what BGP defines as a non
optimal route. An ISP can't just send all their customer traffic through any
foreign network they choose. The peering location happens to be the "best"
path available to Netflix and it won't change until Comcast gets off their
high horse and installs Netflix's free caching servers on their network.

~~~
Dylan16807
An ISP can't send data through any networks at all without some kind of
agreement. But that's what makes them an _Internet_ Service Provider, the fact
that they have those agreements. When they fail to even properly connect to
all Tier 1 networks, that starts to be a misnomer.

------
m_darkTemplar
I still see a problem with his testing in that he might be connecting to a
server in a different area which is less congested when he connects through
his school's VPN.

A proper conclusive test would ensure connection to the same Netflix IP
somehow over a VPN as his home connection.

~~~
jlgaddis
I live in the same town as the author. Comcast is not my ISP but is the ISP
for several of my friends, including friends whom I stayed with for about six
weeks recently when I was (for the most part) incapacitated. I can't be 100%
certain (I'm going from memory) but I'm almost positive that Netflix traffic
did come from the same facility that the author is hitting when connected to
the school's VPN. (If it wasn't 2:33 a.m. I'd go verify.)

As I mentioned in another comment, however, the .edu is almost certainly
better connected to this facility than Comcast is.

------
adventured
Being an ISP _is not_ a natural monopoly. This premise is part of the problem,
ISPs are being protected from competition when they should not be.

From the Wiki entry the author quotes:

"A natural monopoly is a monopoly in an industry in which it is most efficient
(involving the lowest long-run average cost) for production to be concentrated
in a single firm."

Competition in the ISP market drives costs lower as it forces the companies to
become more efficient, and increases the value proposition offered to
customers (greater speeds at an equal or lesser price, thanks Google Fiber!).
In fact, I think it would be easy to prove comprehensively that ISPs are in no
way natural monopolies.

If the author was right, and ISPs were a natural monopoly, then there would be
nothing to discuss, the only practical solution would be to install government
regulated monopolies in every location across the nation to achieve the
absolute lowest cost and consumer price. Google demolished this line of
thinking very handily, they showed how instantaneously the government
protected telecoms respond to competition in a positive way.

The solution is to make it illegal nationally, via Federal law, for any
municipality to ever create or encourage an ISP monopoly; and specifically the
exact opposite should occur, competition should be heavily encouraged
everywhere. The FCC should be mandated with keeping competition lanes wide
open in the ISP market, at all times, across the nation. Standards for how
cities and towns handle multiple ISPs for infrastructure should be
established. Simply: you shall create no law restricting ISP competition or
granting monopoly status, period.

~~~
kalleboo
The OP may be using it wrong, but I still think at this point, the last mile
network is a natural monopoly. Where I live, city-owned power companies have
built single fiber-to-the-building networks that are the open for ISP
competition inside the network (I.e. the ISP you pick just delivers the
bandwidth/peering). In my view this is the optimal compromise. This wasn't
true just a few years ago when DSL and Cable tech was still competitive, but
at this point it seems pretty clear that fiber infrastructure isn't going to
be beat.

------
kev009
A list of some things that can go wrong:

* You could have gotten a lousy route due to poor traffic management or a network issue by your ISP, the transit providers, or a CDN

* You could have been given an overloaded server or POP by a CDN's DNS

* Your ISP could have a netflix or third party caching appliance, and it could be overloaded

* Your ISP could be under-backhauled, under-peered, or under-transited in any points along your route

* Your ISP's transit provider(s) could be under-peered, under-backhauled, or under-transited if they are not a tier-1 network. See also poor traffic management and network issues.

* Any party along the way to the CDN and to the origin could be under-peered, under-backhauled, or under-transited. See also poor traffic management and network issues.

Using a VPN proves very little. Various BGP and DNS tricks are used in the
bullets above, and a VPN can drastically change the parameters of all of them.

~~~
tyw
sure, all of that is possible. but honestly how likely is it that some or all
of those things are the case on the direct path from netflix/youtube through
the ISP in question to so many people's homes? and yet the problem spot is
mysteriously avoided when using a VPN (not any particular VPN, but pretty much
any of them, judging by most of the people who have spoken out about this
problem recently). the problem so effectively avoided by changing the route
from the optimal(ish) path to one that is almost guaranteed to be worse from a
network flow standpoint.

sorry, I don't buy any apologizing for comcast/verizon failing to deliver HD
video from netflix/youtube yet somehow able to do it under the same network
conditions (same time) just this time through an encrypted VPN. if it was an
isolated report... maybe. this is far too widespread to be anything but
intentional.

~~~
kev009
Well, under-provisioned backhaul, peering, and/or transit would cause exactly
this. And the blame trail could be quite complex. I'm not apologizing, but
commentary by almost anyone other than the network architects and peering
coordinators of these organizations are pretty baseless speculations.

Disclaimer: I work for a major CDN. Netflix is trying to ween off of us and
our competitors, which has contributed to lesser experience. Their model of
co-locating gear for free doesn't make much sense to ISPs in the grand scheme
of things.

------
bifrost
So, let me tell you a story about Comcast and network engineering. Actually,
no, I'll skip that.

Basically - Comcast runs all of its ports hot. I would say its safe to say
what you're seeing is Comcast sucking at paying for upgrades rather than
anything specifically malicious.

------
jlgaddis
FWIW, the author's school has tens of gigabits per seconds (and it may even be
100 Gbps by now) to a facility in which Netflix has a presence. Comcast, AIUI,
doesn't.

------
jlgaddis
It's really simple, people. The links that Comcast is carrying Netflix traffic
over is getting saturated during peak hours. I am certainly not a fan of
Comcast but I don't think they are actively doing anything to screw with
Netflix traffic, they just aren't doing anything to fix it.

~~~
roye
This could be tested easily by pinging multiple destinations (besides Netflix)
on peak hours

~~~
jlgaddis
Only if you can ensure that those pings are traversing the same circuits that
the Netflix traffic is.

(And, even if you can, you can't be sure of the reverse path -- something so
many people don't understand when doing traceroutes.)

------
eurleif
Comcast is still supposed to be bound by net neutrality rules, unlike other
ISPs, because they agreed to follow them until 2018 as part of getting their
merger with NBC approved.

------
jijojv
While comcast sucks for its price and monopoly, i think netflix is at fault
here. I did a similar test a while back using my company cisco vpn which i
personally administer :) and found netflix sucked big time with or without vpn
around 9pm any day. Could never get even DVD quality on my 1080p despite being
in San Jose, CA with 25Mbps+ crapcast

~~~
bifrost
I wouldn't be so sure, I AnyConnect sucks and Comcast has been heavily
congested in SJC for months. I have some graphs I should show you...

------
eyeareque
The author points out that netflix cannot see what is going on over the
tunnel: Actually, if Comcast wanted to they could tell what you are doing by
using deep packet inspection. Certain firewall vendors have found ways to
figure out what you are doing over encrypted tunnels based on protocol
pattern/finger printing.

~~~
zwily
They could probably guess he was watching streaming video. It'd be tough for
deep packet inspection to tell that it's netflix though.

~~~
GhotiFish
how could they guess he's streaming a video and not downloading a large file?

~~~
adrianpike
Timing of packets going both ways. Netflix is a bad example since they use
HTTP to wing packets, but with a UDP connection you won't see the regular ACKs
heading back upstream.

~~~
jlgaddis
In this case, the VPN being used is a Juniper SSL VPN, which runs over
443/TCP. Comcast would see TCP ACKs being sent back to the VPN endpoint but
they obviously can't see the encrypted data.

------
jlgaddis
One other note: the author's school (the entity he brought up the VPN session
to) manages many of the largest R&D networks in the country. As such, it is
extremely well-connected to, well, just about everything (including Netflix's
delivery networks).

