

Netflix Performance on Top ISP Networks - carson
http://techblog.netflix.com/2011/01/netflix-performance-on-top-isp-networks.html

======
charlief
Rogers is the top ISP in the set, but according to some official statements,
their systems inadvertently began severe DPI shaping on encrypted and P2P
traffic since September 2010 around when Netflix was rolled out to Canada.
This issue still remains today.

I have a 50 megabit plan with Rogers and all my SSL traffic throttles the
entire line with an exponential dropoff to 25 Kilobytes a second. The packet
loss is severe and persists for all packets even HTTP. It takes a long time
sometimes an hour for the connection to recover. You can imagine how many
applications this would impact (syncs with large files, P2P, Dropbox make the
Internet virtually unusable).

I don't speculate on the matter, but here is some more information.

[http://www.christopher-parsons.com/blog/isps/rogers-
network-...](http://www.christopher-parsons.com/blog/isps/rogers-network-
failures-and-third-party-oversight/)

[http://torrentfreak.com/rogers-bittorrent-throttling-
experim...](http://torrentfreak.com/rogers-bittorrent-throttling-experiment-
goes-horribly-wrong-101213/)

------
bbatsell
This is a fantastic idea.

There are a few elements to it that one could use to dispute its value — for
example, Netflix has no way of knowing what level of service a customer has
paid for. They also don't appear to differentiate between different types of
service from a single ISP (e.g., they lump together AT&T's old ADSL customers
in with ADSL2/U-Verse, and, presumably Verizon's DSL with FiOS). I would hope
that they are also filtering for sessions where a higher bitrate is requested,
which Netflix's Silverlight app doesn't do unless the window/screen is large
enough to benefit from the extra resolution.

Of course, one could argue that those differentiators don't ultimately matter,
and that if its customers are opting for older or slower service levels, that
can tell you something about the ISP's pricing competitiveness.

~~~
TomOfTTB
I'm sorry but I don't see how this is a fantastic idea at all.

First, this is next to useless from a consumer perspective. Because anyone
whose actually tested a single provider can tell you speeds vary by geography.
Time Warner might be middle of the pack overall but the fastest in your
particular area.

Second, they're really p#ss#ng off the ISPs here. If you want faster bandwidth
out of the ISPs its better to privately talk to them than publicly embarrass
them.

Finally this reinforces the reputation Netflix is already getting among
content producers which is the company doesn't play well with others. There's
a reason why the head of HBO and Time Warner are taking hard lines against
Netflix.

Bottom line: Netflix is part of a content delivering ecosystem. If they offend
every other part of that ecosystem it will be much harder for them to survive
in the future

~~~
thetrumanshow
I respectfully disagree. This is a fantastic move by NetFlix.

NetFlix knows that the ISPs are already pissed at them and are moving to
throttle them, so they fire back by showing consumers which of the ISPs are a
better fit for their NetFlix viewing habits. NetFlix is flexing its muscle a
bit here to show it can redirect customers to choose different ISPs and affect
their bottom line.

~~~
draebek
> NetFlix is flexing its muscle a bit here to show it can redirect customers
> to choose different ISPs and affect their bottom line.

I, too, like that Netflix is giving out this kind of information, but let's
remember that in the USA many people have two or even just one choice. I have
my choice of Comcast cable or AT&T DSL for broadband. That's it. And I'm not
100% sure that AT&T DSL would actually work in my building, and it's top speed
is 6Mb/s compared to Comcast's 12Mb/s (or maybe 15Mb/s, I forget). Plus, AT&T:
the only company I can think of with worse customer service than Comcast.

So while I like that they're giving out this information I'm not sure it's
going to be a good idea to piss off ISPs with this near monopoly situation in
many parts of the USA.

~~~
thetrumanshow
A BIG fight is brewing, and NetFlix is summoning allies to its cause. If they
don't, the ISPs will eventually have their way anyway. So, while I agree, I
think NetFlix has almost no choice but to go down this path.

------
rdoherty
I'm glad someone is showing average bandwidth rates for this many ISPs; I'm
not sure I've ever seen any tech site test that many.

I gotta nitpick the graph though, it's incredibly hard to figure out which
line is which ISP (too many lines and similar colors). A regular bar chart
with the ISP names under each one would have worked fine (since the avg.
bandwidth didn't change much over the course of the study).

~~~
mthoms
This community could make some very interesting visualizations if Netflix
would simply provide the raw data used to make the chart. This would be even
better if the data were broken down by zip code. C'mon Netflix, let us have at
it!

------
wildmXranat
I am on their fastest tested network in Canada, Rogers. Just few months ago,
Rogers upped our prices and dropped our monthly quota to 60 gb/month. That
means I can use up my monthly quota by watching Netflix for about 3-4 days.

Yeah, we are getting raped either way. It's pretty much impossible to win this
argument as a consumer, even if Netflix wins their with the ISPs.

I used to be with Teksavvy where 200 gb/ month was the norm. I just might have
to convince my g/f and get her to switch.

Just to compare, Rogers is $59 for 60 gb/ month, Teksavvy is $29.99 for 200
gb.

~~~
kmfrk
Please don't use "raped" like that.

~~~
lincolnq
Offtopic, but do you find this especially offensive? (The GP's quote was
"Yeah, we are getting raped either way").

If "fucked" was substituted for "raped", would you still be offended? What
about "violated"? "shat on"? "murdered"?

I'm trying to determine whether it's the content of the metaphor, the
language, or something else which is offensive. (FWIW, my friends use "raped"
a lot, and I sense that some are offended by it, so I'm trying to distinguish
why.)

~~~
kmfrk
You* shouldn't trivialize a term and crime like "rape" - and doing it makes
you look like a clueless, insensitive asshole, because there might be a rape
victim who reads and hears you.

Feel free to say "fuck", "shit", etc. for all I care.

    
    
        *The grammatically general "you".

~~~
lincolnq
Thanks! I want to know what you think of "murder" though. Why isn't it the
same (or is it)? I feel like someone whose dad was murdered would be offended
by the same principle.

~~~
Natsu
> I feel like someone whose dad was murdered would be offended by the same
> principle.

Actually, my mom was murdered gruesomely. I have better things to do than
getting offended too much over people's word choices when I know that they're
just venting and not actually trying to insult me personally.

That said, there's plenty of awkwardness when people assume that a person your
age should have living parents and you always end up having to explain what
happened when you get to know new people.

------
thamer
In comparison, YouTube has a page with your connection and your ISP compared
to others: <http://www.youtube.com/my_speed>

~~~
Cyndre
Thanks for that. A nice little tool for isp admins to check how they are doing
in general.

------
sachinag
They really should have labeled the lines themselves on the right side instead
of just using the default legend in Excel. I can't really tell who the hell is
on the top since there are like three different greens (let alone the
colorblind).

------
trotsky
I think people are misunderstanding this report, which is surely their intent.
This isn't a chart of how well each ISP performs in general, or even how well
they perform in relation to each other.

As a baseline, all of the ISPs on this list that I've used (ftth, coax and dsl
providers) have been able to push to 90%-100% of quoted line speed 99% of the
time.

What this chart shows is netflix performance, which is quite different than
general network performance. What goes into netflix performance? Total netflix
service load, CDN pop load, CDN circuit size, SLA/QOS terms, the choices the
netflix client makes about adaptive bitrates and shaping the provider applies
to the traffic.

Take a look at the US chart and note the correlations in spikes and dips
across all providers at certain times - 10/21 peak, 10/23 valley, 11/13 valley
etc. These are indications of aggregate stream load netflix wide and are an
indication that netflix manages their peak throughput to be pretty close to
their max throughput (as well they should). But it's also a strong indication
that if 10% of netflix customers switched to a competitor overnight average
stream performance for netflix would jump noticeably, perhaps even 10%.

How their adaptive streaming works is another huge factor. First they measure
your current performance to the CDN POP and then explicitly chooses to use
only 60% of that[1]. So add 40% to all the ISP numbers out of the gate.
TCP+HTTP overhead is another ~4%. Downloading something else while you stream?
If the streaming sees more than a few headroom faults (ie 1000ms worth of data
takes more than 600ms) the system will quickly adapt down in bitrate trying to
do its best to never hard fault (buffering). It likely is considerable more
conservative in moving the stream speed back up. Customer using 802.11b? Max
throughput there is ~5mbps, so netflix will use a max of 3mbps. In an urban
area on a congested channel? You may only see 1-2mbps at times. Microwave on?
etc.

Lastly, with the news that netflix is 20% of ISP traffic at night the network
providers have every incentive to manage that traffic down. Look at the best
US performer at ~2700kbits. The fastest HD stream is 4800kbits - Nearly twice
the speed. Worst case if an ISP allowed full rate streaming they might
increase their peak network load by 10% or more.

Who was it that announced netflix is now 20% of night time traffic? Our old
friend sandvine, provider of the DPI gear made famous by comcast using it to
throttle bittorrent. This tells us two things - 1) sandvine is on net at at
least a few major broadband providers and 2) by announcing that number they
are implicitly telling network managers that they can help reduce that
traffic.

Netflix makes it very easy to do fine grained control on bitrates. If you are
watching the HTTP headers you know exactly which customers are on which stream
rate without having to count the throughput. Most users won't notice a QOE
change from 4800kbits to 3200kbits. Want to cut your network traffic? Use DPI
to identify the higher rates and label them bulk-toofast. Then either
explicitly rate limit them, give them a low queue priority or simply route the
acks through a big buffer under performing router or indirect route. Once the
adaptive bitrate slows down to a rate you like switch the label. The more
subtle of these methods are much, much harder to detect than the comcast RST's
were, and wouldn't technically violate any net neutrality rules in place.

[1]
[http://www.breakingpointsystems.com/community/blog/emulating...](http://www.breakingpointsystems.com/community/blog/emulating-
netflix-delivering-video-through-the-cloud/)

------
jerf
Ah, interesting thing I just barely noticed: Compare the vertical scales on
the US and Canadian graphs.

~~~
InnocentB
You have to bear in mind that the Canadian graph is measuring kilbots per
second, which is a far scarier metric :)

~~~
jerf
Wow, I was just talking about the numbers on the scale, I didn't even notice
that.

~~~
pmchiu
It's funny that on average Canadian broadband performs a lot better that most
American broadband providers.

------
losvedir
Wow, interesting, but a textbook example of how not to make a chart. I had to
break out the DigitalColor Meter utility in OS X to tell which was which.
Here's the list, fastest to slowest based on the leftmost point of each line:
1)Charter 2)Comcast 3)Cox 4)Time Warner 5)Suddenlink 6)Cablevision 7)Cable One
8)Verizon 9)ATT 10)Bellsouth 11)Windstream 12)Embarq 13)CenturyTel 14)Frontier
15)Clearwire

edit: Ah well, I guess that will do, can't make them stand up in a column.

------
danilocampos
For all the shit I give Comcast, I must take my hat off to them for being at
the top of the heap. And I'm not surprised – my Comcast service here in the
Bay Area has been awesome. I get every last bit of my 20 Mbit pipe, Watch
Instantly starts _almost instantly_ and I always get the HD version of my
programming when available.

I hasten to add that part of this must be a function of competition here. I
can choose from Comcast, DSL or AT&T's U-verse thing, so they have some
incentive to provide a compelling bandwidth offering. Still – at least where
data service is concerned, Comcast delivers.

~~~
noodle
comcast's end product is not bad at all, compared to what is available in
america.

their problem is customer service and their corporate policies. i avoid
calling them at all costs and only do so once i know for sure that my problem
is unfixable on my side.

~~~
Cymen
Did you know they have a direct support forum at BroadBandReports.com? I used
it a couple days ago and it is useful for smaller issues that you don't want
to call in. Must be a subscriber and only you and Comcast can read your posts
in the direct forum:

<https://secure.dslreports.com/forum/comcastdirect>

~~~
noodle
i knew about it, but every single issue i've ever had to call for has needed
to be resolved by a service order. its never something technical.

i've had my cable mistakenly disconnected a few times, or a line cut from
utility work.

i'd also note the irony of having online support for internet connectivity
issues.

------
ebbynezer
It's things like this that make me love this Netflix. By opening up and
showing their customers the courtesy of their inner workings, they invite them
to become more than customers.

By showing that they care ALOT about what their users care about, content
delivery performance, they demonstrate credibility, a key component of trust.

Consistent trust building exercises such as these does more than a temporary
sale ever could for long term customer loyalty, even if it can’t be perfectly
measured.

I would like to see more companies try the same, even if they can’t prove the
ROI on a spreadsheet before hand.

Why use a spreadsheet when you have a perfectly good gut you can check to see
if it is the right thing to do?

------
jjguy
Welcome to the opening salvos of the Age Of Preferred Internet. By publishing
the streaming performance metrics of consumer ISPs, Netflix is making sure
providers know they can influence the ISP choice of their customers. This as a
warning, pure and simple.

p.s., in case you missed it -- 21 Dec FCC ruling:
[http://www.thedailybeast.com/blogs-and-
stories/2010-12-21/ne...](http://www.thedailybeast.com/blogs-and-
stories/2010-12-21/net-neutrality-ruling-the-fcc-splits-the-internet-in-two/)

------
mmaunder
They should also filter for geographies that providers serve. e.g. If one
provider is serving predominantly rural areas, most of their trunks will be
using directional wireless, leaving them at a severe disadvantage which is no
fault of theirs.

If we were then to rank ISP's on throughput, even though a rural ISP is
serving a critical and under-served part of our community, they would be
rewarded with negative perception.

~~~
chc
A lot of people I've heard from in rural areas have much better Internet
connections than I can affordably get living in San Diego.

~~~
chadgeidel
I'll second this. When I lived in Mitchell, SD (15k people - look it up) I had
3 ISPs to choose from (in city limits). I had pretty good throughput from 2 of
them - one was wireless.

------
mmaunder
This definitely fits with my experience using Qwest DSL for my business in
Seattle and Comcast cable at home. Comcast has consistently high throughput,
low latency to my Seattle data center, fewer hops to the local DC, and for
some reason Comcast routing seems to be smarter about routing via the higher
quality of our 4 X 1 gigabit providers when I'm connecting to our local DC.

------
aidenn0
My ISP (cox) is near the top, and dealing with their customer service has been
a breath of fresh air compared with any other ISP I've dealt with.

On an absolute scale, it's so-so customer service, but on a relative scale
it's amaizing how many things they don't do wrong compared to other ISPs I've
worked with (Comcast in particular)

------
adestefan
There's a major flaw with this chart: companies that offer more than one
connection service have all their services lumped together. The reason
Comcast, Cox and Charter are at the top is because they're cable-only systems.
Most of the other companies on that chart offer a mix of DSL, wireless and
FTTH services.

------
tlrobinson
This is great. I'd also love to see some sort of "ISP score" based on how
"net-neutral" each one is.

------
bnycum
All the traffic seems to follow the same curve, with the same dips and spikes.
The Canada curve is different than the US curve, but there is still a curve
there too. I'm curious why that is.

------
btucker
I'm a very happy Charter customer in Massachusetts. I pay $30/mo for a 10Mbit
connection

------
jarek
A graph of throughput vs bandwidth usage limit per ISP, pretty please?

------
anonymous246
This is not a very useful graph to assign blame. :) I've always wondered why I
don't always get movies in streamed in HD even for movies available in HD.

Almost anytime of the day, I can saturate my DSL line's download bandwidth
from fast sites such as Amazon S3. But Netflix doesn't always stream in HD
even though my link is faster than what's required (4800 Kbps).

I suspect the bottleneck is at Netflix's end.

~~~
SoftwareMaven
Are you using a dns provider like opendns or google? I am and it drastically
affects streaming rates because you get sent to the wrong place in the cdn. I
changed my streaming box's dns server to my isp's and got much better service
from Netflix.

~~~
anonymous246
Thanks a lot. Yes, I use Google DNS. I will be sure to try this out.

------
HilbertSpace
My ISP says I get 15 Mbps download and 2 Mbps upload, and speed tests confirm
I do. For sustained flows, I get a solid 15 Mbps when downloading software
from Microsoft.

So the 15 Mbps is way above anything in the Netflix graph.

So, the ISP can't always be the bottleneck keeping data rates below 3 Mbps.

~~~
wmf
Netflix (apparently) can't measure anything above 4.8 Mbps, so 15 counts as
4.8 and the average is brought down by people who are on the el cheapo 1.5
Mbps plan.

