
Netflix responds to Verizon - chrisacky
http://www.scribd.com/doc/228871116/Response-to-Demand-Letter
======
dctoedt
Nice jab with (what seems to be) an allusion to the New Jersey bridge-closing
scandal that engulfed Gov. Chris Christie: _"... like blaming drivers on a
bridge for traffic jams when you're the one who decided to leave three lanes
closed during rush hour."_

~~~
awda
I'd only change it to "... blaming drivers on _your toll bridge_ when you're
the one ..."

The drivers (customers) are paying for the bridge!

~~~
roc
Sure, but they're only using the bridge to drive to McDonalds. It's McDonalds
generating all that traffic -- they should pay the bridge operator too!
/sarcasm.

~~~
rodedwards
They do. Property taxes.

~~~
wpietri
I'm not sure you mean to argue for a publicly funded Internet, but that's
basically what this implies.

~~~
tankenmate
You've already got that as the government already pays subsidies. I'm all for
basic infrastructure to be collectively owned; be it co-ops, collectives, not
for profits or even the government (local, national etc). This is the way it
is for roads, rubbish, sewerage, etc. If you want a premium / luxury service
then feel free to pay extra. Contract out support, maintenance, sales for the
infrastructure by all means, but the ownership should be a collective of the
users.

------
ColinDabritz
I hope Netflix makes this a standard feature for all ISPs. Bonus point for
more transparency, e.g. network graphs comparing peak to off-peak.

This sort of visibility into the real problems means the ISPs can't hide
behind their lies.

~~~
jobu
What is the messaging they're displaying to Verizon customers?

~~~
stonemetal
Verizon network is crowded right now. Adjusting video for smoother playback.

~~~
SoftwareMaven
Should be "degrading image for smoother playback". We know that's what's
happening; do most people?

~~~
stuaxo
Even if they don't, they will be annoyed enough at seeing the message that
says they have to wait.

------
Osiris
I believe the core problem here is that ISPs and infrastructure are owned by
the same companies. Based on evidence from other countries, like Japan, these
problems would largely go away if it was illegal to own both the
infrastructure and provide service to customers.

Is it possible that anti-trust laws could be used to force these large ISPs to
break up into separate companies, one that owns and provides the network and
the other that only leases the lines and provides service?

Such a breakup would allow new, smaller, ISPs to leverage the infrastructure
while provided superior service and potentially better pricing models.

Of course there would still be the problem of competing infrastructure
companies. Some of that could be alleviated by increasing competition through
public/private fiber installation projects.

~~~
seynb
>I believe the core problem here is that ISPs and infrastructure are owned by
the same companies.

Exactly. MSO (Cable) & Telco ISPs that offer video products are already
prioritizing their own VoD (Video-on-Demand) offerings over data to ensure QoS
(quality of service). Whether by over-provisioning the virtual circuits to
their customers, not amortizing the data consumed against a hard or soft cap
limit, or directly peering the VoD servers with the edge routers, ISP video
gets the priority because that is where the big profits are. There is no value
proposition for BIG ISP to provide free peering with their competition.

~~~
mindslight
It should also be noted, that this makes perfect sense with regards to the way
the telcos see the world - selling discrete well-provisioned services. This is
the classic datagrams versus circuits battle, still burning strong decades
letter. And of course that's because this is where the money is -
differentiated services lead to price discrimination and high profit margins,
while commodity bits are a highly competitive race to the bottom. And as
computing power keeps dropping in price, they both require ever-similar
infrastructure buildout (contrary to "End-to-End Arguments in System Design")

------
higherpurpose
I hope Netflix doesn't give in, and calls Verizon's bluff. It's _highly
unlikely_ Verizon will sue Netflix, and even if it does just because they are
so angry with Netflix over it, it will most likely come out in _favor of
Netflix_ , once the Court orders Verizon to show what's really happening
behind the scenes and who's fault really is.

In fact, from what I've noticed, Verizon has already lowered their tone about
this, and is backing away from threatening Netflix with the lawsuit. So carry
on!

These ISP's promised _good service_ to their _paying customers_ , regardless
of the conditions. It's their responsibility to live up to those expectations.

~~~
mcintyre1994
Someone on Reddit pointed out that it would be a huge PR win for Netflix if
Verizon did sue and they got the right documents in discovery. Depending on
the scope maybe they could possibly get things as wide ranging as actual
customer speeds as well as the peering points/capacity etc.

------
jusben1369
Written to Verizon but written for everyone but Verizon. This is a PR battle
started by Verizon but I think Netflix just went up 3 - 0.

~~~
Russell91
Netflix lands more squarely on the customer's side on this topic. They have
everything to gain from PR coverage and Verizon everything to lose. Seems like
a bad move to me that Verizon decided to boost the awareness of this issue in
the first place. Perhaps that wasn't the intention, but it was the effect.

------
taylorwc
Ugh. Good for Netflix. Any consumer who doesn't have a strong opinion in favor
of net neutrality should read this sort of thing. It's hard to think of a
relationship I detest more than my relationship with my ISP/MSO.

~~~
markbnj
Yes, indeed, good for them. The ISPs are counting on general ignorance and a
PR battle to win them the right to charge incoming tolls on their networks.
This has to be resisted. However, I read this morning that Netflix caved and
agreed to remove the messaging early. I hope it's not true, and that they keep
up the fight. Three Netflix accounts in our house.

------
gnu8
How about a link to the document instead of some fly-by-night web host with a
cheesy flash PDF viewer?

~~~
dpcx
Scribd isn't exactly a "fly-by-night web host". They've been around for almost
10 years already.

~~~
revelation
That is exactly the problem with Scribd? No growth no pivots, so all that is
left do to is plaster the thing with ads and have someone keep it running at
night.

It's terrible exactly because it is too old now.

~~~
cheeze
But it's a ycombinator company, so it doesn't matter. People will blindly
support it due to the nature.

~~~
i80and
Anecdotally, I've never heard from anybody who didn't resent Scribd. At best,
people use it because of entrenched habits.

~~~
duskwuff
The whole business of "you must buy a paid account to download this PDF" is
pretty galling.

------
dmethvin
They are saying "The Verizon network is crowded right now" but in the letter
they say that the problem is with "interconnection congestion". So yes the
problem lies with Verizon and its hesitancy to open up peering points, but if
I were a Netflix lawyer that wording would have me concerned since Verizon's
network is _not_ crowded. It's like having a crowd outside a bar waiting to
get in.

~~~
Angostura
The congested interconnection point is part of Verizon's network, surely.

~~~
jdmichal
Interconnection points are by definition part of two networks. The question is
whether the congested interconnect involves Verizon at all. I would hope that,
before writing this letter, Mr. Hyman would have done his research and
verified that it is indeed an interconnect that Verizon is involved with.
However, it is an unanswered question.

~~~
at-fates-hands
>>> Mr. Hyman would have done his research and verified that it is indeed an
interconnect that Verizon is involved with

Is it even possible for NetFlix to know this?

~~~
noselasd
Heck, even I write mails to my ISP and tell them which interconnect points
they're having issues at which a simple traceroute/mtr show me. And I mostly
get positive feedback.

With a bit more sophisticated tools, you'll get more confidence. Or if you're
Netflix, I'd assume you can just call up level3, ask them to resolve the
transit issues you're having with Verizon, and get feedback and reports about
it being verizon not upgrading their interconnect to level3. I'd not be
terribly surprised if verizon is one of the 6 peers mentioned at
[http://blog.level3.com/global-connectivity/observations-
inte...](http://blog.level3.com/global-connectivity/observations-internet-
middleman/)

~~~
zero_intp
Traceroutes from a single source are poor tools for positively determining a
source of latency. A traceroute only tells a unidirectional story, and an
incomplete one at that.

In reality in is essential to use multiple provider/observer based tools to
determine point(s) of congestion. However, it is entirely possible to
empirically determine who the congested peers are between, which, and when
specific geographical connections are overloaded.

------
chernevik
Kudos to Netflix, this is great strategy. But I wish the letter were a touch
better written. They might attach a written-for-laymen description of the
interconnection problem (so they can keep the response itself punchy).

As I understand the problem, the congestion builds at the interconnection
because Verizon's routers at that point at working at capacity. Verizon could
solve the problem by adding routers at that point. I honestly don't understand
what Verizon wants Netflix to do, unless it is to extend the Netflix data
provision deeper into the Verizon network -- essentially adding routers at the
point where Verizon is supposed to be maintaining routers. (I wonder if that
is what this Open Connect program is about?)

If that's right, I wish Netflix would provide some further detail explaining
the problem. Otherwise they risk having readers -- and politicians -- go into
glazed eyes and presume that this is just some inscrutable battle among
corporate giants.

~~~
wmf
Netflix paid Verizon to upgrade their network, but Verizon hasn't finished the
upgrades because it takes them months to order and install routers. Netflix
knows that these upgrades are coming but is still publicly shaming Verizon,
perhaps trying to encourage them to work faster. Presumably Verizon would
prefer Netflix to be patient.

~~~
wmf
Some citations for those of you who are confused:

[http://arstechnica.com/tech-policy/2014/04/netflix-and-
veriz...](http://arstechnica.com/tech-policy/2014/04/netflix-and-verizon-
reach-interconnection-deal-to-speed-up-video/)

[http://online.wsj.com/articles/netflix-brushes-off-legal-
thr...](http://online.wsj.com/articles/netflix-brushes-off-legal-threat-from-
verizon-1402004231) (may be paywalled)

[http://www.cnet.com/news/verizon-vs-netflix-whats-this-
reall...](http://www.cnet.com/news/verizon-vs-netflix-whats-this-really-
about/)

------
crazy1van
It is so aggravating to have Verizon pitch an upgrade to FiOS Quantum(!) every
time I turn my DVR on. If my 25mbps connection can't stream a 5mbps video why
would I pay extra for 50mbps?

No thank you, Verizon.

------
uptownhr
I'm thinking from a Consumer's stand point right now. No matter how I see it,
it looks like ISPs are at fault in this matter but I've been questioning what
the outcome will be for the Consumers in the end.

If Netflix doesn't pay, wouldn't Verizon and other service providers transfer
the cost to Consumers? Or will this also be not allowed for ISP to do? If ISPs
want to make more money and charge more, why would they not transfer the cost
to Consumers if they cannot charge Netflix and the likes.

~~~
delinka
You make it sound like we, the customers, aren't already paying; like upgrades
to hardware would never be reflected in the prices we pay. It really doesn't
matter which company actually writes the check, their costs are reflected in
my bill. As well as their profits, executive bonuses...

In this case, what matters is that one company is already taking my money for
a particular service, half-assing the service itself, taking those profits,
and trying to blame someone else.

~~~
uptownhr
That's actually my point. Since upgrades, etc... all are reflected in our
bill. When ISPs cannot charge Netflix and the likes in the long run, wouldn't
it be reflected on the Consumer's bill? Or will the ISP business become less
profitable than expected (from the eyes of ISPs)?

I see that ISPs (companies) generally want to make more money. Attempting to
charge Netflix is an attempt at that. If they are not able to charge Netflix
this extra amount, wouldn't it be logical for them to charge consumers more?

As it appears in Netflix's statement, their network is congested. If ISPs want
to keep their profit ratio and offer better services to their consumers.
Somebody has to pay more. I doubt they would settle for anything less.

Again this is just me. I would be interested to hear if there would be
anything preventing ISPs from charging their customers more...

~~~
aredington
Verizon has so much bandwidth to get in and out of their network for broader
internet bound and broader internet originating traffic. Netflix has to come
in the off ramp that all the torrents, hacker news, and etc. traffic has to
come in.

Netflix has offered to build a new offramp for Netflix traffic that goes
directly into Verizon's network. Netflix is willing to shoulder ALL of the
burden of this infrastructure cost, and to maintain it.

Verizon has refused the offer, and instead wants to charge Netflix a toll so
that their traffic gets to use an EZ-Pass lane, instead of going through the
normal slow toll booths that everyone else's traffic goes through.

------
gdulli
Am I the only one who doesn't use streaming video and doesn't want my ISP
rates to go up because a vast majority of the traffic they now have to deal
with is video and their architecture needs to be built out in a way that it
otherwise wouldn't?

~~~
barrkel
You're absolutely correct, IMO.

This argument is over whether or not non-Netflix users will end up subsidizing
Netflix watchers.

The bandwidth needs to be paid for, at the end of the day. The only thing
anybody is arguing about is how the balance of payments work out.

~~~
gdulli
I don't know the right answer, but I think the issue is more complex than the
typical Netflix-good/ISP-bad dogma which is usually all I see in response to
it and seems to be driven more by consumer sentiment for the respective
companies than anything else.

------
nhangen
Is that real? It doesn't read like it was written by an attorney, and there
was even a grammar mistake in the first paragraph.

~~~
danielweber
Real attorneys make grammar mistakes all the time.

~~~
nhangen
You're telling me that no one proofreads documents like these?

------
panabee
hopefully netflix continues this push toward broadband transparency. as
justice louis brandeis said, "sunlight is said to be the best of
disinfectants." it seems unreasonable for verizon and broadband providers to
advertise, and charge consumers for, high-quality networks -- then try to pry
money from data suppliers like netflix when usage rises. that said, is there a
fair counterargument that justifies verizon's desire to charge both consumers
for high-speed access and netflix for increased network usage?

~~~
rqebmm
> is there a fair counterargument that justifies verizon's desire to charge
> both consumers for high-speed access and netflix for increased network
> usage?

The closest approximation is "as a publicly traded company they owe it to
their shareholders to make as much money as possible"

------
cipherzero
What's the Open Connect program they mention? how does it work?

~~~
hackerguy0217
ISPs can directly connect their networks to Open Connect for free. ISPs can do
this either by free peering with us at common Internet exchanges, or can save
even more transit costs by putting our free storage appliances in or near
their network.

Major ISPs around the world have already connected to Open Connect, including
Frontier, British Telecom, TDC, Clearwire, GVT, Telus, Bell Canada, Virgin,
Cablevision, Google Fiber, Telmex, and more

[https://www.netflix.com/openconnect](https://www.netflix.com/openconnect)

~~~
Touche
Can you explain why ISPs should pay to run your servers in their network and
not do the same for my personal homepage?

~~~
nathos
Because your homepage doesn't attract many TB of traffic?

~~~
Touche
So, bandwidth. What is the threshold with which ISPs should start giving out
free hosting?

~~~
msandford
Econ 101: all things being equal cheaper is better than more expensive.

If I have a network and my customers use Netflix and aren't going to not use
Netflix because I don't want them to, then part of my job is to ensure that my
customers get the bandwidth I've promised them. (This isn't how it ACTUALLY
works but how it's SUPPOSED to work)

As an ISP I have really rather large amounts of bandwidth in the last-mile at
least in aggregate. Let's say that I can reasonably offer 20Mbps to each of my
1mm customers from my POPs to their houses. That's 20Tbps in aggregate. I
probably don't have 20Tbps worth of back-haul from all my POPs to all the
peering stations where I actually get the customers connected to the internet
at large.

If network traffic is all long-tailed and the biggest use of bandwidth is 1%
of capacity and it goes down from there "free hosting" doesn't make sense. But
what if traffic to one company makes up 30% (or 80%) of total back-haul
utilization at peak hours? I'm spending a lot of capacity for a single
destination.

Now what if that place offered to create a magical wormhole from their servers
to my customers at my POPs such that a large fraction -- say 80% -- of my
customer's traffic from/to them never hits my back-haul it just appears out of
thin air at the POP. Would I consider this a good deal? Depends on how much I
pay for the back-haul versus how much electricity they're going to use at the
POP. All-in I would suspect that it is a good deal thinking in these terms.

That's precisely what the Netflix appliance is. It's a way to give the
customers Netflix without costing any bandwidth on the back-haul network that
ISPs operate.

The reason that ISPs aren't all jumping right on this (despite the likely
cost-savings) is that they view Netflix as the competition and they're prefer
an adversarial relationship that hopefully puts Netflix out of business rather
than cooperating and in their minds speeding their own demise.

So the rule of thumb you're looking for is that ISPs should start giving out
free hosting when it's cheaper to give free hosting than to pay for the back-
haul.

~~~
devrelm
> rather than cooperating and in their minds speeding their own demise

It makes sense, then, that one of the few large cable companies that uses Open
Connect is Cablevision, who's CEO is on the record saying "Ultimately over the
long term I think that the whole video product is eventually going to go to
the Internet."[1]

He's one of a few that has accepted the eventual fate of cable TV, and so his
business decisions aren't biased by a need to delay the inevitable.

[1]:
[http://online.wsj.com/news/articles/SB1000142412788732342060...](http://online.wsj.com/news/articles/SB10001424127887323420604578647961424594702)

~~~
hga
That's a nice theory, but the impression I get from the (extra biased against
last mile ISPs) DSLReports is that Comcast is continually spreading usage caps
and plans to make them nationwide in 5 years, e.g.
[http://www.dslreports.com/shownews/128987](http://www.dslreports.com/shownews/128987)

Lots more with this search:
[http://www.dslreports.com/nsearch?cat=news&q=comcast%20caps](http://www.dslreports.com/nsearch?cat=news&q=comcast%20caps)

~~~
Xylakant
There is no inherent problem with usage capped plans, as long as the price is
fair. If you'd like an uncapped plan, pay more.

The problem is that the last-mile providers try to extract money from netflix
for a service that the consumer already paid for (deliver those video bytes)

~~~
hga
You don't see a problem with e.g. Comcast implicitly charging the consumer
extra for using Netflix (by exceeding the cap and paying by the drink
afterwords) and not charging anything "extra" for using Comcast's video
services?

There is an argument there, in that it costs more real money to deliver video
bits the a la carte Netflix way than the cableco broadcast way, but I believe
the conflict of interest remains an issue, and the prices I see for exceeding
these rather small caps don't strike me as fair.

(Albeit AT&T's, the only choice I have aside from a not so reliable WISP, are
particularly ridiculous: 150 GiB/month including I'm not sure what overhead
for a continually rising price 2nd from the bottom "up to" 1.5 Mbs down/300+
Kbs up line that currently costs $36/month, each additional 50 GiB costs $10.
We'd get a faster line so my father could watch video, at $5/month extra each
increment, if the cap wasn't so low and the overages so high.).

~~~
Xylakant
That's not a capped plan. A capped plan charges you more if you exceed a
specific bandwidth threshold - no matter what you spend the bits on. As long
as Netflix, Comcast PPV, Youtube and my private cat videos get charged the
same price - I'm fine with that.

The issue you're pointing out is the lack of net neutrality - but that is
orthogonal to the pricing.

------
tieTYT
I don't know how Netflix figures out that it's Verizon that's slow, but lets
say it's with some software that anyone could use. Wouldn't it benefit the
entire Internet to open source that software so that anyone could use it? Or,
barring that, provide an API that anyone can query.

This way, any company can use the software and report the same thing to their
customers. Lower the barrier to entry for Amazon/Google/Hulu/your startup to
join the fight and inform the people.

~~~
warfangle
mtr

probably over time, comparing transit during and not during peak viewing times
for a given isp

------
bryondowd
Maybe I'm nitpicking, but in their bridge analogy, wouldn't the drivers be
more analogous to the users than to Netflix? If anything, Netflix would be the
city full of employment opportunities that drivers are commuting to and from
across the bridge. So a more fitting comparison would be 'like blaming the
city at the other end of the bridge for traffic jams..'

Either way, brilliant response, and hopefully Verizon gets the hint.

~~~
rmrfrmrf
I thought it was a reference to the Gov. Chris Christie bridge scandal,
alluding to Verizon's corrupt nature.

~~~
bryondowd
Most definitely. But Verizon hasn't quite gone so far as blaming it's own
customers for using Netflix, just blaming Netflix for attracting so much
traffic.

~~~
rmrfrmrf
Maybe you're just trying to read too much into the analogy. I think Netflix is
basically saying that the infrastructure is there (the bridge), but despite
the fact that usage has increased (rush hour), Verizon is artificially capping
speed (keeping lanes closed) to its users (the cars) by demanding that Netflix
pay extra for Verizon to open more lanes on the bridge.

~~~
bryondowd
Yes, I certainly read too much into it. Just pointing out a flaw in the
analogy. Verizon demanding that Netflix pay extra for users to access its
content would be more like the Christie demanding the city of New York to pay
for more lanes to be opened on the bridge. While simultaneously charging the
drivers (the users) for access to it.

~~~
jdmichal
Analogies are always flawed. Must we really continue to dive deep into every
one and point out all the flaws, or can we start to just read them at the face
value they were written at and move on?

------
thatthatis
They might as well have addressed that letter to hacker news with a cc to
verizon. Well executed PR move.

------
oldmanjay
What I find most interesting about these topics is how many people come out to
contribute to them. Nothing gets people riled up like vaguely threatening the
free flow of entertainment.

I wish I know how to direct this energy into something a little less
disappointing.

------
asc123
Netflix should just stop serving to Verizon and Comcast. the outlash would
force the FCC and companies to recognize.

~~~
Moral_
That would be a very bad idea for Netflix. They would be breaching contract on
thousands and thousands of customers and open them self to probably a very
large class action lawsuit. Although I could be wrong IANAL.

~~~
hga
While I agree this is a bad idea---the whole incident shows they have a direct
link to the hearts and minds of their subscribers using these ISPs _and are
not afraid of using it_ (the biggest development in the net neutrality fight
since AT&T's Ed Whitacre opened his foolish mouth and started it 8 years
ago)----they could punt subscribers on these ISPs by simply not renewing
subscriptions as they finish.

I'm sure they're not representing they'll be providing their service forever,
and e.g. issues with content providers means they can't even promise they'll
be supplying access to any particular stuff beyond what they outright buy or
produce themselves.

------
autokad
doesnt this reek of hoax to anyone? Not just the 'response' letter, but the
initial twitter 'screen shot' as well. Why in the world would an executive
scan a letter like that opening them up to litigation? even if they were
'right', they would probably get fired for it.

~~~
lstamour
I don't think they scanned it. The scribd account, at least, shows other
similar documents uploaded by the same user, none of which were original up to
now. I've no reason to doubt its authenticity. I bet it just spread outside
the company by people who are interested in net neutrality.

------
gettingreal
Is this just a USA problem?

Why for example has SKY or TalkTalk in the UK not tried this? What is stopping
them?

~~~
jamesbrownuhh
Sky and TalkTalk both rather like having customers, and would be unhappy if
any significant number of customers perceived Sky or Talktalk's service as
being poor, and chose to move to a different ISP because of it.

UK ISPs do/have used streaming video and buffering (or the absence of it) as a
battleground of quality - you see companies making claims of fastest or "most
reliable" broadband, and it gets them customers as a result.

Mucking up people's ability to access Netflix would bring no benefit - people
would just choose a different ISP.

At least in the UK, where people have that choice. This situation may not
apply in other countries.

------
knodi
Google needs to bring its ISP services to all US markets faster. Kill these
shitty/pos ISPs. I'm really hoping Google is willing to throw enough capital
at this problem and change some of the laws to open up the market to all new
comers.

~~~
callahad
I'm terrified of Google (or any single player) owning the entire stack from
infrastructure to devices to services. Here's to hoping that Google Fiber (and
municipal fiber) spur competition, rather than a new monopoly.

------
dokem
I'm all for what Netflix is doing, but this read like a cheesy publicity
stunt. The whole thing seems like it was written to be read by the public, not
Verizon's lawyers. I mean, it contained a simile/metaphor.

~~~
yazaddaruvala
Ideally everything legal entities write, that can be written for the public,
should be written for the public. And when I say "for the public" I mean in
terms the public can understand. So yes, including simile/metaphor.

------
mantrax5
It's becoming a habit that every week either telco responds to Netflix, or
Netflix responds to telco.

I'm slightly sick of these silly public letters. Why the hell either Netflix
or said telcos feel like they need to bother the public with their internal
problems, instead of solving them privately like adults?

This is not about net neutrality. It's about who foots the bill for upgrading
the network to handle the capacity that Netflix users need.

Meter the damn traffic and let Netflix (and other heavy) users pay the telco
and be done with it. God damn.

------
ajainy
Is it only me, who thinks, this letter might be FAKE. Though it's
professionally written but it doesn't look like real one. For example: three
DOTs right after word doorstep. I tend to use multiple dots in my personal
emails but even in office, talking to clients, using multiple dots, is not
professional. How we can confirm, this is REAL letter not just nice job done
faking it.

~~~
alphakappa
The three dots is the equivalent of an ellipsis. There's no reason for that to
be unprofessional.
[http://en.wikipedia.org/wiki/Ellipsis](http://en.wikipedia.org/wiki/Ellipsis)

Now, random capitalization...

