
Verizon's accidental mea culpa - shimshim
http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa
======
bitJericho
"If that’s the case, we’ll buy one for them. Maybe they can’t afford the small
piece of cable between our two ports. If that’s the case, we’ll provide it.
Heck, we’ll even install it."

That's what I would call a mortal wound to Verizon's arguments.

~~~
sheetjs
Clever technique to draw attention away from what is most likely a contractual
dispute.

Verizon's argument is clear:

> Netflix chose to attempt to deliver that traffic to Verizon through a few
> third-party transit providers with limited capacity over connections
> _specifically to be used only for balanced traffic flows_

Level3 hasn't refuted this point (the imbalance of the traffic flow). If the
traffic is not balanced, the contract between Verizon and level3 probably has
clauses requiring Level3 to pay Verizon for the imbalance (in this case, since
Level3 is pushing more data, they would be the ones paying). That is the real
problem, most likely, and the technical argument merely serves to confuse

~~~
msandford
> in this case, since Level3 is pushing more data, they would be the ones
> paying

No. That's not how the internet works. Nobody PUSHES data. People PULL data.

Verizon has customers who pay for internet access. Those customers make
REQUESTS to Netflix for data and Netflix RESPONDS with data.

They happen to be streaming movies which use a lot of bandwidth, but on
average it's about 3Mbps per concurrent stream. That is WELL below the 25/3 or
50/5 or 100/10 that Verizon advertises for purchase.

If someone were pushing data it would be called a DoS or DDoS. An attack is
when someone sends unrequested data to try and break your network.

But this data isn't unrequested. Verizon's customers have requested it from
Netflix as they are within their rights to do since they have literally paid
for it.

This is Verizon wanting to bill their customers once and then their customers'
vendors as well. Double billing for a single service is a neat trick if you
can pull it off. But it tends not to engender goodwill.

~~~
sheetjs
You still haven't refuted the point. This isn't about the relationship between
Verizon and its customers, at all.

The agreement between _Level3_ and Verizon probably has some balanced traffic
requirement, with penalties for asymmetry. If that is the case, then the
nature of the imbalance would force Level3 to make payments to Verizon.

Verizon gave a very clear suggestion for Netflix: work with other providers as
well. If Netflix worked with the overwhelming majority of third party
providers, then you could argue that Verizon is not playing fair. But I don't
see any commentary that Netflix is actually working with most of the third
party providers.

~~~
Alupis
Verizon is charging me for access to the internet -- then charging the
internet for access to me. That is double-billing, and it's egregious.

Verizon is selling me a 50Mbps connection, but if I use it, for even a little,
they get upset and say they are over capacity. This is the same company that
over-subscribes their network _on purpose_ , as most ISP's do. They will come
into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes
that not everyone uses the connection at the same time, nor for long duration.

~~~
danielweber
_Verizon is charging me for access to the internet -- then charging the
internet for access to me. That is double-billing, and it 's egregious._

Both me and my brother get billed when we call each other on our cellular
phones! Help me!

~~~
staunch
Which is not how it works in the rest of the world. Only U.S. consumers are so
used to getting screwed that they would defend it as normal. Verizon loves
double-billing so much for cell service that they want to do the same with the
internet.

~~~
viraptor
There are other countries that bill for incoming calls. I know of at least
Singapore that does it.

Also, your phone provider does get charged for each incoming call. They just
eat/redistribute that cost, since for normal phone user it's likely to be less
than a penny per month. But try running a conference service and you're
definitely going to get billed for incoming calls (apart from some weird
force-subsidised areas where you can get paid to receive calls)

~~~
cflee
The difference is that in Singapore, you can get charged for incoming calls
depending on your contract, but you _don't_ get charged for incoming SMS.

From a Singaporean perspective, that's the most bizarre part of mobile service
in the US. I need to get an unlimited texting plan so that I don't get billed
for people sending me texts? Whoa!

------
revelation
Theres a common theme in this discussion. We're stuck discussing analogies.
Instead, we should be reasoning from "first principles" as Musk suggests.

If we do that, we find that physical link speeds of any variety have improved
by many many magnitudes over. As this statement suggests, 10GBit is absolute
jellybean in the networking world. Yet, last mile bandwidth available to
customers has stagnated. Imagine a perfectly paved autobahn, and here we are
discussing if we should be able to drive 20 mph on it, and who should pay for
the extra damage that will cause to the surface. Incredulous.

There are some regulatory systems, where in a goldilockes moment, these basic
truths are recognized and acted upon. Consider the EU drastically cutting
roaming charges. What was so blatantly obvious that even the senile, crooked
decision makers at the EU couldn't miss it? All the providers already roam
over the internet, at basically zero cost for both parties if you consider
that speeds are limited by the over-the-air interface at either end. Yet they
kept charging customers the price for a specially erected microwave link from
1960.

~~~
bitJericho
I'm pretty sure that was an analogy! (A good one though)

------
amalag
I have to agree with this assessment. If Verizon is claiming they have green
in their network with 34% utilization to Level 3, then the connection between
Level 3 and Verizon is lacking.

Is it a simple fix like Level 3 & Netflix claim? If so, what is Verizon's
advantage to continue the slow Netflix. Netflix is really slow for Verizon
users, it is very obvious for all their subscribers.

~~~
mikeash
Verizon's advantage should be obvious: they also sell TV service which is in
direct competition to Netflix. If Netflix works poorly, that's more customers
potentially signing up for FiOS TV.

~~~
lbotos
I feel like this is what people often forget in the net neutrality debate. The
pipe providers have a horse in the race (FiOS TV On Demand, XFinity On Demand)
and we need to make sure they aren't rigging the race so to speak.

~~~
mikeash
Going beyond that, they typically have a monopoly or duopoly as an ISP, while
they're subject to competition as a TV provider.

Much of net neutrality can be reduced to "don't let monopoly ISPs leverage
that into a monopoly in other services".

If Verizon didn't offer TV, this wouldn't really be a problem. If there were
17 other broadband ISPs to choose from, this wouldn't really be a problem. But
Verizon does offer TV and they're one of two choices for me and most of their
other customers, so it's a problem.

------
bcRIPster
Let's not forget that those providers that Netflix chooses to use are global
edge service providers for high bandwidth streaming content services. Cogent,
Level 3, Limelight, etc... Their whole business is providing high capacity
back-end network transit which they do well, otherwise they wouldn't be in
business for very long. High-five Level 3 for calling B.S. on this!

~~~
MichaelGG
Funny you'd mention Cogent and L3 together, seeing as L3 has done worse to
Cogent (depeering) for the exact same reason Verizon is using (unbalanced
traffic).

~~~
bcRIPster
Oh I didn't say they weren't bastards. That's a whole other discussion.

------
wil421
It doesnt matter what content is coming down the wire, if I pay for internet I
expect to be able to access or connect to anything thats not forbidden in the
EULA.

With comcast my data cap is 300GB a month, I should be able to use 300GB of
whatever I want Youtube, Netflix, hell even torrents or file sharing. Doesnt
matter where it comes from I'm paying for 300GB of data at X speed. How hard
is that to get?

~~~
MichaelGG
Cool! Now tell Comcast you want to download 300GB from my server, which I've
got connected on a single 300 baud channel. Will they come give me a free 1GE
connection?

~~~
wil421
It doesnt work like that because Netflix has already paid level 3 who does
most of the heavy lifting. Similar to if you had a 10GE connection at your
sever but comcast is still running a 300 baud channel on the connection they
have to your server.

~~~
MichaelGG
OK, so I'll buy a 10G port to another random ISP. And they'll go to the ICX
and say "hey Verizon, hit me up with the 10G port". Oh wait, it doesn't work
like that. It works with peering arrangements, hmm.

~~~
wil421
Why should netflix have to pay for last mile when they already pay their
backbone provider, who may own most of the wire? We can come up with any
scenario or one off situation but still the problem remains.

Should net neutrality go away? And is it fair what Verizon and others are
demanding from Netflix/Youtube... Also, who should foot the bill, the backbone
providers, the content providers or the last mile providers.

You talk of peering arguments yet you dont seem to back them. If netflix is a
level 3 customer and level 3 has a peering argument with Verizon, then whats
the problem?

~~~
MichaelGG
L3's peering contract with Verizon requires roughly equal traffic. Maybe
that's dumb, shouldn't be the case, unfair to Verizon customers, etc., but
that's literally the issue at the moment.

L3 did the exact same thing to Cogent before, for the same reasons.

------
troels
One question: There aren't that many large edge providers around. What would
Verizon do if Netflix signed a contract with a dozen of the larger ones and
routed their streams through whichever had the best connectivity for a given
endpoint? If Verizon throttle all of them, they are going to get a lot of
collateral - essentially degrading the Internet at large for their customers.
Or would they just do packet level filtering then?

~~~
josho
I've wondered this as well. I suspect that it would be cost prohibitive for
Netflix to have a contract with several edge providers like Level 3. I'm
certain that Verizon's price demands to Netflix would be cheaper than this
option.

Think about your own ISP. I pay $80 for service, but to get the same bandwidth
shared between 2 ISPs I'd have to pay $120 plus now I'm doubling my hardware
costs (routers to each ISP).

~~~
troels
True. But I would assume that the contract is largely a pay per usage or at
least expected usage. So if they just make an estimate of how many Verizon
customers they have and buy that block from someone else than Level 3. Then
route all that traffic through the other provider, while keep using Level 3
for the majority of their traffic.

Or perhaps they could ask Level 3 (who are clearly pissed at Verizon) to make
the deal for them. These edge providers already have business with each other
(peering), so it should be relatively simple for Level 3 to purchase some
bandwidth from another large provider and then route some of their packets
(e.g. the Netflix ones) through there.

In any case, the edge providers have good reason to collaborate on thwarting
Verizon on this, so why don't they?

------
mikecb
Is it just me or is this not at all inconsistent with Verizon's arguments?
They've been saying the whole time that this link is meant to be balanced with
roughly equal upstream and downstream, and they showed it was not. What am I
missing?

~~~
msandford
You've missed the part where they're trying to double-bill for the same
network traffic.

Verizon wants to bill it's customers for the bandwidth that they use. And then
for some of that traffic, if it's from Netflix, they also want to bill
Netflix.

Verizon is the company which is selling asymmetric internet plans so it should
come as no surprise to Verizon that their customers are requesting more data
than they're sending.

But they use this asymmetry to suggest that what's happening between Verizon
and Level3 isn't business as usual (which it is) and that they should be able
to double bill.

Verizon is doing a decent job of spinning but if you really look at what's
going on it becomes clear that they're really torturing definitions to make
their arguments.

~~~
mikecb
I agree to an extent. But there's a point at which Level3 stops becoming a
peer and starts becoming a subscriber. That point is somewhere between equal
utilization of Verizon's peering point and (near) total upstream into
Verizon's network, just as any Verizon subscriber would have. Where is that
point?

I'm a Verizon subscriber and I get apoplectic when Netflix stutters, because I
know exactly what's happening. But I also don't know what's the exact right
model for peering points, and I think net settlement might be a better model.

~~~
eridius
Netflix isn't a Verizon subscriber. Netflix is what Verizon subscribers pay to
access.

The problem is that Verizon has apparently been successful in spinning the
story that peering between Verizon and Level3 is supposed to be balanced. As
others have already said in these comments, that doesn't make a lick of sense.
Peering between backbone providers should be roughly balanced, but Verizon
isn't acting as a backbone provider, they're acting as a residential provider,
and there's no way peering with Verizon will _ever_ be balanced. That's a
fundamental consequence of the asymmetric plans Verizon sells and the fact
that their customers are expected to download significantly more than they
upload. This means that Verizon will absolutely be receiving a lot more
traffic into their network than they send out.

Basically, I doubt Verizon actually has balanced traffic with _any_ other
provider, except perhaps other residential providers. But they're only making
a stink about Level3 because they're trying to use this claim to double-bill.

Note again, Verizon made the intentional choice to service the residential
market, and to sell asymmetric plans. They knew going into all of this that
they wouldn't have balanced traffic with providers like Level3. And that's
perfectly fine, because the only reason they're receiving this much traffic is
because their paying customers are _requesting_ it. Verizon has already been
paid to receive this incoming traffic.

Also, and this is something I haven't seen anyone really address, balanced
peering agreements between backbone providers is typically meant to prevent
one provider from routing traffic through a second provider's network,
destined for a _third_ last-mile provider. In that scenario, the second
provider doesn't gain anything from the traffic, and hasn't been paid for it.
That's why the balanced peering agreement exists, to ensure that no provider
gets taken advantage of that way; if you one provider wants to route traffic
through another provider's network, they have to be prepared to receive just
as much traffic.

But that argument doesn't apply to last-mile providers. The traffic isn't
being routed through their network to a third destination. The traffic is
being delivered to the network because that's it's destination. If the traffic
was unsolicited (for example, a DoS), then it's reasonable for the last-mile
provider to try to charge the sender for it. But if the traffic was explicitly
_requested_ , which is generally the case (and certainly is for Netflix), then
the last-mile provider has no justification for charging the traffic sender.
The traffic has already been paid for, by the subscriber who requested the
traffic.

~~~
mikecb
You make a much better point than I've seen here. Verizon acting both as a
backbone and as an ISP. This creates an internal conflict between those
businesses.

------
exabrial
I think a natural solution to this is to have Netflix charge all Verizon
customers a $5/mo surcharge to pay for 4 more 10gbps ports.

But after reading this... am I the only one that thinks peering agreements are
done bass ackwards? If Verizon's customers are _requesting_ Netflix access,
shouldn't Verizon pay Level3, not the other way around?

------
staunch
This is the entire net neutrality debate in one simple blog post. Verizon is
selling internet access to their users but not actually providing real access!

------
BryantD
Point of information about Verizon as a purely residential carrier: Verizon
owns MCI/Worldcom and Terremark. They're not just a residential carrier. I
don't have any idea what the ratio of residential to business traffic is, of
course, but I think Verizon is a real Tier 1 carrier. Happy to have someone
with more definite data chime in if I'm wrong.

~~~
msandford
They are but the residential and commercial sides of the business are VERY
different:
[https://news.ycombinator.com/item?id=8050089](https://news.ycombinator.com/item?id=8050089)

~~~
BryantD
Do they use different backbones? Which side of the business runs the
interconnects?

~~~
msandford
Verizon might technically be one company but in reality there are several
business units which are actually separate companies in all but name.

When you buy business backhaul from Verizon you get guaranteed bandwidth
provisioned on their network from point A to B. They don't over provision
because they can't: contractually. If they don't meet their SLA the customer
is out of the contract and can go elsewhere.

On the residential side things are very different. We have no SLA other than
"pfft whatever we'll roll a truck next Thursday I guess"

Verizon might share long distance backhaul between residential and commercial
sides of the company but if so it's done with strict quotas to ensure that the
commercial side of things isn't impacted. It's not as though the core network
division lets everyone have a free-for-all on the available bandwidth through
the core.

If I had to guess I would venture that the consumer group either runs largely
or entirely separate equipment. They might share fibers but I'm not all that
confident about that. They're very, very close to entirely separate
businesses.

------
drderidder
How did people think this wouldn't happen when ISP's are allowed to be content
providers, too. Of course Verizon wants to choke Netflix and promote their own
service instead. Letting cable and content providers be ISPs was always a bad
idea.

------
shock
Seems to have been taken down. Here's the link to the cached version (a bit
hard to read):
[https://webcache.googleusercontent.com/search?q=cache:http:/...](https://webcache.googleusercontent.com/search?q=cache:http://blog.level3.com/global-
connectivity/verizons-accidental-mea-culpa/)

~~~
basseq
Works for me. Those who can't see it: are you on Verizon?

~~~
shock
It's working for me now as well. When I posted the cached version the page on
Level3 blog was giving me a 404 although the post was on the "Recent posts"
list.

I'm not getting why I'm being down voted for posting the cached version when I
was getting the 404.

------
gcb0
i recently gave up my netflix and amazon prime because i was tired of
streaming hanging up in the middle of an episode/movie. it was just annoying
and embarrassing if i had guests.

now i just torrent everything so i can watch uninterrupted when i want. I
consider i am still legal since i'm paying for both subscriptions yet and they
pay the content. if not, not my problem.

but that raises a question... why the RIAA/MPAA is not using its bullying
powers to harass ISP like verizon?

as i just showed, they are the reason lots of people are going back to
torrents that do not make money to the studios like streaming services does.
Why do they rather sue john does instead?

~~~
FireBeyond
"but that raises a question... why the RIAA/MPAA is not using its bullying
powers to harass ISP like verizon?"

Because it presupposes that whether or not in fact, streaming via Netflix is
profitable or not, the studios didn't have to be dragged kicking and screaming
to allow it at all.

------
phkahler
I've always been suspect of cable and wireless internet connections. Anything
that shares bandwidth with all subscribers is going to get maxed out if a
large number of customers start streaming video. Is it possible that Verison
(and Comcast and etc...) know that opening up the gate to the outside world
will crush their network under the increased load? I like the milk-our-
monopoly-on-the-last-mile theory as much as the next guy, but is it even
possible for the technology used to deliver the necessary bandwidth if they
open up the pipe?

~~~
shimshim
Even so, as a paying customer I would expect them to calculate for this and
know how to scale.

~~~
phkahler
My point is was that spectrum doesn't scale. There's X gigabits per second
available over the air, and a similar X over a cable. For cable companies they
can push equipment closer to the home and have fewer subscribers on the same
line. I suppose Verizon could do the same with more towers, but imagine
doubling the number of towers to support double the bandwidth (or something
like that). As people have pointed out though, the congestion is specific to
connections that carry netflix.

~~~
PeterisP
There's a limit for mobile bandwidth, where also the number of towers won't
help; but cable is different - sure, at one point they have to put proper
fiber to the home (and it's economically feasible to do so, if there's actual
competition instead of monopolistic ability to charge huge prices for weak
service), but after that you can have as much bandwidth as your routers can
service.

------
abalone
Summary: The bottleneck is the the interconnection point. Netflix _is_
saturating the interconnection point, and Verizon wants them or their transit
provider to pay more to "ensure a level of capacity that accommodates their
volume of traffic".

But Level 3 claims it would be _very inexpensive_ for Verizon to increase
capacity there. Just a few port cards and a few thousand bucks for a major
city like L.A. They even offered to pay Verizon's costs.

~~~
msandford
Nope, Verizon's customers are causing the interconnection point to be
saturated. Netflix doesn't send data unless Netflix's customers request it. If
they sent data without requests Verizon could (and probably would) interpret
that as some kind of a network attack.

This is Verizon wanting to double-bill for traffic. Plain and simple.

> Verizon wants them or their transit provider to pay more to "ensure a level
> of capacity that accommodates their volume of traffic".

That's Verizon's job. To go out and get by any and all means necessary enough
bandwidth to satisfy their customers, namely the residential customers who
seem intent on using Netflix. At no point do they have a moral imperative to
hold their customers hostage to extort money out of someone else that their
customers are trying to access. It's already been paid for!

~~~
res0nat0r
> Nope, Verizon's customers are causing the interconnection point to be
> saturated. Netflix doesn't send data unless Netflix's customers request it.
> If they sent data without requests Verizon could (and probably would)
> interpret that as some kind of a network attack.

All of your comments keep ignoring or misunderstanding what peering is. It is
supposed to be roughly equal data traffic for both sides. Once it is lopsided
in one direction, it is no longer peering.

It doesn't matter who requested what, the fact that L3 traffic is consuming a
majority of the bandwidth now is the issue, and Verizon wants L3 to pay to fix
that.

~~~
gergles
You don't peer with residential ISPs, because they aren't backbone ISPs.
Residential ISPs need to buy transit from Level 3.

If anything, Verizon should be L3s customer, and L3 giving them SFP is doing
_Verizon_ a favor.

~~~
res0nat0r
This doesn't really have anything to do with ISP's.

Verizon is a Tier 1 provider:
[https://en.wikipedia.org/wiki/Tier_1_network](https://en.wikipedia.org/wiki/Tier_1_network)

Yes much of the traffic ends up at end user destinations, but this doesn't
negate the fact that the peering arrangement is no longer roughly equal.

~~~
msandford
Verizon Business is NOT Verizon Residential.

Those companies couldn't be more different.

~~~
res0nat0r
It really doesn't matter what the company name is.

At the peering point the arrangement is no longer mutually beneficial for both
parties due to one side of the peering arrangement now using more than its
fair share or traffic. This is what Verizon has issues with.

------
ninv
Greed! It would not stop here. Today it is Netflix and youtube, tomorrow it
will be image heavy sites. Then they will go after google, mail provider and
bank sites etc.

------
adamman
Doesn't Verizon realize they lose customers for crap like this? My phone
contract with Verizon is up in November. I can't wait to be done with them.

------
macspoofing
Great post. Frankly Level3 should be pissed off. Verizon essentially called
them out for providing crap service when in fact they are the ones
responsible.

------
brownbat
What would happen if Netflix stopped allowing access to its service to Verizon
customers in areas where there's another provider? Netflix could argue that it
doesn't want to provide an inferior product if there's a choice, and maybe
leave a number on the homepage for the cable company in your area.

I wonder if Netflix or Verizon would lose more customers...

~~~
wpietri
Then a) Netflix becomes the bad guy, b) they lose money from those customers
leaving, and c) they no longer have the opportunity to influence those people.

They should continue to allow access, but they should also keep saying when
the problem is with the network provider. Customers love Netflix (NPS of 54)
more than Verizon (NPS of 32), so they'll be more inclined to believe Netflix.

------
trjordan
The claim from Verizon that still stands after this blog post is the fairness
argument to peering. If you peer between symmetric networks, it's typically
free. That's not the case here, so Verizon wants to be paid for the asymmetry.

I actually don't know. Is that legitimate?

~~~
allenlavoie
One solution would be for Netflix to modify their clients to send useless and
quickly discarded UDP packets back to balance out the traffic. Verizon then
has tons of data to push back to Level3 and everyone's happy.

~~~
ktsmith
That wouldn't work though. Verizon sells unbalanced services with larger
download capacity than upload capacity. So it's simply not likely/possible
that their customers can send matching data for what they are requesting.
Additionally, while this might not negatively affect their FIOS customers
their DSL customers are on asynchronous connections meaning that the uploaded
data will negatively impact their ability to receive data at the same time in
effect degrading their ability to stream and the quality of the video they can
receive.

~~~
wtallis
What DSL cusotmers have are _asymmetric_ connections, not half-duplex. Even on
ADSL, you can watch a HD video stream while uploading stuff. The only way the
upload matters is if it prevents timely delivery of ACKs for the video being
downloaded, and that's only a problem on the lowest speed tiers and when you
don't have decent QoS.

~~~
ktsmith
Thanks for catching that error, as I typed out asynchronous I was saying to
myself "this isn't right, the a is in the dang acronym" and then I was too
lazy to double check.

------
ck2
_This congestion only takes place between Verizon and network providers chosen
by Netflix._

 _The providers that Netflix does not use do not experience the same problem._

This needs to be put in front of the FCC since it is the bare minimum language
they should be able to understand.

------
eyeareque
He makes great points. But this is what the verizon marketing team is paid to
repond to. They'll come up with something, just wait for it.

------
knodi
If only we have a government body that did something about stuff like this...
o wait FCC... o wait :(

------
slyrus
More like an accidental "Fuck you" than an accidental "mea culpa".

------
ajmurmann
The Internet is a crucial part of our infrastructure and a important backbone
of our economy. If there is any indication that someone like Verizon in this
case is tempering with it intentionally, they should be charged with
terrorism. This is no different than someone going out and sabotaging Hoover
damn on purpose. In fact it's worth!

