
Observations of an Internet Middleman - staticsafe
http://blog.level3.com/global-connectivity/observations-internet-middleman/
======
cs702
In short: if you're in the US, you're paying your ISP for a certain amount of
bandwidth, but your ISP is _not_ giving it to you, because its connections to
middlemen like Level 3 are maxed out. Level 3 proposes to split the cost of
expanding those connections (as is common), but your ISP refuses unless it
gets additional payment from Level 3, Netflix, or someone else. Meanwhile, you
don't get the bandwidth you've purchased from your ISP.

That doesn't seem right to me.

~~~
orik
Disclosure: I work at a small ISP. We mostly compete with Frontier DSL and
Comcast Business Class.

If you're buying residential it's safe to assume you're purchasing a burst-
able speed and downloads/uploads aren't going to sustain that for hours on
end.

Much in the same way that a Utility Company probably isn't gathering enough
water for everyone to be maxing their pipes 24/7, residential ISP's don't
operate under the assumption that every consumer will have maxed their
connection at the same time, and they shouldn't have too. Smart traffic
shaping and peering arrangements gives ISP's room to compete.

If consumers could afford a dedicated 35x15 connection they'd have a T1. When
you're buying copper it's safe to assume it's going to be over saturated.

The real problem is ISP's for the most part don't compete. They zone of
sections for eachother and rack in as much dough as they can. Consumers aren't
informed about the quality of different types of connections vs others, and
buy based off the cheapest Mbps down/up they can get.

~~~
twic
_in the same way that a Utility Company probably isn 't gathering enough water
for everyone to be maxing their pipes 24/7, residential ISP's don't operate
under the assumption that every consumer will have maxed their connection at
the same time_

That's not what's happening, though. What's happening is that the ISPs have
not provisioned enough capacity to serve the actual aggregate demand at peak
times. The hydraulic analogy would be if water pressure dropped every day
between 0800 and 0900 because everyone was having their morning shower. That
would not be acceptable to customers, and nor is the situation with ISPs.

~~~
cs702
Exactly. Great analogy!

What's more, the utility company is getting water through a pipe that goes to
a middleman near a river, but that pipe doesn't have enough capacity to
deliver the water necessary for all utility customers to take their morning
shower at the same time. So what does the utility company do? It _blames the
middleman and the river_ for "sending too much water"... and demands payment
from them!!!

------
johngalt
I enjoy seeing those MRTG/RRDtool graphs everywhere. I find it so surreal that
$X Billions in infrastructure, that is forwarding $Y Billion of internet
traffic. All monitored by a single tool created by one guy from Switzerland.

Since this is HN: yes I realize that there has been substantial work
subsequently, and that cacti/munin/nagios etc.. are more commonly used.

~~~
zero_intp
It's crazy to think that Rancid, MRTG, & RRD are all fundamentally
underpinning of all of these major carriers.

~~~
NDizzle
It's not so crazy when you consider a huge portion of peering agreements took
place over IRC.

~~~
voltagex_
Got any articles on this?

------
wdewind
Here's the part that doesn't make sense to me, and hopefully someone can
explain it:

Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast
claims that only the Level 3 connection is saturated, and that Netflix is
sending all their bandwidth over Level 3 because it's cheaper for them.

Doesn't Level 3 buy a contract from Comcast that says "we get to send this
much data per month"? If Netflix (or Level 3) tries to push more data through
that, which causes congestion, it seems like a contractual fact that Comcast
is either holding up their end of the bargain or not.

If that's true, and assuming that Comcast is transferring the contractually
agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it
reasonable to assume that Netflix would need to either use another entry point
into Comcast's network, or build one?

If Netflix/Level 3 congest one entry point, so movies stream slowly for
Comcast users, it doesn't seem like Comcast is being unneutral, in the sense
of packet inspection and routing based on content/source.

The problem seems upstream from the last mile networks, and it's very unclear
to me whose fault it is, and if this even has anything to do with net
neutrality at all (it doesn't seem to).

Please correct me where I'm wrong, I'm sure I am!

~~~
exelius
I think Level3's issue is that Comcast is unwilling to increase the capacity
of their links without payment from Level3. If Level3 engages in settlement-
free peering with Comcast and their network traffic graph looks anything like
the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in
traffic in vs traffic out.

What's happening here is that Level3 and other transit providers are starting
to see their industry be squeezed by the big ISPs. If you have 5 ISPs that
serve 90% of the customers in the US, and 10 service providers that generate
90% of the bandwidth, why do transit providers even exist? I think they see
their market share decreasing significantly as more of their customers follow
in Netflix's footsteps, so they're trying to pile on and start a grass-roots
outrage like Netflix did. Problem there is that most people haven't heard of
Level3, so it makes it more difficult to get people behind them.

And you are completely right about the whole Netflix issue being Netflix and
Level3/Cogent's problem. Level3 and Cogent have business models based on
selling transit acquired through settlement-free peering. Settlement-free
peering assumes that bandwidth usage is roughly symmetric: when it's not
symmetric, the side sending more bandwidth has to pay. Level3's position here
seems a bit hypocritical: if they had a customer that was routinely sending
data at a 20:1 ratio, they would charge that customer for sending more
bandwidth. But they're expecting the big ISPs not to do the same to them?

~~~
NoMoreNicksLeft
The big trouble is that we _need_ settlement-free peering, or we'll have a
very different sort of internet, or maybe none at all. Though it's hard for
some to remember (or to believe), there were big consumer networks prior to
the advent of consumer-oriented Internet. Compuserve and AOL were their own
networks. And they were abysmal. Peering not only ruined that business model,
it managed to co-opt those networks and assimilate them into the Internet
proper.

If peering is untenable because it leads to outcomes like the one we have now
(Netflix) where it is fundamentally unfair to one party (assuming that is the
case) or the other, then we're all screwed. There might not be a fair solution
that still manages to resemble the Internet.

In one alternative, the Comcasts and the AT&Ts are smited and no longer exist,
and Level 3 goes into the consumer internet business, connecting things end-
to-end. That's not something we could trust them with. Same if Comcast
replaces Level 3.

In another, Netflix continues to pay the extortion (if it is that) which we
have little doubt that Comcast will continue to ratchet up the price for. Or
Comcast has to shoulder that burden alone (and can't charge customers extra
for doing so, without people screaming "net neutrality!").

Or maybe services like Netflix just can't exist in such an environment. Also a
bad outcome.

Or god help us, internet infrastructure is nationalized, and the same people
who manage our roads and traffic lights take over.

Am I blowing this out of proportion?

~~~
toyg
You could have a situation where the government mandates net neutrality and at
the same time compensates the likes of Comcast in some way (recognising their
role as public utility and basically sanctioning their monopoly, but without
taking over their operations entirely). You could have an anti-monopolistic
legal action that splits Comcast but prevents Level3 from taking over / being
present in that market. Etc etc...

There are many ways to skin an onion, and full-scale nationalisation is
probably the least likely outcome (cash-strapped governments don't need
another headache right now). In fact, the real problem here is that one player
is leveraging a monopolistic position; remove that position, and the Free
Market should start working its magic again.

~~~
paganel
> time compensates the likes of Comcast in some way

Serious question, how would one find the exact sum with which Comcast should
be _compensated_ in the absence of a functional market?

~~~
lmm
Analyse the available facts and make a best guess, just like we do with
private companies operating as regulated monopolies in e.g. water supply.

------
sergiosgc
If this is a game of chicken, Level 3 and Cogent should play it to the end
game. They are powerful enough to get Comcast and the other ISPs to concede.
Just null route any interconnect that consistently drops packets and fails to
get negotiated for an upgrade. The catch? It must be done by both L3 and
Cogent.

Comcast is playing the monopoly card against what is an oligopoly of backbone
providers. Comcast isn't strategically as well positioned as their actions
would indicate.

It is one thing for Comcast to have degraded Netflix service. It is a
different thing to present their customers with access to only half of the
Internet .

~~~
DontGiveTwoFlux
Except that it would probably violate their contracts to just stop services at
an interconnect. It's not like Level 3 is losing customers or money from the
congested interconnects. Their incentive to play hardball just isn't there.

~~~
sergiosgc
If it violates their contract, doesn't a soft fail violate it also? Or will
violate, whenever the rate of dropped packets goes above a given threshold. I
view the hard fail as just speeding up the interconnect failure, with the
positive effect of prompting action from the last mile ISP.

Whenever their performance drops towards part of the internet, their value
proposition gets lower. You can view the value chain as

    
    
      (residential customer) <-> Comcast <-> Level 3 <-> (hosting customer)
    

Both Comcast and Level 3 have customers, in opposite ends of the currently
centralized internet. The status quo is that each provider charges to the
customers on their end. Comcast (and each of the other non-net-neutral ISPs)
are eyeing the revenue stream on the other end of the pipeline. This revenue
stream has historically not been owned by the last-mile ISP.

Level 3 and equivalents, by cutting off non-performant interconnects will hurt
their own customers, of course. However, they are just speeding up the same
effect being applied softly over time, by Comcast and equivalents. A softly
failing interconnect moves revenue over to last mile ISPs, thus permanently
lowering Level 3's value proposition. By failing the interconnect sharply,
instead of softly, they actually improve the steady state of the system, in
general (good side effect), and in particular (the real objective).

~~~
mfisher87
I don't understand what's so hard about "You can roll with us as long as you
treat our rented infrastructure with respect." That sort of agreement exists
in many industries. I get kicked out of my apartment if I trash it, even
though I paid for it. I don't ask _my landlord_ to split the costs of an
upgrade to hardwood floors because I fucked up the linoleum (because "I ended
up with more pets than I expected" or something), much less pay for it
himself. I pay for it or I'm booted. If he did anything else, such as let
destructive renters stay on his property or paid for their damage they were
responsible for, what would be the point of renting the property? That's a
terrible business model. Same thing here... Comcast is damaging Level3's
customer relationships on purpose and refusing to pay reasonable level-of-
service costs to rectify the problem and should be thrown off the network.

------
josephlord
Summary of situation

Comcast cripples customers internet services to extort money from companies
connecting Comcast to the internet for free.

~~~
LandoCalrissian
It's completely crazy and just blatant extortion on Comcast's part.

The current system is pretty straight forward.

Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User

Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server

Now Comcast wants to charge everyone in the chain that isn't them. They are
attempting to do this by degrading (or at the very least not upgrading) their
services for their customer.

The techno-libertarians that try to defend this nonsense are going to sink the
whole ship and make every pay to receive even less.

~~~
erichocean
Ahem. "Techno-libertarians" aren't the ones who gave Comcast its monopolies on
the last mile. Please leave them out of this.

~~~
nullc
No, physics and economics did. When you're talking about building
infrastructure that will take 20 years to pay itself off _absent_ competition,
only the most profitable of the profitable locations can afford extensive
physical plant competition.

In communications networks monopolies are natural, especially in last mile
networks, and will exist absent any government action. That in some cases some
governments have mildly strengthened these monopolies in exchange for
commitments to cover otherwise unprofitable areas, its not really that
important this subject.

------
_stephan
Does anyone have an idea what the European ISP is that refuses to provide
enough peering capacity?

~~~
revelation
They said its not from the UK and judging from their connection points, I
guess its Deutsche Telekom.

~~~
josephlord
Don't know they described them as a monopoly position and cable is pretty
strong in Germany. I think Spain or Italy might be better guesses.

Telefonica in Spain might be a reasonable guess. Ono the cable company isn't
huge.

I think Telecom Italian is dominant in Italy and there is virtually no cable
at all so that would be my guess.

~~~
josephlord
Too late to fix typo: Telecom Italia (not Italian).

------
MaysonL
Level 3's filing with the FCC about this:
[http://www.level3.com/~/media/Assets/legal/l3_openinternet_m...](http://www.level3.com/~/media/Assets/legal/l3_openinternet_march2014.pdf)

------
jeremycole
If Comcast doesn't like the amount of bandwidth their customers use watching
Netflix, they always have the option to notify their customers and null-route
or otherwise stop Netflix from working at all. However while it "works" but
sucks they appear to be doing their jobs while their customers blame Netflix
for sucking while largely being unaware that it's actually Comcast that sucks.

------
chrisBob
If you are experiencing a slow connection and packet loss, what is the best
tool for identifying the issue? There must be a good way to find out where the
problem is.

~~~
wpietri
I use mtr. But reading it takes experience, so for a month or so just fire it
up everywhere you go, especially when things are working well.

~~~
chrisBob
Thanks. MTR looks interesting.

