
Double paid traffic - revelation
http://wiki.hetzner.de/index.php/Double_Paid_Traffic/en
======
Pyxl101
Frequently it seems as though statements by neutrality advocates mention
"providers" without naming any. Who are the providers that misbehave, and what
are they doing? Can we see a graph of their traffic and a depiction of the
misbehavior? (Saturated links, offers to peer and expand the links that are
refused, etc.) I'd love it if the evidence were laid out. Unfortunately at the
moment, it seems like there's a lot of ambiguous finger-pointing.

What do (presumably) Comcast and these other companies say when you
communicate with them about peering, and you ask them why they won't? Anyone
with insight into these issues care to share the DLS/cable provider
perspective? (Is there some kind of reasonable argument there we're not
hearing - "Infrastructure costs are rising, you see, and we built the
infrastructure we have now with a government subsidy that's no longer
available. So when we expand, it's going to cost 10x as much ... " just to
make something up. Is there another side to the story, or is it really just
supply/demand where the providers exploit their market power? "Our customers
are not likely to drop our Internet service for a competitor just because
Netflix is slow. In fact, our data show that they start watching cable TV
more, which is good for us - we want cable TV to be sticky. So we'll charge a
fee since we're nearly a monopoly and the market will sustain it. Further,
since Netflix is where all the traffic is going, we'll just charge them
directly rather than expand pipes all around".) [edits]

> For this reason, interfaces are being operated at their highest capacity --
> to make paid access to the DSL and cable providers' networks attractive for
> content providers. The end consumer is then forced to pay double for
> unlimited access to the Internet.

I don't entirely follow this part. Could someone clarify? Is "operating at
their highest capacity" a bad thing? Is the implication "operating at their
highest capacity [without expanding capacity when they should]"?

~~~
Ao7bei3s
> Who are the providers that misbehave

The article is about T-Systems (AS-DTAG), which is the largest ISP in Germany
and basically the company we mean when we talk about net neutrality violation
here.

Look at the speeds of their exchange points:
[https://beta.peeringdb.com/net/196](https://beta.peeringdb.com/net/196) \- 34
Gbps total: 1x 20G, 1x 10G, 4x 1G. Seriously, 1Gbps!

Now compare with another ISP with only 25% the revenue:
[https://beta.peeringdb.com/net/997](https://beta.peeringdb.com/net/997) \-
890 GBps total.

~~~
ullus12
Yet you can expect Cable service (Kabeldeutschland/Vodafone) to be to be
lossy/congested at every days peek hours.

Whereas DTAG _does_ way better.

I dont like it, but as others have said: Relying on transit peering will not
work well once you reach certain traffic levels.

Also you can get Servers from Strato (DTAG Sub) for about the same price as
from Hetzner and pump your Traffic directly into a Tier1.

~~~
merb
That is simple not true (telekom customer). And I know a lot of people on
Telekom.

We even needed to contact Github to change our peering since the default route
from Github to DTag just sucks.

Our overall bandwidth with Telekom simply sucks. Most networks are really
really slow, compared to Kabel Deutschland.

~~~
ullus12
> We even needed to contact Github to change our peering since the default
> route from Github to DTag just sucks.

and they were like "sure cosider it done"?

~~~
merb
> and they were like "sure cosider it done"?

no. they tried to find a way with us. but after that it didn't worked they
changed the routing and it worked.

------
kennysmoothx
I hope this does not fall on deafs ears but I am experiencing this EXACT thing
with OVH(Canada's Mega Datacentre).

I have compared speeds directly to OVH's servers, and indirectly through a
reverse proxy, and AT&T throttles direct connections to OVH. When I prompted
OVH about it they confirmed to me that AT&T requested that they pay them for
priority bandwidth and they refused (as is their right) and now my customers
get throttled to 200 kb/s when accessing my servers on OVH.

PS. OVH has arguably the largest data center in North America so this means a
lot of services are being affected by these ISP's.

~~~
ehPReth
This may be a bit unrelated, but OVH provides a publicly accessible weather
map of their network (something I wish more ISPs/providers did):
[http://weathermap.ovh.net/](http://weathermap.ovh.net/)

------
bakhy
what if, for example, Netflix did something like this, and charged a higher
subscription fee to, e.g., Comcast customers? though i don't believe this
would solve the problem, it might bring the issue of net neutrality closer to
home for many customers, and weigh on their decision which ISP to take (if
they even have the choice, which would be an issue in the US, i presume?). and
it's ultimately fair - why would all users pay for this through their
subscriptions, if only some ISPs violate net neutrality.

~~~
rayuela
This doesn't work when Comcast offers services that directly compete with
Netflix, it just makes Comcast streaming services seem relatively less
expensive.

------
viraptor
I'm not sure how to interpret this post. Is this basically: "we believe in
network neutrality, but here's reality for today: your ISP won't expand
capacity, so you can pay us to add special uplink"? It looks like testing the
ground for idea of "pay per peering" that becomes a separate line on your
invoice. (and I mean for any peering, you rent server and connectivity
separately)

~~~
j_jochem
I think this might actually a good idea. Instead of just paying the peering
fee quietly, they give customers the choice to vote with their money. Alas,
"customers" here are not end users, but companies who provide content on the
internet.

I really like that paying up to DTAG is not the default and is more
bureaucratic. This might already be enough to kill the ISPs' profiteering
schemes, similiar to what happened with the Leistungsschutzrecht (see
[https://en.wikipedia.org/wiki/Ancillary_copyright_for_press_...](https://en.wikipedia.org/wiki/Ancillary_copyright_for_press_publishers))
in Germany.

~~~
digikata
I think more visibility is better, and in this case maybe would give tech
companies some food for thought re: how they approach net neutrality issues in
politics.

------
metalliqaz
This is the correct approach. The ISP pulls their bullshit. Service passes the
cost on to customers. Customers move to different ISP. ISP eventually learns
lesson.

Of course, this only works if there is competition between ISPs.

~~~
tajen
Yes, and where are the JS libraries that show a warning to customers of
networks operating at lower capacity? All ISPs have available bandwidth for
speedtest.com, but when it comes to Youtube or Netflix, it requires a warning.

~~~
bduerst
SpeedOf.me works pretty well, but it would be nice to have a browser extension
that notified me whenever a site loaded more slowly for me than others in the
region on different ISP.

~~~
herbst
Thats a great idea actually. I may also try to track ISP and times on one of
my more popular websites.

------
CodingGuy
I'm hosting my server with Hetzner and using Cloudflare in front of my sites.
So the connection for my site visitors looks like this: Hetzner => Cloudflare
(local pop) => visitor. Works perfectly without paying for an additional
uplink as the connection between Cloudflare and Hetzner is quite fast.

~~~
Pyxl101
Cloudflare has written about running into challenges regarding net neutrality
too:

[https://blog.cloudflare.com/net-neutrality/](https://blog.cloudflare.com/net-
neutrality/)

Their article about the relative cost of bandwidth around the world made the
rounds on Hacker News when it came out:

[https://blog.cloudflare.com/the-relative-cost-of-
bandwidth-a...](https://blog.cloudflare.com/the-relative-cost-of-bandwidth-
around-the-world/)

> Australia is the most expensive region in which we operate, but for an
> interesting reason. We peer with virtually every ISP in the region except
> one: Telstra. Telstra, which controls approximately 50% of the market, and
> was traditionally the monopoly telecom provider, charges some of the highest
> transit pricing in the world — 20x the benchmark ($200/Mbps). Given that we
> are able to peer approximately half of our traffic, the effective bandwidth
> benchmark price is $100/Mbps.

Cloudflare supposedly buys transit from transit providers. However, since it's
serving cached data locally from each POP, presumably that means that
connections from a site visitor to Cloudflare are served locally, for cached
assets, rather than being passed on across transit to Hetzner.

------
Mojah
Aren't they basically saying "We support net neutrality but since we have no
choice, we'll pay for priority access"?

~~~
blueflow
More "We support net neutrality but since we have no choice, we'll allow you
to pay for priority access for your server". Its opt-in.

------
thomaskcr
> They now also want to collect revenue by charging large content providers
> fees for access to their network

Isn't what they are advocating for the exact opposite of net neutrality?
Everyone's connection is saturated, but they argue that because they are a
large content provider they should get special treatment and not need to pay
for direct access. Smaller companies are equally affected during peak hours,
but they would need to pay for that same privilege? If not, what is too small?

There seems to be a trend of adding "free peering" into the net neutrality
debate. Unless you're carrying equal bit-miles, free shouldn't even enter into
the equation. If data flows unequally, money should flow the other direction.

I think a lot of companies are leveraging the current hate for cable companies
to get out of properly engineering their infrastructure. People have leased
lines, built out additional datacenters and paid for colos around the world
for years and it was never "extortion" until a popular company twisted the
definition of net neutrality a bit so they wouldn't need to build their own
datacenters.

~~~
omgtehlion
> If data flows unequally, money should flow the other direction

Ok, data flows (mostly) this way:

Hetzner (or any other DC) --> ISP --> customer

DSL companies want money to flow this way:

Hetzner --> ISP <\-- customer

~~~
thomaskcr
I'm not sure I understand the difference between Hetzner and a customer. They
are both utilizing the ISP for the same purpose, transporting data. We still
need to pay for an Internet connection for our servers. You're not paying the
ISP for access to a specific person's server, you're paying to be part of
their network which happens to be connected to other networks. Everyone pays
to be part of the network.

The end user is not paying for access to content, so the fact that Hetzner is
providing something requested is irrelevant because that's not what the
internet ever was. In your diagram, Hetzner is an end user and whether it's up
or down data you still need to pay.

The end users are not Hetzner's customers - the people paying them for hosting
are. If they want to lease a line and offer access to that leased line at a
premium to their customers for better connections during heavy times they are
free to do that. It's not the cable companies job to make sure Hetzner's
customers are happy though. There is already a framework in place to achieve
what they want.

~~~
Cakez0r
But Hertzner aren't pushing data onto DTAG's network. The ISP's customers are
requesting it.

~~~
thomaskcr
The internet is much more than just HTTP traffic - the purpose isn't to
provide access to content so the fact that these are content providers (i.e.
data is requested) doesn't matter. I can host my backups via a web server at
home and wget them from my remote server, the fact I "requested" that from the
remote server means nothing.

The ISPs customers are paying for a connection to the ISPs network, nothing
more. Every connection to the network is a node - in a network neutral world
the fact they are a content provider is meaningless. If they need to push more
data, they need to pay.

I really don't see the difference between this, and a large company attempting
to send their backups via the web. We could easily set it up so the remote
servers are "requesting" that data -- but it's not the cable subscriber's job
to subsidize my network infrastructure so we lease a line to ensure the speed
we need. We have over 100mbps connection on both sides, but that is only a
100mbps connection to the ISP. If we need to move data at that speed all the
way from point A to B, then we need to pay for the infrastructure to do that.

~~~
Natanael_L
Then the correct solution is to use those peering agreements in between ISP:s
to get that money to expand the infrastructure as necessary, and if the ISP:s
with customers creating the traffic can't pay then they can chose between
stopping the traffic or charging their own clients more.

In no circumstance should an ISP be directly charging the customer of
_another_ ISP.

~~~
thomaskcr
I agree but that's not what's happening here. They are just directly passing
on the cost to provide service - the customers are paying Hetzner not the ISP
(unless I'm reading wrong - but I did double check).

In this situation, Core-Backbone pays for peering with DTAG and Hetzner pays
them to transport their data. I'm not seeing how that is an ISP billing
another ISPs customer? Hetzner is paying Core-Backbone, Core-Backbone is
paying DTAG. Same as Netflix paying Cogent to transport their data.

~~~
Natanael_L
I'm primarily thinking of cases here like consumer ISP:s wanting to charge
Netflix directly. This particular case might be quite different from what I
had in mind.

------
Pyxl101
Could we speed the Internet up with more generic caching?

I'd theorize that a decent percentage of all Internet traffic to residential
ISPs is made up of HTTP requests for highly cacheable resources. However, it
seems like the mythical idea of an automatic caching HTTP proxy at the ISP
level never really materialized. Perhaps that was possible in the HTTP days,
but HTTPS makes it a challenge.

How much better would Internet congestion be if there was a way for ISPs to
cache commonly-referenced resources? They could even be video resources - if
Netflix requests its videos across an HTTP connection by identifying a single
common resource (a URL everyone will access who plays the video), then an ISP
could simply cache the URL and serve it locally.

It seems like HTTPS is a barrier to schemes like that. Instead of an ISP
running a generic cache for commonly-referenced resources, each website
provider needs to run a local POP that's trusted and knows just their content.
(Though maybe Cloudflare like solutions are possible where the POP is not
fully trusted.)

If we set HTTPS aside for a moment, could we speed the Internet up with
caching? How much of the traffic to saturated links are requests for the same
content over and over again?

Is there a conceptual way to achieve some of the privacy benefits from HTTPS
while allowing an ISP to cache content in a generic way? Or could an ISP offer
a discount for using HTTP traffic instead of HTTPS, and then cache content in
a protocol-compliant way?

~~~
throwaway2048
the entire point is ISPs are acting in bad faith looking for a payday, the
limitations are not technical, but purely artifical rent seeking bullshit.

They are not interested in solutions that do not involve paying them money.

------
amazon_not
A simple way to avoid paying double for traffic is just to host your servers
directly at Core-Backbone, instead of funneling your traffic from Hetzner via
Core-Backbone to DTAG.

------
zuckerei
I have a Server in Germany. But my visitors are from all over the world. Will
I have to pay to every ISP out there so my data gets to their customers quick?

~~~
Nyr
No, since you will be forced to use transit to reach many of your visitors.

You don't even have the choice of direct connections to all of the world with
just a single server in Germany.

~~~
scurvy
There are reach products that will help you connect and peer to providers in
remote IX's. You can peer with them directly, then have their traffic ride on
your network all the way back. MPLS is a wonderful thing.

Yeah, you'll still need a lot of transit to reach people, but you can also
peer with who you want.

~~~
fiber
Provided "who you want" wants to peer with you. Peering is a mutual
relationship after all.

~~~
scurvy
Most public peering IX's have route servers which have "most" of the routes
available on the exchange. Peer with that and you have almost all of the
routes available on the exchange. Of course, some entities might not advertise
everything to the route server, but most do in my experience.

~~~
Nyr
> Most public peering IX's have route servers which have "most" of the routes
> available on the exchange.

Doesn't matter when we are talking about DTAG, which is the dominant ISP and
will not peer for free. Yeah, you can peer with a lot of smaller networks
which is nice, but not enough.

Same applies for the US market where the big players refuse to peer.

------
theklub
We need an Open ISP movement... Maybe there is one already but I mean like a
non-profit ISP type deal.

------
ullus12
DTAG is the Network Operator for most of German Business and Private
Customers. Most DSL-Resellers are Business Customers of DTAG.

So maybe there point of view, beeing more important than any public exchange,
is not far fetched as it is most certainly a business reality.

Hetzner making a stand for net neutrality is a joke. They are asking your
money for better service.

~~~
zAy0LfpBZLC8mAC
If DTAG asked for 1 EUR/Gigabyte, would you still maintain that position? 10
EUR? 100 EUR? Is there any point where you would consider the behaviour of
DTAG inappropriate/unfair?

BTW, nobody is denying that they in fact are an important ISP, where
"important" means "lots of customers", which in turn means "lots of power".
Someone holding a gun to your head also has a lot of power and thus is
important to you - but power in neither case automatically implies that any
offer the powerful party is making is a fair offer, be it that they ask you to
give them all your money to avoid being killed, or that they ask you to pay
massively above-market rates to avoid being unreachable for their customer
base.

------
scurvy
I would take this a lot more seriously if Hetzner were not home to about 20%
of the banned/fraud IP's on my network. I find it highly ironic that they get
up on their high horse and start preaching about how to properly run a network
(when they completely ignore what's going on inside theirs).

------
ullus12
Hetzner is not just saving on private peerings.

Most of their Servers are in east-germany where

    
    
      * labor,land,power,darkfibre is cheaper
      * they can harvest developement subs from the goverment [1]
    

Whereas peering (DE-CIX) is most dense in Frankfurt, West-Germany.

[1]
[https://translate.google.de/translate?hl=de&sl=de&tl=en&u=ht...](https://translate.google.de/translate?hl=de&sl=de&tl=en&u=http%3A%2F%2Fwww.datacenterpark.de%2F)

~~~
zAy0LfpBZLC8mAC
1\. What's the logic here? Because a business is trying to reduce its
expenses, it's wrong on any matter by default?

2\. You are aware that DTAG does not peer at DE-CIX, so whether "peering is
most dense" there has exactly no influence on whether Hetzner gets good
connectivity to DTAG?

3\. You are aware that Hetzner does peer at DE-CIX with 200 Gbit/s, with an
open peering policy?

4\. You are aware that Hetzner even peers at AMS-IX in Amsterdam with 100
Gbit/s? That link almost certainly is even more expensive than the one to DE-
CIX.

see also, peering list close to the end:
[http://wiki.hetzner.de/index.php/Rechenzentren_und_Anbindung...](http://wiki.hetzner.de/index.php/Rechenzentren_und_Anbindung/en)

~~~
ullus12
1\. The idea was to make an example of the company favoring cost effevtiveness
over quality. which may not be bad ... but it should underline the cheap-
assiness.

2\. dtag is ofc present in frankfurt and at de-cix, with little capacity for
public peerings as mentioned.

This and your 3/4 each make sense, as they represent a legit business interest
for the different sides. Hetzner preffering cheapass open and dtag preffering
scalable private peerings.

Problem i see is Hetzner externalizing the cost of private peerings. Which in
return plays into 1. and the example of them externalizing other cost factors.

~~~
zAy0LfpBZLC8mAC
1\. How exactly is labor, land, power, or dark fiber lower quality in east
Germany (if we ignore the fact that Nuremberg doesn't really qualify for being
cheap nor for being in east Germany)? If anything, the subsidies might suggest
that operating a business is actually more expensive, which is why it's being
subsidised.

5\. You are also aware that they operate some 170 Gbit/s of private peering?

6\. Which costs exactly are they externalizing with regards to peering in your
opinion? So far, your argument seems to be "they don't pay as much as DTAG
would like them to, therefore, they are externalizing the difference between
DTAG's wishes and what they are willing to pay" \- which isn't exactly how you
determine externalities.

------
rmoriz
Did anyone verify the information? I think it's a hoax to drive cusomers
mad/against Deutsche Telekom.

\- It's not possible to announce a specific (premium) routing for just a
single IP address.

\- Hetzner has no DTAG Transit anymore. none. Not even for specific customers.
See AS24940

\- Current DTAG->Hetzner traffic uses AS33891. See
[https://f-lga1.f.de.net.dtag.de/](https://f-lga1.f.de.net.dtag.de/)

Of course this does not object the non-peering-policy claims and high prices
of DTAG in any way. It's well known in the German ISP scene.

~~~
amazon_not
> I think it's a hoax to drive cusomers mad/against Deutsche Telekom.

You don't need a hoax for that. Everybody in the industry is already pissed at
DTAG.

> \- It's not possible to announce a specific (premium) routing for just a
> single IP address.

Wrong. Google source based routing. You'll get asymmetric routing, but that's
rarely an issue for content providers.

> \- Current DTAG->Hetzner traffic uses AS33891

That's exactly what the article states.

~~~
rmoriz
> That's exactly what the article states.

No. Hetzner is not able to provide direct Telekom traffic. Their offer is
missleading.

Hetzner makes a profit of 36mio EUR/year. It's a shady move. Telekom sucks but
Hetzner is not any better.

~~~
amazon_not
What on earth are you going on about?

The article in no way states that Hetzner has a direct connection to DTAG. It
very clearly states that they route the traffic to DTAG via Core-Backbone.

------
kintamanimatt
"We at Hetzner Online do not support such policies and declare our support for
full net neutrality."

Bullshit. In the same breath they offer something that is entirely
antithetical to their stated ethos, no matter how they spin it.

This page demonstrates what they say and what they do are two different
things. I interpret this as they're making a goldbrick statement of support in
favor of net neutrality with the hope that they won't face a backlash.

I will never touch Hetzner so long as they act against net neutrality.

~~~
0x006A
They are transparent about DTAG not participating in Tire 1 peering for free.
They offer a way to pay for this on a per server basis if _you_ don't want to
participate in this standoff. I much prefer this to Hetzner silently paying
DTAG and increasing the hosting costs for everyone.

