Hacker News new | past | comments | ask | show | jobs | submit login
Double paid traffic (hetzner.de)
356 points by revelation on Nov 30, 2015 | hide | past | favorite | 103 comments

Frequently it seems as though statements by neutrality advocates mention "providers" without naming any. Who are the providers that misbehave, and what are they doing? Can we see a graph of their traffic and a depiction of the misbehavior? (Saturated links, offers to peer and expand the links that are refused, etc.) I'd love it if the evidence were laid out. Unfortunately at the moment, it seems like there's a lot of ambiguous finger-pointing.

What do (presumably) Comcast and these other companies say when you communicate with them about peering, and you ask them why they won't? Anyone with insight into these issues care to share the DLS/cable provider perspective? (Is there some kind of reasonable argument there we're not hearing - "Infrastructure costs are rising, you see, and we built the infrastructure we have now with a government subsidy that's no longer available. So when we expand, it's going to cost 10x as much ... " just to make something up. Is there another side to the story, or is it really just supply/demand where the providers exploit their market power? "Our customers are not likely to drop our Internet service for a competitor just because Netflix is slow. In fact, our data show that they start watching cable TV more, which is good for us - we want cable TV to be sticky. So we'll charge a fee since we're nearly a monopoly and the market will sustain it. Further, since Netflix is where all the traffic is going, we'll just charge them directly rather than expand pipes all around".) [edits]

> For this reason, interfaces are being operated at their highest capacity -- to make paid access to the DSL and cable providers' networks attractive for content providers. The end consumer is then forced to pay double for unlimited access to the Internet.

I don't entirely follow this part. Could someone clarify? Is "operating at their highest capacity" a bad thing? Is the implication "operating at their highest capacity [without expanding capacity when they should]"?

> Who are the providers that misbehave

The article is about T-Systems (AS-DTAG), which is the largest ISP in Germany and basically the company we mean when we talk about net neutrality violation here.

Look at the speeds of their exchange points: https://beta.peeringdb.com/net/196 - 34 Gbps total: 1x 20G, 1x 10G, 4x 1G. Seriously, 1Gbps!

Now compare with another ISP with only 25% the revenue: https://beta.peeringdb.com/net/997 - 890 GBps total.

This reflects only the peering capacity available for public peering and only what has been documented by each ISP for that network. Private peerings (that are direct connections to another network) are much much more cost-effective and easier to scale for a large ISP. In any case you can safely assume that DTAGs peering capacity by far exceeds one or the other Tbit, but it's also far from being sufficient capacity after all. They may also be the largest ISP in Germany by user count, while other metrics may vary ;).

Private peerings are very cost-effective for DTAG because DTAG usually takes money for them.

While that is certainly true, that is not what I meant. For one, it's very cheap to reach such exchanges for companies like Hetzner, because they usually operate their data centers on the same campus, to safe exactly on that backhaul cost to the exchange. By implication all the ISPs are expected to extend their network into these facilities to pick up traffic. While it's true that some ISPs operate big POPs near exchanges, it's not where their customers are. And both users and data center operators rightfully expect a professionally built network with redundancy etc. In that respect it's not necessarily unfair that they charge for the traffic. (While it is certainly unfair that they don't upgrade their connections to extort outrageous prices from other market participants and what's much worse to contain internet growth on a large scale, which is why Germany is a fairly under-developed country in terms of internet connectivity).


But also, people who work for exchange point operators are the only ones left flying business class in the telecom industry. At the same time public exchanges are a single point of failure in the network that one should(!) design around anyway. There are much cleaner solutions you can design into a network at a fraction of that cost.

For our reference, you may want to check that list, it is really outdated/onesided [1], and not reflecting the reality. Better refer to ookla/netindex or other similar services [2]

[1] https://en.wikipedia.org/wiki/Talk:List_of_countries_by_Inte...

[2] http://www.huffingtonpost.com/bruce-kushnick/americas-ugly-t...

Yet you can expect Cable service (Kabeldeutschland/Vodafone) to be to be lossy/congested at every days peek hours.

Whereas DTAG _does_ way better.

I dont like it, but as others have said: Relying on transit peering will not work well once you reach certain traffic levels.

Also you can get Servers from Strato (DTAG Sub) for about the same price as from Hetzner and pump your Traffic directly into a Tier1.

That is simple not true (telekom customer). And I know a lot of people on Telekom.

We even needed to contact Github to change our peering since the default route from Github to DTag just sucks.

Our overall bandwidth with Telekom simply sucks. Most networks are really really slow, compared to Kabel Deutschland.

> We even needed to contact Github to change our peering since the default route from Github to DTag just sucks.

and they were like "sure cosider it done"?

> and they were like "sure cosider it done"?

no. they tried to find a way with us. but after that it didn't worked they changed the routing and it worked.

Vodafone doesn't just offer cable service but also DSL and VDSL, and that's rock solid.

The cable service slowdowns are technology inherent. But that's a last-mile problem and not a backbone problem. Also, these providers aren't trying to use their customer base to extort money from hosters.

Agree on the last-mile part

yet i doubt that buying from a (DSL-)Reseller is improving service quality and prices are roughly the same.

I'm a Unitymedia customer in Germany (200mbps down, 10mbps up for 42€/month including VoIP) and they have been pretty good in terms of speed. I haven't noticed any slowdowns. They also make sure that you get a little extra bandwidth so even with overhead you will actually see those advertised transfer rates in the wild.

> Is "operating at their highest capacity" a bad thing?

Yes, it's a bad thing. Internet traffic is bursty so usually you have to overprovision a lot to deal with the spikes. Operating at full capacity means that they get money for a service that they cannot fullfil. This is their intended goal such that later they can ask money from whoever wants to have priority.

That is just a different way of saying their (DSL) customers are their product, not their customers.

I think you are right about the implication. The providers tend to try and double dip by doing one of three things:

* Not upgrading their network as they should

* Discriminating against certain traffic, see Comcast and Netflix speeds

* or by not allowing content providers to connect their CDNs free of charge, which should be a win-win situation if the providers acted in the customers best interests.

I honestly believe that these bussiness practices are the providers digging their own grave. Best case scenario, they make a few quick bucks before laws get passed against this sort of thing, worst case scenario: the big boys get tired of dealing with them and start competing directly (like Google fiber).

The latter. There is no incentive to pay for additional bandwidth if there is enough capacity for everyone, so supply is restricted. Thus the ISP saves on network infrastructure and gets additional income to prioritize higher-status traffic. What a clusterfuck, I'm embarrassed by my government.

My question is: why aren't the companies being asked to pay for priority telling their users about the problem? Where are the JS libraries that show a warning to the users of those ISPs?

In an (admittedly different) enterprise scenario, I see consistent 50-60% peak utilization as right-sized. 75% or more is tagged for an upgrade or at least analysis to understand the traffic patterns. For a big network to deliver 100 Mbps vs. 1 Mbps connectivity to most locations is a very small cost. The big costs are the fixed costs the initial buildout.

Marginal capacity changes are pretty cheap. There's no technical reason to treat network links as constrained resources. So as a network provider, you only create constraint to make money.

Think of it as food & drink in a movie theatre. A bag of popcorn at the Target snack counter is $1.50. A bag of popcorn at the movies is $9. Same product, both high-margin scenarios -- the difference is you're captive to the movie theatre.

We start the process to add more peering ports once we see sustained traffic at 40% of port capacity (in PPS or BPS). The process takes a while depending on who you're dealing with, and before you know it, you're up to 80% right before that new port comes online.

> I don't entirely follow this part. Could someone clarify? Is "operating at their highest capacity" a bad thing? Is the implication "operating at their highest capacity [without expanding capacity when they should]"?

Yes. Basically existing links (thru transit providers) is saturated, if you(content provider) want good connectivity with these end-user networks you gotta pay more for direct connection.

> Is "operating at their highest capacity" a bad thing?

Yes. This allows providers to say "guys look, we can not increase the traffic going over the lines, so we have to priorize things. Thats 3.50€ a month per uplink"

I hope this does not fall on deafs ears but I am experiencing this EXACT thing with OVH(Canada's Mega Datacentre).

I have compared speeds directly to OVH's servers, and indirectly through a reverse proxy, and AT&T throttles direct connections to OVH. When I prompted OVH about it they confirmed to me that AT&T requested that they pay them for priority bandwidth and they refused (as is their right) and now my customers get throttled to 200 kb/s when accessing my servers on OVH.

PS. OVH has arguably the largest data center in North America so this means a lot of services are being affected by these ISP's.

This may be a bit unrelated, but OVH provides a publicly accessible weather map of their network (something I wish more ISPs/providers did): http://weathermap.ovh.net/

Speeds in which direction? To OVH or from OVH?

what if, for example, Netflix did something like this, and charged a higher subscription fee to, e.g., Comcast customers? though i don't believe this would solve the problem, it might bring the issue of net neutrality closer to home for many customers, and weigh on their decision which ISP to take (if they even have the choice, which would be an issue in the US, i presume?). and it's ultimately fair - why would all users pay for this through their subscriptions, if only some ISPs violate net neutrality.

This doesn't work when Comcast offers services that directly compete with Netflix, it just makes Comcast streaming services seem relatively less expensive.

I think this would settle the net neutrality debate practically overnight and personally back this suggestion 100%!

I'm not sure how to interpret this post. Is this basically: "we believe in network neutrality, but here's reality for today: your ISP won't expand capacity, so you can pay us to add special uplink"? It looks like testing the ground for idea of "pay per peering" that becomes a separate line on your invoice. (and I mean for any peering, you rent server and connectivity separately)

I think this might actually a good idea. Instead of just paying the peering fee quietly, they give customers the choice to vote with their money. Alas, "customers" here are not end users, but companies who provide content on the internet.

I really like that paying up to DTAG is not the default and is more bureaucratic. This might already be enough to kill the ISPs' profiteering schemes, similiar to what happened with the Leistungsschutzrecht (see https://en.wikipedia.org/wiki/Ancillary_copyright_for_press_...) in Germany.

I think more visibility is better, and in this case maybe would give tech companies some food for thought re: how they approach net neutrality issues in politics.

I don't think it's quite as you put it. I think Hetzner is saying more: === We believe in net neutrality, but some of our customers have noticed that users of these telecom companies cannot access their services during peak hours, or the service is quite slow.

Since we cannot change market dynamics, we are offering our customers a way to offer their customers faster service during peak periods with this "pay per peering" option. ===

My understanding (which may be incorrect) is that Hetnzer is telling existing customers that instead of having to install servers within the ISP network, they can instead pay a marginal fee per month to bypass the congested peering links to these providers.

It's more like "Your customers/users ISP (in practice DTAG, Deutsche Telekom AG) doesn't participate much in cost-neutral Tier-1 peerings and instead offers an expensive DTAG peering for additional bandwidth". DTAG can do this because they are (one of) the biggest ISP in germany.

> Make sure to clearly state that you would like to order the DTAG uplink in the text field of the request and that you agree to pay the additional € 4.20 a month. The add-on will be called "Double Paid Traffic" on your bill.

I think the name "double paid traffic" is a very clear statement. They are not trying to sugarcoat this or anything. They just don't want to loose customers over this.

Well, it's not really a case of net neutrality. Packets aren't being given priority. It's just a simple case of: ISPs are at capacity, connection slow, use these servers instead.

This is the correct approach. The ISP pulls their bullshit. Service passes the cost on to customers. Customers move to different ISP. ISP eventually learns lesson.

Of course, this only works if there is competition between ISPs.

Yes, and where are the JS libraries that show a warning to customers of networks operating at lower capacity? All ISPs have available bandwidth for speedtest.com, but when it comes to Youtube or Netflix, it requires a warning.

SpeedOf.me works pretty well, but it would be nice to have a browser extension that notified me whenever a site loaded more slowly for me than others in the region on different ISP.

Thats a great idea actually. I may also try to track ISP and times on one of my more popular websites.

Wait, but the customers here are Hetzner's customers, right? How would they move to a different ISP?

Hetzner is not an ISP but a hosting/servers/colocation company. The ISPs are the companies end customers are using to access internet in the first place (Deutsche Telekom, AT&T, etc)

I'm hosting my server with Hetzner and using Cloudflare in front of my sites. So the connection for my site visitors looks like this: Hetzner => Cloudflare (local pop) => visitor. Works perfectly without paying for an additional uplink as the connection between Cloudflare and Hetzner is quite fast.

Cloudflare has written about running into challenges regarding net neutrality too:


Their article about the relative cost of bandwidth around the world made the rounds on Hacker News when it came out:


> Australia is the most expensive region in which we operate, but for an interesting reason. We peer with virtually every ISP in the region except one: Telstra. Telstra, which controls approximately 50% of the market, and was traditionally the monopoly telecom provider, charges some of the highest transit pricing in the world — 20x the benchmark ($200/Mbps). Given that we are able to peer approximately half of our traffic, the effective bandwidth benchmark price is $100/Mbps.

Cloudflare supposedly buys transit from transit providers. However, since it's serving cached data locally from each POP, presumably that means that connections from a site visitor to Cloudflare are served locally, for cached assets, rather than being passed on across transit to Hetzner.

Aren't they basically saying "We support net neutrality but since we have no choice, we'll pay for priority access"?

More "We support net neutrality but since we have no choice, we'll allow you to pay for priority access for your server". Its opt-in.

> They now also want to collect revenue by charging large content providers fees for access to their network

Isn't what they are advocating for the exact opposite of net neutrality? Everyone's connection is saturated, but they argue that because they are a large content provider they should get special treatment and not need to pay for direct access. Smaller companies are equally affected during peak hours, but they would need to pay for that same privilege? If not, what is too small?

There seems to be a trend of adding "free peering" into the net neutrality debate. Unless you're carrying equal bit-miles, free shouldn't even enter into the equation. If data flows unequally, money should flow the other direction.

I think a lot of companies are leveraging the current hate for cable companies to get out of properly engineering their infrastructure. People have leased lines, built out additional datacenters and paid for colos around the world for years and it was never "extortion" until a popular company twisted the definition of net neutrality a bit so they wouldn't need to build their own datacenters.

> If data flows unequally, money should flow the other direction

Ok, data flows (mostly) this way:

Hetzner (or any other DC) --> ISP --> customer

DSL companies want money to flow this way:

Hetzner --> ISP <-- customer

I'm not sure I understand the difference between Hetzner and a customer. They are both utilizing the ISP for the same purpose, transporting data. We still need to pay for an Internet connection for our servers. You're not paying the ISP for access to a specific person's server, you're paying to be part of their network which happens to be connected to other networks. Everyone pays to be part of the network.

The end user is not paying for access to content, so the fact that Hetzner is providing something requested is irrelevant because that's not what the internet ever was. In your diagram, Hetzner is an end user and whether it's up or down data you still need to pay.

The end users are not Hetzner's customers - the people paying them for hosting are. If they want to lease a line and offer access to that leased line at a premium to their customers for better connections during heavy times they are free to do that. It's not the cable companies job to make sure Hetzner's customers are happy though. There is already a framework in place to achieve what they want.

You're supposed to pay your ISP. The services you use are supposed to pay their ISP:s. The ISP:s that interconnect are supposed to agree on payment for created load via peering agreements in between them.

Your ISP is not supposed to both ask you, the other ISP and also the client of the other ISP, your subscription service, for money, independent of actual load.

But Hertzner aren't pushing data onto DTAG's network. The ISP's customers are requesting it.

The internet is much more than just HTTP traffic - the purpose isn't to provide access to content so the fact that these are content providers (i.e. data is requested) doesn't matter. I can host my backups via a web server at home and wget them from my remote server, the fact I "requested" that from the remote server means nothing.

The ISPs customers are paying for a connection to the ISPs network, nothing more. Every connection to the network is a node - in a network neutral world the fact they are a content provider is meaningless. If they need to push more data, they need to pay.

I really don't see the difference between this, and a large company attempting to send their backups via the web. We could easily set it up so the remote servers are "requesting" that data -- but it's not the cable subscriber's job to subsidize my network infrastructure so we lease a line to ensure the speed we need. We have over 100mbps connection on both sides, but that is only a 100mbps connection to the ISP. If we need to move data at that speed all the way from point A to B, then we need to pay for the infrastructure to do that.

"The ISPs customers are paying for a connection to the ISPs network, nothing more."

They are also (usually) paying for a specific connection speed. If a customer is paying for a 100mbps connection and Hertzner has 100mbps of spare bandwidth, but the ISP consistently only has capacity to relay 20mbps per second from Hertzner to my computer, then I would argue that it is the ISP's responsibility to upgrade their hardware so that they can fulfil their obligations to customers that they have sold 100mbps connections to.

Then the correct solution is to use those peering agreements in between ISP:s to get that money to expand the infrastructure as necessary, and if the ISP:s with customers creating the traffic can't pay then they can chose between stopping the traffic or charging their own clients more.

In no circumstance should an ISP be directly charging the customer of another ISP.

I agree but that's not what's happening here. They are just directly passing on the cost to provide service - the customers are paying Hetzner not the ISP (unless I'm reading wrong - but I did double check).

In this situation, Core-Backbone pays for peering with DTAG and Hetzner pays them to transport their data. I'm not seeing how that is an ISP billing another ISPs customer? Hetzner is paying Core-Backbone, Core-Backbone is paying DTAG. Same as Netflix paying Cogent to transport their data.

I'm primarily thinking of cases here like consumer ISP:s wanting to charge Netflix directly. This particular case might be quite different from what I had in mind.

What is the internet protocol by which you push data from one node to another without the other node's consent? Every one I can think of would generally be called a DoS attack.

There's no such thing as "push" data, it's like trying to push a rope. Any reasonably sized data stream has the consent of both parties on the network.

The internet is not the web. SMTP is "push", VoIP is "push", scp from my local machine to a remote machine is "push" ... sure, there generally should be some form of consent on the part of the receiving party, but just because HTTP is limited to pulling stuff, doesn't mean there aren't less limited protocols around.

SMTP: nope, the server accepts your connection before you do anything

VoIP: generally negotiated somehow before the UDP "pushing" happens

scp: just because in your mind it's a "push" doesn't make it so. In reality the other machine is sending you acks, without which your machine wouldn't continue "push"ing and thus, again consent is required. finally you're logged in to that other machine which nearly universally equates to consent

It doesn't matter that the customers are requesting it or if they want it.

I request my Amazon package, but amazon still pays for delivery--even if amazon charges me for the shipping, UPS or Fed Ex get paid by Amazon.

The internet works sort of like that, except small ISPs will still pay to receive since their network doesn't interconnect with everyone else. ISPs have always charged CDNs for access to their network.

As a practical matter, not all traffic is requested. Which is one reason that the receiver shouldn't pay. Also, big data centers sending massive amounts of data are the in the best position to negotiate good rates and be efficient. It makes sense to lay the cost at their feet.

It's more like if you pay for next day delivery but the delivery company decide that their delivery trucks aren't large enough to keep up with next day deliveries, so they then ask Amazon to buy them larger trucks or their customers' shipments are going to be late.

>There seems to be a trend of adding "free peering" into the net neutrality debate. Unless you're carrying equal bit-miles, free shouldn't even enter into the equation. If data flows unequally, money should flow the other direction.

I think the difference here with CDN provided traffic is the bit miles are basically zero so I don't think that argument really holds up.

I see your point - but it's their content (or content they are being paid to serve at least). Cogent carries data between ISPs regardless of origin/destination for many of their runs -- while they are being paid by their clients for access to their network, Comcast can also push a significant amount of data destined for non Cogent customers through their lines. If you're only offering routes for your customers, you're not offering the ISP anything that you don't benefit from.

I would think rules enforcing net neutrality would explicitly not allow this. If I am a small time host with 20 clients, a host with 5000 clients shouldn't get a direct connection for free while I would need to pay. Both my clients and their clients are equally affected.

> Cogent carries data between ISPs regardless of origin/destination for many of their runs -- while they are being paid by their clients for access to their network, Comcast can also push a significant amount of data destined for non Cogent customers through their lines. If you're only offering routes for your customers, you're not offering the ISP anything that you don't benefit from.

All ISP will only carry traffic that's either from their customers or to their customers.

Heztner isn't arguing that they are large enough to be worth getting a free connection but that their providers Level 3 and NTT are worth it to get a connection to Comcast's customers (not every ISP connected to Comcast) at a reasonable expense.

The issue is that Comcast / insert other residential ISPs have a monopoly on access to their own customers and the large ones can charge for that.

This is not about the size of some content provider, but about not providing the service they bill their customers for. The DSL subscriber pays for an internet connection at n Mbit/s. Note: The DSL subscriber does not pay for a line to their local central office, the product telcos sell to end customers is internet access. Now, these customers who have bought internet access request some data transfer from some server at Hetzner (for example), and Hetzner essentially says "here, we have a few gigabytes of data for your customer, where should we drop them off so you can deliver them?" - and the telco's answer is "we won't, unless you pay us!"

This is not about moving the data to the telco's network, but about the telco delivering the packets to their customers. Cost-neutral peering does not mean free peering, those "content providers" complaining about net neutrality violations do not complain because they have to pay for a line from their datacenter to some location where the DSL telco's network is, so that they can interconnect (they generally are more than willing to pay for that, at least if it's not some location where the respective telco is the only one who could provide the line and would bill unusually high rates for it) - what they are complaining about is that, essentially, even if they were to offer delivering their traffic directly to all the central offices of the telco using their own lines, so the telco only would have to connect them to the DSLs of their customer, those telcos would refuse unless they were paid for it.

And I am not exaggerating: There have been cases where CDNs offered to place cache servers in the telco's network, to pay for the rack space and the power and their servers at normal hosting rates that other hosting customers would pay, so that the traffic originates as close to the subscriber as possible, thus reducing the network costs for wide-area transport by the telco as much as possible, and they refused.

They want to be paid for access to their customers - nobody would complain if they wanted to bill for wide-area transit at normal market rates, or even if they wanted to have the traffic delivered closer to their subscribers, as long as the content provider could choose their transit provider freely, thus buying the transit at market rates. But in practice, they want to be paid unless you installed your own DSL to their customer.

Also very interesting, though german only, unfortunately: http://netzneutral.init7.net/de/situation-init7.php - they report that they tried to get an offer from DTAG for direct peering, and what they got not only didn't match what they requested, but also had progressive pricing: The more capacity they would have wanted to buy, the more they would have had to pay per unit. That is to say: a 10 Gbit interconnect would have been more than 10 times as expensive as a 1 Gbit interconnect. That certainly is not due to costs, hardware for bigger pipes never gets more expensive the more you buy.

> That certainly is not due to costs, hardware for bigger pipes never gets more expensive the more you buy.

That depends. It's currently the case for 100gig (minus xconnect costs) depending on vendor platform. Cisco's yield on their CPAK modules is rumored to be in the low single digits. If you can actually get your hands on a working module, they're crazy expensive right now. People are hoarding them and saying "which customer is this really worth or will pay me the most?"

The Cisco 100gig situation is so bad right now, that there is a very large international carrier looking to replace their Cisco edge with Juniper.

Could we speed the Internet up with more generic caching?

I'd theorize that a decent percentage of all Internet traffic to residential ISPs is made up of HTTP requests for highly cacheable resources. However, it seems like the mythical idea of an automatic caching HTTP proxy at the ISP level never really materialized. Perhaps that was possible in the HTTP days, but HTTPS makes it a challenge.

How much better would Internet congestion be if there was a way for ISPs to cache commonly-referenced resources? They could even be video resources - if Netflix requests its videos across an HTTP connection by identifying a single common resource (a URL everyone will access who plays the video), then an ISP could simply cache the URL and serve it locally.

It seems like HTTPS is a barrier to schemes like that. Instead of an ISP running a generic cache for commonly-referenced resources, each website provider needs to run a local POP that's trusted and knows just their content. (Though maybe Cloudflare like solutions are possible where the POP is not fully trusted.)

If we set HTTPS aside for a moment, could we speed the Internet up with caching? How much of the traffic to saturated links are requests for the same content over and over again?

Is there a conceptual way to achieve some of the privacy benefits from HTTPS while allowing an ISP to cache content in a generic way? Or could an ISP offer a discount for using HTTP traffic instead of HTTPS, and then cache content in a protocol-compliant way?

the entire point is ISPs are acting in bad faith looking for a payday, the limitations are not technical, but purely artifical rent seeking bullshit.

They are not interested in solutions that do not involve paying them money.

I do think it could be neat with the support for a tag for "this is can be cached" in TLS streams, which could be applied to privacy unsensitive material, which identifies the ciphertext blob and thus lets the ISP tell the server sending it "I'm now going to cache this blob, tell every client on my network to use the same decryption key for this blob for X time and I'll hand them my cached copy".

Presumably you'd have to use a signed hash of the ciphertext as the identifier, and decouple the encryption of these blobs from the rest of the TLS stream (use independent key material) to ensure it remains secure. But this way, any ISP that sees the same blob appearing frequently can chose to cache it automatically and quickly start saving bandwidth.

Also, for maximizing privacy, I think this tag should be applied by the client only (can be disabled if you want to reduce metadata leaks) when the server tells the client "this particular data with identifier X can be auto-cached on the network level". One potential neat advantage here is that a LAN router/proxy could implement a cache for its own nodes by having the nodes send these cache requests directly to this router instead of letting the ISP see it, so you don't leak this metadata outside your own network.

A simple way to avoid paying double for traffic is just to host your servers directly at Core-Backbone, instead of funneling your traffic from Hetzner via Core-Backbone to DTAG.

I have a Server in Germany. But my visitors are from all over the world. Will I have to pay to every ISP out there so my data gets to their customers quick?

No, since you will be forced to use transit to reach many of your visitors.

You don't even have the choice of direct connections to all of the world with just a single server in Germany.

There are reach products that will help you connect and peer to providers in remote IX's. You can peer with them directly, then have their traffic ride on your network all the way back. MPLS is a wonderful thing.

Yeah, you'll still need a lot of transit to reach people, but you can also peer with who you want.

He was asking about Hetzner, he has a single server and obviously doesn't operate the network.

Provided "who you want" wants to peer with you. Peering is a mutual relationship after all.

Most public peering IX's have route servers which have "most" of the routes available on the exchange. Peer with that and you have almost all of the routes available on the exchange. Of course, some entities might not advertise everything to the route server, but most do in my experience.

> Most public peering IX's have route servers which have "most" of the routes available on the exchange.

Doesn't matter when we are talking about DTAG, which is the dominant ISP and will not peer for free. Yeah, you can peer with a lot of smaller networks which is nice, but not enough.

Same applies for the US market where the big players refuse to peer.

We need an Open ISP movement... Maybe there is one already but I mean like a non-profit ISP type deal.

DTAG is the Network Operator for most of German Business and Private Customers. Most DSL-Resellers are Business Customers of DTAG.

So maybe there point of view, beeing more important than any public exchange, is not far fetched as it is most certainly a business reality.

Hetzner making a stand for net neutrality is a joke. They are asking your money for better service.

If DTAG asked for 1 EUR/Gigabyte, would you still maintain that position? 10 EUR? 100 EUR? Is there any point where you would consider the behaviour of DTAG inappropriate/unfair?

BTW, nobody is denying that they in fact are an important ISP, where "important" means "lots of customers", which in turn means "lots of power". Someone holding a gun to your head also has a lot of power and thus is important to you - but power in neither case automatically implies that any offer the powerful party is making is a fair offer, be it that they ask you to give them all your money to avoid being killed, or that they ask you to pay massively above-market rates to avoid being unreachable for their customer base.

I would take this a lot more seriously if Hetzner were not home to about 20% of the banned/fraud IP's on my network. I find it highly ironic that they get up on their high horse and start preaching about how to properly run a network (when they completely ignore what's going on inside theirs).

Hetzner is not just saving on private peerings.

Most of their Servers are in east-germany where

  * labor,land,power,darkfibre is cheaper
  * they can harvest developement subs from the goverment [1]
Whereas peering (DE-CIX) is most dense in Frankfurt, West-Germany.

[1] https://translate.google.de/translate?hl=de&sl=de&tl=en&u=ht...

1. What's the logic here? Because a business is trying to reduce its expenses, it's wrong on any matter by default?

2. You are aware that DTAG does not peer at DE-CIX, so whether "peering is most dense" there has exactly no influence on whether Hetzner gets good connectivity to DTAG?

3. You are aware that Hetzner does peer at DE-CIX with 200 Gbit/s, with an open peering policy?

4. You are aware that Hetzner even peers at AMS-IX in Amsterdam with 100 Gbit/s? That link almost certainly is even more expensive than the one to DE-CIX.

see also, peering list close to the end: http://wiki.hetzner.de/index.php/Rechenzentren_und_Anbindung...

1. The idea was to make an example of the company favoring cost effevtiveness over quality. which may not be bad ... but it should underline the cheap-assiness.

2. dtag is ofc present in frankfurt and at de-cix, with little capacity for public peerings as mentioned.

This and your 3/4 each make sense, as they represent a legit business interest for the different sides. Hetzner preffering cheapass open and dtag preffering scalable private peerings.

Problem i see is Hetzner externalizing the cost of private peerings. Which in return plays into 1. and the example of them externalizing other cost factors.

1. How exactly is labor, land, power, or dark fiber lower quality in east Germany (if we ignore the fact that Nuremberg doesn't really qualify for being cheap nor for being in east Germany)? If anything, the subsidies might suggest that operating a business is actually more expensive, which is why it's being subsidised.

5. You are also aware that they operate some 170 Gbit/s of private peering?

6. Which costs exactly are they externalizing with regards to peering in your opinion? So far, your argument seems to be "they don't pay as much as DTAG would like them to, therefore, they are externalizing the difference between DTAG's wishes and what they are willing to pay" - which isn't exactly how you determine externalities.

Without any additional details your contrasting of "cheapass open" vs "scalable private peerings" seems completely baseless. Why is DTAG not the cheapass for skimping on their public pairings?

Actually... both of the Hetzner datacenters are in (former West) germany.

was reffering to geographical, not political east/west

That's not how it works. While there is an east/west divide in Germany, it's along the former border. Bavaria is amongst the most expensive places in Germany in regards to labor and land. Also not more government subsidies than anywhere else in the former West.

Sure, you save a lot by going east. But that means the real east, i.e. the new states.

I'm really dubious of the claim that labor in .de is super expensive. I know that most software devs there make way less thank 100K euro's/year. That's much, much lower compared to the US. Housing costs (buy/rent) in .DE are much lower compared to the US as well.

I have coworkers in Berlin who are pitching me to move over there because it's so easy to live well there.

Edit: I think the problem with "high labor costs" is not monetary, but bureaucratic. It's damn near impossible to get people to work near 40 hours a week. It's hard to get them to do on-call shift work. It's also really difficult to fire people in .de

Note that in this subthread we were just discussing differences in costs within Germany, nothing more, nothing less. Nobody claimed that labor and land was expensive compared to say the US. Only that it is much, much cheaper in Chemnitz than in Nuremberg. Just 200km away from each other but worlds apart in operating costs.

Ahh, OK. Thanks!

Did anyone verify the information? I think it's a hoax to drive cusomers mad/against Deutsche Telekom.

- It's not possible to announce a specific (premium) routing for just a single IP address.

- Hetzner has no DTAG Transit anymore. none. Not even for specific customers. See AS24940

- Current DTAG->Hetzner traffic uses AS33891. See https://f-lga1.f.de.net.dtag.de/

Of course this does not object the non-peering-policy claims and high prices of DTAG in any way. It's well known in the German ISP scene.

Verification for which part? Hetzner used to have DTAG peering but dropped it a year ago because Telecom wants to have a premium paid for it. Hetzner was not willing to carve. This is just the response to some customers wanting to have faster connection to T customers. This story is unfortunately not new.

//EDIT: this thread goes back more than two years: https://telekomhilft.telekom.de/t5/Telefonie-Internet/Perfor...

> I think it's a hoax to drive cusomers mad/against Deutsche Telekom.

You don't need a hoax for that. Everybody in the industry is already pissed at DTAG.

> - It's not possible to announce a specific (premium) routing for just a single IP address.

Wrong. Google source based routing. You'll get asymmetric routing, but that's rarely an issue for content providers.

> - Current DTAG->Hetzner traffic uses AS33891

That's exactly what the article states.

> That's exactly what the article states.

No. Hetzner is not able to provide direct Telekom traffic. Their offer is missleading.

Hetzner makes a profit of 36mio EUR/year. It's a shady move. Telekom sucks but Hetzner is not any better.

What on earth are you going on about?

The article in no way states that Hetzner has a direct connection to DTAG. It very clearly states that they route the traffic to DTAG via Core-Backbone.

> No. Hetzner is not able to provide direct Telekom traffic. Their offer is missleading.

Now you are just outright lying and not reading the text of the article:

> This option costs € 4.20 a month for all of our servers. With this add-on, we will connect an uplink to DTAG via Core-Backbone

> - It's not possible to announce a specific (premium) routing for just a single IP address.

As someone else stated, its technically possible.

Similarly, they could require you to add an IP that is part of the block they've setup solely for customers that pick this option.

> Did anyone verify the information? I think it's a hoax to drive cusomers mad/against Deutsche Telekom.

It isn't a "hoax", DTAG does charge for private peering and their public peering is subpar.

> - Hetzner has no DTAG Transit anymore. none. Not even for specific customers. See AS24940

Its via Core-Backbone as they stated.

> This option costs € 4.20 a month for all of our servers. With this add-on, we will connect an uplink to DTAG via Core-Backbone

Core-Backbone sells private peering to DTAG. Given its an experimental setup, its not surprising Hetzner isn't committing to a contract with DTAG and going through what is essentially a reseller.

> - It's not possible to announce a specific (premium) routing for just a single IP address.

If DTAG allows it, they can accept an advertisement for a single /32 if wanted in their BGP tables, this would allow traffic to that IP address to flow from DTAG -> Private Peering-> Hetzner.

For the reverse, for traffic leaving Hetzner's network, Hetzner would have to set up source based routing which specifically sets the next hop for certain sources/destinations to the router that is peering with DTAG.

This is technically very possible, just complicated.

In order to deliver traffic to DTAG, you don't need to announce anything. This is about traffic from Hetzner to DTAG, not the other way around, Hetzner will accept traffic destined to it whereever it is delivered without asking for payment.

"We at Hetzner Online do not support such policies and declare our support for full net neutrality."

Bullshit. In the same breath they offer something that is entirely antithetical to their stated ethos, no matter how they spin it.

This page demonstrates what they say and what they do are two different things. I interpret this as they're making a goldbrick statement of support in favor of net neutrality with the hope that they won't face a backlash.

I will never touch Hetzner so long as they act against net neutrality.

They are transparent about DTAG not participating in Tire 1 peering for free. They offer a way to pay for this on a per server basis if you don't want to participate in this standoff. I much prefer this to Hetzner silently paying DTAG and increasing the hosting costs for everyone.

When Hetzner say "some Germany ISPs" they actually mean Deutsche Telekom – Germany's largest ISP with almost 50% market share and also the only Tier 1 network.

Deutsche Telekom has been squeezing everyone for transit bandwidth. They're also the only network and ISP to NOT peer with DE-CIX. (they only added their first 10Gbps uplink just a few months ago) I've even seen Akamai bottleneck for weeks on end until they agreed to pay for more 10Gbps ports to DTAG.

What Hetzner Online is offering is merely an effective solution to alleviating a real problem. Supporting net neutrality won't fix your paying users video streaming QoS issues.

As frustrating as it is as a matter of principle, many companies will want to utilize an offering like Hetzner's.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact