Hacker News new | past | comments | ask | show | jobs | submit login
AWS Frankfurt, Germany Region (amazon.com)
293 points by ostrowski on Oct 23, 2014 | hide | past | web | favorite | 107 comments

This is pretty significant, because in Germany many corporations do require their data being hosted on German soil and protected under German consumer protection laws.

As a result the Cloud provider market is currently split into three categories: German corporations (e.g. Telekom) promoting themselves as truly compliant, US corporations with German hosting (Microsoft and Oracle) that self-promote themselves as compliant and US corporations such as AWS and Google that are aggressively attacked by German Cloud providers as violating German consumer protection law.

In the past I personally have lost customers in Germany because my services use App-engine and CloudSQL in Ireland. Thus, I hope Google follows with a German server for their cloud services.

I can see in-EU vs. out-of-EU making a legal distinction, but how is it possible for German consumer protection law to require servers actually in Germany, vs. say in Ireland or Denmark? That seems like it would violate EU treaties on freedom of trade and services within the common market, which usually prohibits both direct and indirect restrictions (e.g. Denmark can't ban EU food imports by imposing unique food-safety requirements). Or is that just a marketing position (people prefer German servers) rather than an actual legal position?

Basically Germany has their own set of consumer protection laws (Bundesdatenschutzgesetz) that are more stringent than the EU laws. There is an ongoing struggle if these laws should be regulated at EU level or at national level since some countries vastly differ in their attitudes in this space and common ground is difficult to establish.

In my personal experience the 'requirement' of some companies that data must be hosted in Germany is their own internal policy rather than something that is prescribed by their reading of the law. However there have been some court rulings where servers hosted in Germany are applied additional restrictions over servers hosted in Ireland.

A lot of this may also have to do with own interests. In the 80s and 90s every EU country wanted US silicon fabs in their underdeveloped regions. Now every country wants hosting farms.

There is an ongoing struggle if these laws should be regulated at EU level or at national level since some countries vastly differ in their attitudes in this space and common ground is difficult to establish.

Oh dear god let it be at the EU level. The only way small EU countries full of semi-corrupt conservatives get decent consumer protection law is when it's an EU law.

As far as I know it is already an EU law and, as with the vast majority of EU laws, each member state can decide to override with additional restrictions (usually under some limitations).

In this specific case, I think Germany is the only exception, but to me it makes perfectly sense: sensitive personal data like medical records should not circulate outside the country.

Even at EU level, how could Germany as a nation guarantee privacy if data is physically maintained in Ireland, where most of the US companies have offices just because of cheaper taxation...? As a country I would't promise that, and as a citizen I wouldn't trust such a promise (side note, I'm Italian so I have no gain/part into this).

> In this specific case, I think Germany is the only exception, but to me it makes perfectly sense: sensitive personal data like medical records should not circulate outside the country.

Having a computer on German soil does not mean that

1) the packets themselves will not travel outside Germany (hint: AMS-IX);

2) people interested in the data contained in that computer will not read it, store it, use it outside Germany.

From the technology point of view, country borders do not exist. (Unless you force a country-wide firewall).

If sensitive data must be properly encrypted. Once it is encrypted you can store it everywhere you want.

I see your point, but the reality is different. Data stored is not data in transit. Both must be protected, but the attacker models are different.

Unfortunately real-world (meaning, not theoretical) encryption is not perfect, thus the sole fact to encrypt is not sufficient to let you store any data wherever you want. At least not in Germany, and at least not from my pov.

To remain in topic with this specific law, data in transit can exit Germany soil, provided that the recipient gives guarantees on its usage (including not store it). This kind of laws should be seen to regulate sensitive (user) data as managed by (big, multinational) organizations, that are thus required to enforce security for both stored and in transit data.

I understand that this may seem silly, but without such laws the landscape would be way worse (consider, e.g., how many personal data are actually traded across the world for ads reasons).

That's not the point. The point is, how is the US government going to bring a case against a German company based on data they illegally intercepted sniffing traffic? If the data were on US soil, they could simply seize the computers once they knew there was offending data on them, and claim it was through an "anonymous tip" that they caught wind of the illegal activity. If the data is on German soil, they can pound sand.

As far as I know it is already an EU law and, as with the vast majority of EU laws, each member state can decide to override with additional restrictions (usually under some limitations).

Yes, but they usually have minimums. Like "Employees must get at least X weeks paid holiday", or "Customers must have a right to return something within at least X days". In countries where X was 0, or there was lots of exceptions, a minimum brings those laws forward.

I'm in the UK, and the only thing holding both the Conservatives and Labour back from using 1984 as a manual is the EU (the Lib Dems may act as a little bit of a brake, but realistically they'll get gutted in the next election)...

The prospect of the UK leaving the EU terrifies me.

Speaking as someone who grew up in Sweden I have a hard time believing the EU government is "less corrupt" than ours. Removing the ability to legislate (on anything) effectively forces every country to a lowest common denominator (that can be voted through) of legislation.

Then again I tend to be biased against centralisation of power so maybe don't listen to me when it comes to the EU.

Not a big fan of the EU or centralisation either, but when it comes to matters of privacy and standing up to US interests, I'll take the EU over Sweden's recent track record.

When it's just about minimum standards, nothing's stopping your legislative to vote for stricter policies -- right? Although I'm often worried that national legislators will point to implementations of the EU minimum standard as being enough and/or more stricter laws being a competitive hindrance.

> small EU countries full of semi-corrupt conservatives

Which ones would those be? From an American perspective, they all look like they're full of semi-corrupt liberals ..

> they all look like they're full of semi-corrupt liberals

What? Sweden and Finland and the Baltic states?

That was my question to OP.

Despite what Fox News may have told you, "Europe" is not coextensive with "France"

I watch Fox News more often than CNN. Which is 'never'.

But good try with the stereotyping and the assuming.

IANAL, but last time I looked, german data protection law regarded the level offered by other states within the european economic area as adequate.

Indeed - although the Microsoft vs US gov nonsense regarding Ireland is really the US gov taking their own cloud businesses out and shooting them, because it makes no difference where the data is if the operating company is HQed in the US.

That said, I wouldn't be so naive as to trust any government not to be attacking cloud providers, but it's important for companies making decisions to understand the jurisdictions their data will end up being regulated by.

You'd also have to be pretty naive to think that the german intelligence apparatus isn't out there actively attacking and subverting cloud datacenters. Google formerly operated datacenters in Munich, Frankfurt, and Berlin, but they're all shut now.

http://www.quora.com/How-many-data-centers-does-Google-have http://www.datacenterknowledge.com/archives/2012/05/15/googl...

Just putting the data on german soil does not seem to help at all, as long as these data are accessible to Amazon personnel located in the US. See this case about data located on Irish servers for example:


In Germany, the individual states have different privacy laws, which are often stricter than the EU minimum. I'm looking forward to AWS being forced to open a datacenter in every German state.

The Bundesdatenschutzgesetz (federal privacy law) is, as the name says, federal law. I have not heard of states imposing their own data privacy laws on businesses. It may however be true that the administrations of each state have slightly differing privacy requirements for their own systems.

The german states have their own different data privacy laws, e.g. see http://byds.juris.de/byds/009_1.1_DSG_BY_1993_rahmen.html for Bavaria.

Those regulate how personal data has to be handled within the state and communal administrations. Those regulations concern private businesses in so far as administrative tasks are delegated to the corporate sector.

My lawyer told me that individual German states have individual privacy laws. This was in reference to my company and the user data of German citizens.

how does that work with eu trade free trade directives seems an out and out illegal protectionist measure

> AWS is fully compliant with all applicable EU Data Protection laws

As long as the NSA can request data from US companies in foreign countries this is not at all compliant with EU Data protection laws at all. Under the current situation ANY US company providing services is not compliant and German companies with sensitive data would be stupid to put this data on US owned servers - wherever they are.

Yep, and exactly that is why I completely moved away from US companies for my hosting. Will I be a target for 'surveillance'? Probably not. Do I want to do everything I can to make the US fuck off from my data? Hell yes.

That's true, EU Data protection law protects Data Centers IN the European Union. Germany signed this as well. http://www.kimeralive.com/privacy

That's great news! As a german based SaaS company, we get many requests from customers asking where the data is stored. Even hair dressers (one of our main customer segment) are very conscious about where their data is stored. I'm looking forward to migrate.

Curiosity peaked. What do you do where one of your main customer segments is hairdressers, yet requires a cloud platform?

Yes, Cutters Lounge (yes, a weird name for native english speakers :) is an appointment booking software. In the beginning we focused on hair dressers, but as we've learned from them in the past years we are about to expand to other industries as well.

This is a fantastic example of how to build a service that doesn't target the usual tech-savvy crowd, but regular brick and mortar businesses, where pen and paper is still the default toolset. These are often overlooked.

I see it everyday: I work at a company that manufactures running shoes in Germany and the retailers we sell to are mostly small, very competent running stores for enthusiasts - not your average national chain like Runner's Point. However these stores barely have any digital inventory/order/customer management solution, use way overpriced point of sale systems and often resort to fax machines when submitting an order. Well, what I'm trying to say is that there are potential customers for useful niche services left and right. It's just not always very obvious.

What you're doing, looks great though.

First of all, thank you for your feedback :) We are very proud, that we bootstrapped Cutters Lounge, other than our competitors which are well funded. This allows us to grow slowly but steady while offering the service for free (we only charge for reminders and invitations).

Germany is horrible at e-commerce. There are still so many opportunities to "disrupt" this sector.

Nice to see something that well made from around here. Currently sitting 800m away from you @ Combinat 56. :)

I remember someone posting in a recurring revenue thread on HN about a similar service for medical doctors some time ago, was that your inspiration?

Look outside the window. I'm less then 30m away from you. 4th floor, other building. Let's meet for a coffee

You've got mail.

So you are competing with patio11, although currently targeting a different vertical ?

Edit: and a different linguistic area

FWIW, this is one of those cases where two services might appear competitive unless you're in one of them. There exist other US-focused companies which do "booking." This is a customer-facing function, and they have to compete on ease-of-use, conversion rates, embeddable widgets, and the like. AR does not do booking and will never do booking.

Why not? Well, you have to have very standardized services which the customer understands to adopt a booking solution. For example, if you're a customer and can say "I want a 45 minute shoulder massage from Cindy", then Cindy's shop can use a booking platform. Most AR customers can't, because the client can't predict how long a dental appointment last, doesn't know that Joe can't come out to his house unless Frank gets the van back in time, etc etc. This is disproportionately the case for upmarket services businesses, which is where AR is moving. (e.g. We want customers with a $100+ value per appointment -- more "professional services" like accountants/medical/HVAC than "personal services" like hair care/massage therapy/etc.)

(I should mention that, even in the hypothetical case that a HNer were in direct competition with AR, I'd be more than happy to see other options available.)

My company is somewhat involved in booking and we've found it to be very specific to the vertical. We looked at building something that integrated with Quickbooks, but found that it's nearly impossible to come up with something that works for enough customers to make it profitable. From what I can tell, people on the Quickbooks team tried it too and decided against it. The long tail just has too many differing needs.

The problem you've listed (not knowing how long a dental appointment will last) isn't really a problem for a booking system...a dental office with a receptionist scheduling patients will run into the exact same issue and the same rules that the receptionist uses can be programed into a booking engine. The bigger problem for a dental office is that the calendar is locked in a management system that's probably running on a Windows computer somewhere in the office. Maintaining two calendars is almost never going to work and the one in the cloud will never be the calendar of record. Short of going the ZocDoc route and having practices reserve certain spots for appointments booked online (businesses hate doing this), you're always going to run into problems with conflicting appointments. The interesting thing is that most dental practices won't care about conflicting appointments since the only patients that will book online will be new patients and patients that have fallen out of the typical schedule. Everyone else will schedule their appointments with the receptionist at the end of their previous appointment. So most dental practices will happily juggle appointments to fit those specific types of patients into their schedule.

But that's the dental industry and almost every other industry has just as many quirks as the dental industry does, if not more. And that's why the market will most likely be filled with smaller, specialized vendors that target either one or possibly a handful of verticals. I'm betting the winners will be the companies that make the management systems used by the businesses, but that's not happening quickly since most of them are small ISVs that only understand Windows development and think cloud computing is something that meteorologists do.

I wouldn't say we are competing. Our service has some features similar to patio11's, but we mostly focus on the appointment planing it self (the calendar).

Hairdressers have to manage customer appointments, accounts, payroll, CRM, inventory etc. Some of that is done online. Source: my wife owns a hairdressing shop in Paris.

The word is "piqued", not "peaked", though the latter does make a certain level of sense.


Both words make sense in context.

Phone number and contact details of all their customers? If that's breeched, then you have a serious data protection issue.

Piqued (just FYI)

The US can still request data from Amazon, regardless of where the server is located, since they are an American country.

I guess, you wanted to write "American company"?

Yes! Dang autocorrect.

Will having servers physically located in Germany really satisfy the privacy concern of German clients given that Amazon is still an American company subject to american laws?

Amazon operating in Germany is subject to German laws.

Legally, all asses are covered, and for 90% that's all that matters, regulatory compliance.

Germany, unlike Ireland, comes with the added bonus that if privacy protection is violated, shit will hit the fan. But that's all it is, a bonus.

Most assume the NSA can get to the data wherever it is, and those very few genuinely worried about that look for protection in encryption rather than legal jurisdiction.

I think the clients care more about the ability to comply with state issued contracts than the actual privacy.

Most likely: no

But better latency is something nice to have.

For those who are curious, this is called "eu-central-1" (Ireland is "eu-west-1").

These namings seem to suggest they don't plan a lot of future expansion (not down to the national level)?

These namings seem to suggest they don't plan a lot of future expansion (not down to the national level)?

Not at all. Look at the US, which has:

us-west-1 US West (N. California)

us-west-2 US West (Oregon)

Or Asia/Pacific:

ap-southeast-1 - Asia Pacific (Singapore)

ap-southeast-2 - Asia Pacific (Sydney)

You're correct. My mistake. :)

For those in London wondering where is best for UK based customers, here's an EC2 ping [1] comparison of Frankfurt and Ireland AWS:

    Europe (Ireland: 25 ms   27 ms   24 ms
    Europe (Frankfurt):	 39 ms   39 ms   42 ms
Suggests Ireland is slightly faster. Obviously just a sample of 1 (more data required), but given Dublin is roughly 300 miles away, and Frankfurt is roughly 400 miles away, it makes sense.

[1] Hitting ec2.eu-west-1.amazonaws.com vs. ec2.eu-central-1.amazonaws.com.

The channel is a good rule of thumb. For consumer isps eu-west should be faster in the uk. On continental europe, africa, and the mideast eu-central will probably be better. The notable exception is france, there it depends on the network as to whether eu-west or eu-central is less latent. Either way its normally +/- 20% or 5ms.

Or just ignore all that and use route 53 latency based routing for your dns records. It will return the record for the least latent endpoint, per client.

Its an 18.57% premium up over US-EAST-1/US-WEST-2 and a 8.57% premium over EU-WEST-1

In case anyone cares

There seems to be an error on the DynamoDB pricing page: http://aws.amazon.com/dynamodb/pricing/

Selecting EU (Frankfurt) I get:

Write Throughput: $0.000702 per hour for every 10 units of Write Capacity

Read Throughput: $0.0001404 per hour for every 50 units of Read Capacity

This is strange as every other region has equal pricing for Write (10) versus Read (50).

Also, Frankfurt's Writes would be ~10 times cheaper than Ireland (Write Throughput: $0.00735 per hour for every 10 units of Write Capacity)

Hi nnx, we are fixing this ASAP.

Maybe I overlooked, but I can't seem to find any information regarding how many Availability Zones it has.

Edit: thanks for the replies, it seems that the '/pt/' localized version of the page hadn't been updated yet. I was able to find the informatin on '/en/'.

The global infrastructure map mentions 2 availability zones: https://aws.amazon.com/about-aws/global-infrastructure/

This is great from a data storage perspective, but i've always struggled to figure out the best approach for utilizing multiple regions to comply with legal issues like this.

That brings me to my question: How do you store your data so that you comply with the laws of a country, when you actually export your product to several countries? Having multiple instances of your system seems impractical and sharding data by country across regions could be rather hard. I.e. I am in Canada, we have US clients who desire their data to be in the US and Canadians who want it in Canada. Either we add complexity or someone doesn't get what they want.

I've always wondered why Amazon put their first EU DC in Ireland, so far away from everything. While Germany is great and all, somewhere more central like Amsterdam would have looked like the obvious choice.

Whatever the location, it's still terribly expensive. Just looking at the Internet traffic charges makes my wallet hurt. I could not affort to serve traffic at any volume from AWS. Luckily there are a lot of other options in Germany.

Ireland was probably chosen because of low corporate tax rates.

This might have been more important when the first location was chosen than it is now, and latencies in Europe are mostly pretty minimal anyway.

Also, highly skilled English speaking work force. A huge amount of other US firms based in the area.

> Also, highly skilled English speaking work force. A huge amount of other US firms based in the area.

It's not like English is a problem in the Netherlands, but if you absolutely needed native English speakers then London with LINX would have been a much better choice for a datacenter.

erik_sub, you have been hellbanned and your comment (reproduced below) can't be seen by anyone.

> Ireland was probably chosen because of low corporate tax rates.

Perhaps you are right. It just sounds incredibly short sighted if true. That's like, to use notax's example location, locating your startup in North Dakota because the rent is cheap over there.

Amsterdam would have been a much better location with better and cheaper bandwidth from a much larger selection of providers. In addition to better infrastructure, Amsterdam has a wider and deeper talent pool for datacenter talent. I also have a hard time really buying the tax argument as Facebook and Google have European datacenters outside Ireland and they seem to manage their Irish tax strategy just fine. Even Apple is reportedly eyeing a datacenter in the Netherlands and I doubt they would consider one if it messed with their Irish sandwitch.

> This might have been more important when the first location was chosen than it is now, and latencies in Europe are mostly pretty minimal anyway.

From experience I can tell you that there is plenty of latency to go around in Europe. Part of that is Amazon's dubious choice of location and part of it is their network. Cloudping easily gives you latencies comparable to east coast - west coast ping times when testing from various European location to AWS Ireland.

Unlike popular entertainment would have you believe, Europe in not a country and neither is it the size of a postage stamp. You'd also be advised to consider that previously Ireland was Amazon's closest location to Russia and that in itself is a pretty big country.

Amusingly enough Amazon's Irish location is almost exactly like putting your datacenter in North Dakota as notax quipped.

Just look at this map: http://www.powercycle.net/wp-content/uploads/2012/03/FireSho...

Frankfurt is home to DE-CIX, which is one of the largest traffic exchanges in the world - i.e. it's pretty central.

Yes, DE-CIX is the second largest IXP, and it's an obvious SECOND choice.

What's not obvious is why Ireland was Amazon's first choice and not Amsterdam which is the premier location with AMS-IX.

In other words I was referring to Amazon's odd first choice in my first message. Sorry if that was unclear.

Logical locations for first batch of EU DCs: Amsterdam, Frankfurt.

Odd locations: Dublin, Frankfurt.

Even starting with London and LINX would have made far more sense than Dublin. Choosing Dublin as your first DC, is like putting your first US DC in some place like North Dakota.

It depends on who youre serving. Ams is nice for western eu traffic. Fra is certainly well connected to the continent also. Additionally fra has much better connectivity to the periphery of europe, including mid east and north africa. Additionally dub already covers the uk, where ams would cut 10ms from your fra times.

So if youve already got dub and you want to cover more of the map fra makes a lot of sense.

> So if youve already got dub and you want to cover more of the map fra makes a lot of sense.

Aye and there-in lies the rub. Frankfurt is a fine second choice and even an excellent first choice. But of all the excellent choices available, why does dubious Dublin have the honor of first choice in AWS EU locations?

It just does not make any sense. Not only it is bad for Amazon, it's very detrimential to AWS users. Given a choice between Dublin and any other major IX location in Europe, I doubt anybody would have chosen Dublin. AWS users just put up with it since they had no choice.

Nowadays DE-CIX is slightly bigger than AMS-IX.

erik_sub, also this comment was invisible due to hellbanning.

erik_sub 19 hours ago | link [dead]

They are fairly equal. According to self-published statistics AMS-IX is nominally bigger in peak traffic and number of members.

Sources: https://ams-ix.net/ https://www.de-cix.net/about/statistics/

A couple of years ago, traffic cost was still high compared to the US, but nowadays it's only a bit more.

I have many servers in Germany and I am able to offer 4 TB / month per VM at no extra cost.

In my experience Germany is far cheaper in traffic costs than the US.

Welcome to Frankfurt. I've found a first sign of a upcoming germany zone some month ago. Here is the posting of it: http://www.nilsjuenemann.de/2014/07/new-aws-region-eu-centra...

You can check your latency to the new region with http://www.cloudping.info/

I'm getting 165 ms from San Francisco to AWS Frankfurt eu-central-1.

More ip ranges to block - are they published yet?

Don't see them here or the subforum, yet


Two important things:

1 - The forum post has been updated and now includes a "/16" IP address range for the new Region.

2 - Please consider taking the time to report suspected AWS abuse using the form at http://portal.aws.amazon.com/gp/aws/html-forms-controller/co... .

Why do you block AWS?

They have unlimited free inbound.

Some people there will destroy your servers with insane spidering rates.

iptables conntrack helps but there are just too many and eventually the firewall takes more resources than the rest of the services you are providing

Did you look at the iptables hashlimit match? It can be used to rate limit per source (inbound) or destination (outbound), without needing conntrack.

Are you hosting stuff that's particularly prone to crawling (and by crawlers that don't respect robots.txt)? Of the spider traffic we see, the vast majority of it comes from Google and the other major search engines.

One example: there are several people who apparently scrape the front page of HN (and proggit, etc.) and then proceed to download all of those links repeatedly every minute (or second!) for several hours. Same link, over and over and over. I can only imagine what get rich quick scheme would require such behavior.

Woah, that suddenly explains why sometimes websites go down so quickly after they get linked on reddit. Surely most hosting won't be able to host 100's of requests, but some times I've seen it happen that websites linked from smaller subs went down quickly.

I do crawling from EC2, and yes, I would not like a 1Gbps traffic spike myself.

Do you deal with generic webpage crawlers that way, or targeted API abuse? Because the first ones can be smoothly shaved away with the help of Cloudflare, for instance.

Maybe they want to control indiscriminate acquisition of infrastructure across many different departments. You'd be surprised how many CFOs/CIOs don't know there is an invisible budget item somewhere in every small department which if added up would be a big item for the whole company.

Of all the services not (yet) available in this reason, the absence of Elasticache seems the most conspicuous. It's a stable mature services with no regional complications.

Can anybody think of any reason for that?

(Maybe it's just me, it's the only missing piece that would stop me from migrating from eu-west-1 to eu-central-1.)

Same here. Maybe they just haven't done it yet.

This is great news.

Does anybody know if there are significant differences between Ireland and Germany, concerning things like privacy and copyright protection? Perhaps there are same laws in EU, which are just enforced less in one country?

Does anybody know if there are significant differences between Ireland and Germany, concerning things like privacy and copyright protection? Perhaps there are same laws in EU, which are just enforced less in one country?

Yes. Irish data protection law is not as strong as other countries. And the government only care about jobs, and promoting the "smart economy". If big tech companies get annoyed at data protection law, they can tell the government that they'd pull out unless things quiet down. The Irish government don't want to "destroy jobs".

Thanks, but I've heard stories about the german Hetzner hosting lots of suspicious stuff and not caring. Especially in minds of russians and other eastern-europeans, Germany has been considered a safe harbor for grey and outright illegal stuff. That was during the 2000s as far as I know.

Is there an up to date german side to this story?

You can even host a Tor relay on Hetzner without issues, but if you receive a SPAM complaint, you'd better reply within the 24 hours they give you...

The s3 buckets in this new region require v4 signing of API requests, FYI. I found that out when adding Frankfurt support to Arq.

Interesting that they don't have m1 instances, which is a problem if you need cheap disk space.

I think some German corporations will still be reluctant to use it due to replication, etc.

I'm not sure what you mean?

Data is usually replicated across datacenters.

any IPs for pinging? I'm curious what the latency is compared to Ireland from here (Eastern Europe).

Pretty rad!

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact