Hacker News new | comments | show | ask | jobs | submit login
Hetzner Private Cloud (hetzner.de)
60 points by stemuk on May 21, 2017 | hide | past | web | favorite | 83 comments

We have a lot of experience with Hetzner servers and I used and recommend this hoster since at least 10 years intensively with tens and hundreds of servers.

The most important thing with Hetzner Servers is to monitor everything very closely:

- CPU Temperature


- Disks (SMART)

- Software- and Hardware raids

- Network (interface) errors

The servers are usually consumer-grade hardware components which have more often issues under heavy load so you have to expect down-times and broken components. However, if you are aware of that and you can easily shift around that with your software Hetzner will save you serious money (10 to 15 times cheaper than GCP and AWS). Also to mention is that their customer support is first class if you tell them all required details and exactly what to do. Usually they respond in minutes and do hardware replacements within an hour and small downtimes.

What do you think of their more 'server-like' offerings like the PX61 line [0]? What do I have to expect in terms of downtime and component replacements?

[0] https://www.hetzner.de/de/hosting/produkte_rootserver/px61nv...

Those are actually (I think Fujitsu) Workstations. You should expect the same downtime you would expect from your own workstation if you were to use one at home. Maybe a bit less downtime since there would be less startups and the temperature differences would be smaller as well. Obviously, server grade hardware will perform and last longer under heavy loads.

I believe if you don't go with the Server auction thing, then you get quite new hardware. So I guess you would not really have to expect a lot of downtime. But as always, downtimes can happen. Always plan accordingly.

Absolutely, the new servers are ok but after a while of permanent load (a year or so), you'll also have to expect issues, at least in our experience. The server auction itself is worse but they replace hardware without any issues and most often the replaced hardware is new as well.

In bit more detail, PX61 has Fujitsu D3417-B1 motherboard.

What do you mean by RAM monitoring? Just keeping an eye on how much RAM you are using? If so how does this relate specifically to Hetzner?

As for temperature monitoring: how do you act on events? If the server runs heavy workloads for a while it will get hot, yes. Heck sometimes it goes above a threshold (say, 80 C) for a minute for seemingly no reason (maybe it is the period software update job?). But what do you do when you get a temperature notification? Shut the server down? For every such event? I am currently leaning towards not monitoring temperature at all because if the hardware breaks Hetzner will replace it anyway (I have backups).

I've also stumbled upon that. However the question remains, how to add more compute nodes? Is there a easy process to do that? How about updating it? Isn't it carless to actually say:

    You will have sole and unrestricted administration
    rights to the dedicated hardware with root access.
    Hetzner Online will not have access to the servers, and
    will therefore not be able to provide server
    administration support.

I mean if I can't provision it, it's probably problematic to update it.

You are right, it's not so much private cloud on bare metal as bulk metal you manage.

OTOH, if that's what you want (many enterprises think they do), this positioning and landing page puts Hetzner into consideration.

This isn't quite hitting the spot to me (non-server hardware, you still administer the OS, single location), but it does seem indicative of a potential new wave of low-cost cloud providers.

Someone like Vultr, Linode, or DO is positioned well to deliver an "AWS lite" offering. They all have decent hardware, lots of locations, and a good delivery history. A bit of work to put together ELB/EC2/Lambda/S3 equivalents and a control panel would open up a new market. Especially if they offered low egress pricing.

For me it's on spot, I don't like Amzn, goog, msft blackbox. This is a decent HA system for 90% startup adding just backup will make it perfect. It's better then ELB/EC2/Lambda/S3 as it uses openstack api under your own control. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).

I just don't see how it's any different than what they had yesterday. You can buy two cheap non-server-hardware servers and put openstack or coreos on them and administer it yourself.

All they did here was pre-install openstack, but leave you to keep it updated. If you can't install it yourself, good luck upgrading it when a security release comes out.

I mentioned ELB/EC2/Lambda/S3 equivalents for 2 reasons. First, AWS is clearly the market share leader, so having a similar pattern might get more buyers. Second, it's a bit easier pattern for apps that aren't cloud aware. But, they could just offer hosted K8S with some add-on ingress controllers.

I also don't get what this offers over renting these machines individually. It can't really be some preinstalled software with no support whatsoever as installing usually is the easy part.

Installing is easy, yes, but installing right can be tricky for many folks. There's a ton of conflicting info on best practices. Not detracting from your point, I agree it doesn't make sense, just saying installing right can be hard too.

Vultr, DO, OVH are moving towards "AWS Lite". They offer or are rolling out their own solutions for object storage (S3). Block storage (EBS). Load Balancers (ELB).

Check out OVH Labs they are in the process of deploying many more services https://www.runabove.com/index.xml

OVH labs routinely spins stuff up and then shuts it down, so that's not a great path. It seems more like a product testbed, and not many products make it.

Which of these has S3 or ELB equivalents? I haven't seen that. I saw DO has failover ip addresses.

Linode has load balancers available. I wouldn't call them ELB equivalents, but they do work as adequate LBs for many needs.

Ahh, yes. That's nice. Did a bit more searching and DO has them as well. So they are headed that way. Will be interesting to see if they both fill the other gaps.

HAProxy is so easy guys... its like configuring nginx

Sure. The context here, though, is rooting for someone to make a low cost AWS alternative. We can roll our own cloud, but that's a different topic.

I'd argue the config syntax is easier with HAProxy, but that's not the point. Folks using Linode / DO typically don't want to manage yet another thing.

Nowdays. In the 1.5 days regidel and such was hard.

I have been happy with using Hetzner services in the past and I wish I had a business use for this. I used openstack for a while on IBM bluemix and found it convenient enough. I would like a good backup recovery setup. I hope that they configure the systems for some redundancy between the two servers, but it is not clear from the ordering page.

It's totally unsupported. If you were not able to just rent the tin and configure this yourself then you're probably not going to be able to maintain it long term. This product is dangerous and some people will learn that too late.

As far as I understood it the OpenStack software will be automatically updated on the releases of the LTS channel, so calling the product dangerous seems like an exaggeration to me.

Have you run openstack? There are million dollar companies out there that just support maintaining any updating openstack for a reason...

Their list of pieces and parts gives a good insight to how complicated the beast is: https://www.openstack.org/software/project-navigator#tiles

I worked in/on Openstack for 3.5 years. It's next to impossible to run it without patches, so it's questionable if they can do auto-updates on releases of the LTS channel. Someone needs to sit down and migrate patches and remove any breakages.

auto-updating openstack on production systems people don't know how to run themselves sounds pretty much like the most dangerous thing you could possibly come up with.

why not just hand your systems over to a smart 15 year old kid to run, and go take a nice beachside vacation in the meantime? probably would work out better.

What do you mean by 'tin'?

Physical servers, with an implication that it's litrally just the hardware and no service level on top. Similar to when people talk about bare metal servers.

Why 'tin' though? Do you mean tin as in the element tin? Why tin specifically? There's hardly any tin in a server is there? Except for the solder.

It's as in "tinpot", British slang meaning low value or little importance.

For many the detail of the hardware does not matter Eg. "Once the packet hits the tin it's no longer my problem". Also talking about a £50k pile of servers as just "tin that needs to be racked" could be a backhanded way of signalling confidence in the task, where others might be intimidated by the cost of the equipment they are dealing with.

Desktop processor without ECC memory plus very slow SATA hard drives - not a setup for reliability and performance

I don't get it for 90% of startup this setup is more than enough to start and far cheaper then goog, Amazon, azure. Just the problem is most modern devops are lazy to learn the basics of setting a HA system with proper backup. Hertzner make it easy but one needs to know what they are doing. I am happy customer of hetzner for 4 years which is better given personal data privacy better in Germany then in USA(initially that was the primary reason of choosing them, to get away from usa based data center after Snowden).

So what, I save $100 over Google but spend 3h a week futzing with configs?

I’ve run the numbers with Hetzner, GCP, AWS, and even more managed products such as Firebase, Heroku, etc.

Bare Metal compared to GCP or AWS will save you about 75% of your operating costs.

Bare Metal compared to Heroku or Firebase will save you around 90% of your operating costs (a large part of this being caused by bandwidth, which is massively overpriced at Google and Amazon).

With those savings you can usually serve 5 to 10 times as many customers.

That might not be worth it if you’re in SV, and pay your devs the same wages Amazon or Google do (as then you’ll just pay more than with AWS or GCP), but if you’re in places where you pay half the wages Amazon or Google do4 , you can actually save a lot with this arrangement.

Total server spend on my last startup was like $150 a month on GCE. So saving 75% is useless if it cost me even 2h.

Fwiw I started on DO and migrated off to get better stability and block storage / proper load balancing. Which DO has since added.

You mist have fit into the free egress tier. Maybe something that could be cached by Cloudflare or similar?

The egress costs for AWS, GCE, and Azure are usually the big cost driver.

Was a b2b app with ARPU of say $100. Our total all in hardware cost for that same user was maybe $.10.

So saving on server cost literally did nothing. Our employee cost where 20x, 100x our hardware costs.

Not all startups aim for millions of free uses you will make money off of later.

Definitely. But if hardware is the only expense that scales per user – as is with most B2C apps, and even many B2B products – the difference between GCP and bare metal can make you last a lot longer on the same seed round.

(The show Silicon Valley actually showed that part quite well)

Except as I told you not always.

If we optimized and saved $100 per month on hardware at the cost of even 1 engineer hour, it would be a net loss.

I think a LOT of startups are very simple from the hardware level. You can go real far on just a few servers. And at that level, who care who manages them? Do what is easier.

I am really reluctant on setting up my own server cluster on bare-metal right now, can you recommend some specific open-source tools to manage and monitor these clusters?

Kubernetes on CoreOS is a good option. Tectonic makes the setup part relatively easy.

Ok, thanks a lot. Since we're on it, is there any open-source software available that is specifically targeted at easily setting up storage servers (maybe with an S3 like SDK)? I found myself pretty much forced to use GCS because of a lack of good alternatives.

I haven't used it personally, but Minio [0] is something I've heard mentioned more than once.

[0] https://github.com/minio/minio/blob/master/README.md

Minio is very nice, but sadly it only allows one set of credentials – I’d have preferred being able to separate the access permissions of each service, but that’s not supported.

You will likely save a lot of headache to just use AWS or GCE. Running your own kube is a multiple week project.

I was a total noob at this, and started setting up a kube cluster on 10th of May, and I managed to set up a working cluster by the middle of the week. It’s actually not as complicated as expected.

I still agree with you, because no one should take setting up and running kube lightly – it’s actually a lot of work.

This is good, it means kube is about 100x easier to set up than openstack.

On the other hand, Google will run your Kube for free up to 5 nodes.

There’s actually even a hobbyists guide for running kube yourself: https://github.com/hobby-kube/guide

And there’s a few guides for setting it up automatically in vagrant, or on aws, or on Juju-managed clouds.

Sadly, Kube is very barebones – only doing scheduling, if you add a storage provider such as portworx (only block storage) or rook (block and object storage) you also gain that, but it doesn’t do any user auth functionality.

That’s something I’m atm still searching for, something allowing me to easily register and remove users, add them to groups and do RBAC, but without using ldap as backend, and with a way to get it working with OpenID Connect and OAuth 2.0, as well as with Shibboleth, Kerberos, and an LDAP compatibility layer (but with app-specific passwords there).

You can give Proxmox a try too.

Sorry to break your bubble a vm with this configuration+bandwidth will be $1000 or more. So it's 1/10th to 1/5th over Amzn,goog or msft. Also if you are a startup you need to understand technology driving it. Blackbox doesn't help, what's necessary for goog isn't first priority for startup.

> Sorry to break your bubble a vm with this configuration+bandwidth will be $1000 or more.

But why would you buy a VM with this configuration? 32gb of RAM for two boxes, with two i7s that have 4 3.40ghz cores? Lots of people have the impression that cloud is more expensive than owning your own iron, and it probably is, but the difference is made a lot larger if you don't consider TCO and you make no effort to actually adapt your needs to the cloud. Cloud's economics are based around horizontal scaling and provisioning on-demand. If you're hosting in the cloud, you don't get two big machines like this, you shard your app over a bunch of cheap machines and you spin up more/bigger instances to meet peak demand. If you're buying cloud hosting but you go into it with "have everything we need to meet peak load running 24/7," of course it's going to cost you an arm and a leg.

Most startups I know don't need anywhere near that full-time processing power. So what does it matter if it's 10% of the price if it's 85% more than I need?

most modern devops are lazy to learn the basics of setting a HA system

Absurd nonsense.

Our Buildsystem (Jenkins) needs CPU and RAM. Couldn't bother less with SATA Performance.

Frontendserver also: Only needs CPU and RAM.

depends, if you do have a database heavy test suite, you probably need fast disks. our application really needs a good i/o performance. I tried to run 3-4 runs on the same machine with containerized cpu/memory, however as soon as 3 tests are running the i/o latency/performance goes numb, only thing that helped was either using different disks or faster disks.

OpenStack. Way overkill. I would advice always against OpenStack unless you have very good reasons to go into this direction.

The company I'm working for moved away from hetzner coz it has a single location and far too many noisy neighbourhood. Hetzner was often target of DDoS and our connectivity was also affected.

I used Hetzner like 10 years ago and didn't had good experience. I doubt anything changed over time. They offer home grade hardware and low priority support in return of cheap prices. They only deal with hardware and network issues (which is quite normal for Dedi service with normal support option)

- Hardware they offer is more prone to fail because of using home grade hardware for long time (especially HDD).

- It's almost impossible to convince them HDD is failing even with showing SMART logs. Hardware needs to fail so they will replace it.

- Hardware replacement times are quite fast (thanks to SLA). They replace it with another used HDD, if you want something newer, than they ask some money for replacing with less used HDD.

- They scan their network regularly for hosted malware, trojan etc. so if one of your sites get hijacked and has iframe viruses etc. Hetzner will null route your server.

- If your IP gets DDOS, null route.

- If you get DMCA warning, null route without waiting 24 hours.

- If your NAT leaks your internal traffic to WLAN, null route.

- It takes almost few day to lift null route ban on your server when you get in contact with support. It's okay for support tickets to wait in queue for long time because of service level but I believe null route tickets needs priority no matter what.

We decided to move over to another provider after having problems.

Hetzner also owns few other brands like Serverloft.

> Hetzner also owns few other brands like Serverloft.

"serverloft ist ein Produkt der Host Europe GmbH."


doesnt say so on the english website thou

Hetzner do not own Serverloft. Serverloft are owned by Host Europe. Hetzner are entirely independent. https://www.serverloft.eu/company/

Why would a German company do DMCA?

good question ...

another german hosting provider also required you to at least respond to dmca + three strikes

DMCA 24h?! It means I could 'troll' someone out of business doing bogus complaints? At least out of their server.

Here I was enthusiastic, but it's not-quite-there. If they took up administration of the Openstack setup this would be a very interesting product.

the auction servers seem like better bang for the buck

Depends on what you want. It's used (for many years usually) Hardware which increases the probability of failure and you only get whatever single configuration is currently available. It's nice though that they offer you to cancel at no charge for 14 days and the variety of configurations available which are often more asymmetric compared to their usual offerings, e.g. if you primarily need a lot of disk space you don't have to pay for a lot of RAM and a fast CPU.

That doesn't sound like an issue though if you just need something cheap for now to just get going. You can keep in mind that it could fail any time and just plan around that (for now).

But they're aimed at individuals who want to buy something cheaply. Larger business customers will want to be able to say we want X of Y machine and will pay the premium to get it.

True, but for the sake of easy management I would still imagine the OpenStack setup to be better. Hopefully they will add some configuration options that make it possible to choose every server type in their lineup to be configured as OpenStack cloud.

We've noticed a larger-than-normal number of disk failures on second-hand auctioned systems. Be sure to get one with a fresh, "enterprise-grade" drive.

Hetzner: Dedicated Root Servers AMD Ryzen : https://www.hetzner.de/us/hosting/produktmatrix/rootserver-a...

* AMD Ryzen 5 1600X

* AMD Ryzen 7 1700X

But it is in germany - sort of police state in term of Internet. No thanks.

In Autumn 2017 there's going to be a new Hetzner datacenter in Finland [0].

[0] https://www.hetzner.de/it/hosting/presse/info-0916

I used Hetzner services until I received an email from there about some of their servers being infected in the RAM.

Here is the email that I received in 2013:

Dear Client

At the end of last week, Hetzner technicians discovered a "backdoor" in one of our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration interface for dedicated root servers (Robot) had also been affected. Current findings would suggest that fragments of our client database had been copied externally.

As a result, we currently have to consider the client data stored in our Robot as compromised.

To our knowledge, the malicious program that we have discovered is as yet unknown and has never appeared before.

The malicious code used in the "backdoor" exclusively infects the RAM. First analysis suggests that the malicious code directly infiltrates running Apache and sshd processes. Here, the infection neither modifies the binaries of the service which has been compromised, nor does it restart the service which has been affected.

The standard techniques used for analysis such as the examination of checksum or tools such as "rkhunter" are therefore not able to track down the malicious code.

We have commissioned an external security company with a detailed analysis of the incident to support our in-house administrators. At this stage, analysis of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database as Hash (SHA256) with salt. As a precaution, we recommend that you change your client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.

Hetzner technicians are permanently working on localising and preventing possible security vulnerabilities as well as ensuring that our systems and infrastructure are kept as safe as possible. Data security is a very high priority for us. To expedite clarification further, we have reported this incident to the data security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and trust in us.

A special FAQs page has been set up at http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further enquiries.

Kind regards

Martin Hetzner

This really needs a website that has a more modern feel to it. It doesn't exactly evoke trust.

Check out Berkshire Hathaway's website. http://www.berkshirehathaway.com

You've really done a great job when the site explicitly has to state "Official Website"...

They are one of the largest server hosts in the world.

that doesn't have anything to do with my comment.

if you care more about websites with a "modern feel". I'm sure companies like amazon and google are more than happy to charge you 10x more money for it.

Chridal is right though, there's lots of SQLi bugs on Hetzner. Not something you get on AWS or GCP.

Hetzners website looks like an old shitty PHP hack because it very much is one.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact