
How to destroy someone who hosts stuff at Hetzner dedicated server - turshija
I&#x27;m using Hetzner services for several years so far (luckily only for personal stuff and friends minecraft server), and had this problem few times. Every time I said to myself &quot;I will get away from Hetzner ASAP&quot;, but I always stay there. I would NEVER imagine to run a business hosted there at all, and here is why...<p>DDoS is a common problem many companies are facing, but Hetzner&#x27;s policy on that is really crap.
If someone starts DDoS on your dedicated, after several minutes they just shut down your dedicated from network, and send you an email like &quot;We disabled your network because you have DDoS attack on your server. Write us an email to reenable your network&quot;. And of course, several hours later I saw that email and tell them &quot;Okay, please enable my network&quot;, but boom, I will have to wait Monday, because their support that can ACTIVATE network on a dedicated works only from Mondays to Fridays ... 
And then the person who attacked me sends me anonymous email like &quot;lol, I bought 5$ packet at [some random booter&#x2F;network stresser website], and I have put you offline for few days for only 15 minutes of DDoS, HAHAHAHA&quot;<p>So basically yea, start small flood from random VPS&#x2F;dedicated or whatever that is 100mbit or more, leave it on for several minutes until Hetzners system automatically disable network from person you are attacking, and look at them being offline for few days :)
I&#x27;m ordering a new dedicated from someone else now, no more Hetzner...
======
MehdiEG
It's worth putting this in context. Hetzner provides really beefy dedicated
servers for ridiculously low prices [1].

You get great support (always had phone calls answered pretty much instantly
and emails answered within a few minutes and all the techs I've dealt with
knew what they were doing).

You can issue automated hardware resets and even get a remotely-controlled KVM
attached to tweak the BIOS or regain access to your machine if you messed up
the networking config (usually only takes a few minutes to get the KVM
attached).

Orders for new hardware are also really fast - dealt with within the hour and
often in under 15 minutes.

But there's no such thing as a free lunch. If you host at Hetnzer, you have to
be aware of the reasons why they're so cheap, namely:

1) The servers are 100% unmanaged. They'll install new hardware for you if you
ask them but everything else is up to you.

2) A lot of their hardware is desktop-grade, e.g. Intel Core i7 CPUs and non-
ECC RAM. They do have some server-grade hardware in their high-end range
however.

3) Their servers are in Germany. So you get quite a bit of latency if accessed
from Asia or the West Coast of the US (see [2]).

4) They don't have any DDoS protection. In case of a DDoS, your server will
get null-routed (but they tell you first). Again: 100% unmanaged. Up to you to
deal with it. I've been lucky enough to not have to deal with a DDoS but my
first port of call would probably be CloudFlare it it happened.

Provided that you're happy to do some sys admin, Hetzner is brilliant for a
personal server, a CI server or even a prod server for a bootstrapped startup.

For literally next to nothing, you get a really powerful machine that will
easily handle big traffic spikes without a breaking a sweat. And dedicated
machine means that you get excellent and consistent CPU performance and disk
I/O. If and when your startup takes off and you get funding, you can then
choose between hiring a sys admin or moving to a more expensive host that
offers a more managed setup.

[1] [http://www.hetzner.de/en/hosting/produktmatrix/rootserver-
pr...](http://www.hetzner.de/en/hosting/produktmatrix/rootserver-
produktmatrix-ex)

[2]
[https://news.ycombinator.com/item?id=3898714](https://news.ycombinator.com/item?id=3898714)

~~~
wazoox
ovh.com and online.net provides even cheaper servers, but with a DDoS filter
by default. I'm very happy with both.

~~~
MehdiEG
I actually went to OVH a couple of days ago. I needed a new server and wanted
to give them a try. But they're not accepting new clients anymore:
[http://www.ovh.co.uk/a1186.SoldOut](http://www.ovh.co.uk/a1186.SoldOut)

I didn't know about online.net. Shame I didn't see this a few days earlier. I
would have given them a try.

~~~
dt3ft
OVH does not take new orders the usual way, but if you are buying as a company
you can still get your order through by calling them or emailing them and
saying how you plan to stay longer than, say, 3 months.

------
Duckeh
A lot of the people commenting don't seem to understand how hard it is to fend
off such DDoS attacks. You either need some serious infrastructure (cloudflare
style) or you need to buy equipment to mitigate attacks (like radware devices)
or route it via a DDoS mitigation service (prolexic style). The one thing all
these solutions have in common is that they are insanely expensive. People can
buy a 1 gigabit DDoS for only a few bucks, whereas mitigating a 1 gigabit DDoS
will cost you either $20K+ dollars for a mitigation device or some stupid
amount of money to have a service like prolexic mitigate it for you. Services
like cloudflare are a whole load cheaper but only provide basic reverse proxy
protection and still leave your server vulnerable for attacks directed at it's
IP instead of DNS name.

I can't say I've ever heard of Hetzner, but from the comments I'm reading they
apparently offer servers for cheap. Bearing in mind how much money DDoS
mitigation costs I don't see how they could handle this any other way without
having to make some pretty serious investments (which in turn would make their
hosting less cheap as the money has to come from somewhere, right?)

~~~
patrickg_zill
You can do some of it via BGP, which is a standard method for handling routing
once you become any sort of server provider with multiple bandwidth providers.
It is builtin to some Juniper devices already, for instance:
[http://njetwork.wordpress.com/2013/04/30/mitigating-ddos-
att...](http://njetwork.wordpress.com/2013/04/30/mitigating-ddos-attacks-with-
bgp-flow-specification/)

There are other ways to do it via BGP also. Plus there is null-routing,
bandwidth limiting, etc.

~~~
devicenull
juniper is the only one that provides that and flowspec is not going to be
able to block everything. Other then that BGP is not really going to help with
attacks.

------
metabrew
IRCCloud had to move off hetzner for this reason. We were continually getting
ddos'ed, and hetzner showed no interest in working with us to try and
mitigate.

At one point they just suggested we "ask the responsible parties to stop", and
closed the ticket.

Now we are on Black Lotus. Expensive, but the regular 50mb-10gbit ddos attacks
are mitigated just fine.

~~~
metabrew
...however, if you aren't concerned about ddos, I still recommend hetzner.

Excellent value for money dedicated servers, with good automated systems. You
can remotely reboot a dedicated server into a recovery image and fix problems
yourself. You can run the install process yourself too, so you get exactly
what you want... except ddos mitigation.

~~~
turshija
Yes, and also except for the hard drives ... I got dedicated server which had
hard disk problems after less than 1 month (both hard disks in raid were
almost dead), and they offered me to PAY additional fee to put hard disks
which is not new o_O (they claim it has less than 1000 hours operation, and
asked me like 20-30 eur per HDD, I'm not sure honestly), or to give me another
one for FREE but it was possibly also crap ... I asked them to put free one,
and I had to reinstal everything and I had to manually backup all data,
reinstall system and set everything up from scratch because the RAID was not
an option with 2 "damaged" hard disks ...

~~~
mb0
Did the disks have SMART errors to begin with? They're a budget provider, you
can't expect brand new hardware with every new installation. Though, I'd agree
that if the drives had a high number of reallocated/offline sectors, CRC
errors, and the like, then hetzner was at fault. However, given that the raid
array initialized & ran fine for a month, it seems that it could have just
been a case of disk failure.

As for making backups & reinstalling the system yourself. You should have
expected that. Hetzner is not a managed provider.

------
spindritf
Yup, pretty much. Those attacks have become a real problem because they can be
ordered so cheaply and easily that even kids use them in Minecraft feuds. The
channel takeovers of the 21st century.

OVH's much more tolerant in that regard (ie. they keep your server online if
battered) and all their servers now include a mandatory anti-ddos
protection[1]. Unfortunately, they're fighting turn-over and don't accept new
orders.

[1]
[http://forum.ovh.co.uk/showthread.php?t=6661](http://forum.ovh.co.uk/showthread.php?t=6661)

------
level09
That sucks. I have moved many websites recently from EC2 to Hetzner. what they
offer is really impressive and the difference is clear (probably 5x more
resources/power for 25% of the Amazon price).

I guess I will still keep the server, but will have to work on a quick
migration/failover plan in case I encounter something similar.

I have also started using cloudflare as my default DNS host, so that could
also be a possible solution.

~~~
turshija
Cloudflare doesn't help if they DDoS your server's IP directly ... You can
also "hide" your IP by activating CF on all subdomains (the orange cloud
thingy), but people always find a way to find server's IP and attack it (the
CF doesn't help there at all, they only filter packets that are going through
their servers which your domains resolve to).

~~~
x3sphere
How would they find the IP if you don't have it used in any DNS records?
Unless Cloudflare exposes the real IP at times and you've taken all the proper
preventative measures I don't see how this is possible...

~~~
zzzcpan
There are ways. You could use services like domaintools and get IP history if
you did use any of the IPs in the past. You could get the IP from e-mail
headers, if the website sends e-mails during registration, password recovery,
etc. You could look for ways for a server to make a request somewhere and log
its IP, like posting an image on a forum, some forums do that. And this is
just off the top of my head.

~~~
x3sphere
Right, in most cases though those holes are easily plugged. When switching
over to CF don't use an IP that was ever public-facing for your site, use
distributed systems like Amazon SES for sending email, etc. I imagine the
things you mentioned do go overlooked by some when fighting off an attack,
though.

------
oellegaard
So I manage quite a few servers at Hetzner and we were DDOS'ed quite a few
times. First, they warn you and if you don't get back to them in 12-24 hours,
_then_ they will shut down your server.

Sounds like you were unfortunate, but this is not generally what they do.

~~~
leokun
What would you do after they warn you? It's not really under your control to
fix is it.

~~~
manmal
You can boot up some EC2 instances.

~~~
darklajid
Help me understand...

What would those instances do, exactly?

~~~
klapinat0r
Depending on the DDoS attack, they'd either help you load-balance or keep
serving "real" users. That is, either the attack is on Hetzner only, in which
case actual users would be redirected (e.g. you have multiple A records for
your domain, some for Hetnzer, some for those AWS instances) to a working
site. Or the attack is on everything related to you, in which case you'd
utilize some load balancing to mitigate the volume of the attack - and
depending on AWS' DDoS protection, the AWS part of your site might still be
up, serving real users.

~~~
turshija
What would happen if I put for example 3xA records to 3 different IPs (in
different locations), and one of them gets offline ... Does that mean all
other traffic will go to remaining 2, or some of them will try to load the
site from the offline IP ?

~~~
agwa
Web browsers will try the other A records if one of them fails, though it
could potentially take them a while before they realize the host is offline.
If the client receives an immediate error when trying to connect (such as
connection refused or ICMP destination unreachable) the failover will be
instantaneous. However, if packets to the downed host are just being dropped,
the browser might sit there for 30 seconds waiting for a timeout before
failing over. It's therefore best that you remove the downed host from the DNS
as soon as possible (ideally from an automated monitoring process) and that
the A records have reasonably short TTLs so the bad record doesn't remain
cached for too long.

~~~
addandsubtract
So load balancing via DNS records is the easiest way to deal with the problem,
at the cost of (potentially) long failover times. What would be the next step
to load balance your servers? I know Amazon offers elastic load balancing for
their platform, but if I'm not using AWS or don't want to rely on them
exclusively, then what would be my best course of action to load balance
between 2-3 different VPS?

~~~
agwa
You should think of load balancing and geographic redundancy as two separate
concepts, because they have two different best solutions.

For load balancing, the best solution is to put all of your servers at the
same provider, in the same datacenter, and proxy connections through a load
balancer. On AWS, you can use ELB. Outside of AWS you can roll your own load
balancer using software like haproxy.

For geographic redundancy, DNS round robin is good, but I must emphasize that
to do this properly you really need to have a short TTL and an automated
monitoring process that removes downed servers from the round robin. If speed
of failover concerns you, you can set your TTL really low (like 30 seconds) at
the cost of slower DNS lookups. You have to strike the balance that suits you.

You can of course combine these - use DNS for redundancy between geographic
locations, and at each location use a loud balancer. Note that each location
needs to be able to handle more than its share of the traffic, not only
because DNS round robins produce a very unpredictable load distribution, but
also so a location doesn't get crushed under the load if another location goes
down. Theoretically each location should be able to handle 100% of the
traffic, but you can play the odds and skimp a little.

(Note: the very best way to do geographic failover is to get your own portable
IP address allocation and an AS number, and use BGP to announce your IP
address allocation from your active datacenter. If it goes down, you start
announcing from your backup datacenter. However, only big players can afford
to do this. For the rest of us, DNS-based failover is as good as it gets.)

------
Qantourisc
Here is a simple solution and everybody is happy: re-enable it every hour, if
DDoS continues, disable again.

Everybody is probably "happy" then: Customer-> their unusable DDoSed server is
disconnected, but wasn't reachable anyway. But once the DDoS is over, it's
back online. Provider -> they have their traffic routed to null. However, they
will have to do some more work to get this working too. And not to mention
happier customers.

------
codexon
Here is a forum that sells DDoS attacks. Attacks are much cheaper than
protection.

[http://www.hackforums.net/forumdisplay.php?fid=232](http://www.hackforums.net/forumdisplay.php?fid=232)

~~~
verroq
You are kidding me right, nobody takes skidforms seriously.

~~~
turshija
[http://quantumbooter.net/](http://quantumbooter.net/) this one actually
works, that one is used by many kids, and even the one who attacked my server
... :S the prices are very cheap, and I've bought lowest packet to test it on
my 1gbit server, opened ssh, started iftop to check for traffic and see ddos
strength, activated ddos using SSYN type, and BOOM, connection time out from
my dedi ...

------
andrew_wc_brown
I had to do deal with DDOS attacks in the past and DDOSArrest worked like a
charm to mitigate the problem.

~~~
qohen
Link for convenience:

[http://www.dosarrest.com/](http://www.dosarrest.com/)

BTW, does anyone know how what their prices are like?

(Their site doesn't have seem to have pricing info, just "Get a Free Quote"
forms.)

------
csense
How can DDoS mitigation devices distinguish between legit and malicious
traffic? I'm not a networking expert, but it seems to me that if you're a
website hosting a big file like the latest Ubuntu release, a legitimate client
will say:

    
    
        GET /ubuntu-13.10-server-amd64.iso
    

and cost you 500 MB of traffic (or however big the ISO file is).

A DDoS is nothing more than thousands or millions of machines saying:

    
    
        GET /ubuntu-13.10-server-amd64.iso
    

How do the solutions others are talking about in this thread (DDoS mitigation
provider or specialized hardware) tell the difference between DDoS traffic and
legitimate requests?

~~~
turshija
That is something different, it is only used to waste bandwidth from someone
(or potentially clogging server's upload, but its easy solvable), but in big
DDoS attacks the attacker usually has several hundred thousands of zombies
infected in his botnet, and then he orders all those zombies to spam packets
at an IP he orders ... Every infected PC uses his maximum upload to target IP,
resulting into something like this: [http://d.pr/i/kmAn](http://d.pr/i/kmAn)

If I'm online during the attack and check iptraf or tcpdump, I can see
literally hundreds of different IPs spamming random stuff at me, completely
overflowing my download until I get totally disconnected from server (time
out), and I can do nothing about it, just watch it being offline ...

------
lb0
Wow, they detect the DDoS, but instead of blocking this they take off the
servers?? Sounds ingenious..

Or are they unable to properly detect a DDoS and would also take off a server
that hosts a web page mentioned on Hacker News?

How do other hosters handle this situation?

~~~
4hthth4
From what other people are saying, it sounds like Hetzner is the Walmart of
service providers. You wouldn't see a traffic jam in the Walmart parking lot
and then become indignant that they didn't have valet parking automatically
start up to clear the parking lot traffic.

------
_s
Use cloudflare or a similar service provider to mitigate such attacks?

~~~
swinglock
Not everything on the Internet is a web server.

------
linas
We had the same problem at Hetzner, the server was attacked on Saturday. We
moved out. Hetzner is very cheap and you get what you pay for.

~~~
turshija
Yep, I would like to move out my files from Hetzner at this moment, but my
server is locked, and I will have to wait Monday to get access to it ...
Luckily I'm not hosting anything important on it and my business doesn't rely
on them, or else I would be screwed very hard ...

~~~
jorgeregula
You can unlock a IP address over the admin panel to get access to your server.

------
ianhawes
Great tip. Does anyone know who Hetzner's largest customers are? Or at least
major web services that host with Hetzner?

~~~
aroch
They have a "notable customers" page:
[http://www.hetzner.de/en/hosting/unternehmen/referenzkunden](http://www.hetzner.de/en/hosting/unternehmen/referenzkunden)

There are a couple NBA.com subdomains and Der Speigel appears to be hosting
'local' CDNable content for their Germany-based readers

------
AznHisoka
Does this apply to servers that do NOT host websites? I host databases in
Hetzner that aren't hosted in the same server as the website(they're in
another provider)

~~~
vertis
In theory if they can find it, it can be DDOS'd. But not hosting public facing
servers makes it much less likely to be a problem.

------
Demiurge
well this is good timing, just moved to hetzner last month and server
mysteriously went awol yesterday until a reset...

~~~
turshija
Google this: hetzner hard disk failure They are putting faulty hard drives in
their servers, and if you notice its faulty and tell them, they replace it
with another faulty one (less faulty if you are lucky) ... Make regular
backups to servers outside Hetzner network...

------
bolder88
FWIW, This is fairly standard.

Linode for example will null-route your linode for 24 hours if it's attacked.

It's quite irritating that hosting companies seem to see null-routing as a
solution to a DDoS attack.

~~~
devicenull
What alternative would you propose? With a virtual machine an attack on one
instance can effect everyone on the same machine. Also actually blocking the
attack is very expensive.

~~~
toast0
It's worse than that; a DDoS can also overwhelm the networking infrastructure,
affecting other machines on the same switches, or in the same facility
(depending on the magnitude of the attack, and the capacity of the
networking).

Null routing does a good job of mitigating impact to other servers, but
obviously causes problems. If there's enough capacity, filtering at the border
would probably work. I think most of the attacks these days are DNS reflection
(because it's easy and effective), so if the border routers could be
configured to drop incoming udp from port 53 to the IP under attack, that
would get you most of the way there (just make sure the server under attack
doesn't need to get port 53 replies from the internet). That sounds simple,
but it has three big problems: a) You need a lot of spare input bandwidth. b)
You need to be able to filter on border routers. c) You need to be able to
safely change the filters on the border routers.

~~~
devicenull
DNS reflection, chargen reflection, and SNMP reflection. You can block them
all fairly easily, but you still need enough upstream bandwidth to deal with
them.

------
patrickg_zill
If they can detect the DDOS, they should be able to mitigate it, right?

(EDIT: of course Hetzner could choose to mitigate the DDOS by any number of
methods - but they choose not to, because they have made a conscious decision
based on cost.)

~~~
lucb1e
It's like a traffic jam. You can solve it by stopping the incoming cars at the
ramp or by making the road wider, but both are outside of your control when
you're just running the toll gates at some point ahead. You can only ask your
"host" to do that (in this example, the owner of the road).

