
DDoS protection - benevol
https://wiki.hetzner.de/index.php/DDoS-Schutz/en
======
nik736
I'm from Germany but Online.net is the way better alternative to Hetzner for
me. Server grade hardware, cheaper, no compromises, internal network, etc.
etc. I think they even hired a lot new support people recently so you get an
answer pretty fast. Hetzner always has this "cheap feeling" even though the
hardware looks good on paper.

~~~
comboy
Have you been using online.net for long? Their offer looks interesting except
for availability rate 99.5%. That's almost 4 hours of downtime every month.
within availability range.

~~~
sgt
So they're willing to provide a SLA with 4 hrs of downtime every month. This
does not however mean that you will have 4 hrs of down-time per mont - it's a
managed risk.

Also, depending on the nature of your application, you can often get away with
that kind of downtime. Something trivial (yet extremely popular) like Twitter
had lots of down-time and it made the users love the service even more.

~~~
Drdrdrq
Huh, never heard of that. Care to explain how that happened?

~~~
wtracy
The claim that downtime "made the users love the service even more" is
probably a stretch, but Twitter's error page became a bit of a phenomenon in
its own right for a while:

[http://www.theatlantic.com/technology/archive/2015/01/the-
st...](http://www.theatlantic.com/technology/archive/2015/01/the-story-behind-
twitters-fail-whale/384313/)

I think the main lesson here is that your error messages are part of your UX
and need to be treated as such.

------
davidspiess
We are switching away from hetzner managed server as we had ~30 hours of
downtime this sunday. They suddenly swapped our valid ssl certificate with a
self signed one. They offered no immediate support (i called them three times)
and open tickets were left unanswered for another ~20 hours. As i ranted about
their customer service on twitter, they finally took care of it. So please
stay away from them.

~~~
paride5745
> we had ~30 hours of downtime this sunday

30 hours in a 24 hours day? :O

~~~
davidspiess
From Sunday 11 pm to tuesday morning

------
fivesigma
Cloudflare is a major source of centralization.

The more providers offer something like this, the merrier. I understand that
this isn't a layer 7 solution, but that has it's downsides as well -
Cloudflare (or any other reverse proxy) will MITM all your TLS traffic, for
example.

It's also time to address the elephant in the room: AWS. "Oops, you got
DDOS'ed? Here, have a $50k invoice"

~~~
majke
[disclaimer: I work for CF]

CloudFlare also regularly speaks about attacks and mitigations, therefore is
helping the community to build better defences. Other providers stay shy and
never disclose their magic. We believe DDoS is an internet wide problem and
one of the ways to solve it is to spread the mitigation know how.

Examples:

\- DNS attacks
[https://www.youtube.com/watch?v=UcAygzNSxlI&t=2h13m20s](https://www.youtube.com/watch?v=UcAygzNSxlI&t=2h13m20s)

\- Iptables is great
[https://www.youtube.com/watch?v=pCVTEx1ouyk](https://www.youtube.com/watch?v=pCVTEx1ouyk)

\- Our DDoS mitigation pipeline
[https://www.youtube.com/watch?v=XiK4643YdOk](https://www.youtube.com/watch?v=XiK4643YdOk)

\- BPF for DNS [https://blog.cloudflare.com/introducing-the-bpf-
tools/](https://blog.cloudflare.com/introducing-the-bpf-tools/)

\- BPF for SYN [https://blog.cloudflare.com/introducing-the-p0f-bpf-
compiler...](https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler/)

\- Kernel bypass with netmap [https://blog.cloudflare.com/single-rx-queue-
kernel-bypass-wi...](https://blog.cloudflare.com/single-rx-queue-kernel-
bypass-with-netmap/)

\- NTP attacks [https://blog.cloudflare.com/technical-details-
behind-a-400gb...](https://blog.cloudflare.com/technical-details-
behind-a-400gbps-ntp-amplification-ddos-attack/)

\- DNS amplification [https://blog.cloudflare.com/deep-inside-a-dns-
amplification-...](https://blog.cloudflare.com/deep-inside-a-dns-
amplification-ddos-attack/)

\- Recent attack trends [https://blog.cloudflare.com/a-winter-of-400gbps-
weekend-ddos...](https://blog.cloudflare.com/a-winter-of-400gbps-weekend-ddos-
attacks/)

\- L7 attack with ad networks [https://blog.cloudflare.com/mobile-ad-networks-
as-ddos-vecto...](https://blog.cloudflare.com/mobile-ad-networks-as-ddos-
vectors/)

~~~
scrollaway
Cloudflare gets so much undeserved hate on HN. You guys do amazing work,
provide an incredible service and your writeups are awesome. Thank you.

~~~
kristofferR
Yeah, but they've also the "Wifi Captive Portal" of the internet, which
frequently creates trouble (if you're using Tor for example).

~~~
mschuster91
As a site operator you basically must use Cloudflare and highly aggressive
policies regarding Tor.

I like Tor as a concept, the problem is that there is no way to stop people
massively abusing Tor.

Whatever you want - nazi/hate speech, swatting, trolling, DDoS (by hitting
expensive render paths or by forcing cache bypasses) - operate any kind of
site with user interactions and you will get messed around with by mostly Tor-
using scum, since trolls have understood by now that exposing their real IP
will lead to them being v&d.

I don't like Cloudflare as a SPOF for half the internet, but for now they seem
to be the only one able to reduce the impact of crap you have to deal with as
a site op to a manageable level.

Trolls are destroying the Internet.

~~~
Shank
As a site operator, I have Tor (T1) whitelisted on all of my CloudFlare
websites and have no additional spam problems or anything else. CloudFlare
doesn't even catch all of my malicious traffic (spammers, mostly) -- even
StopForumSpam doesn't do a 100% perfect job.

~~~
meowface
Counter-anecdote: I run a fairly small community and deal with an absurd
amount of problems (spam, child pornography, threats, attempted attacks)
sourcing from Tor and VPNs like PrivateInternetAccess.

I'm very glad CF exists. It certainly doesn't catch all bad traffic in my case
either, but it helps. I don't think any cloud security service can ever stop
all spam or bot activity.

------
cosmins
I used Hetzner, they have one of the worst support teams out there. Everything
is next to unlimited and blah blah blah but the reality is really painfull for
their customers including myself as a former one.

~~~
seqizz
Sadly I have the same experience. They lost me after a small ddos (which
results null routing our ip).

------
esseti
It was about time. So far the approach for DDoS on hetzner was not really
business frendly (let's say so). (e.g.
[https://news.ycombinator.com/item?id=6577465](https://news.ycombinator.com/item?id=6577465))

------
rmdoss
That's a great move by Hetzner. Glad to see that OVH is not the only main
player doing it anymore.

The only issue with both is that they don't handle l7 DDoS, which seems to be
getting more common. I also don't like that they leverage TCP rst's for syn
floods, but I guess thats better than going down.

But so far, for l7 attacks you still need ddos mitigation strategies or
something like CloudFlare.com or [https://sucuri.net](https://sucuri.net) in
front of your site.

thanks,

~~~
kuschku
It seems you’re affiliated with Sucuri, I wonder why you don’t include that in
your profile?

~~~
rmdoss
Because I am not. Just like their service and use it along with CloudFlare
(which I always recommend).

~~~
kuschku
It’s just surprising when more than two thirds of all the comments you ever
made were somehow telling people to use it.

That just creates a wrong impression.

~~~
rmdoss
I can see that. I just tend to engage on threads about ddos/security where I
share what I do and tools I use.

------
stemuk
What always strikes me is the amount of free traffic Hetzner includes in their
plans, always 20TB and upwards.

AWS charges a whooping 90$ per TB, it keeps me wondering why their traffic is
SO much more expensive than Hetzner's...

~~~
corobo
Can you actually use that traffic though? I can run AWS bandwidth at max for
the entire month and (assuming I'm not spamming or whatever) I just get a
bigger bill.

With places like Hetzner you may be shut down for abusing their services if
you use that limit too frequently.

~~~
Medowar
nah, wrong. If you go over the allowed bandwith, you can get either limited to
10Mbit/s or pay 1,37€ per TB. In comparison, this is pretty cheap, but that is
about waht you pay, when you buy transit. Source:
[https://wiki.hetzner.de/index.php/Traffic/en](https://wiki.hetzner.de/index.php/Traffic/en)

~~~
Medowar
correction, 1,17€/TB

------
zAy0LfpBZLC8mAC
Free? How about mandatory? I hate blackboxes in my communication channel that
decide which communication is acceptable and which isn't. Plus all the
security vulnerabilities and other bugs that the complexity of such systems
probably brings with it, making it harder and harder to debug network
problems.

And all that is especially true in the case of Hetzner, who have repeatedly
demonstrated how to build hilariously broken networks. Even ignoring that they
were vulnerable to ARP spoofing for a long time and other things that have
since been fixed, just some blunders of their current offerings: virtual
servers behind v4 NAT (yes, you can't make that shit up ...) and IPv6
assignments of a single /64, even for dedicated machines (with an option to
extend it to a /56 for money). So, if their new blackbox screws something up,
what are the chances that they will care?

------
Kephael
Hetzner announced this a couple weeks ago on their German language forums.
What's concerning is they still have not stated their filtering capacity and
the bandwidth and throughput protection levels being provided. If it's only 5
gbit and 500k PPS of filtering it won't be all that helpful.

------
Namidairo
If I remember correctly, didn't this host have their entire address space
banned from a couple IRC networks because their support wouldn't act on abuse
reports?

------
StanAngeloff
We used to run some basic infrastructure on Hetzner. The support was appalling
– any requests for help were swiftly met with short answers such as
"Unfortunately we can not help you here". That makes me question how
responsive and understanding Hetzner's staff will be in case of an on-going
DDoS that their automated systems are unable to detect and take care of.

What sort of scenario would benefit from the announced DDoS protection? People
are surely not running websites on commodity hardware.

~~~
latch
What type of support issues? I had servers with hetzner for years and I never
had any problems, which also means I never contacted support, so I can't say.

I'm trying to figure out if you were expecting support for things that most
dedicated, colo and cloud providers wouldn't support (OS updates, install
nginx, ....) or if these were clearly provider issues (hardware and network).

    
    
       > People are surely not running websites on commodity hardware.
    

Huh? That's been the growing trend for...decades? It feels like the two places
where pure server-grade equipment is used, high end dedicated hosting and
collocation, are losing market. I'd go as far as to say high-end dedicated
hosting is flat out dying (yay!).

~~~
StanAngeloff
It didn't come across in my comment – my amazement was regarding WordPress,
etc. websites being run on commodity hardware without proper backups. E.g.,
get that ludicrous 128GB machine and run Apache on it with a single website
for years. What could possibly go wrong?

The most recent issue we had with Hetzner was when we requested a LARA remote
console to connect to a failing machine. The standard keyboard on the remote
is German and much of the non-Latin keys are remapped. It was impossible to
log in to a shell. All we got from support was "This is a standard Generic 104
key layout". That was it.

Before that Hetzner rebooted a whole raft of machines and on one of them this
corrupted ext2. We asked what prompted the reboot (not bothered by the dead
server) and we got (as above) "Unfortunately we can not help you here".

There were many more incidents before that and at the end it was just too much
to swallow so we moved away.

Just out of interest, what's your use case for Hetzner?

~~~
latch
Read-heavy APIs, which I like to setup as:

A central location to take writes. Writes to this location go into a database
and a durable, at-least-once, queue (writing to the queue is as important as
writing to the database).

Then you put your API servers in different geographies (and preferably
different hosting vendors). They listen to the queue and update their own
storage (which could itself be a full relational database).

Edges aren't caching layer, since all the data is expected to be there (so, no
miss). But certainly some of the data could be loaded in this way. The edges
can also take some writes and send it back to "central", but you have to not
care too much about the consistency (and maybe even accuracy) of that specific
data.

Throw a latency-aware DNS on top of it (anycast, geo) with a short TTL and
good health checks and you don't care too too much about uptime, allowing you
to focus on raw price and performance.

Ironically, the only time I remember a serious outage (at a single location)
was during a DDoS against a specific location at a much more expensive
provider (not-rackspace-but-the-other-one-you're-thinking-about) which, of
course, null routed us.

~~~
StanAngeloff
Sounds like a well thought out CQRS system. Thank you for taking the time to
reply.

------
psynapse
I host at Hetzner and have noticed reduced Fail2ban notifications and shorter
Logwatch emails recently.

If this has kicked it, I guess that might explain it.

~~~
ryanlol
That'd be somewhat worrying, DDoS protection shouldn't be dropping non DDoS
packets. If the chinese bruteforcers can't get to your server there's a very
good chance your users can't either.

------
bogomipz
Can someone from the EU or DE talk about where Hetzner sits reputation-wise
for those not familiar with them. Are they a solid provider?

~~~
easychris
We run a SaaS business there on ~25 servers with Hetzner. Besides small issues
we are pretty happy. Feel free to email me if you want to know more details.

Funnily I saw a thread on the customers forum at forum.hetzner.de today about
why Hetzner does not have a better reputation social-media wise. Seems not to
be on their priority list. There is even an English subforum, you may try to
register there and ask for experience by other customers.

~~~
discopicante
Social media is a wet rag in Germany: few companies see the point to invest
with the relative low user engagement. It's typically seen as a distraction
and/or too compromising on privacy.

There is also the 'German' view of customer service to consider (i.e. the
customer is not always right).

------
vizzah
Does it mean Hetzner won't pull the cable off your server now if/when it's
under heavy DDoS? I read horror stories about customers being attacked and
Hetzner disabling their servers. Never experienced it myself, but I host my
playground server with them and chose OVH for a more serious project due to
DDoS policies..

------
DisposableMike
I wish that Hetzner would provide me with DDoS protection from THEIR users.
I've blacklisted their entire IP space for several different customers because
of the relentless onslaught of malicious and aggressive attacks originating
from their networks. They have never responded to a single abuse notification.

------
jpalomaki
I see great future for companies like Hetzner which are providing dedicated
hardware with reasonable prices. While Amazon and Azure are making the live of
virtual server providers difficult, I believe it is harder for them to compete
with companies like Hetzner without eating their own business.

What these companies need are just additional services and nice management
tools which make the whole dedicated server and network stuff look more like
what we see on AWS or Azure.

My assumption is that at some point more people will come to the conclusion
that _part of their workloads_ such actually run on cheap dedicated hardware.
On Hetzner €60/month buys you 4 cores, 32GB, 480GB SSD, 30TB traffic. Compare
that to Azure A2, which at €64/month gives you 2 cores, 3.5GB, 60GB.

------
anc84
Official announcement: [https://www.hetzner.de/hosting/news/hetzner-online-
bietet-um...](https://www.hetzner.de/hosting/news/hetzner-online-bietet-
umfassenden-ddos-schutz) (german)

~~~
onestone
Also in English: [https://www.hetzner.de/ot/hosting/news/hetzner-online-
bietet...](https://www.hetzner.de/ot/hosting/news/hetzner-online-bietet-
umfassenden-ddos-schutz)

------
znarfor
This is what I expect from any serious hosting company know: DDoS protection
for every customer as part of the service. It's part the hosting duties and
should not be offloaded to any third party company, say Cloudflare.

------
ex3ndr
Recently moved from Google Cloud (8x less price - 4x more power) and we got
~100kb/s of constant traffic from countries like china. fail2ban usually locks
me for day or two until i disabled password-based auth and fail2ban.

~~~
dx034
Disabling fail2ban doesn't sound like the best solution. Why did it lock you
out? If your username is not public and root has no password the locks
shouldn't affect you.

------
cosmins
I wouldn't buy DDoS protection from Hetzner, not to talk about using it for
free. There is no such things as free ddos protection which means tons of free
bandwidth, this cheap marketing.

------
jensC
I always liked Hetzner and I have a few dedicated servers there. Their service
was always fast and very competent. However they are not soo cheap compared to
other hosters anymore.

------
daap00
I'd have to say that webtropia.com or even myloc.de great service great value
myloc has some real nice bells and whistles. webtropia has great vps and the
dedi servers too. I'm at 2 locations east side USA and Deutschland. Check it
out, I'm happy.

