
Hetzner Servers Compromised - martindale
http://wiki.hetzner.de/index.php/Security_Issue/en
======
925dk
As part of the registration process with hetzner.de, you have to send them
scans of personal documents (such as passport, drivers license or similar).

I asked them just now if these systems were compromised and they promptly
replied:

"The system that stores scans of ids, credit cards and so on was not
compromised. In addition to that, we delete that information after 21 days."

~~~
larrys
"you have to send them scans of personal documents (such as passport, drivers
license or similar)."

What? Seriously?

"In addition to that, we delete that information after 21 days."

Hmm, do they delete the info that ends up on backup copies? How do you know
they even actually delete it in 21 days? It's not like there is a third party
even auditing which you can rely on. (Not that I'd ever do that for something
like this anyway, I would just find another provider.)

~~~
925dk
Yes seriously, here is what they asked me on first order with them:

"Since you're a new customer with Hetzner, we ask you for a scan of your
passport or ID card (authenticity check). It's only necessary for your first
order with us.

Please send the scan by fax or as an email attachment."

When they say they delete it after 21 days, as they did in the mail I've just
received, I trust them. I find their communication on this matter, as well as
previous matters, open and serious.

~~~
warrenm
not true for all customers - I needed to send them nothing of the kind (in the
US)

~~~
925dk
This was dedicated servers (root servers they call them) - and I'm from
Europe.

~~~
werid
they haven't been doing this forever, so if you've been a customer for a
while, you might not have been asked.

they also don't ask business customers (might not be true for all countries)
if they supply certain details about their business.

------
ChuckMcM
One of the differences between servers in data centers and desktops is that
servers don't reboot a lot. So the bad guys can build an in memory system that
doesn't change anything on disk (which avoids the configuration management
system from flagging it) and if it cloaks itself as a normally long lived
process then active monitors might be fooled as well. It then runs,
effectively undetected, until the server reboots which can be months (or even
years).

Tools to protect against those threats are going to have start taking into
account process activity and footprint.

------
raylu
Hetzner is very open, transparent, and exact in their communication about
this. I'm not a customer but I really like that they didn't try to dress
anything up and were forthcoming with information.

~~~
chappi42
Doesn't surprise me. I'm a customer and the whole experience feels like this.

------
pekk
Several events with Linode, now Hetzner. These are relatively "premier," high-
quality hosting companies, you can count on thousands and thousands of
companies to pay even less attention.

Yet every time, the discussion is only about one specific company, without
seeing any broader pattern.

When are we ever going to draw the conclusion that popular hosting companies
(and, actually related, facilities like RubyGems) are especially attractive
targets, and that the approach of waiting for an exploit and then shaming the
targeted company is not an effective way of getting better security?

~~~
saturdayplace
Is it just the customer account details that apparently make hosting companies
attractive targets. If that's the case, I'm wondering why we're not seeing
more breaches from all over the e-commerce world. Why just hosting companies?

~~~
ahi
Maybe the hosting companies are simply more likely to detect a breach. Not a
happy thought I know.

~~~
gizzlon
And more likely to disclose it perhaps?

There are a lot of breaches that you will never hear about because they go
undetected or undisclosed.

------
gokhan
The info in the mail regarding the safety of credit card info contradicts with
the linked FAQ.

FAQ: _Bank details are encrypted (two-way) in the database. However, it cannot
be excluded that the attacker/s have also been able to obtain access to the
key._

Mail: _With credit cards, only the last three digits of the card number, the
card type and the expiry date are saved in our systems. All other card data is
saved solely by our payment service provider and referenced via a pseudo card
number. Therefore, as far as we are aware, credit card data has not been
compromised._

~~~
perlgeek
Hetzner probably has many customers who pay with debit card/recurring direct
debit instead of credit card. It's quite common in Germany.

~~~
thejosh
They also allow PayPal, which I recently changed to with Hetzner. 99% of my
payments to websites are PayPal, for this very reason.

------
FooBarWidget
Sigh, another hosting provider hack. As if Linode and OVH are not enough. This
is the reason why we use full disk encryption, where we enter the key manually
during boot. This way we're protected against many types of hosting provider
hacks.

~~~
epochwolf
How does this help? The key is still stored in memory which I assume the
hypervisor has access to.

~~~
tbh
Hetzner provide a lot of physical machines too, I believe this is what the
other poster was talking about.

~~~
FooBarWidget
Physical machines don't prevent keys from leaking out. A physical attacker can
analyze power usage usage patterns to extract the encryption key. :)

~~~
lawl
I don't think you have that sidechannel with AES-NI. Besides, as a physical
attacker cold boot attack would be much easier Or if the server has any
interfaces with DMA, like PCI or something, that's even easier.

------
anoother
Full text of the email sent to cutomers:

 _Dear Client

At the end of last week, Hetzner technicians discovered a "backdoor" in one of
our internal monitoring systems (Nagios).

An investigation was launched immediately and showed that the administration
interface for dedicated root servers (Robot) had also been affected. Current
findings would suggest that fragments of our client database had been copied
externally.

As a result, we currently have to consider the client data stored in our Robot
as compromised.

To our knowledge, the malicious program that we have discovered is as yet
unknown and has never appeared before.

The malicious code used in the "backdoor" exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
been affected.

The standard techniques used for analysis such as the examination of checksum
or tools such as "rkhunter" are therefore not able to track down the malicious
code.

We have commissioned an external security company with a detailed analysis of
the incident to support our in-house administrators. At this stage, analysis
of the incident has not yet been completed.

The access passwords for your Robot client account are stored in our database
as Hash (SHA256) with salt. As a precaution, we recommend that you change your
client passwords in the Robot.

With credit cards, only the last three digits of the card number, the card
type and the expiry date are saved in our systems. All other card data is
saved solely by our payment service provider and referenced via a pseudo card
number. Therefore, as far as we are aware, credit card data has not been
compromised.

Hetzner technicians are permanently working on localising and preventing
possible security vulnerabilities as well as ensuring that our systems and
infrastructure are kept as safe as possible. Data security is a very high
priority for us. To expedite clarification further, we have reported this
incident to the data security authority concerned.

Furthermore, we are in contact with the Federal Criminal Police Office (BKA)
in regard to this incident.

Naturally, we shall inform you of new developments immediately.

We very much regret this incident and thank you for your understanding and
trust in us.

A special FAQs page has been set up at
<http://wiki.hetzner.de/index.php/Security_Issue/en> to assist you with
further enquiries.

Kind regards

Martin Hetzner_

~~~
amarraja
Have all Hetzner customers received this mail? I currently have a couple of
servers with them and have received nothing yet.

~~~
moepstar
AFAIK yes - it might take a while though, i've gotten mine quite a while after
the first ones seemed to have gotten them..

------
tbh
Considering <https://twitter.com/omgtbh/status/337567604887658496> I can't say
I'm surprised...

(tweet text reproduced here: "I asked Hetzner if they plan to support 2 factor
auth & was told that they already do - they require a username _and_ a
password. Seriously.")

~~~
aw3c2
You don't seriously think that a server admin said that, do you? It was
probably some low rank customer service peon.

~~~
tbh
Of course not, but regardless of who in the company said it, the official
support response was as quoted. Not exactly encouraging!

~~~
MrCheese
To be fair, Hetzner has very "google translaty" English support for simple
matters, so it is not impossible that they simply didn't understand the
question

~~~
PaulFreund
Exactly, and somebody who didn't yet hear the term can easily come to the
conclusion that it is an authentification with two "factors" which means
basically two things and that would be username and password. In the past
there were system with only one factor...

------
berkay
Things seem to be getting worse there. A server we have there just went
offline, and their management servers are not responding. I wonder it's part
of the exploit or as a result of attempts to "fix" the problem

~~~
zapt02
I had 5 minutes of downtime, but everything seems to be back in working order.

------
alexvr
Am I an overly-cautious nubcake, or do any of you also refrain from clicking
the link when it says, "We've been compromised. Come visit our site!"?

If I were a mean dude who hacked into some prominent site, the first thing I
would do is submit a link to HN that points to a malicious "article" and laugh
as malware gets thrust into thousands of computers.

~~~
ricardobeat
Way too much mork. If you have a way of infecting computers by simply visiting
a webpage, just tag it some kind of porn and post on reddit & other forums.

------
X4
Seriously, again??

I think such a fauxpas shouldn't be tolerated twice.

~~~
M4v3R
Do you have link to their previous hack?

~~~
atesti
In October 2011, unfortunately these links are in German only:

[http://www.heise.de/security/meldung/Web-Hoster-Hetzner-
geha...](http://www.heise.de/security/meldung/Web-Hoster-Hetzner-
gehackt-1356501.html)

[http://www.heise.de/ix/meldung/Datenleck-bei-
Hetzner-1356468...](http://www.heise.de/ix/meldung/Datenleck-bei-
Hetzner-1356468.html)

[http://www.heise.de/security/meldung/Passwortklau-bei-
Hetzne...](http://www.heise.de/security/meldung/Passwortklau-bei-Hetzner-
ueber-FTP-1358172.html)

There was unauthorized access to customer data, even passwords. Someone claims
to have accessed it via an FTP-server where he found a root password to a
management server

~~~
X4
Thanks for posting the links, I've forgot to mention it in all my wisdom,
duh..

------
jtchang
The lesson to take from this is you can't rely on someone else providing
security for you. You need to make sure you "layer" your security. If your
database is compromised make sure you are salting your passwords or at least
using a hash that is designed to be resistant against cracking.

This doesn't protect against someone who has root access to your box and
manages to capture passwords in the clear as they are being transferred but it
does limit the surface of a breach.

------
rebelde
Something was happening with their routers, too. Using PingPlotter to one of
my servers shows severe packet loss on the hos-bb2.juniper2.rz19.hetzner.de
router.

I see it just cleared up for now. Still, I'm moving my traffic to other
locations till this blows over.

~~~
PaulFreund
That might be because various parts of germany are flooded at the moment. One
of Hetzners datacenters is located in Falkenstein near Regensburg and others
might me affected too

------
muxxa
I've got a login for robot.your-server.de and have changed my password but
don't seem to have one for konsoleh.your-server.de.

Is the konsoleh login/account something that would have gotten created for me
automatically? i.e. do I need to worry about it?

~~~
linx
You need a konsoleH account for the registration of 'special' domains not
covered by the robot. One doesn't have one by default.

I think webspace users get a konsoleH account, too.

------
sgt
I use the Robot to access a VNC console of my leased server. I can even do a
CTRL-ALT-DELETE and boot the OS in single user mode from it, if I only have
access to the Robot.

Doesn't this imply that my server is potentially compromised as well?

------
tonetheman
hmmm... what does Hetzner do?

~~~
dspillett
They are a provider of hosted dedicated servers, colo and virtual machines
(with or without server management services), based in Germany.

~~~
samspenc
They're really inexpensive, which explains their niche popularity...

~~~
Gravityloss
They're very big, not just a niche. Also reliability / support is not
necessarily always so much worse than some higher price solutions.

~~~
dspillett
Aye, they seem to have a very good reputation for a cheaper provider. I've not
used their services myself though.

------
illumen
Their SSL certificate is fucked for their security page... oh dear.

<https://wiki.hetzner.de/index.php/Security_Issue/en>

Hope everyone has fun with their identity theft cleanups. They ask people for
ID scans.

Bank details are compromised.

boo!

This was announced AFTER the German work day was done. Really nice for admins.
Thanks.

~~~
StavrosK
You would have preferred that they waited until tomorrow?

