I asked them just now if these systems were compromised and they promptly replied:
"The system that stores scans of ids, credit cards and so on was not compromised.
In addition to that, we delete that information after 21 days."
"In addition to that, we delete that information after 21 days."
Hmm, do they delete the info that ends up on backup copies? How do you know they even actually delete it in 21 days? It's not like there is a third party even auditing which you can rely on. (Not that I'd ever do that for something like this anyway, I would just find another provider.)
"Since you're a new customer with Hetzner, we ask you for a scan of your passport or ID card (authenticity check).
It's only necessary for your first order with us.
Please send the scan by fax or as an email attachment."
When they say they delete it after 21 days, as they did in the mail I've just received, I trust them. I find their communication on this matter, as well as previous matters, open and serious.
they also don't ask business customers (might not be true for all countries) if they supply certain details about their business.
If you're worried about someone stealing your entire backup system then you have bigger issues.
This shit is annoying. I think it have been only 6-8 months since the managed server part of Hetzner (KonsoleH) got hacked. Now the VPS/root server part (Robot) got hacked. I understand that both incidents are completely different and it seems that they might've learned a thing or two from the KonsoleH-hack, but still. My address data and my bank data are very likely to be compromised.
But then, changing the hoster doesn't make any sense. My data is somewhere out there, can't get any worse I guess.
Tools to protect against those threats are going to have start taking into account process activity and footprint.
Yet every time, the discussion is only about one specific company, without seeing any broader pattern.
When are we ever going to draw the conclusion that popular hosting companies (and, actually related, facilities like RubyGems) are especially attractive targets, and that the approach of waiting for an exploit and then shaming the targeted company is not an effective way of getting better security?
I do appreciate them being forthcoming and sharing some security details that do make me feel pretty safe. I don't care if they have the last few digits of my CC number, and I've already reset my password. Stinks I had to even think about this, but again, Chevy Malibu.
Compromises will happen sooner or later when running a large number of public facing services. With good policy, the breaches will be well contained, which seems to be the case here. What more can be expected out of a premier hosting company, let alone Hetzner which is very cheap budget provider?
There are a lot of breaches that you will never hear about because they go undetected or undisclosed.
FAQ: Bank details are encrypted (two-way) in the database. However, it cannot be excluded that the attacker/s have also been able to obtain access to the key.
Mail: With credit cards, only the last three digits of the card number, the card type and the expiry date are saved in our systems. All other card data is saved solely by our payment service provider and referenced via a pseudo card number. Therefore, as far as we are aware, credit card data has not been compromised.
account numbers / routing numbers / company names / po boxes/ physical addresses / account history and other billing info.
I'd assume they're saying some of these might be compromised, but specifically that credit cards aren't.
I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part.
But I'm not a security expert. Maybe I've overlooked something.
Properly hard problem, as you've pointed out.
I strongly doubt that:
Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices).
As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard.
I still think that it mostly defends against someone stealing/cloning the physical disks though.
Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.
I suppose the sanest middle ground is to wait for the dust of an event like this to settle (or have someone investigate, in case of "just" an unexpected reboot) -- then wipe the server(s) in question, reinstall (hoping the install media is ok) -- and importing the disks/backup.
I still think it is little gain for a lot of work.
Basically if your server software is secure, you've avoided data compromise due to a reboot into single user mode or similar -- but you have to assume (for the sake of guaranteeing the safety of the encrypted data) -- that attackers have a copy of the data at rest (which they can get if the can boot into a recovery image etc.
So now you have to make sure that you can boot a safe environment in which to import the data to, without leaking your key(s).
All the while, it is rather likely there are a few vulnerabilities in your software stack that all this never did mitigate.
I'm not saying don't do it -- I'm in favour of doing it for the (relative) peace of mind that no that will get lost/out as the servers are recycled -- I'm not sure it makes much sense to avoid hackers -- the online threat seems to dominate for most cases I can think of.
It's relatively easy on Linux, where /boot is usually not encrypted.
This is the same methodology I use, and I think about these attacks a lot. There is no good/cheap way to verify a remote execution environment right now with commodity hardware. :/
It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it.
If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks.
If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.
At the end of last week, Hetzner technicians discovered a "backdoor" in one
of our internal monitoring systems (Nagios).
An investigation was launched immediately and showed that the administration
interface for dedicated root servers (Robot) had also been affected. Current
findings would suggest that fragments of our client database had been copied
As a result, we currently have to consider the client data stored in our Robot
To our knowledge, the malicious program that we have discovered is as yet
unknown and has never appeared before.
The malicious code used in the "backdoor" exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
The standard techniques used for analysis such as the examination of checksum
or tools such as "rkhunter" are therefore not able to track down the malicious
We have commissioned an external security company with a detailed analysis of
the incident to support our in-house administrators. At this stage, analysis
of the incident has not yet been completed.
The access passwords for your Robot client account are stored in our database
as Hash (SHA256) with salt. As a precaution, we recommend that you change your
client passwords in the Robot.
With credit cards, only the last three digits of the card number, the card type
and the expiry date are saved in our systems. All other card data is saved
solely by our payment service provider and referenced via a pseudo card number.
Therefore, as far as we are aware, credit card data has not been compromised.
Hetzner technicians are permanently working on localising and preventing possible
security vulnerabilities as well as ensuring that our systems and infrastructure
are kept as safe as possible. Data security is a very high priority for us. To
expedite clarification further, we have reported this incident to the data
security authority concerned.
Furthermore, we are in contact with the Federal Criminal Police Office (BKA) in
regard to this incident.
Naturally, we shall inform you of new developments immediately.
We very much regret this incident and thank you for your understanding and
trust in us.
A special FAQs page has been set up at
http://wiki.hetzner.de/index.php/Security_Issue/en to assist you with further
I just wish that they used bcrypt instead of a salted SHA256 but at least it's salted (and anyway in my case I never reuse passwords so no big deal).
If they fall under PCI:DSS, they might not be able to use bcrypt since it isn't an official recommended standard. (I am of course assuming that they mean PBKDF2 when they say salted SHA256.)
From http://security.stackexchange.com/questions/4781/do-any-secu..., I gather this: bcrypt is neither a NIST or FIPs standard, but PBKDF2 is.
> For any business required to comply with U.S. NIST or FIPS standards, bcrypt is not a valid option. Check every nation's laws and regulations separately if you do business there, of course.
However, from a comment on this other answer: http://security.stackexchange.com/questions/11552/would-it-m..., someone mentions this:
> As far as I know, PCI doesn't specify actual technology, especially Encryption specifications. The legal cases I've seen support what's technically feasible, not technically ideal or perfect. I'm sure either one [bcrypt or PBKDF2] meets PCI requirements
An opinion shared by many people that deal with PCI, is that you follow PCI not to secure your systems, but to limit your exposure to PCI fines and remedies. It seems like it is a rock-solid defense for PCI compliance to choose the algorithm that is certified by NIST.
The difference is that one way your lawyers won't even need to make an argument and the other way they might have to make an argument.
Lawyers "do you use blabla standard to encrypt the data?" You> "yes" - and it is secure.
The user Thomas Pornin knows his crypt which is why he has 86k reputation.
..not that I'm a fan of Nagios' architecture though. I'm not.
rkhunter is out of date.
We use it as well but I'm not sure it is an example of a serious security practice when given with only one other "checksum or tools such as" type statement.
Nice email, one improvement would be to host the wiki behind a proper https-site -- especially since we already know the attack appears somewhat sophisticated, and that the attackers gained control over servers on the Hetzner network. Now, the faq doesn't contain anything obviously bad (email your credit card info to... etc) -- but one small thing that could be improved.
Other than that I appreciate how Hetzner have handled this so far.
(tweet text reproduced here: "I asked Hetzner if they plan to support 2 factor auth & was told that they already do - they require a username and a password. Seriously.")
If I were a mean dude who hacked into some prominent site, the first thing I would do is submit a link to HN that points to a malicious "article" and laugh as malware gets thrust into thousands of computers.
I think such a fauxpas shouldn't be tolerated twice.
There was unauthorized access to customer data, even passwords. Someone claims to have accessed it via an FTP-server where he found a root password to a management server
This doesn't protect against someone who has root access to your box and manages to capture passwords in the clear as they are being transferred but it does limit the surface of a breach.
I see it just cleared up for now. Still, I'm moving my traffic to other locations till this blows over.
Is the konsoleh login/account something that would have gotten created for me automatically? i.e. do I need to worry about it?
I think webspace users get a konsoleH account, too.
Robot = VPS/root servers
Doesn't this imply that my server is potentially compromised as well?
But since this seems to be affecting only the main/german division, i didn't bother to mention them.
Hope everyone has fun with their identity theft cleanups. They ask people for ID scans.
Bank details are compromised.
This was announced AFTER the German work day was done. Really nice for admins. Thanks.