Hacker News new | comments | show | ask | jobs | submit login
OVH has been compromised (bitcointalk.org)
95 points by mrb 1632 days ago | hide | past | web | 88 comments | favorite

This type of shit will keep happening until people stop trying to store money in relatively insecure consumer-grade hosting environments. The entire bitcoin economy is built on sand. Look into how much expense real banks go to in order to secure their systems. Then look at this never-ending parade of surprised victims who lost their bitcoins just because somebody found a vulnerability in some hosting company's bespoke password reset form.

I totally agree with this. I was working on a proof of concept for a back end system for a financial advice business. This needed to go to financial institutions and retrieve the latest price of a portfolio, that's it, not 'write' to the system, just updating data.

Before the companies would even talk to us, we needed to fill out a 20 page questionnaire about everything from where the company was located and registered, etc, to our security practices. We needed to have individual user accounts, with full logging and auditing so that everything could be traced. The datacenter we were in needed to be ISO27001 compliant, with yearly penetration tests carried out by an external company. Our systems needed to have penetration tests carried out by an external company... the list went on.

In the end, as a small business, we concluded we couldn't do it.

I've got some friends who work for a company that does secure datacenter hosting, for government stuff, bank stuff, etc. It is not cheap. It is seriously highly priced, and you don't get anything like Amazon AWS, it's all about managed servers and dedicated firewalls.

People need to start taking this stuff a little more seriously.

> it's all about managed servers and dedicated firewalls

The product I spend most of my time working on sells as a managed service to the investment/advice banks, and even though we only deal with the T&C side of things (no money, very little by way of client data (it should be zero client data really, but as names and other identifying info sometimes creep into file-check records and such they have to take precautions based on the service having more than just employee details within)) we have to keep with that sort of standard: ISO27001, everything dedicated including firewall boxes between "their" machines and us (& the rest of the DC, obviously), regular penetration tests, they even require background-checks and other vetting of our staff.

One of the other products worked on from here is used by the Met Police and several local councils. Even though it is only their procurement departments (bulk order of paperclips, ...) the hosting security standards are similar (in some ways more stringent).

> People need to start taking this stuff a little more seriously.

For bitcoin definitely. Companies like our clients already take it very seriously (the regulators would fine them heavily if they didn't, and if they didn't and information leaked out because they didn't the PR could be disastrous). What many of the individuals dabbling in bitcoin don't realise is that if they want a decentralised currency where no bank or government department has control, no bank or department can protect them so they need to make sure they provision sufficient security themselves which means properly vetting any suppliers or keeping the wallet in their own environments.

It surprises me that people running towards bitcoin because it is decentralised then immediately run to a centralised coin store to make life easier... They seem to want their cake and its icing without taking precautions to defend it from ants.

(OK, so many are running to it ATM because of the hype, rather than because of the decentralised nature, but they often claim otherwise)

People cannot simply stop using consumer-grade hosting and jump on a very expensive BTC vault.

There must be a demand proportional to the cost of any solution. And demand takes time to grow. Bitcoin is not bootstrapped by anyone, it grows from absolute zero, step by step. This type of shit will gradually happen less and less, but people cannot just decide to use more expensive and robust hosting.

I see your point, but if I were running a consumer-grade hosting company, I'd seriously consider updating my terms of service to disallow running mining operations or keeping cryptocurrency wallets in the servers.

That would be your choice. Just like I don't provide any kind of hosting to anyone because I'm not good at it. But the miners and users want to do their job somewhere and if we don't provide them with the hosting, someone else will. And only after some time some seriously better services will emerge.

Get your own damned server. They're not expensive. Get a connection you can cut with a pair of scissors if you need to. It's not hard.

Having a cloud-hosted front-end isn't a big deal. Having your BTC wallet on a system you do not have 100% physical control over is amateur hour.

A Raspberry Pi on a cable modem connection could be made more secure than whatever off-the-shelf hosting most of these companies are crazy enough to use.

Well banks generally refund stolen money, so you won't see a lot of noise from victims... but anyway:

How do you think banks got more secure? Trial and error, incremental progress. These repeated break-ins should make consumer-grade hosting environments more secure.

These repeated break-ins should make consumer-grade hosting environments more secure.

Not going to happen at the price points people are accustomed to - they'll simply become professional-grade hosting environments, with a price tag to match. See Amazon's recent cloud HSM announcements - $5000 set-up fee before you get started. Then people will whine about how expensive it all is, and some bright spark will come up with ways to make it cheaper by compromising here and there, and we're back to square one.

Honestly, the high tech industries are shocking at learning from history - "Oh, those constraints from 10 years ago don't apply to us anymore, technology has moved on." Sure, but people haven't, and most of the real problems are sociological problems - fraud, greed, stupidity, stubbornness. Companies that deal with money or payment processing come up against this faster than ones that don't, and they adapt (see PayPal's anti-fraud department, so successful they spun off Palantir) or die (80% of all Bitcoin exchanges to date).

It's just not that easy. People running these things completely lack the understanding of just how seriously security needs to be taken. Various sites handle millions of dollars worth of bitcoin with "patch all flaws" style security. People need to understand that millions of dollars of easily disposed of goods are worth killing people for.

It won't be long now until we'll hear the first case where some employee of a third-rate exchange or something will wake up with a barrel of a gun pointed at him, forcing him to go to work and turn over all the float at his company or have his family killed. It's been done in the USA, and for much less than what these exchanges and mining pools routinely manage.

Banks have security against this sort of thing. They manage it by making actually keeping that money after obtaining it without getting caught really hard. I have no idea how it can be protected against when it's bitcoin.

Banks have security against this sort of thing. They manage it by making actually keeping that money after obtaining it without getting caught really hard. I have no idea how it can be protected against when it's bitcoin.

Fiat currency has government buy-in, and government support for loss prevention. Bitcoin not so much. The general anti-government attitude that goes with BTC means there's little incentive for law enforcement to care. (Yes, a server break-in is cybercrime, but so is someone hacking your Wordpress or Facebook accounts, and that's the level of attention it will be paid)

>They manage it by making actually keeping that money after obtaining it without getting caught really hard. I have no idea how it can be protected against when it's bitcoin.

Well, you can trace bitcoin more easily than you can trace other currencies.

"People need to understand that millions of dollars of easily disposed of goods are worth killing people for."

If this were my site you would be banned.

Care to explain why?

It looks like cryptocurrency brings an incentive to finally take computer security seriously for consumer grade software and hardware. It is the first time something is really at stake apart for intangible qualities such as "privacy" and "intellectual property". It is an interesting and unexpected(?) second-order effect of cryptocurrency.

But is the cost of taking security seriously more than the entire lifetime revenue of the Bitcoin industry?

It is impossible to predict the cost and/or gains from now to an undefined time in the future. But the good thing is that this will help non-cryptocurrency security as well.

I kindly disagree - the fact its consumer grade hardware doesnt change anything here. The fact its a hosting company and not own datacenter does influence the security, but not as much as you'd think, just count all these huge companies, gov institutions brought to their knees... It was, is and always will be a matter of effort vs value and risk, determined and skilled attacked will get what he needs one way or another and hosting choice in this case is more like OS choice, that matters a bit, but not a lot. It is not a boolean, where it does or doesnt make you secure. Security is just a state of mind.

IMO, it shows that we really aren't that good at securing our data. BC is basically data that is worth cash. And that's a huge incentive for criminals to go after it, just like a thief would prefer stealing $100 cash than stealing a bike they can sell for $100.

Exactly. Why not using a host witout connection to internet to host the database ? One could use a simple serial or usb link to exchange information with the host with a specific protocol. There is no way someone could hack ino the database.

A "simple" transport medium won't prevent security issues itself (you can run PPP over RS-232), but it will force you to define a fairly minimal API, which is possible with HTTP too.

Notice how all these leaks were not because of HTTP server security issues or the like, but because the critical parts of the application were not isolated.

No need to force obscure interfaces, just add an extra NIC and connect the critical part over Ethernet.

You are right.

A different strategy could be to use a set of distributed server hosted at different hosting providers and use shared secrets. This would require that k hosts over n are compromised for the full system to be compromised.

Such system is more difficult to design so that it is as secure as the shared secret. But the benefit is that it would be distributed and thus avoid the single point of failure of a server behind a dedicated link and in a cage.

Wells Fargo, establishing connection: https://news.ycombinator.com/item?id=5305925

What is this response supposed to mean in context of the actual discussion here?

Just because they use a cheap trick to make the user feel more secure doesn't mean the system underneath it isn't, in fact, secure (it doesn't mean it is secure either, it says nothing one way or another).

I don't hear about big bank sites (even ones using psychological trickery on their users) getting cracked very often while bitcoin exchanges/wallet services/etc are falling over seemingly daily).

To make it easier for everyone reading the HN Post Title, they're talking about "Their Server(s) at OVH" being compromised, with no evidence of "OVH" itself (along with all customer servers) being compromised.

Now you may go ahead and read the post.

The author suggest the manager interface may have been used to get access to the servers. He saw a password reset mail he didn't asked for. So he suspect that there is a security weakness in the manager. Lets wait OVH manager explanation before calling back into question OVH's security.

Most strange thing is that he says the email was untouched so he concluded that the attacker somehow must've bypassed the email - if such a thing is possible..

On OVH's manager you can add secondary email addresses to a single account. I for example have three emails linked to my account. Upon logging in all those email addresses will get an email notifying me, AND upon doing a password reset all three email accounts will get the password reset link.

At the first compromise which may have been external to OVH's systems the attacker may have added their email address to the list of valid emails, and may have received the password reset email at the other email address. This could explain why the second reset password email was unread...

The attacker could have intercepted the email somewhere between OVH and the recipient unless the SMTP session was using SSL (which, unfortunately, isn't always the case).

Obviously this is highly speculative.

There are multiple managers in use at the moment, I've used v3 and v5 within the past 6 months. It's a shame they didn't mention the version they're using.

AFAIK v3 and v5 serve different product line so if they were mining bitcoins it was certainly on a dedicated server, so v3 it is ! And I don't think the manager was broken, my bet is the client got his email hacked or his machine keylogged. As OVH is the biggest European hoster if the manager was hacked there would be way more evidences than a few BC theft.

The linked post doesn't make it very clear, but the evidence that OVH itself was compromised is that two otherwise unrelated bitcoin services running on OVH (slush's pool and bitcoin-central) were compromised at around the same time.

I think the method of compromise doesn't depend on this specific customer; it could happen to any customer of OVH.

OVH founder, Oles, posted on Twitter that OVH has not been compromised https://twitter.com/olesovhcom/status/328845993867083776

He just said "no" and "I will post a longer answer to @hosting".

What is "@hosting"?

It sounds like Oles has just been informed of the potential compromise, despite it happening 6 days ago (Apr 23) and 5 days ago (Apr 24) for respectively slush's pool and bitcoin-central. He better has irrefutable proof that the compromise did not happen at the OVH level. Two highly technical clients (slush has been running a pool for more than 2 years so he has countered many attack attempts during this time, his mailbox linked to his OVH account is secured via OTP, etc) compromised at just the same time, with their OVH manager password reset with no explanation, and no email?

The main communication medium between OVH and its clients is by private mailing list, so hosting@ matches the list hosting@ml.ovh.net IMO Oles hasn't just been informed of the compromise, it is a common practice for OVH to resolve this kind of problems in private. But as the bad rep is growing he will soon provide explanations to the mailing list.

Well OVH do install backdoors on all the servers, its a surprise how many people dont know about it, here is how to remove it

echo "" > /root/.ssh/authorized_keys2 rm -rf /usr/local/rtm echo "" > /etc/crontab killall -9 rtm

more here > https://news.ycombinator.com/item?id=4839414

Link to official documentation: http://help.ovh.co.uk/InstallOvhKey

This won't protect against OVH being compromised.

In the manager you can select the netboot / PXE option and "boot from rescue mode". From there you get e-mailed the logins from the server booted onto a Debian rescue image. You have full access to the OS / hard drives.

Regardless of whether the backdoor-building of theirs is documented, someone obtained unauthorized access to the machines, be it an employee or not. This means OVH really is compromised, even if they documented the installation of this backdoor.

I'm not entirely sure why they bothered with the root SSH key. If they really wanted they'd just pop open a serial console on the virtual machine wouldn't they?

At the serial console, you end up with a password prompt. It's also easier to just ssh over than to go over to the server and connect a serial cable.

a virtual serial cable.

Please don't go blowing away your entire /etc/crontab! In most cases it isn't even necessary.

As for their keys, they are locked down to a single IP address, so unless the attackers gained control over that IP the attackers wouldn't be able to access your machine using those keys even if they were compromised.

Are they blaming OVH for their own server being poorly configured and/or insecure?

I host with OVH and the only way to reset a password is with the email or the customer number which both send a reset link to the email address.

This isn't OVH's fault.

Also in the forum post...

"I asked OVH support to provide some additional information and restrict Manager access to my IP range."

This is already available.

It sounds like the problem here was the OVH account's password was reset without the account owner's authorization.

Why isn't critical infrastructure for Bitcoin hosted on a first-rate, known, trusted provider like Amazon AWS, Google App Engine, Rackspace, Terremark, Joyent, etc.?

Amazon? Trusted? Don't get me started. I got my credit card info and e-mail addresses stolen from Amazon database twice in the past 5 years.

Hmm, I've placed 291 orders on Amazon since 2004, used upwards of 6 cards there, and not had one compromise. About as anecdotal as your story.

Well, I could write down card numbers, but that would not prove anything would it. I had cards used exclusively on amazon.com for 3-4 years. Suddenly, one day, I got a report from my bank that my card number is among some numbers stolen in USA and they cancelled it. As I only used it with Amazon, I can only conclude that it must have leaked from there. Luckily, I did not lose any money, as the card was blocked by the bank immediately.

E-mail addresses are even better. At first I used an email I rarely used for anything else for some 4-5 years. The day I completed my first purchase on Amazon using this e-mail to open an account, was the day I got first spam message on it. And it kept flooding with more and more messages each day. Once it reached about 200+ spams a day, I decided to ditch the address and created a new one for myself and a specific new one on my domain for Amazon exclusively. It was amazn123894@[mydomain]. Anyway, when I got the first order using that e-mail, the same story happened. Now, I don't think hackers have a got a hold on the Amazon servers, it's more probable that you have employees selling the data. Especially since I never heard anything like this happening to some of my friends in western countries. I guess it's easy to decide that nobody would care or notice if a guy from eastern Europe gets screwed.

I'd really like to see Amazon's internal rules of data access clearance.

so anecdotal my sides hurt

OVH is the leader for Hosting in Europe. If they messed up, they should fix it quick, but the other provider you mentioned are not perfect either.

Because the BitCoin industry is terrible at security.

OVH just confirmed, they were compromised.

http://forum.ovh.com/showthread.php?t=88277 (in french).

Nutshell: - attacker brute forced the unique ID used in the reset URL - they could do it because the unique ID was not random enough - they analysed 3 years worth of logs (still ongoing to 10) and concluded that only 3 clients (all bitcoin related) were affected.

OVH is one of the biggest hoster in the world, are there any third parties confirmations of this ?

People, keep bitcoins in your own, preferably determenistic wallets, such as Electrum.

Trust other services only for a few minutes when you need to exchange bitcoins. Once done - transfer everything back to your own wallet.

I had my finger over the buy button with these guys yesterday.

I have several servers at OVH and absolutely love them. I have a feeling that if there is a flaw in their manager then it will get fixed fast.

That being said, I wonder if the guy had any alternate email accounts listed on his OVH manager account. Emails get sent to both accounts when some action occurs in the manager that requires an email to be sent.

I tried to get a trial with them, their signup gateway 500'd out every time and a couple of hours later I got a bunch of emails to say my trial was ready...meh.

By the time they got back to be on my ticket I'd gone with Hetzner, so far can't fault them and the price is pretty much the same, extra bonus I signed up for an i7-2600K/32Gb and got an i7-3770K/32Gb, pretty nifty increase (I figure they have a mix of 2600's and 3770 so it may be random which you get).

I would go with Hetzner, but principle tells me not to. I was with them for about 3 months. I was talking to an account manager (or w/e the official title is) and asked them to cancel my account for a short period of time until I need to launch (and remove the server of course.) They agreed happily.

Then 3 months later, I found out they were still charging me. I forwarded them the conversation and they just kept responding with canned messages refusing to acknowledge the wrong on their part. It's a shame because they are by far the best value :-(

Just in case you don't know: there's an customer-only internal forum at http:/forum.hetzner.de/ for which you can sign up.

There's no guarantee that anyone from their staff will look at your post but even Martin Hetzner, the founder, frequents the board and occassionally answers & helps out, so that's another route to get support there.

Thanks, I'll use it if I do decide to go back!

> It's a shame because they are by far the best value :-(

If you are buying on kit/price and don't need a lot of CPU umpf then the cheaper unmanaged servers through OVH's value brand (kimsufi.co.uk/kimsufi.com) are fairly hard to beat.

That's an awesome price, but I doubt I could ever go with anybody that cheap. I'm not sure how anybody could ever sell servers that cheap.

I have one of there machines that acts as part one of my off-site backup sites (amongst a few other things) and it has been completely reliable over the last ~18 months.

Though I agree I'm not entirely sure how they do it! (everything on there has an extra redundant copy elsewhere anyway so if it dies unexpectedly I can recreate the content PDQ)

I assume they have a metric shed load of those boxes sat somewhere from when they were considered mid-to-high spec, and that the rack space costs next to nothing, and have absolutely everything possible completely automated (I've not needed to contact them since setup: the control panel can do everything down to a full OS rebuild, and they officially offer zero support for those cheap boxes other than in the case of hardware failure). That is the only way I can see them breaking even on the price, before accounting for the PR bonus of attracting people to look (who then take a higher spec service either straight away or later).

I host several customer projects on Hetzner managed servers. Ports of all the services, MySQL, Postgres, smtp, everything else are open to the world and their support refuses to lock that down in any way.

I accept that for managed servers they need to connect from outside to manage them, but how hard would it be to drop all traffic except for the IP range their management systems use?

l2software firewall

I'm talking about a managed server. No root access. If there was root access, I wouldn't be listening to their "we can't limit access because that's how our management tools work".

They've had problems for one week until last wednesday due to a db crash, and service was sub-optimal. Everything's back to normal.

Have I understood this?

> Today at 3pm UTC I noticed that somebody succesfully resetted the password to OVH manager,


> but at 11pm UTC I realized that there's another succesful password reset at OVH.

> This time I realized that the attacker resetted the machine with the wallet to rescue mode, which means that I lost the control to this machine.*

The machine gets hacked, but 8 hours later the wallet is still on the same machine, which gets hacked again?

I wonder: is there is any user in a variant of bitcoin that disallow's mining pools ? Obviously we can't prevent users for cooperating, but we might be able make it so they have incentives to defect rather than cooperate (i.e. can use the computed hash solution on their own).

If you disallow pools the network might stabilize at a fairly low difficulty (because mining is too risky if you make nothing for months on end) and then it would get 51%'ed.

If the difficulty of a given block is that high,yes. I was thinking something more like litecoin: you'd have way more computational power total witha lot of participants, but each individual amount of work would be smal and so the risk would be low.

Fundamentally if there are N blocks mined per month and, say, 10N miners, each miner will get paid very rarely which will scare them away. The only advantage of Litecoin is that it mines blocks at a faster rate than Bitcoin.

Again, ppcoin.

People Coin is not only about mining but also about having a stake of the money supply. You could attack ppcoin by owning 51% of the currency supply, but by that time you're so invested you don't want to.


True, though doesn't that just replace a mining monopoly with a monopoly of money?

Isn't it funny that people who keep a cryptocurrency behind a website don't use a key pair to sign-in.

Not everyone that has put money into bitcoin is a cypherpunk.

Oles posted on Twitter that he will send an email about what happened tomorrow (Apr 30th): https://twitter.com/olesovhcom/status/328882263913811968

One of the first things I do on a server is disable root logins, disable password authentication, install sudo, add my_user to sudoers, and copy my public key to the server, as well as only allow connections from my IP address.

In this case, the attacker (supposedly) booted the machines into rescue mode through OVH's control panel. None of the steps you listed would've prevented that (I'm not saying it won't prevent other attacks.)

That's six of the first things you do. ;)

Is anyone else affected by this hack? I have a server at OVH and I haven't seen any suspicious activity.

I have 10 servers at OVH and haven't seen anything out of the ordinary.

you likely aren't hosting a BTC exchange too, they were likely specifically targeted.

> stores /currency/ on consumer service

> doesn't have physical access to servers

> "budget" provider

> hurr durr why are my buttcoins gone??!?!?!?!

People are going to draw parallels between buttcoins and /real/ currency, but they're not the same. However, they both hold value. Therefore, people with big buttcoin wallets should value their stash, and get some physical servers and host it in their own locking cab or somewhere more secure.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact