Before the companies would even talk to us, we needed to fill out a 20 page questionnaire about everything from where the company was located and registered, etc, to our security practices. We needed to have individual user accounts, with full logging and auditing so that everything could be traced. The datacenter we were in needed to be ISO27001 compliant, with yearly penetration tests carried out by an external company. Our systems needed to have penetration tests carried out by an external company... the list went on.
In the end, as a small business, we concluded we couldn't do it.
I've got some friends who work for a company that does secure datacenter hosting, for government stuff, bank stuff, etc. It is not cheap. It is seriously highly priced, and you don't get anything like Amazon AWS, it's all about managed servers and dedicated firewalls.
People need to start taking this stuff a little more seriously.
The product I spend most of my time working on sells as a managed service to the investment/advice banks, and even though we only deal with the T&C side of things (no money, very little by way of client data (it should be zero client data really, but as names and other identifying info sometimes creep into file-check records and such they have to take precautions based on the service having more than just employee details within)) we have to keep with that sort of standard: ISO27001, everything dedicated including firewall boxes between "their" machines and us (& the rest of the DC, obviously), regular penetration tests, they even require background-checks and other vetting of our staff.
One of the other products worked on from here is used by the Met Police and several local councils. Even though it is only their procurement departments (bulk order of paperclips, ...) the hosting security standards are similar (in some ways more stringent).
> People need to start taking this stuff a little more seriously.
For bitcoin definitely. Companies like our clients already take it very seriously (the regulators would fine them heavily if they didn't, and if they didn't and information leaked out because they didn't the PR could be disastrous). What many of the individuals dabbling in bitcoin don't realise is that if they want a decentralised currency where no bank or government department has control, no bank or department can protect them so they need to make sure they provision sufficient security themselves which means properly vetting any suppliers or keeping the wallet in their own environments.
It surprises me that people running towards bitcoin because it is decentralised then immediately run to a centralised coin store to make life easier... They seem to want their cake and its icing without taking precautions to defend it from ants.
(OK, so many are running to it ATM because of the hype, rather than because of the decentralised nature, but they often claim otherwise)
There must be a demand proportional to the cost of any solution. And demand takes time to grow. Bitcoin is not bootstrapped by anyone, it grows from absolute zero, step by step. This type of shit will gradually happen less and less, but people cannot just decide to use more expensive and robust hosting.
Having a cloud-hosted front-end isn't a big deal. Having your BTC wallet on a system you do not have 100% physical control over is amateur hour.
A Raspberry Pi on a cable modem connection could be made more secure than whatever off-the-shelf hosting most of these companies are crazy enough to use.
How do you think banks got more secure? Trial and error, incremental progress. These repeated break-ins should make consumer-grade hosting environments more secure.
Not going to happen at the price points people are accustomed to - they'll simply become professional-grade hosting environments, with a price tag to match. See Amazon's recent cloud HSM announcements - $5000 set-up fee before you get started. Then people will whine about how expensive it all is, and some bright spark will come up with ways to make it cheaper by compromising here and there, and we're back to square one.
Honestly, the high tech industries are shocking at learning from history - "Oh, those constraints from 10 years ago don't apply to us anymore, technology has moved on." Sure, but people haven't, and most of the real problems are sociological problems - fraud, greed, stupidity, stubbornness. Companies that deal with money or payment processing come up against this faster than ones that don't, and they adapt (see PayPal's anti-fraud department, so successful they spun off Palantir) or die (80% of all Bitcoin exchanges to date).
It won't be long now until we'll hear the first case where some employee of a third-rate exchange or something will wake up with a barrel of a gun pointed at him, forcing him to go to work and turn over all the float at his company or have his family killed. It's been done in the USA, and for much less than what these exchanges and mining pools routinely manage.
Banks have security against this sort of thing. They manage it by making actually keeping that money after obtaining it without getting caught really hard. I have no idea how it can be protected against when it's bitcoin.
Fiat currency has government buy-in, and government support for loss prevention. Bitcoin not so much. The general anti-government attitude that goes with BTC means there's little incentive for law enforcement to care. (Yes, a server break-in is cybercrime, but so is someone hacking your Wordpress or Facebook accounts, and that's the level of attention it will be paid)
Well, you can trace bitcoin more easily than you can trace other currencies.
If this were my site you would be banned.
Notice how all these leaks were not because of HTTP server security issues or the like, but because the critical parts of the application were not isolated.
No need to force obscure interfaces, just add an extra NIC and connect the critical part over Ethernet.
A different strategy could be to use a set of distributed server hosted at different hosting providers and use shared secrets. This would require that k hosts over n are compromised for the full system to be compromised.
Such system is more difficult to design so that it is as secure as the shared secret. But the benefit is that it would be distributed and thus avoid the single point of failure of a server behind a dedicated link and in a cage.
Just because they use a cheap trick to make the user feel more secure doesn't mean the system underneath it isn't, in fact, secure (it doesn't mean it is secure either, it says nothing one way or another).
I don't hear about big bank sites (even ones using psychological trickery on their users) getting cracked very often while bitcoin exchanges/wallet services/etc are falling over seemingly daily).
Now you may go ahead and read the post.
At the first compromise which may have been external to OVH's systems the attacker may have added their email address to the list of valid emails, and may have received the password reset email at the other email address. This could explain why the second reset password email was unread...
Obviously this is highly speculative.
What is "@hosting"?
It sounds like Oles has just been informed of the potential compromise, despite it happening 6 days ago (Apr 23) and 5 days ago (Apr 24) for respectively slush's pool and bitcoin-central. He better has irrefutable proof that the compromise did not happen at the OVH level. Two highly technical clients (slush has been running a pool for more than 2 years so he has countered many attack attempts during this time, his mailbox linked to his OVH account is secured via OTP, etc) compromised at just the same time, with their OVH manager password reset with no explanation, and no email?
echo "" > /root/.ssh/authorized_keys2
rm -rf /usr/local/rtm
echo "" > /etc/crontab
killall -9 rtm
more here > https://news.ycombinator.com/item?id=4839414
In the manager you can select the netboot / PXE option and "boot from rescue mode". From there you get e-mailed the logins from the server booted onto a Debian rescue image. You have full access to the OS / hard drives.
As for their keys, they are locked down to a single IP address, so unless the attackers gained control over that IP the attackers wouldn't be able to access your machine using those keys even if they were compromised.
This isn't OVH's fault.
Also in the forum post...
"I asked OVH support to provide some additional information and restrict Manager access to my IP range."
This is already available.
Why isn't critical infrastructure for Bitcoin hosted on a first-rate, known, trusted provider like Amazon AWS, Google App Engine, Rackspace, Terremark, Joyent, etc.?
E-mail addresses are even better. At first I used an email I rarely used for anything else for some 4-5 years. The day I completed my first purchase on Amazon using this e-mail to open an account, was the day I got first spam message on it. And it kept flooding with more and more messages each day. Once it reached about 200+ spams a day, I decided to ditch the address and created a new one for myself and a specific new one on my domain for Amazon exclusively. It was amazn123894@[mydomain]. Anyway, when I got the first order using that e-mail, the same story happened. Now, I don't think hackers have a got a hold on the Amazon servers, it's more probable that you have employees selling the data. Especially since I never heard anything like this happening to some of my friends in western countries. I guess it's easy to decide that nobody would care or notice if a guy from eastern Europe gets screwed.
I'd really like to see Amazon's internal rules of data access clearance.
http://forum.ovh.com/showthread.php?t=88277 (in french).
- attacker brute forced the unique ID used in the reset URL
- they could do it because the unique ID was not random enough
- they analysed 3 years worth of logs (still ongoing to 10) and concluded that only 3 clients (all bitcoin related) were affected.
Trust other services only for a few minutes when you need to exchange bitcoins. Once done - transfer everything back to your own wallet.
That being said, I wonder if the guy had any alternate email accounts listed on his OVH manager account. Emails get sent to both accounts when some action occurs in the manager that requires an email to be sent.
By the time they got back to be on my ticket I'd gone with Hetzner, so far can't fault them and the price is pretty much the same, extra bonus I signed up for an i7-2600K/32Gb and got an i7-3770K/32Gb, pretty nifty increase (I figure they have a mix of 2600's and 3770 so it may be random which you get).
Then 3 months later, I found out they were still charging me. I forwarded them the conversation and they just kept responding with canned messages refusing to acknowledge the wrong on their part. It's a shame because they are by far the best value :-(
There's no guarantee that anyone from their staff will look at your post but even Martin Hetzner, the founder, frequents the board and occassionally answers & helps out, so that's another route to get support there.
If you are buying on kit/price and don't need a lot of CPU umpf then the cheaper unmanaged servers through OVH's value brand (kimsufi.co.uk/kimsufi.com) are fairly hard to beat.
Though I agree I'm not entirely sure how they do it! (everything on there has an extra redundant copy elsewhere anyway so if it dies unexpectedly I can recreate the content PDQ)
I assume they have a metric shed load of those boxes sat somewhere from when they were considered mid-to-high spec, and that the rack space costs next to nothing, and have absolutely everything possible completely automated (I've not needed to contact them since setup: the control panel can do everything down to a full OS rebuild, and they officially offer zero support for those cheap boxes other than in the case of hardware failure). That is the only way I can see them breaking even on the price, before accounting for the PR bonus of attracting people to look (who then take a higher spec service either straight away or later).
I accept that for managed servers they need to connect from outside to manage them, but how hard would it be to drop all traffic except for the IP range their management systems use?
> Today at 3pm UTC I noticed that somebody succesfully resetted the password to OVH manager,
> but at 11pm UTC I realized that there's another succesful password reset at OVH.
> This time I realized that the attacker resetted the machine with the wallet to rescue mode, which means that I lost the control to this machine.*
The machine gets hacked, but 8 hours later the wallet is still on the same machine, which gets hacked again?
People Coin is not only about mining but also about having a stake of the money supply.
You could attack ppcoin by owning 51% of the currency supply, but by that time you're so invested you don't want to.
> doesn't have physical access to servers
> "budget" provider
> hurr durr why are my buttcoins gone??!?!?!?!
People are going to draw parallels between buttcoins and /real/ currency, but they're not the same. However, they both hold value. Therefore, people with big buttcoin wallets should value their stash, and get some physical servers and host it in their own locking cab or somewhere more secure.