

OVH acknowledges back office security issue - kerneis
http://pastebin.com/skGagP0X

======
coutud
Translation from french :

Hi, To change your password, you have to go on ovh website and ask for a
change with a given id. An email is sent to the email address associated with
the id which includes a unique url. This url is randomly generated with 21
characters, generated from 3 different algorithms for randomess generating
each 7 characters. The client who receives the email can then click on the
link and get a new password for his id. A confirmation email is sent
announcing the change of password. In all emails sent, ovh includes the IP of
the person who did the request.

This is a procedure that was set up 7 years ago and hasn't changed since.

On april 26th, we have discovered internely an issue with the function that
generates the 21 characters. Two of the three random functions were generating
a not-so-random string. It was hence possible to ask for a password change and
then to brute-force the password reset url. This problem was found by one of
our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin
of this problem was linked to the rand() function used in this part of the
code that hasn't been patched to the same level as the rest of the code when
we activated the script execution cache (?). We have replaced the old function
returning 21 characters from 3 "random" functions with a new function using
two really random functions each providing 64 characters.

We have then started searching in our databases to check if the vulnerability
had been exploited, and when it could have been. To do that, we have looked at
3 years of history of password resets. We are authorized by the CNIL
(commission nationale informatique et liberté, french agency regulating
databases containing personnal data) to archive and use all logs for the last
10 years, specifically for this kind of issue.

We have found 3 active ids which had a password reset done by brute-force.
These three occurences were targeted attacks against a part of the "bitcoin"
community that uses ovh. The hacker seems to have found the flaw on apr 23rd
at 22:00, and did a lot of testing for one hour to make his method work. At
23:00, his method was working and he has hacked the first id, then the two
others ids the following day, still for bitcoin related sites. We have been in
contact with these clients but the quality of our exchanges did not allow us
to get enough information that would help reveal an vulnerability on our side.
Our developpers have in a completely independant way found the issue
mentionned above, and only then have we realised that there was a link between
our flaw and these 3 customers. We certainly have something to learn here
about the best way to ensure dialog with clients about this kind of issue.

It took us some time to communicate on the subject, because we have quickly
realised that the impact was very limited (3 customers) and we wanted to take
enough time to verify everything in depth and make sure that this had not
impacted any other customers. Having finished our search today going back
three years, we can now assert that no other customers have seen this issue.
We are going back 10 years now, but the probability that this happened ealier
is null.

I think that despite the small impact for our customers, it was our duty to
inform everybody of this security incident that we had to deal with last week.
We have added a code-review process on some very old code that hasn't been
rewritten for years, to check that there are no other similar issues. We are
also trying to see how we can improve communication with clients for this kind
of incident, knowing that 2 of the 3 customers are actually customers of our
affiliates/branches.

TL;DR : Yes, we had a security vulnerability allowing a password change using
some complex brute-force procedure. We advise all clients running critical
services to limit access to the manager from some IP only. Yes, 3 clients of
the bitcoin community have been impacted by this security flaw. It is very
important to read the emails that ovh sends automatically, especially the
notification of password change emails, when they aren't initiated by you. If
you see a request for a password change that you haven't initiated, you should
call our incident support 24/24 which will block your account until everything
is clear. No, our clients database hasn't been compromised. No, there was no
impact on other clients than these 3.

We are deeply sorry for the three customers that have been impacted, and
invite them to contact our commercial team (in french).

Regards Octave

~~~
luser001
Really good article on the pitfalls of a weak random number generator. I think
it's been on HN before:
[http://www.cigital.com/papers/download/developer_gambling.ph...](http://www.cigital.com/papers/download/developer_gambling.php)

For me, this was the article that made it crystal clear why you cannot use
current time as a seed for a RNG.

------
aw3c2
A pastebin link is not a source worth reading.

Here is what I found as more official, the oles@ovh.net account is often used
for official announcements: <http://forums.ovh.net/showthread.php?p=548183>

tl;dr: Bad random string generator in password change URLs, 3 customers
affected, all Bitcoin related.

edit: OVH, if you are reading this. The 4 character string for receipt
protection is creeping me out ever since I saw it. Why not simply use a longer
string?

~~~
kerneis
Thanks for the link. I've been looking for the list archives on
<http://ml.ovh.net> (as advertised in the email itself) but the domain doesn't
seem to exist anymore; hence my (unreliable) copy on pastebin.

------
thomc
I'm 6 for 5 at the moment - of the 6 companies I've hosted pet projects with
over the years, 5 have been compromised at the control panel / infrastructure
level while I was a customer:

    
    
      * Layered Technologies, dedicated OpenBSD (Sep 17 2007) [0]
      * Hetzner, dedicated Linux (Oct 21 2010) [1] 
      * Dreamhost, shared Linux (Jan 19 2012) [2]
      * Linode, VPS Linux (Apr 15 2013) [3]
      * OVH, VPS Linux via reseller (Apr 23 2013) [4]
    
      [0] http://what.entwinedstudios.com/?p=21
      [1] http://mybroadband.co.za/news/internet/16021-Hackers-hit-Hetzner.html
      [2] http://www.dreamhoststatus.com/2012/01/20/changing-ftpshell-passwords-due-to-security-issue/
      [3] http://blog.linode.com/2013/04/16/security-incident-update/
      [4] https://news.ycombinator.com/item?id=5632479
    

Food for thought.

------
mrb
There we go. I suspected that Oles was misinforming/lying to the community
when he initially denied the existence of a security vulnerability, and I was
right: <https://news.ycombinator.com/item?id=5627487>

When you think about it, this whole situation with the hacks of Linode, OVH,
etc, is completely _insane_. Bitcoin has become so successful and valuable,
that bitcoins thieves are compromising _core Internet infrastructure_ that so
many companies rely on.

People say "use more serious/professional hosters", but OVH is the world's
largest server host and is very much a "professional" service. I bet that
within 2-3 years we will see more and more stunning hacks of other hosters
like Amazon EC2, etc.

~~~
DigitalJack
The flip-side is now OVH is more secure than they have been in the past.
Nothing like a pot of gold to motivate people to try to break your security.
This is probably more effective at exposing weakness than paid pen-testing.

Not that you ever want customers to lose money/data/whatever in the process,
but it's not all down-side.

------
doe88
From the post on the french forum: they were generating a 21 characters string
from 3 different sources of random (each source contributing 7 characters to
the final string) and it looks like that among these 3 sources, 2 of them were
flawed and were not really random. They say they fixed this flaw by now
generating 64 characters random strings from 2 sources of random which they
think secure.

~~~
MichaelGG
How hard is /dev/urandom? Or without that, read /dev/random every so often and
mix it with SHA1?

------
adorable
TL;DR: Here's what happened. When a password reset is submitted, the user
receives a random URL by email containing a random 21 letter string. The
algorithm used was bad, so the random string could be brut-forced.

The OVH team researched 3 years of password changelogs and found 3 customers
who had been brute-forced that way, all in the "bitcoin community"

------
laurent123456
"oles" is Octave Klaba, the CEO of OVH. I'm quite impressed by how transparent
they are in dealing with this issue, and the degree of technical details they
are willing to provide.

------
mikegogulski
Vision of the future: Hosting companies which do not provide insurance against
such losses, subject to intensive security auditing, justifiably lose
business.

~~~
rubinelli
If you decide to stash your life savings into your local swimming pool's
locker, don't be surprised when they are taken away.

For $40~$100/month, OVH offers adequate securit for your regular daily deals
site or micro-SaaS. If you expect the level of security that comes with
PCI/HIPAA compliance, expect to pay two orders of magnitude more. At some
point, someone in the Bittcoin community will bite the bullet and pay the
price, and that will be a huge competitive advantage.

~~~
gtaylor
This. OVH is a budget provider. While this doesn't excuse them from fault
here, you definitely get what you pay for in this case.

------
sylvinus
Not sure how relevant this is but there is a huge amount of spelling errors in
this email, which makes it look all the more authentic but very
unprofessional.

Also surprising that this email (if the pastebin is authentic) appears to have
been written by a mail client from 2006 suffering from at least one public
GnuPG-related vulnerability.

~~~
gizmogwai
Octave Kabla, aka Oles is native from Poland, his french has always been quite
approximative (like my english, I guess).

------
luisivan
Oles, what a CEO. I have had direct contact with him, as OVH is a partner of
one of my projects, and it's not that easy to find someone so passionate,
humble and responsive as he is.

------
DoubleMalt
Can someone proficient in French provide a tl;dr?

I have been thinking about renting a server from then so this interests me
greatly.

~~~
jkat
21 character reset was generated via 3 separate functions each generating 7
characters each. 2 of these were old and weak, resulting in a URL that could
be brute forced.

Once discovered, they analyzed 3 years worth of logs to see who was exploited
(not clear how they matched it..maybe scanning for brute forces?) and found 3
sites all related to bitcoin.

The weakness in the algo has been resolved by using a 2 real random functions
to generate 64 random characters

