Hi,
To change your password, you have to go on ovh website and ask for a change with a given id. An email is sent to the email address associated with the id which includes a unique url. This url is randomly generated with 21 characters, generated from 3 different algorithms for randomess generating each 7 characters. The client who receives the email can then click on the link and get a new password for his id. A confirmation email is sent announcing the change of password. In all emails sent, ovh includes the IP of the person who did the request.
This is a procedure that was set up 7 years ago and hasn't changed since.
On april 26th, we have discovered internely an issue with the function that generates the 21 characters. Two of the three random functions were generating a not-so-random string. It was hence possible to ask for a password change and then to brute-force the password reset url. This problem was found by one of our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin of this problem was linked to the rand() function used in this part of the code that hasn't been patched to the same level as the rest of the code when we activated the script execution cache (?). We have replaced the old function returning 21 characters from 3 "random" functions with a new function using two really random functions each providing 64 characters.
We have then started searching in our databases to check if the vulnerability had been exploited, and when it could have been. To do that, we have looked at 3 years of history of password resets. We are authorized by the CNIL (commission nationale informatique et liberté, french agency regulating databases containing personnal data) to archive and use all logs for the last 10 years, specifically for this kind of issue.
We have found 3 active ids which had a password reset done by brute-force. These three occurences were targeted attacks against a part of the "bitcoin" community that uses ovh. The hacker seems to have found the flaw on apr 23rd at 22:00, and did a lot of testing for one hour to make his method work. At 23:00, his method was working and he has hacked the first id, then the two others ids the following day, still for bitcoin related sites. We have been in contact with these clients but the quality of our exchanges did not allow us to get enough information that would help reveal an vulnerability on our side. Our developpers have in a completely independant way found the issue mentionned above, and only then have we realised that there was a link between our flaw and these 3 customers. We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue.
It took us some time to communicate on the subject, because we have quickly realised that the impact was very limited (3 customers) and we wanted to take enough time to verify everything in depth and make sure that this had not impacted any other customers. Having finished our search today going back three years, we can now assert that no other customers have seen this issue. We are going back 10 years now, but the probability that this happened ealier is null.
I think that despite the small impact for our customers, it was our duty to inform everybody of this security incident that we had to deal with last week. We have added a code-review process on some very old code that hasn't been rewritten for years, to check that there are no other similar issues. We are also trying to see how we can improve communication with clients for this kind of incident, knowing that 2 of the 3 customers are actually customers of our affiliates/branches.
TL;DR :
Yes, we had a security vulnerability allowing a password change using some complex brute-force procedure. We advise all clients running critical services to limit access to the manager from some IP only.
Yes, 3 clients of the bitcoin community have been impacted by this security flaw. It is very important to read the emails that ovh sends automatically, especially the notification of password change emails, when they aren't initiated by you. If you see a request for a password change that you haven't initiated, you should call our incident support 24/24 which will block your account until everything is clear.
No, our clients database hasn't been compromised.
No, there was no impact on other clients than these 3.
We are deeply sorry for the three customers that have been impacted, and invite them to contact our commercial team (in french).
Just thinking here. If you sadly used just 26 characters in this space (rather than the 64 you can easily get in a URL), 26^7 ~= 8B codes. So, even if you perfectly got the first 14 characters, you'd have to brute force in the billions of attempts here. If you even more sadly did just hex characters (16^7) you're down to ~250M.
However you cut it, someone had to hit up the reset API a bunch of times even in the worst case. Regardless of the entropy and key length used, if you're going to allow that many attempts, someone will get in.
That and shame on anyone for assuming a basic math lib rand() function creates security grade entropy.
"We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue.
[...]
We [...] invite them to contact our commercial team (in french)."
To ensure dialog, why not start by being able to communicate with clients in English? (or their own language, for that matter).
Hi, To change your password, you have to go on ovh website and ask for a change with a given id. An email is sent to the email address associated with the id which includes a unique url. This url is randomly generated with 21 characters, generated from 3 different algorithms for randomess generating each 7 characters. The client who receives the email can then click on the link and get a new password for his id. A confirmation email is sent announcing the change of password. In all emails sent, ovh includes the IP of the person who did the request.
This is a procedure that was set up 7 years ago and hasn't changed since.
On april 26th, we have discovered internely an issue with the function that generates the 21 characters. Two of the three random functions were generating a not-so-random string. It was hence possible to ask for a password change and then to brute-force the password reset url. This problem was found by one of our developpers on apr 26th at 11:03:14 and was fixed at 12:54:13. The origin of this problem was linked to the rand() function used in this part of the code that hasn't been patched to the same level as the rest of the code when we activated the script execution cache (?). We have replaced the old function returning 21 characters from 3 "random" functions with a new function using two really random functions each providing 64 characters.
We have then started searching in our databases to check if the vulnerability had been exploited, and when it could have been. To do that, we have looked at 3 years of history of password resets. We are authorized by the CNIL (commission nationale informatique et liberté, french agency regulating databases containing personnal data) to archive and use all logs for the last 10 years, specifically for this kind of issue.
We have found 3 active ids which had a password reset done by brute-force. These three occurences were targeted attacks against a part of the "bitcoin" community that uses ovh. The hacker seems to have found the flaw on apr 23rd at 22:00, and did a lot of testing for one hour to make his method work. At 23:00, his method was working and he has hacked the first id, then the two others ids the following day, still for bitcoin related sites. We have been in contact with these clients but the quality of our exchanges did not allow us to get enough information that would help reveal an vulnerability on our side. Our developpers have in a completely independant way found the issue mentionned above, and only then have we realised that there was a link between our flaw and these 3 customers. We certainly have something to learn here about the best way to ensure dialog with clients about this kind of issue.
It took us some time to communicate on the subject, because we have quickly realised that the impact was very limited (3 customers) and we wanted to take enough time to verify everything in depth and make sure that this had not impacted any other customers. Having finished our search today going back three years, we can now assert that no other customers have seen this issue. We are going back 10 years now, but the probability that this happened ealier is null.
I think that despite the small impact for our customers, it was our duty to inform everybody of this security incident that we had to deal with last week. We have added a code-review process on some very old code that hasn't been rewritten for years, to check that there are no other similar issues. We are also trying to see how we can improve communication with clients for this kind of incident, knowing that 2 of the 3 customers are actually customers of our affiliates/branches.
TL;DR : Yes, we had a security vulnerability allowing a password change using some complex brute-force procedure. We advise all clients running critical services to limit access to the manager from some IP only. Yes, 3 clients of the bitcoin community have been impacted by this security flaw. It is very important to read the emails that ovh sends automatically, especially the notification of password change emails, when they aren't initiated by you. If you see a request for a password change that you haven't initiated, you should call our incident support 24/24 which will block your account until everything is clear. No, our clients database hasn't been compromised. No, there was no impact on other clients than these 3.
We are deeply sorry for the three customers that have been impacted, and invite them to contact our commercial team (in french).
Regards Octave