One of our projects used to get ddosed and as a small company the only reason we managed to survive was thanks to OVH. When the attackers kept changing their attacks, we were direct to talk to their engineers and they helped fix the issue, all we had to do was send a bunch of packet captures to them.
Over the years since then we had a few issues and every single time their customer support was really helpful.
I have used paid support from google, azure and amazon, but I have had a better customer service experience with OVH overall.
not unforced, prolly ransom requests before IPO
I imagine this is especially common here since one of the services offered by OVH is DDoS protection. Hence being down from a DDoS would be even worse press.
A state is a sovereign political division, or a subdivision within a federation (France, Arizona).
A nation is a community of people with a common culture, language, ethnicity etc (the Navaho, the Welsh, the Japanese).
When these are the same entity, it's a nation state (Japan), but there are many cases where they aren't the same (Belgium, China, USA, Russia).
Nitpick: The belief of the existence of the said common ground is actually more important than its actual existence. France is considered a nation state even though there are several ethno-linguistic groups in France. A few illustrations:
- less than half of the French population during the French revolution spoke French as their native language.
- The second Nobel Prize of Literature from France (Frédéric Mistral) didn't wrote its work in French, but in Occitan.
- My family has lived in France for as long as genealogy could trace back, yet half of my great-grandparents only learnt French in school, they didn't spoke it at home.
: I don't have an online source for that, nor the exact figure: I read it in Frernad Braudel's L’Identité de la France
Following a human error during the reconfiguration of the network in our DC in VH (US-EST), we had an issue on all the backbone. We're gonna isolate the DC VH then fix the configuration
Those last few days, DDoS attacks intensity has grown a lot. We decided to increase our DDoS handling capacity by adding new infrastructures in our DC VH (US-EST). A bad router configuration caused the network crash.
Short lived incident fortunately...
72 bytes from 22.214.171.124: icmp_seq=1 ttl=249 time=276 ms
72 bytes from 126.96.36.199: icmp_seq=2 ttl=249 time=1041 ms
72 bytes from 188.8.131.52: icmp_seq=3 ttl=249 time=313 ms
72 bytes from 184.108.40.206: icmp_seq=4 ttl=249 time=198 ms
72 bytes from 220.127.116.11: icmp_seq=5 ttl=249 time=226 ms
72 bytes from 18.104.22.168: icmp_seq=6 ttl=249 time=390 ms
72 bytes from 22.214.171.124: icmp_seq=7 ttl=249 time=865 ms
Also, I can access my OVH VPS from an Italian IP, but if I try with a VPN from outside my country, I get error 521.
That seems the wrong way round.
> Unofficial OVH Status Feed
Only the image is conveniently cropped to a circle so that the "Un" part isn't visible and it reads "OFFICI".
Truly, only a professional outfit could have written a Twitter update with spelling like this.
3 2a02:c28:1:6506::106 2.131 ms 2.124 ms 2.117 ms
4 2a02:c28:11:6::100 15.695 ms 15.689 ms 15.682 ms
5 2a02:c28:1:1900::19 15.998 ms 15.991 ms 15.985 ms
6 2a02:c28:0:1819::18 15.259 ms 15.051 ms 16.975 ms
7 2a02:c28:0:1718::17 16.968 ms 16.403 ms 15.619 ms
8 2a02:c28:0:1731::31 15.406 ms 18.620 ms 18.554 ms
9 2001:7f8:4::3f94:2 12.025 ms * *
10 * * *
11 2001:41d0:aaaa:100::5 21.622 ms 38.813 ms 2001:41d0:aaaa:100::3 37.231 ms
12 * * *
13 * 2001:41d0::25f1 19.507 ms 2001:41d0::c68 20.892 ms
14 2001:41d0::513 19.685 ms 19.674 ms 2001:41d0::50d 18.248 ms
15 2001:41d0:0:50::5:10a1 19.092 ms 19.917 ms 2001:41d0:0:50::5:10a5 20.126 ms
16 2001:41d0:0:50::1:143f 19.375 ms 2001:41d0:0:50::1:143b 24.609 ms 2001:41d0:0:50::1:143d 18.933 ms
Don't consumer ISPs hand out /64s or /48s ?
... and yeah, lots of sites impacted
On the other hand something like Gitlab's "We deleted the production database and our backups were not working for months" sounds much more stressful to me.
Thank you! I had almost forgotten about my beloved provider since the last outage :)
It has been  days since the last outage.
EDIT: To those having issues starting/deploying Windows virtual machines in Azure and are using ARM to do so: change the OS type to Linux instead of Windows. This seems to resolve the following issue:
> Error: No version found in the artifact repository that satisfies the requested version '' for VM extension with publisher 'Microsoft.WindowsAzure.GuestAgent' and type 'CRPProd'
3 0.ae21.xr4.1d12.xs4all.net (126.96.36.199) 7.324 ms
0.ae21.xr3.3d12.xs4all.net (188.8.131.52) 5.929 ms
0.ae21.xr4.1d12.xs4all.net (184.108.40.206) 5.864 ms
4 0.et-1-1-0.xr1.tc2.xs4all.net (220.127.116.11) 7.562 ms 5.487 ms
0.et-7-1-0.xr1.tc2.xs4all.net (18.104.22.168) 5.551 ms
5 asd-s8-rou-1041.nl.as286.net (22.214.171.124) 6.034 ms 5.759 ms 5.608 ms
6 ae11.cr6-ams1.ip4.gtt.net (126.96.36.199) 6.620 ms 7.938 ms 6.670 ms
7 188.8.131.52 (184.108.40.206) 7.348 ms 6.170 ms 6.320 ms
8 if-ae-45-2.tcore2.av2-amsterdam.as6453.net (220.127.116.11) 310.705 ms 257.678 ms 257.278 ms
9 if-ae-14-2.tcore2.l78-london.as6453.net (18.104.22.168) 257.217 ms 256.759 ms 259.044 ms
10 if-ae-2-2.tcore1.l78-london.as6453.net (22.214.171.124) 258.850 ms 256.723 ms 256.735 ms
11 if-ae-12-2.tcore2.mlv-mumbai.as6453.net (126.96.36.199) 257.349 ms 255.042 ms 257.338 ms
12 if-ae-16-2.tcore1.svw-singapore.as6453.net (188.8.131.52) 248.550 ms 257.559 ms 329.240 ms
13 if-ae-2-2.tcore2.svw-singapore.as6453.net (184.108.40.206) 307.149 ms 255.582 ms
be101.mrs-mrs1-sbb1-nc5.fr.eu (220.127.116.11) 181.205 ms
14 be101.mrs-mrs2-sbb1-nc5.fr.eu (18.104.22.168) 180.612 ms * 179.738 ms
15 * * par-gsw-sbb1-nc5.fr.eu (22.214.171.124) 186.377 ms
Perhaps I'd tolerate the constant outages if they were actually any good at communicating. Their status page sucks. They rarely update it properly, and when they do, its hard to find the information you need and that you know relates to your servers.
Minus the fire incident, can't recall any outage. Not the cheapest offering out there but been running many things on it on autopilot. Support is solid too, always had a good experience.
OVH kinda won because their VPS pricing was always dirt-cheap compared to the cheap dedicated servers provided by Online.net/Scaleway. Since then OVH VPSes got expensive, and Scaleway started offering cheap VPSes, the prices got aligned. I know a few people who used to be OVH-fanboys and they all migrated to Scaleway and Vultr the last 4 years.
But I agree, the competition for unmetered reliable VPSes is getting thinner and thinner… People are even started to consider metered VPS solutions like Vultr…
Rasmus Lerdorf (the author of PHP) did an objective non-sponsored comparison in 2019, I'm using IONOS personally, but since then strato.de appeared. (Both are German) I'm thinking about adding another node from them to avoid relying on only one vendor.
Following a human error during the reconfiguration of the network on our DC to VH (US-EST), we have a problem on the entire backbone. We will isolate the DC VH and then fix the conf.
is also down which is their WIP tasks dashboard, having worked there I can tell that something definitely went very very wrong
We recovered our OVH sandbox env (yay) but prod is still very down, guess they are rolling back DC by DC
Our call center provider has VoIP provided by OVH as well, they are down as well... It's going to be a very long day...
The web app that configures the monitors is deployed separately, so the web app can go down while the monitors + alerting keeps running.
One of their four data center burned down in march.
A few days ago they sent me an email that they were deleting my account soon, and a few hours later another E-Mail stating that it was an error.
Tomorrow is their IPO, it will be at a very high valuation, lets see how that plays out.
Edit: Our US server also.
Total downtime around 50minutes.
The rule should be host your status page on your competitor's cloud. If you're AWS, host it in Azure, if you're Azure, host it in GCP, if you're GCP, host it in AWS. (Linode, Digital Ocean, OVH, etc can do their own dance.)
The status page is still hosed, but I'll take "my systems are up but the status page is down" over the converse any day ;)
The only problem I've found is that unlike most VPS providers, they don't seem to do gigabit speeds in their cheaper tiers. I'm content with the 100mbit offering (which is 200mbit if you sign up these days, I think) so that's not a problem for my use case.
If you only run very basic stuff (basically a raspberry pi in the cloud) then Oracle offers two VPS servers for free (not "free for a year", "free with a purchase", actually "free forever", which was a huge surprise to me). You get very little in terms of performance but for a website or small project that should be sufficient.
They have many locations and I used their VPS for a long while before I moved to colocation. Had no issues apart from when I accidentally deleted /etc/.
Quite attractive pricing and been an user for about 2+ yrs, never faced any issue so far, probably luck me?
edit: also can't get access to ovh vps via ssh
I became able to reach our servers in London, and www.ovh.com, from UK locations and from DigitalOcean.
3 126.96.36.199 (188.8.131.52) 1.365 ms 184.108.40.206 (220.127.116.11) 1.354 ms 18.104.22.168 (22.214.171.124) 1.549 ms
4 126.96.36.199 (188.8.131.52) 1.179 ms 1.168 ms 184.108.40.206 (220.127.116.11) 1.114 ms
5 * 18.104.22.168 (22.214.171.124) 1.167 ms 1.155 ms
9 vl1272.lon1-eri1-d1-a75.uk.eu (126.96.36.199) 2.594 ms be101.lon1-eri1-g2-nc5.uk.eu (188.8.131.52) 2.654 ms 2.466 ms
14 (184.108.40.206) 2.102 ms 2.091 ms 2.076 ms
If they screwed up routing, BGP was very likely to be involved, but the cause was probably human error, failure of process design, failure of system design, etc etc.
Do we really need these kind of comments?
These issues, one might think, should have turned into tomes of extensive information on why they happened and how to avoid them, and become an integral part of showing the ropes to new sysadmins, or operation persons, you name them. It seems, however, that by and large the knowledge of the actual working systems does get irretrievably lost once some Kevin gets retired.
P. S. Would you be just as cavalier if lives were lost as a result of such incidents?
Maybe the greybeards don't do a good job at educating their successors, or maybe it's the "tl;dr, YOLO" attitude at work. I don't know. I suspect a combination of both.