
All servers at OVH North American datacenters offline due to fiber cut - ck2
http://status.ovh.net/?do=details&id=11304
======
jedisct1
Constructing a brand new dark fiber network over long distance requires an
insane amount of work, plus a ton of legal and administrative issues to deal
with.

OVH had started working on the construction of a second dark fiber network a
long time ago, with at least the capacity of the previous one. Due to a
strange quirk of fate, they had planned to complete the construction within
the next few days.

After the fiber cut, they not only tried to locate and repair the cut pairs of
cable, but also simultaneously rushed to finish the construction of the new
network. They made in hours what was initially planned to require at least
another week.

The services are now back online at full speed, because they managed to finish
installing the new cables, test them and connect the whole data center right
away to the new network. And it worked.

~~~
jagger27
That's a story to tell your kids and grandkids.

I have such great respect for network admins. I have no idea what it takes to
get a network to route properly worldwide on multiple trunks.

------
Tombar
Long time OVH customer here, they had a similar issue on May 2015 for the same
DC in Canada where a car hit one of the poles along the BHS/MTL via the North
route, damaging the optical fiber cable: 2 of the 3 pairs of cables where
down.

[http://status.ovh.co.uk/?do=details&id=9603](http://status.ovh.co.uk/?do=details&id=9603)

At the time, they told customers they were adding new fiber providers and
cables.

~~~
bryanlarsen
Laying new fiber takes years. They turned it on today, so it must have been in
the works then too.

------
smoyer
This is why you have fiber leaving your datacenter in two different direction
and to two backbone providers. Once you do your due diligence, it's really
hard to guarantee your providers never share infrastructure. Good luck guys!

~~~
bcoates
Also, be aware that fiber sabotage is a thing that happens. It's rare, but
it's bad, and in my experience even top-notch providers take days to recover
from it.

Even if you aren't the target, it doesn't matter how many redundant links your
single uplink provider has if the threat model is hostile actors with their
infrastructure mapped out vs. wayward backhoes.

------
Loic
The effect on the weathermap is "clear cut":
[http://weathermap.ovh.net/#usa](http://weathermap.ovh.net/#usa)

It looks like they still have one link running, but a small saturated one.

~~~
ju-st
Yes, technically the DC is not down... I can still ping forum.ovh.us and the
webpage loads, but very slowly (as excpected).

------
imperialdrive
800gbps of connectivity cut! yikes, that would put all hands on deck... I'm
hearing issues with Verizon/Terremark and Wellpoint networks from clients,
wonder if it has anything to do with... godspeed OVH

------
sithadmin
Multihome your edge, folks.

~~~
api
Glad to see this instead of bashing on OVH. This can happen anywhere. Every
data center is at the end of one or maybe a few very thin little cables that
can be cut very easily.

~~~
ChuckMcM
Reminds me of the joke: Traveller to ops guy, "Why do you have a piece of
fiber optic cable in your rescue kit?", ops guy, "That way if I ever get
stranded I just bury this cable nearby and wait for a back hoe to show up and
cut it!"

On a weird note, when evaluating data centers I had one tell me they had
redundant connections to the Internet but what they really had were two
lambdas on the same fiber. Sure its great that you have two providers but _it
is only one fiber_ didn't seem to sink in. Pass.

The problem is very real of course, when a disgruntled AT&T engineer cut lines
in South San Jose they did a huge amount of damage. Looking at their impact
and the time it takes to resplice a fiber, a group of us calculated it would
take no more than a 100 people to take California completely off the cabled
Internet (satellite is really hard to disable from the ground). That seems
pretty risky.

~~~
ck2
There are actually reports of people being spotted testing major connection
points for fiber and power at various places (I think in California).

Not sure if it is real or trumped up, but if real, it would be domestic
terrorism for sure.

~~~
nosuchthing
Maybe this what you're referring to:
[http://www.wsj.com/articles/SB100014240527023048511045793591...](http://www.wsj.com/articles/SB10001424052702304851104579359141941621778)

[https://en.wikipedia.org/wiki/Metcalf_sniper_attack](https://en.wikipedia.org/wiki/Metcalf_sniper_attack)
?

------
jdalgetty
This will certainly mean a temporary drop on botnet activity!!!

------
napsterbr
For customers of dedicated servers, remember to open a support ticket so you
can get compensated accordingly. (default SLA = 5% per hour of downtime)

[https://twitter.com/rpurinton/status/661255883582775298](https://twitter.com/rpurinton/status/661255883582775298)

------
gesman
Now lets correlate this with "bad" TOR sites downtimes to narrow down their
origins...

------
meirelles
The OVH is great. A really good solution for many use cases. Of course you
must know exactly what they are. Isn't a good idea trust a single DC setup,
even worst if they are totally dirt-cheap. Unfortunately people misuse them.

------
jedisct1
Up!

[http://status.ovh.co.uk/?do=details&id=9603](http://status.ovh.co.uk/?do=details&id=9603)

~~~
Zarel
You're looking at the wrong one. The downtime today is this one:

[http://status.ovh.co.uk/?do=details&id=11304](http://status.ovh.co.uk/?do=details&id=11304)

And as of right now (4:17 PM CST), it says it's still being worked on:

> We still don't have an ETA regarding the repairs on the existing fiber.

> However, it's been confirmed that 2 out the 3 new pairs of the Eastern route
> (which were being setup) are now usable. We are working with our providers
> in order to get a full connection as quickly as possible: the only cross-
> connect linking OVH to our provider (in Montreal) is currently missing.

------
skrowl
It's back up as of 16:50 EST (UTC-5)!

~~~
napsterbr
Actually, as you can see at their network map[1], the cables are still out.
Only the three new ones, upgraded today, are working.

[1] [http://weathermap.ovh.net/#usa](http://weathermap.ovh.net/#usa)

------
chinathrow
I wonder, is this (still) common these days? Having a datacenter on a single
physical cable?

~~~
deepsun
At Google -- no. At OVH -- yes, apparently. Google have it's own outages,
caused mostly by a network config change or software update, but not from
fiber cuts.

~~~
tomschlick
Plus Google has a shit ton of redundancy. I remember watching a video about
their site reliability team. Their job is to test worst case scenarios and
they can/will cut an entire datacenter out of a service's (gmail, drive, etc)
pool to see what happens.

~~~
oasisbob
Their exercises are also excellent at including the full organization in DiRT,
not just engineering.

eg, Just because someone is "authorized" to spend six-figures for scheduling a
diesel delivery doesn't mean they will.

_Weathering the Unexpected_
[https://queue.acm.org/detail.cfm?id=2371516](https://queue.acm.org/detail.cfm?id=2371516)

