
Undersea Cable System Cuts in Singapore - methodover
http://status.linode.com/incidents/wwzn1mcmc2fp
======
rwmurrayVT
I know this is probably a slight tangent to the article, but I is anyone else
fascinated by underwater cabling?

My first job out of university is working on the USNS Zeus for Military
Sealift Command. I've been out on the ship and we're working directly with the
cable handling equipment. I've walked in the big empty vats they fill with the
cable. It's over 25 feet deep and 20 feet wide. They actually "walk" the cable
up to prevent it from getting tangled. Seriously!! SO COOL!! I would love to
post pictures, but I like my job a little bit too much to risk it.

~~~
losvedir
Neal Stephenson has a great little non-fiction piece on these cables, both
overland and undersea, talking about the process of laying them down and
negotiating what countries they go through, etc. It was unexpectedly tucked
into the end of my version of Cryptonomicon, but it might be available
separately. I agree, I'm fascinated as well.

~~~
oneubauer
It was originally published in wired. It is indeed a great write up on the
topic!

[http://www.wired.com/1996/12/ffglass/](http://www.wired.com/1996/12/ffglass/)

~~~
peterburkimsher
That literally just took me a week's worth of free time to read. Absolutely
worth it.

Several years ago I read a short book with a similar theme: The Victorian
Internet by Tom Standage.

Another inspiration to me was a 25 minute video about the HVDC lien linking
North and South Island in New Zealand.
[https://hackaday.com/2015/08/19/retrotechtacular-one-does-
no...](https://hackaday.com/2015/08/19/retrotechtacular-one-does-not-simply-
string-up-a-half-million-vdc-transmission-line/)

I wonder if anybody will make a popular game out of estimating cable slack?

------
fosco
Nice map of all cables -->
[http://www.cablemap.info/](http://www.cablemap.info/)

~~~
danjayh
_Sigh_. BP can afford to run fiber to their oil rigs, but Comcast can't be
bothered to lay it through my neighborhood?

~~~
superuser2
Oil rigs make a little more money than residential neighborhoods.

~~~
zeckalpha
I thought they were currently operating at a loss?

------
djfergus
I'm in Kuala Lumpur and use Telekom Malaysia Unifi for my ISP (20 Mb/s
bidirectional for $US62 per month, for anyone who is curious). This fault has
existed for the past few days and is only now being reported.

It not a simple slowdown and is frustrating due to the architecture of the
modern internet content hosting and I suspect however TM is rate limiting its
customers. e.g.

1) some youtube videos play fine, others stall and simply never load.

2) I cannot play anything on soundcloud.com, plays first 2 seconds then never
recovers (however if I switch off wifi and use Maxis 4G mobile connection it
works fine).

3) Animated gifs take ages to load and stutter

4) torrents stream perfectly: I can fully utilise my upstream and downstream
bandwidth

5) Netflix has no problems (I use a dynamic DNS to get USA content)

6) iOS App updates depend on time of day, during the day they are fine, at
early evening peak hours they crawl.

7) Downloading any large file from a website is very slow at all times

It is a pity TM won't purchase/route capacity through an alternative link,
having throttled internet for a month is terrible service...

~~~
nowprovision
Have you considered a VPN to a Singapore provider. e.g. PureVPN or
alternatively PPTP on a DigitalOcean, Linode (maybe not in this case),
Softlayer etc.. Generally VPS provider will have better international transit
than your local ISP, however your local ISP will peer in Singapore so that
part is usually congestion free.

~~~
djfergus
Hmm, interesting suggestion for a temporary fix. I'll have a play around
later.

But I'm loathe to add another point of failure in my transmission route, let
alone one I need to maintain myself and troubleshoot myself (VPS).

------
NiekvdMaas
According to Telekom Malaysia, it will be fixed earliest end of this month:
[https://www.tm.com.my/OnlineHelp/Announcement/Pages/RESTORAT...](https://www.tm.com.my/OnlineHelp/Announcement/Pages/RESTORATION-
WORKS-TO-REPAIR-SUBMARINE-CABLE-FAULT_2-MAR.aspx)

------
PakG1
This reminds me of a funny story. I was a tech manager for the Winter Olympics
in 2010, allocated to the mountain venues. We had this crazy test rehearsal
day where we'd test our systems and processes. Proctors would actually walk
around the various venues at the Olympics and just randomly create both big
and small problems to see how we'd respond. Suddenly, our network connection
to the primary datacenter went down, and we couldn't figure out why. We later
learned that a boat had somehow (I don't know how) cut a main underwater cable
that connected the mountain areas to the city. It was a stupid day. :)

~~~
cperciva
Are you sure it was a boat? The fibre providing most of the connectivity from
Whistler to Vancouver runs along the highway and has been cut by traffic
accidents, but I don't think there's anywhere that a boat would kill it...

~~~
PakG1
All I remember is what I was told. It could have been specific special-purpose
fibre that was laid by Bell for the Olympics, don't know.

------
appleflaxen
Maybe it's the NSA splicing a fiber with the USS Jimmy Carter.

~~~
mschuster91
How does that work in practice, anyways? I mean, cables these days don't just
carry a bunch of fibre wires, but also high-voltage cabling to provide power
to the repeaters. Isn't it risky / detectable to cut open such a cable?

Also, how is the spliced data actually exfiltrated back to land? The obvious
candidate is either a leased dark fiber (which requires intimate knowledge of
how the individual fibres are arranged) and laying a second cable (with the
obvious disadvantage that cable-laying ships are rare and can easily be
spotted).

~~~
cynwoody
>how is the spliced data actually exfiltrated back to land?

During the Cold War, they periodically returned to the site of the tap and
changed the tapes and batteries. One of the taps was installed on a Soviet
telephone cable across the Sea of Okhotsk in several hundred feet of water. It
was a 20-foot long contraption able to eavesdrop without penetrating the
cable. It was designed to fall off the cable if the Soviets raised it for
repairs. It survived nearly a decade, until 1981, when Ronald Pelton sold its
location (among other secrets) to the KGB for $35k.

[http://www.military.com/Content/MoreContent1/?file=cw_f_ivyb...](http://www.military.com/Content/MoreContent1/?file=cw_f_ivybells)
[https://en.wikipedia.org/wiki/Operation_Ivy_Bells](https://en.wikipedia.org/wiki/Operation_Ivy_Bells)

~~~
shultays
An inflation calculator says it is $100k today. That sounds awfully cheap.

------
ernsheong
"TM: Work to restore Internet speed starts March 25"
[http://www.thestar.com.my/tech/tech-news/2016/03/02/tm-
inter...](http://www.thestar.com.my/tech/tech-news/2016/03/02/tm-internet-
traffic-slow/)

~~~
cdf
I'm in Singapore and this is not reported at all. Likely affects only Telkom
Malaysia users only - one Malaysian telco lost their link to Singapore rather
than Singapore losing link to the world as implied by the headline.

~~~
ago
It definitely affects Thailand too, Sinet sent notifications about the
problem, and speed is quite bad to some sites.

------
samstave
If I were a terrorist org with ___deep_ __pockets, I 'd just hire a black
market deep see trawler with a sharp cable hook and have it drive north south
a thousand miles out of Japan. Global everything shall halt.

~~~
whitegrape
Infrastructure damage that causes severe issues is such a low hanging fruit
it's a mystery to me why it isn't attacked more by terrorist / anarchist /
primitivist / et al. groups unless I adopt the Hansonian "X is not about
seemingly related Y" viewpoint...

~~~
contingencies
Thesis: The bogeymen don't really exist.

~~~
goodcanadian
And/or they aren't very smart or very well resourced.

~~~
contingencies
If a terrorist plots in his lounge-room and nobody ever sees repercussions, do
they really exist?

------
yeukhon
I am amazed that the undersea cable is not so thick at all. I was expecting a
huge tub of cable. Just thinking about that and the amount of data we can
equip across continents in less than 200ms is just incredible.

Is there a documentary on deep sea/undersea cable system? I am really
interested in how they deploy the cables across the ocean. Have a ship sail
across the ocean and start laying the cable down?

~~~
jmiserez
This animated GIF shows how it's done:

[https://upload.wikimedia.org/wikipedia/commons/8/8f/Undersea...](https://upload.wikimedia.org/wikipedia/commons/8/8f/Undersea_cable_laying.gif)

~~~
yeukhon
I see thanks. Now wonder how they pick up where it left off :-))

------
wslh
Weird, but in February 29, some of our customers experienced difficulties to
connect to our Linode servers in Dallas (not Singapore) from some Internet
providers in US (e.g: mobile carriers, cable ISPs). That day I checked the
Linode status and it was normal so I forgot the issue until now.

Do you think these events can be correlated? May be they were trying to
mitigate the issue and affected other regions as well.

~~~
jlgaddis
Is it _possible_? Sure. Is it likely? No.

Without _a lot_ more information -- preferably from both sides -- it's
impossible to know.

~~~
wslh
Sorry, but I was expecting an answer from someone at Linode. Since their
update page was not accurate I couldn't rely in their last report. I think the
answer is: it is possible and it is likely.

------
unholiness
It's so startling to see failures so...unabstract. We depend so much on the
layers of the internet fulfilling their protocols perfectly, that seeing a
vulnerability like this is just bizarre.

Of course xkcd is relevant as always:
[https://xkcd.com/697/](https://xkcd.com/697/)

------
myth_buster
On a tangent, I'm curious whether events as the 8.2 earthquake that happened
today, could affect these fibers?

~~~
jlgaddis
I just recently read an article about this...

 _" Around 150 to 200 fiber-optic cable breaks are recorded each year. Between
65% and 75% occur in water depths of < 200 m and result mainly from fishing
and shipping activities. In contrast, breaks attributed to geohazards comprise
< 10% of the world average. However, seaward of the busy continental shelf and
upper continental slope, geohazards account for at least one-third of
breaks."_

\-- _" Insights into Submarine Geohazards from Breaks in Subsea
Telecommunication Cables"_

PDF:
[https://www.iscpc.org/documents/?id=1794](https://www.iscpc.org/documents/?id=1794)

------
mkj
Singapore has a fair bit of redundancy of cables. Perhaps linode are only
using a single provider (seems likely given they're cost concious).

The only Perth-Singapore cable has broken a few times on the past few years,
latency really sucks then going from 50ms to 270ms (routing via the US
instead).

------
nowprovision
Is there a global status of cut/degraded undersea cable systems? That would be
very beneficial. It seems that the connectivity providers Level3,
Hibernia/Atrato, HE, NTT etc.. only relay limited information, perhaps their
notification from cable providers is very limited too, by the time it gets to
downstream customers we're all guessing, e.g. on WebHostingTalk there are
customers trying to determine latency issue to a Hong Kong provider, only to
have a competitor announce a fiber cut 2 days ago in Jakarta may be affecting
it.

~~~
nmc
You are wrong on one part: connectivity providers usually provide detailed
information about failures to their customers [1]. The only thing is: you are
not their customer — your ISP is.

Some ISPs will communicate to their own customers, i.e. you, for instance see
the link NiekvdMaas posted in the thread [2]. Sadly, most ISPs do not relay
any detailed info.

[1] I once saw a fully detailed report from a connectivity provider about a
cable failure and repair, with pictures of scuba-divers included — the cut
happened on fibers running at the bottom of a water canal.

[2]
[https://news.ycombinator.com/item?id=11213555](https://news.ycombinator.com/item?id=11213555)

------
dev1n
I hope this isn't China what with their recent land grab in the pacific [1].

[1]: [http://www.independent.co.uk/news/world/asia/us-issues-
stron...](http://www.independent.co.uk/news/world/asia/us-issues-strong-
warning-to-china-over-land-grab-island-in-south-china-sea-10286206.html)

~~~
Terr_
I wonder if any countries have "just in case" sabotage-devices near the the
undersea cables of possible enemies..

~~~
Phlarp
or a fleet of submarines that run around tapping them...?

~~~
Terr_
If you've installed a tap, it can just as easily be a tap with a limited self-
destruct (for deniability) and/or a bomb (for sabotage.)

~~~
Phlarp
The point is that the comm lines are more valuable as an intelligence asset
than they are as a strategic target.

------
jawngee
This is reported maintenance though.

[http://www.thanhniennews.com/tech/some-bad-days-ahead-as-
vie...](http://www.thanhniennews.com/tech/some-bad-days-ahead-as-vietnams-
internet-cable-will-undergo-maintenance-59782.html)

~~~
notwhereyouare
is it though?

From the link:

The Asia America Gateway, which connects Vietnam and the US, will be serviced
from March 4 to 6, the source told Thanh Nien.

That says, March 4 -> 6\. Also only specifies Vietnam.

Edit: Spelling correction

------
gtrubetskoy
Few people realize that to splice an undersea cable, you need two (big) ships.
This is because if the cut was where the cable is a mile deep, by the time you
bring the ends to the surface they will be approximately two miles apart.

~~~
davb
Aren't the cables run with a bit of slack for just this reason?

------
hellbanner
What caused this? Sabotage? Animals? Submarines?

~~~
jlgaddis
Let me check.

    
    
      $ bofh
      user to computer ratio too high.

------
CoachRufus87
how thick are these cables?

~~~
knowaveragejoe
Depends on the cable, but it's not unheard of to be a foot or more in
diameter. Here's a deep-sea fiber splitter:

[http://i.imgur.com/qh4Gf.jpg](http://i.imgur.com/qh4Gf.jpg)

~~~
amazon_not
This is incorrect. A subsea fiber optic cable is about an inch in diameter.
The splice cases and amplifiers are bigger.

~~~
knowaveragejoe
It's not incorrect, the original comment is vague. I think it is more
reasonable to assume they are talking about the entirety of the cable rather
than just the carrier embedded within it.

~~~
CoachRufus87
Indeed I was. Thanks for the clarification.

------
djsumdog
Fucking sharks

~~~
noobermin
Downvoted for being funny(?) but seriously, any speculation on what could have
cut them? Ocean currents? Debris?

~~~
yeukhon
I think it had happened before
[http://www.slate.com/blogs/future_tense/2014/08/15/shark_att...](http://www.slate.com/blogs/future_tense/2014/08/15/shark_attacks_threaten_google_s_undersea_internet_cables_video.html).

------
alfiedotwtf
Discuss: Kickstarter to fund unmanned submersibles to follow a cable end-to-
end, looking for taps

~~~
fabulist
I'm afraid that you'll only be able to follow them for a short while before
they slip below the ocean floor.

~~~
alfiedotwtf
What about duplicating how they repair broken cables... hook onto it close to
shore, pull the cable up to a boat, then follow the length of the cable across
the ocean.

I'm assuming that's how they're repairing this one?

