I spent a few years supporting these from the financial end. Modern cables are built in a loop so that they automatically fail-over. For instance, most Trans-Atlantic cables have pops in New York, Florida, England, and mainland Europe. If there is a cut in one segment, traffic automatically switches to the other.
Most customers these days are also on products that mux across these cables. So, if your Trans-Atlantic cable has two simultaneous outages, your traffic would automatically route itself across the Pacific. Your latency would go up, but your service would continue.
When I was there, construction was beginning on SMW-4, which had the dubious honor of being the first billion dollar cable. They are typically incorporated via international treaties between states and companies that are roughly about 100 pages long. Each partner has to cover maintenance on the part between the main cable and their drop, while everyone chips in on the main part.
It really is fascinating; outages are typically caused by boat anchors close to shore or large earthquakes. Once a guy in Hawai'i cut the Southern Cross wire with a pair of clippers while doing yardwork.
Also have the privilege of still working on these types of networks; one of the things that fascinated me was the money and effort that telecoms companies are spending these days on tracking boats/ships/fishermen by their transponders.
There dedicated people, equipment and networks just to track these offenders, as the telecoms obviously want to work out who to send the bill onto once they've fixed the cable, or which insurance company ;)
Yeah, the other funny thing is that there are only a few boats in the world that can repair breaks. So if there are too many outages, you just have to wait for them to finish and then sail across the world.
>Modern cables are built in a loop so that they automatically fail-over.
I know of a company that did this on a regional network - ring physical layout. Only to discover that murphy's law never fails to deliver and the SLAs suffered as a result. They switched to multiple interlocking rings soon thereafter.
So if someone can cut them with their clippers by accident, certainly a malicious actor could wreak tons of havoc? Why does this not seem to occur? Even from a juvenile "hey I turned off the Internet lol" kind of level?
As I recall there have already been cases of cutting cables to move traffic to other cables which have better monitoring capability. But as for the lulz, I expect that even the most naive of folks who might do this understand that it would get them some serious prison time, and there would be a huge amount of effort put into catching them and making an example out of them. Not quite worth it I'd expect.
That said, we've had folks take sniper shots at power substations in California[1], which can cut power to big chunks of real estate.
Overloading the 512K BGP tables was a fun way for people to toss a monkey wrench into the Internet. But then, it didn't affect everyone easily and nor would cutting a cable, which not only doesn't affect everyone, but, I imagine, is rather hard to do. It's much easier to mess with the Internet using software, and by doing so remotely, you've a better chance of not getting arrested for it, I imagine.
I recommend 'A Thread Across the Ocean: The Heroic Story of the Transatlantic Cable' by John Steele Gordon. [1]
From description,
"But in 1866, the Old and New Worlds were united by the successful laying of a cable across the Atlantic. John Steele Gordon's book chronicles this extraordinary achievement -- the brainchild of American businessman Cyrus Field and one of the greatest engineering feats of the nineteenth century. An epic struggle, it required a decade of effort, numerous failed attempts, millions of dollars in capital, a near disaster at sea, the overcoming of seemingly insurmountable technological problems, and uncommon physical, financial, and intellectual courage."
Also very enjoyable, and somewhat shorter (though still very long): 'Mother Earth Mother Board' [1], Neal Stephenson's essay on the laying of the FLAG cable, and related history.
That was such an amazing article. I still remember how long that story and how engaging it was. IIRC that issue of wired was like 20 pages thicker than the rest. Somewhere Stephenson talked about being grateful to Wired for not cutting it down to a 6k length. I still tell people the part about the library of Alexandria interconnect.
Isambard Kingdom Brunel was not a man to faf around was he. He never even got to see her sail her first journey as he died of a stroke. You've done a lot if you've died at 53 and achieved as much as he did.
True. Then again my grandfather led the charge on laying one under decidedly difficult conditions and that thing is still live (carries insignificant traffic though...mostly for backup).
Me too but then if you think about it you can lay a cable around the biggest part of the earth the equator that's about 40,000km. If you have a geostationary satellite that's about 35,000km. Pretty close but latency is a tiny bit worse for the satellite T=D/V
Equator at 40,075km / 209,855km/s=0.19s (saw a Stack Exchange question about speed of light in optical fibre slows light 30%).
And for satellites 35,000km / 299,792km/s=0.12s but I'm sure air, humidity, big balls of floating water aka clouds, solar radiation just weather in general must make it far worse I've seen mentions of typical latency of 0.25s.
But most cables are not 40,000km long I don't know what the longest cable is of the typical length probably 20,000km max that would give 0.09s latency and for 10,000km 0.05s.
The cables themselves fail surprisingly often (anchors, fauna, earthquakes, material faults...) but most connections have redundant paths (almost every undersea network is bidirectional ring) and can be readily repaired (which is the thing that still surprises me)
This long article from Wired in 1996, Mother Earth Mother Board [1], would answer your question. It is long, it took me more than a couple of hours to read, but it's also my all-time favourite article from any magazine, and whenever the topic of submarine cables comes up on Hacker News I often see this link.
I read the entire article. It does not go into detail about the deepest parts of the ocean. The closest it gets is when it talked about the calculations required to lay cable from the UK to Africa, but this is still close to shore (I remember seeing a diagram showing how the cable angled down a steep slope into deep ocean, but this archived version doesn't seem to have pictures).
Edit: this is what it actually gives you as an answer in that article:
"The answer has to do with slack control. And most of what is known about slack control is known by Cable & Wireless Marine. AT&T presumably knows about slack control too, but Cable & Wireless Marine has twice as many ships and dominates the deep-sea cable-laying industry. The Japanese can lay cable in shallow water and can repair it anywhere. But the reality is that when you want to slam a few thousand kilometers of state-of-the-art optical fiber across a major ocean, you call Cable & Wireless Marine, based in England. That is pretty much what FLAG did several years ago."
"Beyond 1500 meters from the shoreline, builders typically lay unarmored cable directly on the seabed ... a few reach depths greater than 8000 meters."
Not much of an answer. How the heck do they lay it and control slack at greater than 8000 meters? How do they verify the cable is on the seabed at those depths?
In some cases the cable is buried on the sea floor by a special plow. It's incredible that the cable can be buried up to 6m deep in the mud on the bottom. Cable burial is intended to keep fishing equipment from snagging cables laid on the bottom.
Jets of water are used to bury the cable closer to shore, and if I recall correctly, Google used Kevlar on their latest cables to reinforce them against sharks.
It's interesting to see how much politics weight on the connections. E.g.:
1. There more connections between USA and UK than to non-english speaking European countries.
2. Venezuela is the only country connecting Cuba.
3. Brazil connects with Cape Verde island and will connect with Angola, another portuguese-speaking country. Those will be the only connections crossing south Atlantic.
4. Southern Asia connects to Europe circumventing the Middle East by connecting to Egypt and then crossing the mediterranean.
Since the position of many cables differ strongly in both maps (their position is obviously sketched rather roughly) I wonder what the real position of those cables is? (or if this information is secret)
When a telco hired me (they were the owners of the portal I worked for) I got, as a gift, a book with beautiful maps detailing where their fiber network, down to street corner level. I called it the Modern Terrorist Manual.
With that book in hand (and I don't know how many copies were actually made) any resourceful bad guy could knock off about 60% of Brazil's phone and data network.
Yes, it is scary when you realize how susceptible some of this infrastructure is.
Reminds me of an instance in 2009. I was working in Santa Cruz, CA. I was configuring some of our servers remotely, when suddenly I lost internet connection. The whole office was out. VoIP was down - no signal on my cell phone as well. I went to the front desk and tried the land line. No dial tone.
This was weird.
I saw a few people from other offices milling about the courtyard. I went outside. They were all experiencing the same thing. A few of us went to Starbucks. Couldn’t buy coffee because their registers were down.
At this point, people were starting to leave their offices in droves. Santa Cruz PD actually began to have a few officers walk around the area, since no one could make phone calls.
The only thing that worked was Verizon cell phones. These were being passed around so people could make calls (data did not work however).
All in all, this lasted about 6 hours.
This was caused by a single frustrated former ATT employee. He just went in and clipped some fiber lines and left.
I'm a little surprised Starbucks wouldn't just revert to cash mode? Who cares about exact prices when everyone involved knows that each items costs no more than a few cents. Just round to the nearest dollar, and any competent manager would reward such behaviour given the alternative.
At least this is my experience with pharmacy billing. If we couldn't put claims through due to network problems, we'd just estimate based on your previous claims and keep things humming along.
> I'm a little surprised Starbucks wouldn't just revert to cash mode?
The register - I speculate - is also tracking inventory. Ring up two cappuccinos and a pastry, corporate keeps track and knows what and when to re-order.
Getting the inventory out of whack might be worse than loosing trade during the outage.
It's pretty easy for anyone with half a brain to target infrastructure.
I worked in places that made it difficult even for employees with a need to know to access network data and locations of wan fiber connections. Half the time, the cables are clearly marked on the street -- in New York, you call 611 and a team of people will draw on the street the locations of all telecom, electric, water, sewer and other infrastructure right on the street for you.
I wonder do they just drop the transatlantic cables blindly over the mid-atlantic ridge and hope for the best or check that it's not a hot spot or something.
And four cables connect Alaska to the lower 48? I wonder why.
I live in southeast AK. An earthquake near Glacier Bay this month broke one of the undersea cables, which caused an outage for customers of one service provider. I had no idea there were multiple cables to southeast, until I read about the response to this incident.
The Capital City Weekly has a good article about the recent incident, which also includes a short history of communication cables in Alaska [0]. Basically a few companies built seabed cables in southeast during the dot com boom, and then there was too much capacity for a while. One company went bankrupt and another company attempted to buy all the capacity, but concerns about a monopoly prevented that deal from going through. Now when there's an outage, companies can lease access from each other until the broken cable gets repaired.
The currents in our coastal channels are significant, with 8-12 foot tidal swings. Those currents, combined with the silty and rocky nature of the ocean floor in much of southeast, makes for a pretty abrasive environment. Add to that a lot of bottom fishing and anchoring, and cables get broken.
As a resident of southeast, I'm happy for the redundancy!
Cost estimates for terrestrial cable are harder to come by, and seem depend a lot on things like rights of way (likely not relevant here as a cost), terrain (_very_ relevant), ability to get equipment to the right location, etc. But the numbers I'm seeing on various links from https://www.google.com/search?q=cost+of+laying+fiber+optic+c... are in the $20k to $40k range per kilometer (I took the actual numbers I saw and divided by 1.6) about 5-10 years ago. Figure about 1.4x smaller if you want to adjust for inflation to compare to '90s numbers.
All of which is to say that "zillion" in this case is probably a number somewhere between 1 and 4. Give or take; these are all estimates, obviously. Also not clear to me which is faster to lay, by the way, which might matter in terms of deciding which one to do.
Not to mention that the undersea cable might be shorter, offsetting some of the per-km difference in cost, since it's a lot easier to go straight.
The alcan highway is pretty long, and passes through some pretty remote regions. You'd have to run cables from the alcan down to Skagway, and then you'd have to run cables underwater to connect the various coastal communities that are on islands, or far from developed roadways. Juneau, Ketchikan, Sitka, Wrangell, Petersburg are all communites with 2,000-30,000 people, and none of those communites connect to the outside world by road.
It's easier to lay a subsea cable than to go overland in remote areas. Take a look at the subsea map of Africa as an example. In Africa the subsea links are there because it's too remote to run cables through the jungle or across the Sahara. Additionally terrestrial cables are more frequently cut than subsea cables. In India terrestrial fibers are cut routinely every day to the point that they have to allocate extra repair margin for the links.
> James M. Barrett, a former deputy director of international engineering for AT&T , said that there were 55,000 or 60,000 miles of old undersea cable made out of copper that did not have a single case of shark biting, meaning sharks prefer to snack on fiber optic cables specifically. It is believed that the electrical current in the fiber-optic lines attract the sharks, triggering a feeding reflex.
"electrical current in the fiber-optic" but not copper?
I guess I figured copper wire would also required amplification.
Interesting on the return path. Where does the source connect i.e. one side of the "battery" goes to the amplifier (which returns to the sea, not the battery) and the other side of the battery goes to where? the sea?
I think that electrical cables have pretty low current. The fiber optic ones need to power amplifiers, so they have quite a bit of power going through.
Yes, copper cables have repeaters too. And I would expect them to work at similar voltages.
The only difference I can think of is that the amplifier power is probably a steady DC (and corresponding magnetic field) whereas the signal in copper will be varying voltages - creating a fluctuating magnetic field that repels the sharks ?
I imagine you can use wget or curl to download the images (or other batch download tools that let you put in ranges), then ImageMagick to stitch them together. Not too hard, but all command line tools that may take a bit of experimenting to get right.
Eh.. This map is not useful whatsoever for reference, other than in a general "sales guy" sort of sense.
Speaking for my company, I buy these each year because A) It looks great framed in our lobby and B) I want to support continued development of such things.
When you compare them to actual fiber maps, they are accurate mostly in the general sense and not all that useful for planning specific routes anywhere.
anyone have any idea what happens to user experience when one of these guys breaks? is it a big deal? what about islands who are surviving off of one cable connection?
If only a single cable break occurs, every point is still reachable from every other point -- the worst thing that happens is that ping times might increase because the packets will have to travel a longer route.
As typical L2 technology for such cables is SDH/SONET, the difference in latency after single ring failure tends to not be noticeable for normal IP traffic.
The total length of the cable is 'only' 15,428km. So the maximum one-way latency between any two points is only ~70ms assuming 0.7C. You'd definitely notice if you were gaming, but probably not if you were watching YouTube.
The map notes a "resiliency" factor for each country. A country with a single link has very low resiliency. Most likely a lot of people lose their connection while higher paying customers/more important data gets routed to satellite communication systems.
Was thinking the same, considering that the first few transatlantic landed in Ireland. Recently visited a section of the first transatlantic cable in Valencia Island no idea how they got much of it on a ship.
The cable repair ships send down custom ROVs that can bring up the broken cable ends. The cable cut is repaired on ship and then placed back on the sea bed. Amazing.
At the end of the day these submarine cables are just optical cables, that is a really long strand of glass that you can send optical light (laser) down, and it'll pop out at the other end.
So the capacity is really based on the equipment at either end (and repeaters if required), the cables can degrade overtime due to hydrogen impregnation and damage from ships etc... your also limited by distance before you need to repeat, however basically there will be newer and faster hardware on the market every day meaning there is no way to work out the total capacity.
I've seen unrepeated short span submarine cables only lit/working at 10Gbps, the equipment at either end can support substantially faster (100Gbps+) with some small hardware changes, however simple business sense means your only going to provide capacity for what your customers are only demanding.
Prices are also more relational to the traffic going over them cables (and demand) rather than the distance, for example it can be cheaper to get from London to New York on one of these cables than it is to get around the UK.
The lack of interconnect between Africa and South America is interesting. Wouldn't that be a better route with other transatlantic failures than moving to Pacific transit?
A few weeks ago I read an article on undersea cables somewhere. I think it was posted on HN. It mentioned about a planned cable between South America and Europe for exactly the reasons you mentioned. A quick duckduckgo gave me this link: http://www.networkworld.com/article/2174686/data-center/eu-m.... It mentions the NSA surveillance as one of the unofficial reasons for laying this cable.
Linked by ALBA-1[0], which apparently is carrying public traffic since early last year[1], although everything in Cuba is highly restricted and filtered by the state. Worth reading the Wiki article[2]
all five cables landing at that tiny country in the Horn of Africa, Djibouti? they terminate into half a floor of a single building about 150ft from the shoreline. the NSA has been tapping submarine optical cables since the early nineties.
for the patch panels they can't get to, there's always the covert submarine they are purported to have.
That part of world has been capitalizing on its position with regard to trade routes since the Sumerians pioneered exchange with the Indus Valley civilization. It's remarkable to see this development as the latest iteration of a truly ancient tradition.
This really isn't such a horrible comment. It illustrates nicely that "common sense" is often wrong. There was a similar reaction when radioactive material was stolen in Mexico. People said it should have been escorted with armed guards. But in truth, shipping radioactive material is extremely common. Three million packages of radioactive materials are shipped each year in the U.S. alone. So while it seems like radioactive shipments should have armed guards, it isn't actually practical.
In this case, the truth is that anyone who has the capability to disrupt these cables already knows where they are. But that isn't necessarily intuitively obvious. This is a reminder to all of us that we shouldn't blindly trust intuition, but should try to find numbers or other hard data to back it up.
What? How? It's as dangerous as posting a map of nuclear power plants online. It's not like the cables go into unguarded shacks - the base stations are heavily guarded.
Most customers these days are also on products that mux across these cables. So, if your Trans-Atlantic cable has two simultaneous outages, your traffic would automatically route itself across the Pacific. Your latency would go up, but your service would continue.
When I was there, construction was beginning on SMW-4, which had the dubious honor of being the first billion dollar cable. They are typically incorporated via international treaties between states and companies that are roughly about 100 pages long. Each partner has to cover maintenance on the part between the main cable and their drop, while everyone chips in on the main part.
It really is fascinating; outages are typically caused by boat anchors close to shore or large earthquakes. Once a guy in Hawai'i cut the Southern Cross wire with a pair of clippers while doing yardwork.