Hacker News new | comments | show | ask | jobs | submit login
How Syria Turned Off the Internet (cloudflare.com)
435 points by dknecht 1639 days ago | hide | past | web | 82 comments | favorite



I love these highly technical blog posts on recent events by cloudflare.

Keep up the good work guys.

[1] http://blog.cloudflare.com/why-google-went-offline-today-and...


I swear I just thought the same thing to myself.. Very concisely written, but it's written both interesting and well. Not to mention it's always welcome to see large companies like this share vital information with the public [+1 internet points for CloudFlare] :)

I wonder if it would be possible to hack into these "wall" routers from inside the country (or I suppose outside), but something tells me probably not (I doubt they would even be ping-able, but maybe through other attack vectors [like other internal computers]...). Seriously, I can't imagine how enraged I would be if someone decided to simply block the Internet.


Since BGP is sent over TCP, the routers are definitely pingable. You know how when you run traceroute you see a bunch of IPs between you and your destination? Routers like these are those hops.


Ah, sorry, I meant now, in the current state. I wonder if they just shut them down (the routers), or blocked all incoming connections from the outside...


Probably the simplest thing to do would be to just shut off the ports leading to external networks at the physical layer using management commands. They'd not be pingable from the outside, naturally, and it'd look like a cable cut (until you start doing TDR on the cable anyway), but they could bring it back online quickly once the powers that be choose to do so.


Most providers setup bgp sessions over loopback addresses only reachable from the remote router.. So no, being tcp doesn't imply the should respond to ping.


I came here to post just this. IMO it's one of the best tech blogs on the net right now.

It also seems like it would be a really cool place to work.


Thanks for the kudos. This was a really interesting story to follow and write. We've got monitors setup so we'll see the instant Syrian traffic begins flowing again.

And, if you're interested, we definitely are hiring:

https://www.cloudflare.com/join-our-team


I was just checking if Renesys[1] had any analytical updates, but hey, kudos to Cloudflare for a well-written and visualised post!

[1] http://www.renesys.com/blog/2012/11/syria-off-the-air.shtml


I love it too. I always enjoy reading about post-failure posts from GitHub about their own infrastructure. CloudFlare pushes it to another level in these posts.


That is a frickin' awesome video. Does anyone know what tool that is, I think I want it running on my status displays.

For those who are wondering your edge router (or border router) "advertises" that it can route to a particular subnet. That information propagates around and packets find there way there. So someone in Syria told all of their border routers to stop advertising routes to Syria's IP blocks. Now the fun thing you can do is since they aren't advertising those routes, if you are sitting in a data center somewhere and have peering access and a ASIN id you can advertise those routes and all of Syria's traffic will start heading your way :-) Of course if that monitoring tool is still running it will have all these lines suddenly running off the screen toward your data center.

The traffic that is actually in Syria can't get out. So its not like you could snoop on Syria or anything.



For those trying this tool the IP prefix is 5.0.0.0/18.

29/11/2012 10:20 to 10:30.


Thanks! That is a neat tool.


The traffic that is actually in Syria can't get out.

I'm not really routing savvy, but isn't it a little bit more like, the traffic could get out but no one could reply?


Not being able to get replies back would mean that setting up a new TCP connection would be impossible because you're not getting replies to the initial handshake. The TCP connections that were alive when the routes were dropped would time out pretty quickly if they weren't getting any replies. Whether UDP packets could get out is going to be down to how the connectivity was dropped. If it's just via BGP then there's a chance.


Since DNS (usually) uses udp, does anyone have access to information showing that syrian lookups are coming out?


I guess it depends on the equipment, you would have to configure them to 'default route' non-local IPs to some upstream provider, but since router doesn't know how to return them it probably just whines. But if you're cutting off the country (and it did look like that from the Cloudflare page that someone in Syria configured the routers to stop advertising a network) then you probably wouldn't install a default route to a hop outside the country.


This tells us nothing about how it was turned off, only that the routes were withdrawn from the global routing table. Which isn't "how" it was turned off, by why it's stopped working.

There's some guesses in here, but the title is rather misleading. No one still knows why these routes are no longer being advertised, only that they're not.


Considering Syria is saying cables were cut even though Cloudfare pointed out 4 cables would have had to been cut simultaneously with a couple being under sea it's seriously doubtful this is what happened. The only plausible explanation is that the traffic was stopped and given Syria's reputation for trying to stem free speech and the flow of information about what is really going on kind of makes sense.

The blog post also makes mention of traffic being disrupted a couple of times on varying scales before this major outage occurred which kind of signals that they were testing bringing the whole Internet down beforehand.


Most likely was shut down by the government but 4 guys with cable cutters, wristwatches, and a plan is not totally implausible


Even if that isn't what happened, sounds like a wonderful plot for a movie. Want to help me get started on writing the script?


Kind of muddles up why and how, but it's quite clear that someone made a call to the state ISP, and then someone went around reconfiguring the routers to drop all the published routes into Syria.


How many routers do you think need to be reconfigured to do this? Is it a lot, or would it be easier to simply choke the lines or turn them off?


From the article: "our network team estimates that Syria likely has a small number of edge routers." I'd imagine it was an easy to implement given the appropriate access.


Or the state coordinated to take down the physical lines at the same time.

Which I think is more likely, considering the relatively few points of failure.


I'd say that's more unlikely. It's just a lot easier to sit at your desk and issue a couple of commands, which will guarantee getting the job done. Unplugging things is messy and unreliable in contrast, with redundant links, rings, and who knows what.


In the past, netsweeper has been one of the tools employed by governments looking to accomplish incidents like these.

http://thenextweb.com/me/2011/06/07/this-company-is-helping-.... http://www.thestar.com/news/canada/article/1218965--guelph-t....

They have previously been used by Egypt to close down their internet. I have no insight into how this particular incident was accomplished, but this sort of software is where I'd start looking.


At one point when I was looking to see what was available in the job market, I saw a posting that I was a suitable fit for with their organization (netsweeper). It got me curious, being tangentially aware of their organization and content filtering focus. Their perspective was very contrary to my own, and I was interested in having the opportunity to have them pitch what they did to me. I hadn't really had the chance to talk face to face with what I considered to be an "evil" organization before. Maybe it was unethical for me to waste their time, when I didn't have the intention of working for them, I don't know.

The interview itself was interesting. The technical tests etc were relatively trivial, mostly stuff you'd expect about load balancing and responsiveness (as would be required for routing software). But more than half of the interview was them pitching the company to me -- they're quite aware of the reservations people might have about working for a company whose direction is to suppress free speech.

Their basic pitch was that they believed all 1st world governments would have internet filtering technologies in place in the near future -- that governments would legislate the use of it to stop the proliferation of things like child pornography, and eliminate the zombie-computer defence -- that if there were child porn on your computer, you went and got it.

We talked a little about the technical challenges of instantly shutting down the internet of an entire country, like Egypt, but they more or less blamed that on "improper configuration" of settings. At one point in the interview, the interviewer told me that you have to believe that the people that make the weapons aren't the ones firing the guns, and that dangerous tools can be used for good, but that yes, there were nights in which he cried himself to sleep.

Super interesting interview.


I still find it interesting as someone who if you'd ask how to turn off a country's internet, would give you a blink stare in return.


If you read a little about BGP you could figure it out relatively quickly.


Maybe it doesn't tell us directly and the title is a bit strong, but so is the word 'nothing'. Cloudflare tells us quite a lot. It is unlikely the terrorists cut all the lines, so that is out. That is information I didn't have before this blog post.

That leaves a few options of how to shut off the internet of a country, and they give a pretty plausible one, and who would have the authority/power to do it. And they back it all up with evidence.


This doesn't necessarily explain HOW the network was taken down but, it does highlight the conflicting evidence between a "terrorist" style fire-sale attack, and a state-imposed outage to limit communications between dissidents.

Unfortunately, non of these hosting companies want to give an alternative to HOW to bring the network back up...

Anonymous seems to be the only group oriented at actually helping the citizens of the nation of Syria regain communication via alternative methods such as TCP/IP over HAM radio, and satellite links, personal wireless mesh networks using WiFi on mobile devices.

Everyone can bitch about HOW to take DOWN a nations internet but, it takes real humanitarians & 1337geeks to consider & implement HOW to bring a nations communications infrastructure back UP.

So, what are you waiting for...

HELP.


Syrians can still dial an international number and get online via modem. So you are better off sending money to aid organisations to pay the phone bill.


It was posted that international calling has also been disabled, as well as cellular I believe. Unfortunately, I can't locate the source.

It is the same source in which it stated that the data center that handled the BGP routes was being held by 6 remaining techs whose current location is now unknown.

I wish I could find the link as it proves much of what the OP posted. Not to mention I would love to see someone repair one, let alone four cables in fifteen minutes. Heck, I bet it may take more than fifteen minutes to cut one.


This is probably the best explanation of the Syrian outage I have ever seen. Cloudfare are exceptionally good at explaining things like this and even included a video of the Syrian traffic slowly dropping off. Expect those "cut" cables to miraculously be repaired shortly.


This isn't a good explanation of anything.


Would you care to embellish us all with your expertise then? I'd love a link to your blog post explaining the outage. This isn't Reddit, it's Hacker News and on HN if you dispute something you have to at the very least lightly back it up.

Maybe you're right, what would a company who was voted Most Innovative Network & Internet Technology Company of 2011 & again in 2012 by the Wall Street Journal know about networking and how Internet traffic is routed, right? The very fact they even embedded a video showing the traffic dropping off in the blog post I think proves they know a thing or two about network traffic, they provide content delivery and domain name server services to a lot of happy customers after all.


My wife's first thought when she heard about Syria's lack of internet was that the sitting government is about to start a nasty offensive.

This was a good read and informative. One question, though.

> When the outage happened, the BGP routes to Syrian IP space were all simultaneously withdrawn from all of Syria's upstream providers.

Does withdrawn mean not advertised or was a message sent out saying these routes are no longer available?


Once a route is withdrawn (either from the router going offline, a cable being cut, or someone adjusting the routing tables) upstream peers will drop the path very quickly (typically in seconds).


I believe that once it stops routing most providers will shut you off relatively quickly to prevent BGP from flapping. I'm sure that someone else could articulate this better than me.


There are a few really baseless, attack-y comments on the original post. I wonder if they're some kind of Syrian social media reactionary force.

CloudFlare, if you have access to their data, what's interesting about them?


I have a few questions not knowing all that much about BGP other than thinking of it in terms of a higher level DNS system for IP routing.

What gives Syria the authority to do this? What gives anyone authority to do this? What prevents malicious routing?. Could they route all traffic to 8.8.8.8 and overwhelm another network? If it's ICANN, can they come in and revoke control and give it to a third party intermediary?

It has been rumored a small staff of 6 stayed trying to keep the routes up. Their current status is not known. I understand in those cases no intermediary would help. But from a pragmatic standpoint, I'm curious.

While I understand this goes against the "rules" but if I have a DNS server and the roots drop a zone, and I don't agree, I can add it back in. As a local user I could add to etc/hosts like in the old days.

The above assumes I had a large user base like openDNS or google pDNS to be effective. Can the same be done with BGP? Can major broadband providers decide to ignore the dropped routes and send traffic along?

I understand Syria would just toggle off some other "switch" and terminate core routers but it would at least send a tiny message of sorts.

How are they stopping satellite access?

How are they stopping cellular based access?

Is there any form of TCP over Ham radio? TCP over laser? USB over carrier pigeon (seriously)? What are the bare bones options here for getting data in and out and where is that closest point of access?

No one has put in long range wifi links of the 20 mile line of site type, or is that still too short a distance to get a few users online?

What about dialup?

If CloudFront can see this much traffic, they must be doing pretty well. What is the point of Facebook, reddit, and many others using CDN's and Amazon and such when they could probably half their hardware and push the rest to ClouFront. Or is CloudFront really only best for static sites that are hit hard and need lots if bandwidth. Dynamic sites would still bottleneck at the database/drive/physical/etc layer?

Thanks. Sorry if these are rudimentary questions. I haven't even met that many people who have had BGP access. I'm going to go look up what the format files look like now, just out of curiosity.


  > Can major broadband providers decide to ignore the
  > dropped routes and send traffic along?
IIRC, each router makes it's own decisions. Take the following route:

  A - B - C - Syria
Assuming that there are no published routes to Syria, if node A tries to send a packet to Syria, the only node that can force a packet onto the Syrian routers is node C. If node B decides to ignore the lack of a published route and forwards the packet to node C, then node C will just drop it (possibly sending an error response back). Even if the node C forces data over the Syrian connection, the Syrian routers won't act on it.

This is my understanding of how it works.

  > How are they stopping satellite access?
Presumably very few Syrians have satellite access. If they are from a Syrian provider, then it's pretty easy to cut them off. If they are with a foreign provider, not so much. On the other hand, if you were in Syria when they Internet was shutdown, you would probably be very secretive about your foreign satellite access. If only because men with guns might have something to say about it.

  > How are they stopping cellular based access?
Presumably because the state can go to the cell providers and shut them down. What is Syria's cellular data infrastructure like?

  > Is there any form of TCP over Ham radio?
There are ways of getting Internet over HAM radio, but this probably runs up against the same friction as a foreign satellite connection. Especially since the equipment would be more conspicuous.

  > TCP over laser?
Really?

  > USB over carrier pigeon (seriously)?
USB is a client-server protocol. It probably wouldn't do too well over a carrier pigeon.

  > What are the bare bones options here for getting data in and
  > out and where is that closest point of access?
Probably a directional antenna pointed over the border to a line-of-sight receiving station.


  > What is Syria's cellular data infrastructure like?
Well, there are 2 cellular providers. One is completely state owned and called "Syriatel" and the other is MTN(http://www.mtn.com/). Though MTN is extremely heavily regulated by the state.

Also, Syria enforces a web filter(similar though not as sophisticated as china's) which also affects browsing over the cellular data network. So even cellular providers eventually go through some choke point which is state-owned so that the filter can be applied.

In other words, if the state wants to cut off internet, cell isn't going to save you.

  > USB is a client-server protocol. It probably wouldn't do too well over a carrier pigeon.
I think he meant literally sending a USB thumbdrive via a carrier pigeon. See this: http://www.telegraph.co.uk/technology/news/8007897/Carrier-p...


TCP/IP over optical links may be a good option if they are just triangulating radio.

HERE is a link to an article featuring full-duplex voice over cheap laser/LED optical link over 95 miles.

http://hackaday.com/2012/09/22/communicating-with-a-beam-of-...

If anyone can get that to the "right-side" then, by all means this may very well be an option for establishing high-speed TCP/IP as well.


> Is there any form of TCP over Ham radio?

Yes, there is AX-25 packet radio and there is TCP over that.

http://en.wikipedia.org/wiki/AX.25


They can't stop satellite access or dialup to modems outside Syria (unless they spike the 'phone network), or TCP over HAM radio, or anything else like that. More than likely there are people getting access like this. However, anyone using any of those methods won't have an IP Address from the block assigned to Syria, they'll have an IP address associated with the Internet access provider they are using (Satellite, dialup to another country, etc).


There are surely people online in Syria, just are there are people online in North Korea (very few). They're just not going through the official channels and their IP then appears to come from neighbouring countries/the US etc.

And yes, if the phone system is still operational, the easiest method is to use an international dialup service.


International phone access in Syria is abysmal though. I've been there and to call my parents in neighbouring Lebanon would sometimes take me ~20 attempts to get a successful phone call through. It almost never worked in less than 5 attempts. My hunch is that they artificially limit the number of simultaneous international phone calls so that they can more easily monitor all of them.

Further corroborating that theory, when we first got there, we were told by Syrian friends never to discuss anything that might be possibly misconstrued as political over the phone and especially international calls. I was there when the trouble started(I eventually left around June of 2011, some 3 months later) and during that time I'd have friends/family call to check up on me and I could never give any status updates about what was happening just that my surrounding area was calm and that I was fine.

TL;DR; I would guess that international dialup, if you even managed to keep it up for any length of time given the difficulty of calling international would quickly flag you as "suspicious" as these things are heavily monitored.


They mention that there are four cables "connecting Syria to the Internet."

OK, I'm curious! How do they know? How many cables connect, say, Brazil to "the Internet?"

Is this publicly available somewhere?



That's a very cool map! I see it is part of a larger article that ran in Fortune magazine back in July 2012:

http://nicolasrapp.com/?p=1180

Good find!


At the bottom of the Renesys blog post, the folks there have a map showing the cables into Syria:

http://www.renesys.com/blog/2012/11/syria-off-the-air.shtml


this was posted on HN a while ago... only undersea cables tho. http://www.submarinecablemap.com/

and if you've got an hour (or two) to waste http://www.wired.com/wired/archive/4.12/ffglass.html


They've been blocking traffic in Syria to numerous social networking and email/voip sites since the revolution began, and nulling cell towers in every area where there are protests. This hasn't hampered the free syrian army or activists as since last year they've been passing sdcards to the border of lebanon, jordan, iraq and turkey and uploading their videos wirelessly from there. The FSA is running two border crossing with turkey anyways.

This seems to be like some sort of incompetence, more like they tried to set up some sort of spying choke point and it massively failed. ask nokia-siemens, they helped iran set up their chokepoint, most likely some infosec whitehats with zero ethics are currently flying out there to assist in the holocaust, er I mean rebellion put down, by working with Assad to get the tubes back up.

It only hurts Assad to keep the tubes out, his loyal base needs to buy their louis vuitton bags off alibaba to keep their minds off the constant public shooting of protesters and shelling of entire cities full of "terrorists"


Well, Syria already does have a spying chokepoint in place. It's had it for years(at least since late 2010 when I was there). They have something similar to China's in that it also does filtering. Also, at some point when I was there around may~june 2011 when this was all starting up they seriously started messing with https stuff conducting MITM on facebook and what not.

Also, no, having the internet cut off absolutely does not hurt Assad. Syria is very sectarian in nature. In other words your loyal base absolutely does not correlate with what services you provide, only what religion/sect/tribe people are.

The rebellion didn't happen because people suddenly started hating Bashar el Assad, it happened because those same people have hated this regime for at least the past 40 years. In 1982, Hafez, the guy who established the regime put down an earlier rebellion that had gone on for something like 3 years by completely destroying entire sections of cities(and pretty much all of Hama) and killing, by conservative estimates, at least 10,000 people. This was during a time when the USSR still existed and where such things within its sphere of influence were par for the course, which is why no international response happened at the time.

The rebellion happened now because of a perceived weakness in the regime's freedom to act(no USSR anymore, much more US influence in the area, weak(er) dictator) and because of perceived international support for similar things happening in Arab nations. That's it. So if the internet in any way helps the FSA get international support by exposing atrocities committed against civilians by the government then you can bet it's in the government's best interest to shut it down.


Distance between Cyprus and Syria is < 200km. How much money would it take to setup a wireless link between the two? Wikipedia tells me it's possible (http://en.wikipedia.org/wiki/Long_Range_WiFi#Italy)


Why not just use Turkey, Iraq, or Jordan, which are approximately 0km from Syria, and hate Assad too. And are already serving as basing, conduits for personnel and materiel (particularly Iraq and Turkey), etc.?


I guess the cost of securing your 'bridgehead' would dwarf any cost associated with the actual link technology. And then, what did you achieved? You have your packages going back and forth, but how will you distribute the connection? Will you hijack the presumably uncooperative ISP's network, or just use sneakernet? If you choose the last, then you might as well ferry the information via boats. (Not saying that's easy, just it seems easier.)


This is actually a great application for UAVs, and I think there is some DOD and/or DOS money to do it (not for Syria specifically, but the general application.)


Syria would blow it out of the sky. They also are targeting all satellite signals in their borders with mortar shelling. That's how all those French journalists got owned, they turned on the cameras, set up the uplink and 30 seconds later bam here's a face full of explosive censorship enjoy. They simply locked on to the signal and shelled it not caring who was there.

They've also been targeting wireless uplinks across borders and adhoc cell towers. Syria levelled an entire apartment building full of kids in Turkey to destroy a suspected adhoc cell tower they claimed was being used by "terrorists". This is serious business, no piratebay UAV drone solution is going to work. Mesh networking is also certain death, any building with a mesh router on it is going to have all its inhabitants dragged out of their apartments and shot in the street so they've been using good old walkie talkies and bouncing signals all over the place to confuse the syrian army, and in most cases simply stealing Syrian army comsec devices and pretending to be them talking in slight code with each other to avoid suspicion. Al Jazeera ran a story with some smugglers who used gps disabled cellphones but changed the IMEI every couple of hours to avoid being found


They would be more able to go after terrestrial radio and L-band portable satellite systems (e.g. Thuraya DSL, thuraya sat modems, various forms of BGAN/RBGAN/etc., which is what journalists tend to use) than Ka or Ku band satellite.

I haven't kept up on Syria or Libya (I wasted 2003-2010 on this stuff in Iraq/Afgh/etc., and am trying to do a "normal" tech startup now), but while I think Syria (and Libya) had better European gear than Iraq or the Taliban, it isn't on par with the US, UK/FR/DE, RU, CN, etc. It's basically "good commercial equipment designed for law enforcement", which is very heavily cellphone focused.

The #1 vulnerability with satellite systems remains "operator assistance to the adversary", or "network configured in a way which relays location data of connected terminals to everyone in the footprint", both of which can be addressed if you control the network.


Since Google has also confirmed that Syria is offline now, we should spread the word about the telecomix dialup project: http://dialup.telecomix.org/

How could the information on that page be spread within Syria? SMS, MMS, Phone, Fax, Mail?


after telecomix published probably dangerous and incriminating traffic data, I feel extremely uneasy about their competence and especially about routing any kind of information through them.


Great post, too bad the comments section is already full of syrian trolls trying to derail the subject from freedom of speech to some bullshit conspiracy from "the empire" against syria.

Guess they still give access to those who are loyal...


It's probably not syrians but Russian/Chinese who are worried about having yet another ally switch over to camp USA.


Its great to know how, but here is why: http://www.youtube.com/watch?v=H40EsEVU1Wk NSFW/NSFL

Apparently, bombing people turns out to be bad PR when they tweet about it.


I don't know enough about markets to find the answer to this. Does Syris have a "stock market" or some form of electronic exchange? I would imagine if they have money they do.

Did those routes stay up? If not, what are the repercussions of that going to be come open time? Or any foreign trade, anything that relies on network time, etc. the list is pretty large for services that need Internet at least infrequently in order to keep basic services up.

I can't imagine how many deaths there would be in the USA if this happened. All those televisions stop working, that's gotta be a few million heart attacks right there.


The only form of stock exchange in Syria is the Damascus Securities Exchange. And that's it. Syria is pretty much a Socialist nation. Under Bashar it had been slowly moving toward a more capitalistic model but that was pretty much just crony capitalism full of corruption and the state still pretty much running everything even though it was supposedly private.

So no, no markets in Syria.


Easy way to shut off your internet connection. Quit paying the bill. ;)

Does the government have to pay for those lines? Or is that handled by some other entity?


I wonder what caused them to do that. It doesn't seem like the regime's been pulling any punches so far, and the various YouTube evidence etc didn't seem to cause any increase in outside pressure. Why do it now?


Perhaps things are about to get even worse and the regime wants to eliminate a source for news getting out of the country.


I would imagine that some traffic could get through via modems, or does no-one use them anymore?


You'd need to dial through to ISPs outside of Syria, and it would be fairly easy to disable international calling (probably easier than cutting the internet off, in fact). Satellite is really one of the few ways to get internet access, as you're only reliant on power. Not cheap, though.


But that would show the IP of the ISP, and you won't know it's from Syria.


What happened to Hillary Clinton's giant wi-fi airplane?


"Our thoughts are with the Syrian people and we hope connectivity, and peace, will be quickly restored."

Err... well, that's great. I'm glad that this information free blog post pimping your services ended with your sincere token of solidarity with the Syrian people.

For an article with real information from people who actually understand what a network is, go here: http://www.renesys.com/blog/2012/11/syria-off-the-air.shtml

(Yes, I'm a little annoyed. But that's just because a lot of companies are submitting their "informative" blog posts on HN while really it's just more pimping of said service.)


In no way is this post information-free or service-pimping. The quoted line may have been a platitude, but the post itself was a decent breakdown of events and their interpretation thereof.


How is the article that you are promoting as an alternative any different, from your perspective?

As far as I can tell, both blogs are run by companies in the network infrastructure space. The chief difference appears to be that CloudFlare bothered to express any solidarity at all.


That.

.. but its true his link has less pimping and is more of a tl;dr; than a "oh we rock because we can read a bgp table" ;-)


What the heck are you talking about? There was no pimping of any services here.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: