
How Syria Turned Off the Internet - dknecht
http://blog.cloudflare.com/how-syria-turned-off-the-internet
======
kami8845
I love these highly technical blog posts on recent events by cloudflare.

Keep up the good work guys.

[1] [http://blog.cloudflare.com/why-google-went-offline-today-
and...](http://blog.cloudflare.com/why-google-went-offline-today-and-a-bit-
about)

~~~
Breakthrough
I swear I just thought the same thing to myself.. Very concisely written, but
it's written both _interesting_ and _well_. Not to mention it's always welcome
to see large companies like this share vital information with the public [+1
internet points for CloudFlare] :)

I wonder if it would be possible to hack into these "wall" routers from inside
the country (or I suppose outside), but something tells me probably not (I
doubt they would even be ping-able, but maybe through other attack vectors
[like other internal computers]...). Seriously, I can't imagine how enraged I
would be if someone decided to simply block the Internet.

~~~
jemfinch
Since BGP is sent over TCP, the routers are definitely pingable. You know how
when you run traceroute you see a bunch of IPs between you and your
destination? Routers like these are those hops.

~~~
Breakthrough
Ah, sorry, I meant _now_ , in the current state. I wonder if they just shut
them down (the routers), or blocked all incoming connections from the
outside...

~~~
bdonlan
Probably the simplest thing to do would be to just shut off the ports leading
to external networks at the physical layer using management commands. They'd
not be pingable from the outside, naturally, and it'd look like a cable cut
(until you start doing TDR on the cable anyway), but they could bring it back
online quickly once the powers that be choose to do so.

------
ChuckMcM
That is a frickin' awesome video. Does anyone know what tool that is, I think
I want it running on my status displays.

For those who are wondering your edge router (or border router) "advertises"
that it can route to a particular subnet. That information propagates around
and packets find there way there. So someone in Syria told all of their border
routers to stop advertising routes to Syria's IP blocks. Now the fun thing you
can do is since they aren't advertising those routes, if you are sitting in a
data center somewhere and have peering access and a ASIN id you can advertise
those routes and all of Syria's traffic will start heading your way :-) Of
course if that monitoring tool is still running it will have all these lines
suddenly running off the screen toward your data center.

The traffic that is actually _in_ Syria can't get out. So its not like you
could snoop on Syria or anything.

~~~
eroded
The tool is <http://bgplay.routeviews.org/>

~~~
sparkinson
For those trying this tool the IP prefix is 5.0.0.0/18.

29/11/2012 10:20 to 10:30.

------
muppetman
This tells us nothing about how it was turned off, only that the routes were
withdrawn from the global routing table. Which isn't "how" it was turned off,
by why it's stopped working.

There's some guesses in here, but the title is rather misleading. No one still
knows why these routes are no longer being advertised, only that they're not.

~~~
ivix
Kind of muddles up why and how, but it's quite clear that someone made a call
to the state ISP, and then someone went around reconfiguring the routers to
drop all the published routes into Syria.

~~~
cube13
Or the state coordinated to take down the physical lines at the same time.

Which I think is more likely, considering the relatively few points of
failure.

~~~
trepid
In the past, netsweeper has been one of the tools employed by governments
looking to accomplish incidents like these.

[http://thenextweb.com/me/2011/06/07/this-company-is-
helping-...](http://thenextweb.com/me/2011/06/07/this-company-is-helping-..).
[http://www.thestar.com/news/canada/article/1218965--
guelph-t...](http://www.thestar.com/news/canada/article/1218965--guelph-t..).

They have previously been used by Egypt to close down their internet. I have
no insight into how this particular incident was accomplished, but this sort
of software is where I'd start looking.

~~~
trepid
At one point when I was looking to see what was available in the job market, I
saw a posting that I was a suitable fit for with their organization
(netsweeper). It got me curious, being tangentially aware of their
organization and content filtering focus. Their perspective was very contrary
to my own, and I was interested in having the opportunity to have them pitch
what they did to me. I hadn't really had the chance to talk face to face with
what I considered to be an "evil" organization before. Maybe it was unethical
for me to waste their time, when I didn't have the intention of working for
them, I don't know.

The interview itself was interesting. The technical tests etc were relatively
trivial, mostly stuff you'd expect about load balancing and responsiveness (as
would be required for routing software). But more than half of the interview
was them pitching the company to me -- they're quite aware of the reservations
people might have about working for a company whose direction is to suppress
free speech.

Their basic pitch was that they believed all 1st world governments would have
internet filtering technologies in place in the near future -- that
governments would legislate the use of it to stop the proliferation of things
like child pornography, and eliminate the zombie-computer defence -- that if
there were child porn on your computer, you went and got it.

We talked a little about the technical challenges of instantly shutting down
the internet of an entire country, like Egypt, but they more or less blamed
that on "improper configuration" of settings. At one point in the interview,
the interviewer told me that you have to believe that the people that make the
weapons aren't the ones firing the guns, and that dangerous tools can be used
for good, but that yes, there were nights in which he cried himself to sleep.

Super interesting interview.

------
TaskConsidered
This doesn't necessarily explain HOW the network was taken down but, it does
highlight the conflicting evidence between a "terrorist" style fire-sale
attack, and a state-imposed outage to limit communications between dissidents.

Unfortunately, non of these hosting companies want to give an alternative to
HOW to bring the network back up...

Anonymous seems to be the only group oriented at actually helping the citizens
of the nation of Syria regain communication via alternative methods such as
TCP/IP over HAM radio, and satellite links, personal wireless mesh networks
using WiFi on mobile devices.

Everyone can bitch about HOW to take DOWN a nations internet but, it takes
real humanitarians & 1337geeks to consider & implement HOW to bring a nations
communications infrastructure back UP.

So, what are you waiting for...

HELP.

~~~
ivix
Syrians can still dial an international number and get online via modem. So
you are better off sending money to aid organisations to pay the phone bill.

~~~
biturd
It was posted that international calling has also been disabled, as well as
cellular I believe. Unfortunately, I can't locate the source.

It is the same source in which it stated that the data center that handled the
BGP routes was being held by 6 remaining techs whose current location is now
unknown.

I wish I could find the link as it proves much of what the OP posted. Not to
mention I would love to see someone repair one, let alone four cables in
fifteen minutes. Heck, I bet it may take more than fifteen minutes to cut one.

------
DigitalSea
This is probably the best explanation of the Syrian outage I have ever seen.
Cloudfare are exceptionally good at explaining things like this and even
included a video of the Syrian traffic slowly dropping off. Expect those "cut"
cables to miraculously be repaired shortly.

~~~
onetwothreefour
This isn't a good explanation of anything.

~~~
DigitalSea
Would you care to embellish us all with your expertise then? I'd love a link
to your blog post explaining the outage. This isn't Reddit, it's Hacker News
and on HN if you dispute something you have to at the very least lightly back
it up.

Maybe you're right, what would a company who was voted Most Innovative Network
& Internet Technology Company of 2011 & again in 2012 by the Wall Street
Journal know about networking and how Internet traffic is routed, right? The
very fact they even embedded a video showing the traffic dropping off in the
blog post I think proves they know a thing or two about network traffic, they
provide content delivery and domain name server services to a lot of happy
customers after all.

------
jimktrains2
My wife's first thought when she heard about Syria's lack of internet was that
the sitting government is about to start a nasty offensive.

This was a good read and informative. One question, though.

> When the outage happened, the BGP routes to Syrian IP space were all
> simultaneously withdrawn from all of Syria's upstream providers.

Does withdrawn mean not advertised or was a message sent out saying these
routes are no longer available?

~~~
eastdakota
Once a route is withdrawn (either from the router going offline, a cable being
cut, or someone adjusting the routing tables) upstream peers will drop the
path very quickly (typically in seconds).

------
squeed
There are a few really baseless, attack-y comments on the original post. I
wonder if they're some kind of Syrian social media reactionary force.

CloudFlare, if you have access to their data, what's interesting about them?

------
biturd
I have a few questions not knowing all that much about BGP other than thinking
of it in terms of a higher level DNS system for IP routing.

What gives Syria the authority to do this? What gives anyone authority to do
this? What prevents malicious routing?. Could they route all traffic to
8.8.8.8 and overwhelm another network? If it's ICANN, can they come in and
revoke control and give it to a third party intermediary?

It has been rumored a small staff of 6 stayed trying to keep the routes up.
Their current status is not known. I understand in those cases no intermediary
would help. But from a pragmatic standpoint, I'm curious.

While I understand this goes against the "rules" but if I have a DNS server
and the roots drop a zone, and I don't agree, I can add it back in. As a local
user I could add to etc/hosts like in the old days.

The above assumes I had a large user base like openDNS or google pDNS to be
effective. Can the same be done with BGP? Can major broadband providers decide
to ignore the dropped routes and send traffic along?

I understand Syria would just toggle off some other "switch" and terminate
core routers but it would at least send a tiny message of sorts.

How are they stopping satellite access?

How are they stopping cellular based access?

Is there any form of TCP over Ham radio? TCP over laser? USB over carrier
pigeon (seriously)? What are the bare bones options here for getting data in
and out and where is that closest point of access?

No one has put in long range wifi links of the 20 mile line of site type, or
is that still too short a distance to get a few users online?

What about dialup?

If CloudFront can see this much traffic, they must be doing pretty well. What
is the point of Facebook, reddit, and many others using CDN's and Amazon and
such when they could probably half their hardware and push the rest to
ClouFront. Or is CloudFront really only best for static sites that are hit
hard and need lots if bandwidth. Dynamic sites would still bottleneck at the
database/drive/physical/etc layer?

Thanks. Sorry if these are rudimentary questions. I haven't even met that many
people who have had BGP access. I'm going to go look up what the format files
look like now, just out of curiosity.

~~~
pyre

      > Can major broadband providers decide to ignore the
      > dropped routes and send traffic along?
    

IIRC, each router makes it's own decisions. Take the following route:

    
    
      A - B - C - Syria
    

Assuming that there are no published routes to Syria, if node A tries to send
a packet to Syria, the only node that can force a packet onto the Syrian
routers is node C. If node B decides to ignore the lack of a published route
and forwards the packet to node C, then node C will just drop it (possibly
sending an error response back). Even if the node C forces data over the
Syrian connection, the Syrian routers won't act on it.

This is my understanding of how it works.

    
    
      > How are they stopping satellite access?
    

Presumably very few Syrians have satellite access. If they are from a Syrian
provider, then it's pretty easy to cut them off. If they are with a foreign
provider, not so much. On the other hand, if you were in Syria when they
Internet was shutdown, you would probably be very secretive about your foreign
satellite access. If only because men with guns might have something to say
about it.

    
    
      > How are they stopping cellular based access?
    

Presumably because the state can go to the cell providers and shut them down.
What is Syria's cellular data infrastructure like?

    
    
      > Is there any form of TCP over Ham radio?
    

There are ways of getting Internet over HAM radio, but this probably runs up
against the same friction as a foreign satellite connection. Especially since
the equipment would be more conspicuous.

    
    
      > TCP over laser?
    

Really?

    
    
      > USB over carrier pigeon (seriously)?
    

USB is a client-server protocol. It probably wouldn't do too well over a
carrier pigeon.

    
    
      > What are the bare bones options here for getting data in and
      > out and where is that closest point of access?
    

Probably a directional antenna pointed over the border to a line-of-sight
receiving station.

~~~
entropy_

      > What is Syria's cellular data infrastructure like?
    

Well, there are 2 cellular providers. One is completely state owned and called
"Syriatel" and the other is MTN(<http://www.mtn.com/>). Though MTN is
extremely heavily regulated by the state.

Also, Syria enforces a web filter(similar though not as sophisticated as
china's) which also affects browsing over the cellular data network. So even
cellular providers eventually go through some choke point which is state-owned
so that the filter can be applied.

In other words, if the state wants to cut off internet, cell isn't going to
save you.

    
    
      > USB is a client-server protocol. It probably wouldn't do too well over a carrier pigeon.
    

I think he meant literally sending a USB thumbdrive via a carrier pigeon. See
this:
[http://www.telegraph.co.uk/technology/news/8007897/Carrier-p...](http://www.telegraph.co.uk/technology/news/8007897/Carrier-
pigeons-are-faster-than-rural-broadband.html)

------
bernardom
They mention that there are four cables "connecting Syria to the Internet."

OK, I'm curious! How do they know? How many cables connect, say, Brazil to
"the Internet?"

Is this publicly available somewhere?

~~~
trepid
[http://nicolasrapp.com/wp-
content/uploads/2012/07/world_map_...](http://nicolasrapp.com/wp-
content/uploads/2012/07/world_map_05_DARK.jpg)

~~~
danyork
That's a very cool map! I see it is part of a larger article that ran in
Fortune magazine back in July 2012:

<http://nicolasrapp.com/?p=1180>

Good find!

------
Derpsec
They've been blocking traffic in Syria to numerous social networking and
email/voip sites since the revolution began, and nulling cell towers in every
area where there are protests. This hasn't hampered the free syrian army or
activists as since last year they've been passing sdcards to the border of
lebanon, jordan, iraq and turkey and uploading their videos wirelessly from
there. The FSA is running two border crossing with turkey anyways.

This seems to be like some sort of incompetence, more like they tried to set
up some sort of spying choke point and it massively failed. ask nokia-siemens,
they helped iran set up their chokepoint, most likely some infosec whitehats
with zero ethics are currently flying out there to assist in the holocaust, er
I mean rebellion put down, by working with Assad to get the tubes back up.

It only hurts Assad to keep the tubes out, his loyal base needs to buy their
louis vuitton bags off alibaba to keep their minds off the constant public
shooting of protesters and shelling of entire cities full of "terrorists"

~~~
entropy_
Well, Syria already does have a spying chokepoint in place. It's had it for
years(at least since late 2010 when I was there). They have something similar
to China's in that it also does filtering. Also, at some point when I was
there around may~june 2011 when this was all starting up they seriously
started messing with https stuff conducting MITM on facebook and what not.

Also, no, having the internet cut off absolutely does not hurt Assad. Syria is
very sectarian in nature. In other words your loyal base absolutely does not
correlate with what services you provide, only what religion/sect/tribe people
are.

The rebellion didn't happen because people suddenly started hating Bashar el
Assad, it happened because those same people have hated this regime for at
least the past 40 years. In 1982, Hafez, the guy who established the regime
put down an earlier rebellion that had gone on for something like 3 years by
completely destroying entire sections of cities(and pretty much all of Hama)
and killing, by conservative estimates, at least 10,000 people. This was
during a time when the USSR still existed and where such things within its
sphere of influence were par for the course, which is why no international
response happened at the time.

The rebellion happened now because of a perceived weakness in the regime's
freedom to act(no USSR anymore, much more US influence in the area, weak(er)
dictator) and because of perceived international support for similar things
happening in Arab nations. That's it. So if the internet in any way helps the
FSA get international support by exposing atrocities committed against
civilians by the government then you can bet it's in the government's best
interest to shut it down.

------
papercruncher
Distance between Cyprus and Syria is < 200km. How much money would it take to
setup a wireless link between the two? Wikipedia tells me it's possible
(<http://en.wikipedia.org/wiki/Long_Range_WiFi#Italy>)

~~~
krisoft
I guess the cost of securing your 'bridgehead' would dwarf any cost associated
with the actual link technology. And then, what did you achieved? You have
your packages going back and forth, but how will you distribute the
connection? Will you hijack the presumably uncooperative ISP's network, or
just use sneakernet? If you choose the last, then you might as well ferry the
information via boats. (Not saying that's easy, just it seems easier.)

~~~
rdl
This is actually a great application for UAVs, and I think there is some DOD
and/or DOS money to do it (not for Syria specifically, but the general
application.)

~~~
Derpsec
Syria would blow it out of the sky. They also are targeting all satellite
signals in their borders with mortar shelling. That's how all those French
journalists got owned, they turned on the cameras, set up the uplink and 30
seconds later bam here's a face full of explosive censorship enjoy. They
simply locked on to the signal and shelled it not caring who was there.

They've also been targeting wireless uplinks across borders and adhoc cell
towers. Syria levelled an entire apartment building full of kids in Turkey to
destroy a suspected adhoc cell tower they claimed was being used by
"terrorists". This is serious business, no piratebay UAV drone solution is
going to work. Mesh networking is also certain death, any building with a mesh
router on it is going to have all its inhabitants dragged out of their
apartments and shot in the street so they've been using good old walkie
talkies and bouncing signals all over the place to confuse the syrian army,
and in most cases simply stealing Syrian army comsec devices and pretending to
be them talking in slight code with each other to avoid suspicion. Al Jazeera
ran a story with some smugglers who used gps disabled cellphones but changed
the IMEI every couple of hours to avoid being found

~~~
rdl
They would be more able to go after terrestrial radio and L-band portable
satellite systems (e.g. Thuraya DSL, thuraya sat modems, various forms of
BGAN/RBGAN/etc., which is what journalists tend to use) than Ka or Ku band
satellite.

I haven't kept up on Syria or Libya (I wasted 2003-2010 on this stuff in
Iraq/Afgh/etc., and am trying to do a "normal" tech startup now), but while I
think Syria (and Libya) had better European gear than Iraq or the Taliban, it
isn't on par with the US, UK/FR/DE, RU, CN, etc. It's basically "good
commercial equipment designed for law enforcement", which is very heavily
cellphone focused.

The #1 vulnerability with satellite systems remains "operator assistance to
the adversary", or "network configured in a way which relays location data of
connected terminals to everyone in the footprint", both of which can be
addressed if you control the network.

------
eik3_de
Since Google has also confirmed that Syria is offline now, we should spread
the word about the telecomix dialup project: <http://dialup.telecomix.org/>

How could the information on that page be spread within Syria? SMS, MMS,
Phone, Fax, Mail?

~~~
aw3c2
after telecomix published probably dangerous and incriminating traffic data, I
feel extremely uneasy about their competence and especially about routing any
kind of information through them.

------
JVIDEL
Great post, too bad the comments section is already full of syrian trolls
trying to derail the subject from freedom of speech to some bullshit
conspiracy from "the empire" against syria.

Guess they still give access to those who are _loyal_...

~~~
flyinRyan
It's probably not syrians but Russian/Chinese who are worried about having yet
another ally switch over to camp USA.

------
saosebastiao
Its great to know how, but here is why:
<http://www.youtube.com/watch?v=H40EsEVU1Wk> NSFW/NSFL

Apparently, bombing people turns out to be bad PR when they tweet about it.

------
biturd
I don't know enough about markets to find the answer to this. Does Syris have
a "stock market" or some form of electronic exchange? I would imagine if they
have money they do.

Did those routes stay up? If not, what are the repercussions of that going to
be come open time? Or any foreign trade, anything that relies on network time,
etc. the list is pretty large for services that need Internet at least
infrequently in order to keep basic services up.

I can't imagine how many deaths there would be in the USA if this happened.
All those televisions stop working, that's gotta be a few million heart
attacks right there.

~~~
entropy_
The only form of stock exchange in Syria is the Damascus Securities Exchange.
And that's it. Syria is pretty much a Socialist nation. Under Bashar it had
been slowly moving toward a more capitalistic model but that was pretty much
just crony capitalism full of corruption and the state still pretty much
running everything even though it was supposedly private.

So no, no markets in Syria.

------
gexla
Easy way to shut off your internet connection. Quit paying the bill. ;)

Does the government have to pay for those lines? Or is that handled by some
other entity?

------
nir
I wonder what caused them to do that. It doesn't seem like the regime's been
pulling any punches so far, and the various YouTube evidence etc didn't seem
to cause any increase in outside pressure. Why do it now?

~~~
brown9-2
Perhaps things are about to get even worse and the regime wants to eliminate a
source for news getting out of the country.

------
SeanDav
I would imagine that some traffic could get through via modems, or does no-one
use them anymore?

~~~
objclxt
You'd need to dial through to ISPs outside of Syria, and it would be fairly
easy to disable international calling (probably easier than cutting the
internet off, in fact). Satellite is really one of the few ways to get
internet access, as you're only reliant on power. Not cheap, though.

------
caycep
What happened to Hillary Clinton's giant wi-fi airplane?

------
onetwothreefour
"Our thoughts are with the Syrian people and we hope connectivity, and peace,
will be quickly restored."

Err... well, that's great. I'm glad that this information free blog post
pimping your services ended with your sincere token of solidarity with the
Syrian people.

For an article with real information from people who actually understand what
a network is, go here: <http://www.renesys.com/blog/2012/11/syria-off-the-
air.shtml>

(Yes, I'm a little annoyed. But that's just because a lot of companies are
submitting their "informative" blog posts on HN while really it's just more
pimping of said service.)

~~~
cdata
How is the article that you are promoting as an alternative any different,
from your perspective?

As far as I can tell, both blogs are run by companies in the network
infrastructure space. The chief difference appears to be that CloudFlare
bothered to express any solidarity at all.

~~~
zobzu
That.

.. but its true his link has less pimping and is more of a tl;dr; than a "oh
we rock because we can read a bgp table" ;-)

