
Why the Internet in Sub-Saharan Africa Sucks - dweekly
https://www.facebook.com/dweekly/posts/10101729307500703
======
realusername
We are currently building applications for our shops in Africa (Rwanda, Kenya
and Uganda) at my current company.

Kenya is not even the worst of all three, Uganda is really a disaster in terms
of connectivity. In Uganda it's even difficult to get actual phone signal in
most of the country. And you never have reliable Internet anyway, it works
like half of the time even if you are in the few right places in the country.
Building applications for this kind of environment has been quite challenging.

In a few shops in Uganda, the only way to get data back is by USB keys, so you
have an actual truck going there with data to send to the shop and data to get
back from the shop. We are building connectivity out of trucks and USB keys.

~~~
yxhuvud
That sounds like a challenge! What line of business are you in that can
survive that bad connectivity?

~~~
realusername
We are selling small solar panel kits in small towns on those three countries.
The goal is to have a few lights in places where there is no electricity grid
at all (it's a for-profit company, not charity). We handle pretty much
everything ourselves there since there is not a lot of services pre-built to
help you. The shops are selling the kits on payplan since it's quite expensive
for the local purchase power. We are the equivalent of a car dealer locally to
give a comparison. And the shops are running some software to handle all of
that, so this is where my problems are starting ! :)

~~~
allendoerfer
Sounds like you could branch out and offer these infrastructure services to
other businesses, too.

------
sitharus
250ms to Facebook and that 'sucks', glad to see New Zealand's opinion on our
internet connectivity is validated!

I routinely see 270-300ms pings to facebook. Anything on Amazon's CDN is at
150ms that as is CloudFlare. Akamai does have local nodes, as does Google but
the lack of competition on the cable makes it slow.

Guess I should move to Kenya?

~~~
viraptor
Same for Australia. (at least outside of state capitals) It's surprising how
quickly you can get used to facts like github actually timing out on ssh
connection...

I really hope that something like ppsp / swarm / ipfs gets integrated into
browsers. I'm pretty sure almost every non-user-content file I download every
day has been downloaded somewhere in my town. I shouldn't have to go all the
way to the US to get it.

~~~
afarrell
[https://mosh.mit.edu/](https://mosh.mit.edu/) may be of use to you as an
alternative to ssh

------
jgrahamc
This is why CloudFlare is building out PoPs in Africa. Our goal is to be 10s
of ms from people living there, not 100s, so that people who use CloudFlare
for their sites provide a great experience in African countries.

We are already live in Johannesburg and there will be many more.

~~~
akhatri_aus
Your Johannesburg PoP is always crashing and gives lots of error messages.
Very noticeable every few days with Reddit and HN [clearly shows Johannesburg
as the PoP on the error message].

That being said the speed boost is very obvious :)

~~~
jgrahamc
What sort of error messages are you seeing?

~~~
akhatri_aus
I would have to screenshot it to you when it shows up next and email/tweet it
to you. Something along the lines of overloaded, took too long to respond and
then the Cloudflare location shows up as Johannesburg.

When it's with HN it shows something like the twitter feed of @HNStatus
pointing out the CloudFlare server as Johannesburg with a request id (I
think).

When it's with reddit it shows the reddit mascot crushed up with his orange
beady eyes over a pile of orange stuff. Scary stuff.

~~~
nsomaru
I can confirm this; I'm in Durban, South Africa. It says something about a
gateway timeout and pops the HN error page with the twitter feed.

Next time it happens, will screenshot and send to cloudflare.

~~~
mappu
Lacking any further context i'm almost certain this isn't an issue with
cloudflare, instead with the upstream web server.

I saw a cloudflare gateway timeout on HN within the last hour, and the twitter
feed had a recent post excusing some technical issue.

~~~
akhatri_aus
These don't have the twitter feed excusing the issue, the last tweet was a
days prior when it happened last. It's as if they only affect African users,
the Johannesburg PoP is also used by Kenyan users.

At first, I wondered why the HN Feed never explained the downtime. Then it
made sense, it must be cloudflare.

Given that CloudFlare is everywhere & on everyone's websites, it's a pain when
it behaves this way affecting users all over a continent.

Uptime websites also report the site as up.

------
samuell
This is interesting! It makes me wonder if it could also explain why AJAX
based google apps like Gmail totally don't work in Addis Abeba in neighboring
Ethiopia, even on the speediest of hotel wifis (really, the download speed was
not bad, except for the initial lag). And I mean _totally_ don't work ...
gmail.com just gives a blank page, while the plain html version works.

I have been wondering if it could be something with limits of the number of
multiple connections, or even requests, affecting the AJAX requests in the
background, or some kind of unfortunate timeout limit in the javascript layer.

Not sure, but with this info, I'm of course thinking more about the lag.

Would be interesting with more details on how it affects typical AJAX apps!

------
davidu
We have contemplated a POP in Kenya. We will get there. Johannesburg should be
less than 200ms, so let me see if I can improve that... I would expect about
100ms.

~~~
Havoc
JHB to Europe is generally round the 190ms mark. 170 if you're closer to the
undersea cables. No way you're going to drop that to 100ms.

~~~
tribaal
JHB is much further away from Europe than Kenya, though.

JHB means usually going to Cape Town first, then up the west coast of the
continent, wheras Mombassa uses SeaCom, going around the east of the
continent. That saves a lot of time from a pure distance perspective alone.

~~~
Havoc
>JHB is much further away from Europe than Kenya, though.

Indeed. I was responding to a comment that was discussing JHB though.

SA traffic is split - some of it goes round east (Seacom), some west (SAT3).
Ends up being roughly 200ms east or west.

Source: Got my interwebs over said cables for years

------
chetanahuja
My company provides mobile app content delivery for a bunch of apps. We use a
mobile specific protocol (and backend smarts) that measures and records the
latencies for each of these connections. Unfortunately, due to a lack of
reasonably easy hosting or cloud solution in Africa, all African traffic is
currently served out of Europe. As for sub-saharan countries, Nigeria Kenya
and South Africa are the only countries where we see any significant traffic
at all. And yes, the latencies are horrible, but if we look at our worldwide
traffic, there are bad mobile connections everywhere (even in US and Europe)
and we have specifically designed the protocol to minimize roundtrips and
handle losses more cleverly than TCP, so we do ok under these conditions
(though clearly not as well as we would if we had extensive server presence in
the continent).

There is some slight chance that Amazon or Google might introduce compute
cloud nodes in Africa... as soon as that happens, we'd love to extend our
presence there. And if that's not forthcoming in the near future, we'll
definitely explore other hosting options (as we've done in India for now).

~~~
Sami_Lehtinen
What's the problem with Hetzner?
[http://www.hetzner.co.za/](http://www.hetzner.co.za/)

~~~
chetanahuja
Hmm... that's an intriguing possibility. I always thought of Hetzner as a
Germany only hosting provider. Thanks for the pointer.

------
tribaal
That is suprising to me.

I lived in Tanzania for a year, and while the internet connection there was
not perfect by any stretch of immagination, I had way less ping than this (100
ms to London, where my company's servers are, 120-150 to Boston).

For reference, that is: aerial to my ISP two kilometers away, then fiber to
Dar Es Salaam, then Seacom to Mombassa, then Seacom again to Marseille, then
terrestrial fiber to London via the channel).

This smells a lot like contention on the seacom link (the big fiber optics
carrier that goes up the east coast of Africa through the Suez canal and
accross the mediterranean to Marseille). I would be interested to see a
traceroute.

Most of the times ISPs in west Africa don't buy enough upstream traffic on
undersea cables, creating contention. I can't blame them, it's expensive, but
it's mostly an economic argument, not because the servers are too far away
(also, most ISPs run pretty agressive caches of everything under the sun).

~~~
batbomb
Are you sure that's round trip? It's at the extreme physical limit of being
possible, as the distance to Boston, for example, is greater than 12k km as
the crow flies, and the speed of light in fiber is about 70% that of light.

------
discardorama
Summary: Nairobi needs a datacenter, with Akamai and Cloudflare.

~~~
dweekly
Also, QUIC helps a lot.

~~~
aaggarwal
Is QUIC protocol out of experimental phase?

~~~
scurvy
It's only in Chrome and only supported by Google's web server. They are using
it though. You can see the stats by going to chrome://net-internals/#quic

They are "working" to get it into Apache Traffic Server.

~~~
aaggarwal
Yeah, its communicating over port 443. Apache Traffic Server would sure be a
big step forward, if and when that happens.

~~~
scurvy
Yeah all QUIC connections are on 443. Here's the most recent public talk from
Google about QUIC, given at NANOG 64:
[https://www.youtube.com/watch?v=cSNT88_gedw](https://www.youtube.com/watch?v=cSNT88_gedw)

------
pyvpx
having spent a considerable amount of time in the Horn of Africa, the reason
African internet connectivity "sucks" is mostly obvious: lack of local talent,
substantially increased costs, graft/corruption, and minimal use of peering
infrastructure (plenty of peering fabrics, little to no engagement or use)

while I certainly love the excitement this has generated, it's not as easy as
"we'll PoP Kenya! We'll PoP Tanzania! We'll PoP all of Africa! 50ms for
everyone!"

you would think a place like Djibout with five submarine cables and
terrestrial routes into Ethiopia, Sudan, and others, would have a vibrant
peering fabric or CDN nodes.

Level3 just put a CDN node in there, and Telecom Italia Sparkle has been there
for a while. AFAIK the CDN node is completely under-utilized and Sparkle
provides high cost MPLS services for private and governmental outfits; not as
much general IP transit.

More work needs to be put in getting carriers to peer at existing points of
presence, and costs have to be lowered any way they can.

------
Dylan16807
Taking one second to establish the connection doesn't really explain a 12
second load time. TCP takes a little bit to ramp up, but not 30 round trips.

Plus you have TCP preconnect as a lightweight way to avoid that initial delay.

~~~
hstrauss
It's that, along with local-loop congestion (and eventually packet loss). Then
the retransmits begin.

At least that's my experience with anything cellular-network based.

~~~
Dylan16807
But the article is talking about plentiful bandwidth over wires. Not a
terrible cell link.

------
xenophonf
Is there a way to read this posting without having to log into Facebook?

~~~
witty_username
Google cache?

[https://webcache.googleusercontent.com/search?q=cache:Sv7aOT...](https://webcache.googleusercontent.com/search?q=cache:Sv7aOTA2pjUJ:https://www.facebook.com/dweekly/posts/10101729307500703+&cd=1&hl=en&ct=clnk&gl=in&client=ubuntu)

------
noobie
_sub-Saharan_ * Africa

~~~
dweekly
(fixed, ty)

------
zimtzucker
Type Nairobi in the search box, and you'll see that Akamai has a presence in
Nairobi. By the way also in Kampala (Uganda) and Kigali (Rwanda):
[http://wwwnui.akamai.com/gnet/globe/index.html](http://wwwnui.akamai.com/gnet/globe/index.html)

------
DocG
instead of wires, even between citys the internet is transmitted by air. Im
currently in southwest cameroon, and only means for access is over the air. At
home, I use mobile network. But even in all the cyber cafes, offices, etc they
use different over the air solutions. As far as I know, no one is wired
directly to the network. And apparently even from this region, traffic goes to
douala over the air, where it meets optical sea cable. Oh, and incoming
connection is under monopoly. If it goes down, everyone go dark.

There are startups whose plan is to create local network coverin city for high
bandwith content, without relaying to the outside connection as this is mostly
bandwith limited. So called alternative network of catche servers and wifi, as
often even watching youtube is pain depending on the time and location.

------
kaboro
[https://angani.co/blog/its-not-how-much-fibre-you-have-
its-w...](https://angani.co/blog/its-not-how-much-fibre-you-have-its-what-you-
do-with-it/) a response to this post, highlighting our experience over the
years in Africa

------
scurvy
Bummer about all those IPv4 traceroutes. I'd like to see the output of IPv6
traceroutes or mtr output.

~~~
hstrauss
For .ZA, I've only really seen native IPv6 at jupiter.is and on the TEnet
(NREN) network. My university campus doesn't even have native IPv6 available
for servers I maintain.

From what I've seen running an HE tunnel through London, Youtube and Facebook
make up the bulk of IPv6 traffic (as expected). At least those networks are
the only thing routinely dedicating 10%+ of the total inbound traffic to IPv6
according to my home network flow analysis (I think it's about 18% IPv6 usage
on a monthly basis).

------
zatkin
>Apple Company

Huh? I thought they're Apple, Inc.

------
scurvy
Elephant in the room here -> There's no money to be made there (yet)

~~~
Epenthesis
The question then: Is there sufficient first-mover advantage that a far-minded
company setting up a datacenter there _now_ could reap the benefits when there
is money to be made?

~~~
scarmig
I think this leaves out something critical: information infrastructure isn't
merely something you deliver to a region to extract profits from it, but
something that accelerates growth and helps build a civil society and
flourishing middle class.

If I had a couple billion dollars to blow, I'd be spending it on that. Though,
maybe that's why I don't have a couple billion to blow =)

------
witty_username
Hmmm, I live in India and I get ~280 ms ping to Facebook and 180 ms ping to
Twitter. Seems mostly an issue of "not-living-in-the-US".

------
higherpurpose
Solution: let's replace the "Internet" with only 100 faster-loading websites
that we (and the local governments) approve. Win-win, right?

~~~
dublinben
I can't help but notice that the writer is only checking US-based websites.
I'm sure that local alternatives actually hosted in-country are much faster.
Why should users in Kenya care about loading some Silicon Valley dating app?

~~~
dweekly
Without debating whether or not it's a good thing, it's inarguable that a
healthy portion of the apps that are popular worldwide (outside China) are run
by companies headquartered in the US: Facebook, WhatsApp, YouTube, Google,
Twitter, Instagram. And now Uber.

Given that each one of these has the vigorous majority of their users outside
the US and most have substantial non-US infrastructure build, it gets a little
strange to label them "US-based".

But I think it's fair to point out that if performance lags to the extreme it
may open market opportunities for local competitors to differentiate
themselves with speed. Local hosting options are limited, however, making
things more challenging for indigenous entrepreneurs. I was amused and
appalled to find that many "Kampala web hosting" companies are actually
themselves hosted in Los Angeles.

So: not much is hosted in-country. Very little traffic terminates domestically
outside caches.

------
tekesteg
I think Ethiopia is the worst!

------
MichaelCrawford
In one of Google's lobby there is a 3-D animation of a spinning globe, with
spikes that appear wherever and whenever searches are performed, with the
height of the spikes being proportional to the rate of query submissions.

The recruiter who showed me this was very proud of google's accomplishment and
indeed it was impressive to see the spikes erupting from the east and west
coast of the US, Europe, Japan and India.

But I watched quietly, sadly and solemnly South America and Africa. There were
modest spikes in south africa and brazil. Not many anywhere else.

Both continents are quite populous.

Among the reasons for the choices I've made in my life and my career the last
few years is that I have come to regard the vast majority of the code I've
written in my long career as a developer as being of no real lasting value to
anyone.

Instead I wander around downtown and oldtown portland seeking out crazy people
then finding some way to lift their spirits somehow. It can be a little
frightening as I specialize on the particularly violent sort. Those are also
the most cruelly isolated:

"LEAVE ME ALONE! LEAVE ME ALONE! WHY DON'T YOU LEAVE ME THE FUCK ALONE?"

The gent he was accusing had no clue.

"Hi! My name is Mike. What's yours?"

"Jerry! Pleased to meet you."

Does it have to be so hard?

My point is that those google search spikes would be erupting all over South
America and Africa were someone to take responsibility to enable them.

I intend to contribute somehow, someday, in part by setting up some
nameservers in cybercafes. There are actually lots of them, cybercafes are
quite popular in the developing world.

~~~
RyanZAG
I'm in Africa and... I'm sorry, but we have nameservers already. That's not
really a problem. Are you really talking about DNS servers here? I wasn't
sure.

The issues which prevent those spikes you'd like to see on the maps at Google
are education (primarily literacy, especially English literacy), access to
devices, and culture. Access to devices is being solved by cheap smartphones
becoming obtainable to some degree and constant improvements to data networks
in African countries, but the other two have no easy solution.

EDIT: To maybe make it a bit easier - consider someone in a village (with a
cellphone and access to a data network) who wants to find out why something
happened to the carrot he was growing at home. All of us would pop open Google
and search about some information about carrots. A guy in a village would head
up the road to talk to his friend who's been farming for 50 years and knows
about carrots. If you tried to tell him "hey, you could ask Google that
question" the only reply you'd get would be "I don't really know about that".

I've had someone come to me to ask me to look up bus fares on the internet for
them. They'd hand me a piece of paper with the URL for the bus companies
tariff page. Not too interesting until that same person later pulled out his
Windows Phone smartphone to make a phone call, and he had full internet access
and a good mobile browser on that phone.

~~~
karmazeroed
Yes, the issues are substantially bigger. South Africa can't even provide
reliable electricity in the cities, never mind elsewhere. People living there
tell me it's not a problem. If they're not bothered by that, I doubt they give
a damn about google searches. If memory serves me correctly that's been going
on for about 10 years now.

~~~
hstrauss
South African here.

There's the electricity issue (we call the rolling blackouts "load-shedding")
which affects the cellular base-stations and telco exchanges as much as
anything, but also another more obvious issue for content delivery: the local
loop.

For the past 15+ years (as long as I've had Internet access), the incumbent
telco (TelkomSA) has not been forthcoming with upgrading or maintaining the
ADSL infrastructure. Finally, there seems to be traction with FTTx providers,
but this seems limited to affluent pockets of a few thousand people in the
country. All the new fibre offerings serve the same locations.

While it's hopeful this will improve the situation in the medium-term, the
backhaul from ISP to eyeball is oversubscribed at least 30:1 for ADSL, with
equipment being replaced by (IMO) inferior network equipment. Additionally,
there are reports of automated line-conditioning software which should improve
local loop connectivity, but I've seen DSL sync issues only since the software
has been implemented.

The (ex-public, privatised under Telkom) Exchanges have been neglected and
have recently been divided into three groups: Earners, Maintainers and "Lost
Causes". The latter two will not receive upgrades from ADSL1 and the Earners
will likely migrate to newer technologies. This is largely due to copper-theft
and vandalism of exchanges/cellular base-stations (which are oversubscribed
enough that wireless technologies routinely sustain 3000-20000ms latencies).
The oligopoly running cellular networks have no incentive to improve this,
since the Regulatory Authorities don't really seem to push competition as well
as we'd like.

TL;DR: There are issues, but rose-tinted glasses make me hopeful in the medium
term. Claims are that wireless will fix everything seem bleak, since base
stations become oversubscribed faster than they can be built/upgraded (and the
backhaul from them is measured in megabits/s, rather than gigabits/s charged
at ~US$0.15/MB).

~~~
nroets
I also live in South Africa.

Our mobile networks are in pretty good shape:

* Anyone can buy a prepaid SIM from any supermarket and access the Internet within 10 or 20 minutes. Last time I visited France, it took me 4 days to achieve a similar result.

* Mobile data is cheap when buying in bulk. I pay USD2 per gigabyte.

* LTE coverage is growing fast and RTT is good. For example my ping time to google.com is 30ms.

Conclusion: Private ownership and free markets lead to good service and low
cost. ESKOM proves that state ownership of companies and monopolies lead to
bad service at high cost.

------
MichaelCrawford
Northern China talks to Southern China through Japan, to the extent that they
talk at all. All of China at first had one state telecom monopoly, but then
some party official set his son up with Northern China. What was left of the
state monopoly won't route to it.

My take on what to do about that is to set up servers in both north and south
that have website mirrors, as well as a geographically-sensitive nameserver. I
don't have a clue how to do the GeoDNS but surely it is a solved problem as
big sites require it for load balancing.

~~~
scurvy
China Telecom also has a separate AS dedicated to high party officials, VIP's,
etc. It still goes through GFW, but it goes through a dedicated, higher
performant GFW.

~~~
MichaelCrawford
The US Government, at least at one time had something called FTS, for the
federal telecommunications system. It was a private phone system for Uncle
Sam.

