
Fixing the Internet for Real Time Applications, Part II - rayascott
http://engineering.riotgames.com/news/fixing-internet-real-time-applications-part-ii
======
eloff
The big cloud providers also do similar kinds of things to bring down latency
on their networks, but I wonder how effective they are? I've never seen
latency comparisons between cloud providers - but there must be differences.
None of them seem to use latency as a selling point either. I'd add latency to
one of the criteria when choosing between providers if there were a way to
compare them. I'm not talking about their latency to me or any arbitrary
point, but their latency to internet users in the US in general. Like 50% of
users within 80ms RTT. Also packet-loss rates would be nice to know.

~~~
plq
Yes, I was actually suprised that all graphs were going up :) Number of users
increased -- great but ping times and packet loss should be at an all-time low
now and these are the graphs that we want to see!

~~~
eloff
I think you read the graphs wrong, they are % of players with better than 80ms
ping. So going from 30% to 80% was an incredible improvement!

~~~
PeytonKoran
Hello,

I am the author, you got that right. And thank you for taking the time to read
it.

Peyton

------
BWStearns
I wonder if there's a long-term goal here of becoming the AWS for gaming and
other realtime network needs.

About to go back and read Part I so please forgive if that's covered in the
first part.

------
raarts
I'm from the VoIP world and we share this pain. Kudos, this must have been a
huge undertaking.

~~~
PeytonKoran
It was, but so much fun and really worth it. :) I am the author. Thank You for
reading :)

Peyton

------
surfmike
How is this different from Akamai's IP Application Accelerator
([https://www.akamai.com/us/en/solutions/products/web-
performa...](https://www.akamai.com/us/en/solutions/products/web-
performance/ip-application-accelerator.jsp))? Honestly curious.

I would love to see a consumer videoconferencing service built on top of one
of these networks.

~~~
PeytonKoran
Hello,

I am the author. The difference is that Akamai is working on optimizing
bandwidth to their customer from a pipes size perspective, not a latency
perspective. In other words, they want the widest highway possible, maybe not
the fastest.

Peyton

~~~
surfmike
Interesting, thanks for that.

You guys should really consider making this a platform that anyone could use!

~~~
BillyParadise
I'd be surprised if they haven't already considered it. Building a national
backbone is a large endeavour, and now that it's built, a great way to
diversify revenue. Games only stay popular for a finite period of time.

------
hchenji
Interesting read. Could you blog about the math/techniques/algorithms you used
to determine the best fiber paths?

Consider writing a SIGCOMM paper about it!!

~~~
PeytonKoran
We had to RFP each link, meaning we reached out to every fiber/wave vendor
that was present in the data centers we wanted to be in that we could find,
and had them submit routes with latency data. Then we tested to verify their
data. It is really manual unfortunately.

------
plq
How is this "fixing the internet"? This is just good old network optimization.
And a local one, not "internet-wide" by any means! Colossal undertaking, sure,
kudos to Riot team for pulling it off. But... I don't see what makes this
different.

~~~
ajkjk
I'm pretty sure that phrase is not meant to be taken so seriously.

~~~
PeytonKoran
Hey,

I am the author. We are building it out world wide, but you are correct, this
is for one application. The same blueprint could be used for any other
application though.

Peyton

------
chatmasta
> A reduction in the number of participating routers would lower latency (yay
> for LoL players), but it would also lower the number of customers they could
> support.

This is because ISPs oversell their networks. That's great and it makes
sense... as long as they can get away with it. But when services like online
gaming and netflix gain popularity, the ISP business model is insufficient for
delivering quality connections.

Also, nice "pipe dream" pun in the second paragraph. ;)

------
sargun
What prevents you from using GCE or AWS and running chunks of your
applications near the edge?

~~~
madsushi
1) Databases - if your app servers at the edge need to talk to your database
back at your main datacenter, you've just made performance worse. Replicate
the data to the edge? Now it's harder to keep consistent.

2) Too many edges - if you regionalize too heavily, you end up restricting
your matchmaking pools to smaller groups. Matchmaking theory says that either
(a) quality of matches decreases or (b) time to matchmake increases as your
pool shrinks. At some point, to have a healthy matchmaking pool in a game with
a wide variety of skill levels, you have to consolidate. Riot already had Euro
datacenters (with separate accounts, which solves #1).

3) Performance - cloud low-latency UDP performance isn't nearly as good as you
can get yourself. Plus, you have more control (and leverage) doing it in-
house.

4) Perception - some level of latency becomes imperceptible, 50ms is getting
pretty close to that number.

------
PhaseMage
The TCP/IP internet is irreparably broken for real time applications. The
barriers to new entrants wanting to do the same thing as Riot Games is too
high.

I've proposed a scalable mesh network, see IsoGrid.org

