
Fixing the Internet for Games - gafferongames
https://gafferongames.com/post/fixing_the_internet_for_games/
======
AntiRush
Valve recently announced partner access to their network backbone which
addresses a lot of the problems mentioned here:

    
    
        * Access to our network, giving your players protection from attack, 100% reliable NAT traversal, and improved connectivity.
        * Tools for instantly estimating the ping between two arbitrary hosts without sending any packets.
        * A high quality end-to-end encrypted reliable-over-UDP protocol.
    
    
    

>> Second, clients can select a route that gets off of the public Internet and
onto our dedicated links as early as possible. On our backbone we can ensure
that the routing is optimal, since we have peered with over 2,500 ISPs. We
also prioritize the latency-sensitive game traffic over HTTP content
downloads, which we can afford to do because game traffic makes up a
relatively small percentage of our overall bandwidth utilization. And on our
backbone, a sudden surge of traffic unrelated to gaming won’t degrade the
experience.

[https://steamcommunity.com/groups/steamworks#announcements/d...](https://steamcommunity.com/groups/steamworks#announcements/detail/1791775741704351698)

They started using it in some of their games a couple of years ago
(Counterstrike Global Offensive and Dota2 if I recall correctly).

~~~
Impossible
Which is great if Steam is your only platform and you want to be tied to Valve
forever, but as soon as you want your game on mobile, console or even another
PC store like Epic or GOG you need another solution.

~~~
est31
Microsoft and Apple are far worse IMO by pushing their proprietary graphics
APIs. I see Valve more as their competitor than as a threatening monopoly. In
the future, things can change though, nothing is fixed.

~~~
Impossible
More "graphics API is the worst lock in possible"... With an online game it's
way easier to escape graphics "lock in", especially considering almost
everyone is using a game engine (Unreal or Unity) that hides most
implementation details from you. There are challenges with porting graphics
apis are different shader languages but with glslang and dxc supporting
different languages it's basically a solved problem. The main issue with
service lock-in, and why it's worse than proprietary graphics APIs is that it
blocks cross play. So even if you wrap the API for different platforms Steam
users cannot play with Xbox users, etc. The idea that graphics API is the
biggest bottleneck to game portability (especially across systems that also
have proprietary windowing systems, threads, audio, input, programming
languages, network libraries, etc) because John Carmack said we should use
OpenGL 20 years ago needs to die.

------
stcredzero
_The Internet is optimized for throughput at lowest cost._

 _No amount of good netcode that you write can compensate for this._

 _The problem is the internet itself._

 _The internet doesn’t care about your game._

The Internet did change, such that streaming movies became viable. Given that
games are now bigger than movies, could it be argued that that the network lag
we're seeing is the industry lagging behind the culture? (Which it generally
does, at least a little.)

A part of this might well involve special deals between large companies
involving specialized infrastructure. There are such things now involved with
movie streaming. Google's plans involve the infrastructure part of that
already. What happens to net neutrality?

Once game streaming becomes mature, then it's only a matter of time for App
streaming to follow. We'll be back in the mainframe days, just with much
larger companies with much larger multi-box cloud "mainframes", commanding the
attention of a much larger portion of the population and the culture. _" The
computer is your friend. Trust the computer!"_
[https://en.wikipedia.org/wiki/Paranoia_(role-
playing_game)](https://en.wikipedia.org/wiki/Paranoia_\(role-playing_game\))

~~~
milesward
Wanna build your game on the same network Stadia uses?
Https://cloud.google.com

~~~
Matheus28
And pay an obscenely high amount of money for bandwidth? Those providers all
charge at least $0.05/GB; you can get bandwidth for a tenth of that elsewhere.
Games use a lot of bandwidth at scale (agar.io would use a couple of PB a
month). For F2P games, server costs eat a lot of revenue if you use those
premium providers.

~~~
tree123
Why did you stop making io games? Where did you learn multiplayer game
development?

------
tarikjn
Great reading this, I just want to drop a line to say that I have been working
to solve precisely this problem -- for gaming but also other applications --
at the root. I have been working on a paper on this, here is the pre-proposal
I published a whiled ago:
[https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDC...](https://docs.google.com/document/d/1xWaSB-3VSOMaiyxfjDqTalDCein0V46EbVrO6mFGPQs/edit?usp=sharing)
\-- a lot of progress has been made since and I am looking for funding/grant
before moving to publishing the actual paper and moving to implementation.

The root of the issue is that the IP protocol is incomplete, in that is relies
on private agreement and ISP servicing that cannot react to a real-time, real-
world demand. IPv4 had some attempts at public QoS features that were mostly
dropped in IPv6 because people realized it didn't work in practice.

The only way to fix this for real is to merge ISP servicing and peering into a
per-packet peer-to-peer network market. This solves a great many issues at the
same time and essentially achieves the promise of mesh networking and more.

If you would like to help, please drop me a line: tarik@graphnet.org (sorry no
website at the moment).

------
tschwimmer
This was an awesome write up. The most interesting thing for me was learning
that bottlenecks are not on the edge, but across the backbone because T1
carriers are optimizing for low cost, not low latency.

~~~
kingosticks
The thing I'm still a bit confused about is: is this a limitation with how
their networks fundamentally work, or an artefact of my product choice when I
buy a relatively cheap home Internet service? i.e. if I want a specialist low-
latency and low-jitter link, could I not just pay for those benefits by
purchasing a business-grade service? Or is there no distinction once the data
gets onto a peered-network?

Or is the idea here that I don't have to cough up for all the benefits of that
superior product when I only want some of them and only some of the time?

And this whole thing does rely on the bottleneck not being at then edge, which
I don't see any proof of or reason for. EDIT: is there any proof/data to
support this?

~~~
gafferongames
We are seeing players with Google Fiber get 100ms improvements. It’s not bad
ISPs, it’s poor routing decisions and these are pretty much universal.

------
Insanity
Modern games have a problem that I believe is a direct consequence of
consoles. The lack of a server browser.

Just let me choose a server to which I have a good connection and I will be
much happier. Instead of dropping me in games with random people all the time
with random connection issues.

One of the main reasons I don't play fps on the consoles is missing this. (And
there are social advantages of playing with the same 'random' people as well).

Having a server browser does not solve all issues mentioned in his
presentation, but it does at least inform the player of the connection before
they join a game.

~~~
gsich
Random players can work with proper servers too, and many games just do that.

The problem with server browsers, or user hosted servers are/were arbitrary
rules enforced by the admin of that server. It's what killed Black Ops 1 on
the PC.

~~~
cjslep
The benefit of the server browsers is that there were tons of positive gaming
communities that competed around making the best and friendliest server
possible in the early 2000's. I fondly remember playing Call of Duty as a
tween with other community members that were adults and learned how to have
fun, manage disagreements, and deal with problematic people without screaming
profanities and raging about other peoples' mother.

The death of self-hosted servers means the death of admin powers which means
the death of building a gaming community one wants to see come to fruition.
Those admin powers are key to protecting vulnerable people from general
assholes. And the "modern" online "report a person" is woefully laughable in
comparison.

~~~
gsich
Often times admins are the assholes here (kick for using weapon X, perk Y,
attachment Z) and you have no way prior to connecting to know which stupid
rules are now in place.

(that and the fact that dedicated servers in BO1 meant you could rent them
just from one or two places, no files for you)

~~~
Insanity
At least on CoD4, people would often play the same servers with the rules they
liked.

Banning boring weapons is a good thing. Also, some servers mentioned the main
rules in the name.

~~~
gsich
That's the problem, "some". I partially blame the developers for not having
proper filters in the serverbrowser.

~~~
Insanity
Hard to support all possible mods and changes I guess. The best they might be
able to say is "modded: yes/no". And that is another great thing that is dying
out. Mods. :(

~~~
gsich
That would be enough. Although mods are easier I guess. In BO1 I just wanted
an "all stuff allowed" filter.

------
the_mitsuhiko
I'm just glad to see someone implements a real time bidding market without
dropping the word blockchain. I really like the idea of it.

~~~
gafferongames
I really hate blockchain :)

------
NelsonMinar
See also Haste, which has been doing this for a couple of years now:
[https://haste.net/](https://haste.net/)

One thing that's interesting is how different these services need to be in
different countries.

~~~
grenoire
I think Network Next is developer-targeted, as opposed to Haste which offers a
subscription service for gamers.

~~~
gafferongames
This is correct. We work with game developers to sponsor the best route for
their players. If we can’t significantly beat the public internet for that
player (according to what the game developer specifies in their route shader),
then we take the public internet and that player is free.

------
tinco
This is epic. Great to see that someone who's been such a good support to the
gaming community with his excellent learning resources is also being
successful in his business.

It's also interesting to see these "private internet" endeavours have popped
up to solve centralization issues on the internet. Streaming sites like
Youtube and Netflix have solved it in similar way, striking deals with ISPs,
as have the big CDN's, simply putting their machines at the ISP's datacenters.

Network.Next is solving this for the gaming domain, but there is a more
general preferential routing on the internet problem this idea could be
expanded to.

~~~
gafferongames
Thank you very much.

~~~
jay_kyburz
How about some Tribes Jetpack action!

------
AlimJaffer
As someone who works in the games space, I find this to be an interesting
proposition. My primary concern revolves around similar 'net neutrality'
aspects - more and more of the internet is becoming pay to play, this just
further entrenches that theory if you want the best networking for your games
as a studio.

~~~
politician
That's a good observation. Given the alternative between bidding for
performance from Network Next vs paying some flat fee to Comcast, et al.
directly, I'd go with NN and let them deal with Comcast.

... and then maybe add "faster gameplay" power ups to our loot boxes
(kidding).

~~~
gafferongames
When we work with game developers, they find this aspect of the system
incredibly compelling. Does it really make sense for every game company to
negotiate with every transit provider and ISP to make sure traffic is
delivered properly for their game? We think not.

Plus, how could they get a fair price if they’re the small guy. Suppliers on
Network Next cannot identify buyers and cannot have one price for one buyer
and one for another. So now the small guys get the same price the big guys do
and competition should drive the price down over time.

------
Animats
Games need _some_ low latency data, but not much. Player actions, character
movement, and maybe bullets need low latency. Asset downloading does not. If
you had a dial up modem running over a non-packet phone network, and didn't
overload the buffering, you'd have better latency than the Internet.

Many games make this distinction already. The latency-critical stuff goes over
UDP, and it's limited to data where a missing packet is superseded by the next
packet. Nothing is retransmitted. Bulk data goes over TCP, with reliable
retransmission but more delay.

It's too bad ISDN died off. 64Kb/s end to end, not packetization delay, and
_no jitter_. The ideal gamer network would be an ISDN connection and a IP
connection in parallel.

If we had QoS systems where 1% of your maximum data rate could be at high
priority, this, and the VoIP problem, would be mostly solved.

~~~
gsich
Mostly it's just UDP with some TCP like reliable transfer built on top of it.

~~~
gafferongames
This is correct. Most FPS game protocols are 90%+ unreliable UDP acting as a
carrier wave for a very small subset of reliable/ordered data. The key here is
to get this unreliable time series data across with the lowest latency and the
greatest consistency (least jitter), and packet loss. That’s what we do at
Network Next.

~~~
gafferongames
And also, we only accelerate the part of the game that needs the low latency.
Game downloads and everything else... ehh, that’s a solved problem for a while
now. We accelerate only the game traffic between client and server!

------
robterrell
Glenn, if you're still hanging around -- can the shader be written to account
for the flow of P2P network traffic? Let's say I'm using an old-school
configuration of a centralized lobby server that simply connects players to
each other and the players connect directly over UDP?

~~~
gafferongames
We currently only support client/server topology with dedicated servers.

------
VectorLock
Clever concept. I wonder if they'll publish any more information on how their
service works. Things like what this 'route shader' is that people can read
without having to go through Email-Our-Sales-Team shit.

~~~
politician
It's a custom bidding script that tells their real-time auction platform how
much you want to pay for route improvements for a particular user session.

Typically, shaders are used to move vertices in 3d meshes or color pixels in
the GPU. Network Next is just using the term in a creative way to help connect
to their audience, but it's nothing different than a bidding model.

~~~
gafferongames
Route shaders select the best route out of multiple ones available. Right now
they could be used to for example, specify basic things like: acceptable
latency (eg. I don’t want to pay to optimize a player’s session from 20ms down
to 19ms, 20 is good enough), minimum improvement vs. public internet before a
session takes network next (eg. 5ms improvement or greater), and of course
maximum price for the bid. If we don’t find a suitable route at the price
specified, the session takes the public internet and is free. Right now it’s
fixed function like OpenGL 1.0 (eg. Parameters), but long term, it could even
be LUA bytecode. Hence, route shader is not just a marketing term, it’s a real
thing :)

~~~
kbenson
The really interesting thing I see for this is a possible other route for game
revenue in the future. Instead of paying up front (traditional) pricing or
advertising, or cosmetic (or not so cosmetic) in game purchases, you could
have two tiers, free and optimized latency for a cost.

Or some combination. Cosmetic in-game purchases aren't necessarily bad, but
the amount my son _has_ spent and _tries to_ continue to spend (I strictly
limit what of his money he can waste on it now) on fortnite stuff is
ridiculous and concerning. A sane middle ground would be helpful.

~~~
gafferongames
Our customer is the game developer not the player. They are free to pass the
cost on to the player if they wish, but I think it’s much more enlightened to
simply sponsor your player’s connection and help them have as much fun as
possible. When players are having fun, they like your game more, stick around,
tell their friends, and spend more in your store.

~~~
kbenson
> Our customer is the game developer not the player. They are free to pass the
> cost on to the player if they wish

That's exactly what I was referring to. In this situation, you would be sort
of like a transaction processor.

> I think it’s much more enlightened to simply sponsor your player’s
> connection and help them have as much fun as possible.

I'm not sure I would use "enlightened" to describe anything to do with the
majority of games that use microtransactions, at least when it's a constant
barrage of "special deals that last for a limited time and you'll only ever be
able to get this digital item right now and if you miss out it will never be
back."

When aimed at a very young audience, this is _anything_ but enlightened. I
don't mind my kids paying for the content they use. I do mind them _investing_
in digital content locked to a single platform that they both don't own, and
which becomes useless when they stop playing. At least if they collected cards
for a collectible card game in real life, they could look at a collection,
pass it on, or sell it.

Basically, I'm advocating for revenue streams that don't require hiring
specialists in promoting addictive behavior linked to spending for games where
a large part of the audience includes adolescents.

~~~
gafferongames
Speaking as the CEO, I would think twice about working with any customer who
intends to pass the cost on to the player.

~~~
kbenson
Why? If it's up front, and it allows two classes of players and clearly
defines them (and allows people to sort by them for games), what'st the
problems that would make you think twice?

I can see some problems with the model, where it might not allow for a more
evenly distributed set of revenue from players and certain players would get
the benefit without the cost by having a very good and well connected and
routed ISP that happens to line up with most servers, but what's the problem?
If the accounts were designated "competitive" tier, that seems to clearly
signal what the user gets. Additionally, at least it's then more information
the service can expose about a player advantage, rather than the user paying
for it outside the service and having a similar advantage but without the same
signalling.

If it's about pay to win, I understand the sentiment, but I think there's a
clear difference between pay nothing (or a token amount) and pay something for
a "regular" account without a bunch of microtransactions and extra tiers, and
what we see with games where you literally can buy your character into an
elite level of capability that changes the game at a fundamental level (either
from hard to easier or from nearly impossible to possible). I think making it
so your players can easily separate based on these levels is good as well
(which is another point towards very few differentiating levels). As long as
the price is low enough that anyone could easily afford it for a while, it's
not a real problem. Close to unbounded capability to pay for advantage is
though.

~~~
gafferongames
Pay to win is a serious concern. There’s also the fact that not everybody gets
improvement. For some players, some of the time, their default route is just
fine. Would be a shame to charge those players $5 a month for something that
gives them no benefit.

~~~
kbenson
Well, there's plenty of payment options for games that include multiple
things, some of which many people never find a use for. What use is a novelty
model/skin that's part of a paid package that is never utilized?

As _part_ of a paid account I think it at least has utility for a large
percentage of people, and may provide more or less useful as their play mode
changes or they play on different servers or with different people over time.

I think a $5 a month plan that included your service, plus a visual indicator
or something on the name or player that's customizable, plus early access to
some content such as maps, would be a good sell. Some people might not take
advantage of the player customization, some might never play the maps early
because their group includes people that aren't paying, and some people might
not have a use for the better networking, but all together, it's an
interesting package.

~~~
politician
Explicitly passing on fees to the customer for a capability that may (or may
not) help them see lower latency erodes trust and makes NN look like an
insurance scam. As mentioned, not all users will definitely see performance
improvements, so your proposed $5/mo is more of an insurance policy than a
defined capability -- it could even be pure profit (who would know?)!

The more honest way to do it would be to convert funds to "latency
improvement" tokens that trickle out of customers' accounts when used to
improve performance, and provide the user with analytics of that improved
performance. (NN could take note here and manage this token bucket feature
themselves in a 3-way arrangement between customer, vendor, and NN that
maintains NN's B2B focus. Happy to explain more if you want to spitball the
idea).

I _really was joking_ when I said that players could find latency improvement
tokens in lootboxes.

~~~
kbenson
> Explicitly passing on fees to the customer for a capability that may (or may
> not) help them see lower latency erodes trust and makes NN look like an
> insurance scam. As mentioned, not all users will definitely see performance
> improvements, so your proposed $5/mo is more of an insurance policy than a
> defined capability -- it could even be pure profit (who would know?)!

Plenty of things are offered as value-added services for products all the
time. Sometimes you get a free skin you might never use, or a coupon on
something else you'll never buy. Players are free to buy a similar service
themselves through other providers, but as _part of multiple incentives_ for
pay, I don't think it's that bad, and really, when it comes to the current
status quo of paying for products with advertising or with cosmetic
enhancements that are designed to have no upper limit in purchasing amount and
advertised in a way to make children go crazy over them, I'm happy if the
payment alternatives include more options that don't include that, even if
they aren't ideal in some other aspects.

~~~
politician
It would be reasonable position if we were talking about a free market, and
not a market captured by last mile ISPs operating a cartel on bandwidth
supported and enabled by their regulators.

The near future involves last-mile ISPs openly selling $40/mo bundles for
Facebook, Amazon, and Google. In that environment, flat fees for _sometimes
potentially_ lower latency seems exploitative (albeit no doubt wildly
profitable).

If NN objects to their service being used to further entrench that future,
then being opposed to working with customers who want to do flat-rate
passthrough pricing seems defensible and consistent.

(Regarding your comment on the status quo, I completely agree.)

------
jauer
This reads like a traffic targeting system ala FB's cartographer (c10r) bolted
onto an ad exchange (continuous bidding etc.)

Interesting to see if they can keep the 10-second update intervals while
scaling to more users.

Curious to see if the market actually wants to pay for marginal improvements
in gaming latency. I would have expected something like this targeting VoIP to
be more lucrative.

------
simonsays2
This sort of stuff has been around for a while, it helps some, but
realistically your game needs to be designed from the ground up with latency
and stutter in mind. If you do this, you wont need the complexity and expense
of this sort of thing.

First you need client side prediction and smoothing. Next, you must design
your game so it follows certain very strict rules about player interaction.
These rules allow the players to feel the have real time synchronization when
in fact synchronization is lazy. The specific rules required are different for
each game, and must be nailed down before game design begins.

~~~
kaoD
Are you teaching game networking to Glenn Fiedler? You probably learnt it from
him, or someone who read his articles!

Now seriously: client side prediction is not magic and it always introduces
its own problems, like getting killed after getting into cover (especially in
low TTK shooters) or the opposite: bullets not registering when if they hit in
your screen.

Client side prediction is sleight of hand. It will never replace actually
perfect network conditions and will still feel better the lower your latency
is.

It's necessary, but not sufficient.

~~~
simonsays2
Fuck yeah, I'm trying to warn people to go it right and not wait for a magic
bullet that will never come.

------
mkj
This could be useful for long-distance remote desktop and SSH too. Curious how
to build on top of it for diverse apps though (use it as a VPN somehow?)

~~~
gafferongames
Right now we are an SDK that embeds in the client, server and backend for
applications. It is open source too (BSD license), but not yet publicly
released. Will be released soon, I look forward to seeing what people do with
it, outside of games.

------
ndmrcf
That reminds me of QuakeWorld's proxy servers.

------
petermcneeley
The primary issue with the internet networking is the lack of Quality of
Service (QoS) controls at the application layer.

~~~
gafferongames
This is true, but without a marketplace and somebody paying, how can we tell
whether the QoS requests are legitimate? I mean, “yeah, my traffic is always
real-time priority”. People just will cheat it... so Network Next is a way of
doing this, while saying, OK, for the truly real-time traffic, applications
are willing to sponsor it. Since it’s paid, it won’t be exploited.

------
peteretep
Where can I read more about League of Legends private internet as mentioned in
the article?

~~~
gafferongames
[https://technology.riotgames.com/news/fixing-internet-
real-t...](https://technology.riotgames.com/news/fixing-internet-real-time-
applications-part-i)

[https://technology.riotgames.com/news/fixing-internet-
real-t...](https://technology.riotgames.com/news/fixing-internet-real-time-
applications-part-ii)

[https://technology.riotgames.com/news/fixing-internet-
real-t...](https://technology.riotgames.com/news/fixing-internet-real-time-
applications-part-iii)

------
mjevans
Other things that can help:

    
    
      * A national policy that actually encourages competition
        > Also one that classifies 'the last mile' as public infrastructure to compete on top of.
      * Fiber to the home/MDU (Ethernet the rest of the way)
      * Where cable Internet must be used, the anti-buffer bloat parts of DOCSIS 3.1 (DOCSIS-PIE).
      * Actually having peer to peer focused ISPs, not content consumption ones.
        Upload should be no less than 25% of download; ideally symmetric.
      * Better visibility in to peering between users
        and actual effort applied to removing the bottlenecks.

~~~
weberc2
> A national policy that actually encourages competition

I'm all for improving the Internet and dunking on large ISPs, but what does
this mean? Specifically what kind of national policy would _foster_
competition (regulation tends to favor monopolies, right?)? Also, don't these
private networks and marketplaces constitute competition? Or are you talking
specifically about the network edge while the article is talking about the
backbone?

This article has me thinking that there is potentially a lot of economic
efficiency to be unlocked by these sorts of marketplaces--basically routing
based on economics, but maybe this is antithetical to net neutrality ("pay to
play" internet routing)?

~~~
mjevans
Ideally competition in all areas, but different areas might require different
policy and this is a question that should actually be studied by experts, or
at least in academia summarizing the results of policy in different nations
(as well as how closely they might match similar housing density for actually
lived in areas).

Some places that I think should be at least initially studied / considered for
LAST MILE* competition include:

* LAST MILE (connectivity) being the connection between citizen's houses and peering/point-of-presence locations: connecting citizens to a marketplace of different service providers.
    
    
      * Remove legal barriers that prevent municipalities and utility entities from providing last mile connectivity
        (often laws lobbied (paid) for by existing monopoly or near monopoly companies).
      * Things similar to "Local Loop Un-bundling"
        > Require the existing providers break out last mile connectivity from their back-haul package.
        > Require they sell access to their private PoP at a cost which they also pay
          at a similar rate (to the PoP on the balance sheet).
      * Things equivalent to 'right of way' grants (poles on public land, trenches, etc) must be owned by the citizens.
        > Also leased at FRAND rates to any entities
        > Serviceable with 'one touch' by any entity licensed with the owners (citizens via the City/County/State/etc that owns it)
    

Possibly other things I can't think of in a short break.

