
Making Fast-Paced Multiplayer Networked Games Is Hard - Impossible
http://www.gamasutra.com/blogs/MarkMennell/20140929/226628/Making_FastPaced_Multiplayer_Networked_Games_is_Hard.php
======
tinco
The hardest part is that most game engines are not designed to be networked in
this way. Find an open source physics engine that natively supports
teleportation, easing and prediction, they're not there. UE4 is the first
engine I've used that seems to have a very nice multiplayer API, and it's only
been out for indie developers for a couple of months.

So the hard part is not devising the networked scheme, it's building a whole
game engine (or thoroughly modding one) afterwards, at least in my experience.

I was working on a multiplayer racing game project (like GTA2), and my
approach was to run 2 physics engines in parallel. One physics engine would
always be authoritive, and be in sync with the server, but because of the lag
would always be a frame or two (or more) behind. The other would be working on
predicting what is going to happen. Every frame snapping back to the
authority, and then applying the (predicted and user) inputs over that.

The actual position the user would see would be an averaged position between
the current predicted position, and its previous predicted position (to
prevent too much jitter/snapback).

If you've got a better scheme to do networked physics I'm all ears :)

~~~
aaasux
_> UE4 is the first engine I've used that seems to have a very nice
multiplayer API_

I'm working on a triple a ue4 title and i don't like ue4s replication system
at all. It's a step towards making the networking concerns invisible, and that
always seems like a mistake to me. If the protocol for keeping clients in sync
were more explicit, it would be easier to tightly control what and when things
get sent over the network.

~~~
JabavuAdams
Are you able to contrast it with Tribes' / Garage Games' model? What about
RakNet? I'd love to hear more.

------
ggambetta
Another series of articles I wrote ~4 years ago about the same topic, but
perhaps more detailed and with a live demo:
[http://www.gabrielgambetta.com/fpm1.html](http://www.gabrielgambetta.com/fpm1.html)

The title of this article sounds vaguely familiar ;)

~~~
cinskiy
Your articles were SUPER helpful when I added multiplayer to my game, you may
be the reason I succeeded in it, alongside Valve documentation. Thank you so
much for writing them!

~~~
ggambetta
Happy to hear my articles helped! But the reason you succeeded is you, not me
:)

------
teddyh
See also:

 _Distributed Virtual Reality – An Overview_ , Bernie Roehl, 1995:
[https://ece.uwaterloo.ca/~broehl/distrib.html](https://ece.uwaterloo.ca/~broehl/distrib.html)

 _Characteristics of UDP Packet Loss: Effect of TCP Traffic_ , Hidenari
Sawashima, Yoshiaki Hori, Hideki Sunahara, Yuji Oie, 1997:
[http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM](http://www.isoc.org/inet97/proceedings/F3/F3_1.HTM)

 _I Shot You First: Networking the Gameplay of HALO: REACH_ David Aldridge,
2011: [http://www.gdcvault.com/play/1014345/I-Shot-You-First-
Networ...](http://www.gdcvault.com/play/1014345/I-Shot-You-First-Networking)

~~~
rlefebvre
This last link is a gem. They refer to the TRIBES Networking Model in the
talk, explained here also: [http://gamedevs.org/uploads/tribes-networking-
model.pdf](http://gamedevs.org/uploads/tribes-networking-model.pdf)

There is a beautiful implementation of this in the old OpenTNL that seems to
be still alive in the newer Torque engine:
[https://github.com/GarageGames/Torque2D/tree/master/engine/s...](https://github.com/GarageGames/Torque2D/tree/master/engine/source/network)

The original article does not discuss the aspect of scaling fast-paced games
to handle large numbers of replicated objects and players under constrained
network conditions but I found the approach proposed in the original TRIBES
model to be the most (only?) credible so far. I don't see support for this in
any of the popular, modern network libraries (RakNet, enet, lidgren, ...).
They all seem to have taken the 'multiple reliable channels' direction but
that just doesn't seem to scale to many connected players the way the TRIBES
model does.

I would love to hear from anyone who has had experience with that!

------
hellbanner
In specific contexts like a 1v1 fighting game, you can get clever. GGPO*.
Every move in the game has a specified "startup" time which is generally a)
too fast to react to & b) consistent.

When you jump & throw a punch you send the frame along with your attack. My
client speeds up your character's game state to match what your client
experiences.

[http://www.gamasutra.com/view/news/177508/The_lagfighting_te...](http://www.gamasutra.com/view/news/177508/The_lagfighting_techniques_behind_GGPOs_netcode.php)

------
fredley
Another reason
[TagPro]([http://tagpro.koalabeast.com](http://tagpro.koalabeast.com)) is an
excellent game. It's implementation of this stuff is great - with a reasonable
connection my ping is usually under 10ms, and the game is often won and lost
with the smallest of margins making this kind of thing very important.

Not to mention the excellent gameplay and mechanics - it's very simple to
understand and learn, but very very difficult to play well!

~~~
Luc
Not saying this is the case, but 10 ms is a very low ping that could hide a
lot of issues with the implementation.

------
no_future
Fascinating, I've been interested in what kinds of serverside software real
time online games(WoW, Call of Duty, etc) run for a while now, but haven't
been able to find much info on it.

Seems that they would need to be optimized for many very high stress
concurrent connections with as little latency as possible, so I'd guess that
they run C/C++ and/or Java? Do they use something like Websockets, or do
UDP/TCP or some other persistent two way connection method? There don't seem
to be any publicly available libraries focused on this kind of thing, so I
assume that they develop their networking stuff mostly in-house.

Anyone that knows about this stuff willing to share?

~~~
dividuum
> There don't seem to be any publicly available libraries focused on this kind
> of thing, so I assume that they develop their networking stuff mostly in-
> house.

Have a look at RakNet
([http://www.jenkinssoftware.com/](http://www.jenkinssoftware.com/)) or enet
([http://enet.bespin.org/](http://enet.bespin.org/)).

~~~
no_future
Interesting. Enet seems pretty barebones, but I guess that would be beneficial
for games where there can be a lot of variance in gameplay, what with the
parent post's explanation of the level of precision required.

------
Dove
Just a few months ago, I did an overhaul of the network code for DXX-Retro[1]
-- a source port of Descent. Descent worked much better over the (laggy,
lossy, bursty) net than DOOM, and -- if you're looking to mimic old school
games -- is really worth studying.

Some quick technical commentary:

The bandwidth calculation in the article is predicated on sending updates at
60 hz -- or what we'd in the Descent community call 60 PPS. Probably because
the screen is refreshing at that rate? It's unnecessary. You want a high
framerate for control of your own game, but you don't need it to see enemies
smoothly. Remember, movies only run at 24 FPS. ;)

The highest I allow in Descent is 30 PPS, and really . . . it's seen as a
luxury. 20 is generally fine. Sometimes I play games at 10, and there you can
definitely tell -- even with the smoothing (it sends velocity and
acceleration, too, and interpolates) -- but it's perfectly playable.

Which is something worth remembering. With old school games, "crappy but
perfectly playable" is actually all they were able to achieve.

No, the physics engines aren't perfectly locked, and how tolerable this is
will depend on your game. In a simple FPS, this really isn't a big deal. You
just lag lead (compensate in aim, both for the motion of your enemy, and the
fact that the data is old). :)

Some of how he's proposing to send the data is wasteful. He initially proposes
sending "is weapon equipped" and "is firing" as 1 byte a piece -- and later
concludes he can get those in 1 bit a piece. True. That's a savings by a
factor of 8 right there. But I can do you one better -- don't send it every
update. How often do those states really change?

In Descent, we have two classes of data: position and orientation data that's
sent at a steady rate (10-30 PPS), and event data that's sent . . . whenever
it happens. Equipping weapons is definitely the second type; it happens
extremely rarely -- like, seconds pass between weapon switches. :)

One thing that may surprise you. We don't send "is firing" as a flag with
every packet -- we send one packet per shot taken! Two reasons for this: one,
it's actually lower bandwidth. Shots fire rarely -- our fastest gun fires 20
bullets per second, but the next fastest fires six. Then five, then four, then
. . . one shot every two seconds. And you're not _always_ firing, either.
Sending one packet per shot saves bandwidth. But it also increases accuracy!
We attach those shot packets to a position and orientation update, so -- even
if the receiver has incorrectly interpolated your position -- the shot goes
exactly where you intended it to. This is very important. :)

As a player (and a programmer -- but mostly a player), I'm concerned about the
article's conclusions about the value of prediction, and the recommendation to
avoid position popping by applying smoothing -- to quote the article, "it
ruins the virtual reality of the game".

Ok, yes, it does. But the thing is, you have a choice here. You can present
your players with something pretty and smooth that is fundamentally a _lie_ ,
or with something jittery that is the best knowledge you have about _where the
enemy is_. This is a fundamental tradeoff: versimiltude or accuracy. You can't
have both.

My players overwhelmingly prefer accuracy. To them, the avatars on the screen
are targeting aids, and they understand that the data is old and a bit lossy
and bursty sometimes, and they want the best data possible so they can take
the best shot possible. :)

I suppose your mileage may vary by audience. Mine's been playing this game 20
years and "crappy but playable" is both normal and good to them. :)

But -- I can't imagine this would be different in another FPS -- taking a shot
that you _know_ is good on your screen, and have it not hit the other guy . .
. that's rage-inducing right there!

Yeah, networked games are hard. For sure. And there are fundamental hard
tradeoffs involved in engineering them. For sure. But it's an interesting
problem, and also worth it. :)

[1] [https://github.com/CDarrow/DXX-Retro](https://github.com/CDarrow/DXX-
Retro)

~~~
jl_2014
I wrote the multiplayer code for Descent 2 and Descent 3 (as well as the
graphics engine for D3). Although I can't remember all the details because D2
was back in 1995(!), D2 had a significantly overhauled network layer from D1.
Some examples: Short packets, where position and orientation data was
quantized down into single bytes instead of floats, lower packets per second
(you could go down to 5 PPS if I recall correctly). We were also the first
game where if the 'master' dropped out the game another player would become
the master in a hand-off scheme that was a bit complex. The master controlled
things like notifying other players of new players, end of level stuff, etc.
We had to sweat every byte because we were trying to have 8 players with low
lag over a typical 28.8 baud modem.

D3 changed the overall feel of the Descent series, mostly because we
introduced a terrain engine and that had a cascading effect on the rest of the
game. The speed of the ship, for example, had to be significantly increased
because if we used the ship speed from D1/D2 then going out into the terrain
felt like you were stuck in molasses.

Working on those games was incredibly fun. Ah, to be 25 again.

~~~
lotharbot
> _" We had to sweat every byte ... 28.8 baud modem."_

Yeah, that stuff _really_ mattered in that era. Nowadays we've eliminated a
lot of the compression and upped the packet rate -- a modern "bad" connection
can handle 8 players at 10 pps without compression, and the accuracy is
necessary with pilots who dodge as well as some of the modern masters.

> _" if the 'master' dropped out the game another player would become the
> master"_

We'd love to reimplement that. It was such a great feature! It was removed
when one of the previous coders overhauled the networking code.

> _" D3 changed the overall feel of the Descent series"_

Yeah. It was a fun game in its own right, but it really shifts the emphasis
from dodging to aiming. It didn't play as well in a 1v1 setting, but had some
really compelling team modes. I played CTF every night from about 2004 to when
my son was born in 2009.

EDIT: P.S. there's a Descent LAN coming up this month. If you still keep up
with Gwar, he has all the info.

------
Animats
This problem was first described in Farmer and Morningstar's "The Lessons of
Lucasfilms' Habitat" (1990), which was a MMORPG running on Commodore 64
machines over 300 baud dial-up modems. They referred to this problem as
"surreal time".

Back then, they had latencies of 100ms to 5000ms. The original article here
says they can get latencies of 200ms from 99% of XBox connections. Not much
progress in latency in 25 years. I can understand the LAN party nostalgia.

------
toqueteos
I've been using Bolt [1] for Unity3D for a fast-paced game and surprised me
how far I got with almost no knowledge of the library. It's UDP all the way
and super explicit about everything.

[1]: www.boltengine.com

------
moomin
As apparently, is being on the right side of the GamerGate fiasco. (They're
down at the moment.)

------
benihana
I always found this article about how the Source Engine handles networking
fascinating:
[https://developer.valvesoftware.com/wiki/Source_Multiplayer_...](https://developer.valvesoftware.com/wiki/Source_Multiplayer_Networking)

Really shows how hard just a few of the challenges of making a multiplayer
game are.

~~~
squeaky-clean
I also love the Source Engine networking. Another one I like is The Torque
Game Engine. It has a very similar networking model, I think it may even be a
little better (it's the engine that powered the original Tribes game).

I remember buying a license to the original TGE years ago and digging through
the networking code, it was great. They've open-sourced the latest version of
Torque under the MIT license, and I believe the networking code is nearly
identical to the original code used for Tribes.

~~~
JabavuAdams
Yeah, there was a really interesting paper on the Tribes networking model in
the '97 or '00 GDC conference proceedings.

