
1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond (2001) - tosh
https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php?print=1
======
red_admiral
From another Ensemble Studios interview with Gamasutra (emphasis mine):

"6\. We did not plan for a patch. The version 1.0a patch, even though it was a
success, was problematic in that as a company we had not planned for it. THE
GENERAL ARGUMENT IS THAT IF YOU KNOW YOU ARE GOING TO NEED TO RELEASE A PATCH,
THEN YOU SHOULDN'T BE SHIPPING THE GAME IN THE FIRST PLACE."

Those were the days!

~~~
SketchySeaBeast
Which is a ridiculous statement - I remember the number of 90's games that
were utterly unplayable in that they'd crash constantly or do other ridiculous
things (The remake of Temple Of Elemental Evil would actually delete your C:
drive upon uninstall). I feel like things haven't gotten worse as far as the
need for day 1 patches, they've just gotten more pragmatic on their
deployment.

~~~
gambiting
As a dev on modern AAA titles - the main driving reason behind Day 1 patches
is that you do the first submission to Sony/MS/Nintendo 2-3 months before the
game is about to be released. We just did a 1st sub on a title coming out in
November. That means you still have potentially hundreds of people on the
project for few months until the game actually comes out - so they might as
well keep working on the game and their work is released as day 1 patch.
Obviously we don't purposefully leave bugs and features until day 1 patch -
but in the extra time we might as well add a few things and fix some bugs.

~~~
SketchySeaBeast
I'd never assume you left them on purpose. Have you worked with Steam? What
sort of turn around time do they require?

~~~
gambiting
Steam is super quick, especially with a company of our size, we basically have
the necessary permissions to publish when we want to, it requires very little
from Valve.

------
borellini
And in 2018 with Internet connections a bit better than a 28.8k modem and
hardware slightly more powerful than Pentium 90, the HD remake of AoE2 fails
to deliver a reliable multiplayer gameplay, having lag of multiple seconds and
at least as many out-of-sync bugs as the original game did 20 years ago. It
should be closer to 150 than 1500 archers with the HD edition.

~~~
lucb1e
This is one of the many reasons why I refuse to buy AoE2. The "new" version
does not offer multi-platform support, it does not offer a stable multiplayer,
it requires half a gigabyte of RAM instead of the previous 32MB, and it still
costs as much as I pay these days for some of the smaller but brand new games
(not one they made profit on for almost twenty years already without any
maintenance).

~~~
chaosbutters314
if you play it on voobly versus steam, there is almost no lag and plays super
smooth

------
lordnacho
As a low latency coder, I found it fascinating.

It turns out the game is synchronized by each player having the commands for
each 200ms "turn" a couple of turns in advance, and then playing the actions
so that the same happens on all player's machines. That includes sending
random seeds around. And then there's a load of provisions for lost packets,
slow machines and thus forth.

Is this how most games do it? I would think something like WoW couldn't do
this, and indeed sometimes I'd see glitches where a character would blink
(like the spell) to somewhere new.

~~~
rollcat
Welcome to the world of game network programming :) it's an exciting place!

Yes, you have to be more-or-less real time, so you must compensate for
latency, unreliable/slow connections, jitter, etc. If you're used to the web
and the request/response model, you have to throw all of this out the window.
The 200ms delay "hack" is pretty much standard practice, the window will
differ from game to game (smaller in FPS's), but it's usually there.

Most games use UDP, since transmission of any single package doesn't have to
be reliable, and in case some packets are lost, it's cheaper to re-calculate
the state diff and resend one slightly larger package instead of two (or more)
standard-size packages. Sometimes this can result in a "blink".

With sending around seeds and other "secret" data, you have to make a trade-
off, since sending too much allows for cheating (map hacks, wall hacks, etc),
but sending too little will create unpleasant surprises (enemy "teleporting"
from around the corner).

Also often it's cheaper to run most of the calculations on the client (even
including the critical stuff like hit tests, damage calculation, etc), and
only occasionally verify the results on the server - especially in MMO's.
Clients that are found suspicious get verified more often, and eventually get
penalized / kicked.

Source: never actually wrote a networked game, but love reading about this
stuff.

~~~
pure-awesome
> Source: never actually wrote a networked game, but love reading about this
> stuff.

Got any favourite sources where I can learn more? It sounds pretty
interesting!

~~~
rollcat
Sorry for a late reply! Some good starting pointers:

[https://www.gamedev.net/](https://www.gamedev.net/)
[https://www.reddit.com/r/gamedev/](https://www.reddit.com/r/gamedev/)
[https://www.reddit.com/r/truegamedev/](https://www.reddit.com/r/truegamedev/)

Some interesting case studies are anything by Id Software (Quake etc), and
Lineage (that's mostly tales of a friend who is a hardcore player and a
developer; he'd have the relevant source code open in a separate window while
playing).

~~~
pure-awesome
Cool, thanks.

------
adamson
> Part of the difficulty was conceptual -- programmers were not used to having
> to write code that used the same number of calls to random within the
> simulation

Can anyone grok this? I can't see why this (each player's simulation making
the same number of calls to random) would ever not be the case if all players
are running the same patch of the game and are executing the same commands.

~~~
MarkTerrano
(Original author here) - here was a really hard-to-find bug - in some
instances more than one quantity of fish could be placed in the same location
- that meant the game would work fine until someone fished that same fish the
second time and the fishing boats would diverge in the different simulations.
The world sync check only counted the tile contents so we didn't see it. -
There were actually two RNG's in the game, one was synchronized with the same
start seed on all machines (basically the same random pool) for combat and
whatnot - the other was unsynchronized and used for animation variance, etc
-things that weren't gameplay related. Not knowing when to use one of these
specifically (e.g. animals facing seems like animation but definitely affected
gameplay if they could be hunted) could alter the code path and cause an out-
of-sync condition.

~~~
hughes
Why even keep the unsynchronized RNG? I'm guessing it was more performant when
used appropriately?

~~~
kirkules
If you don't have an unsynchronized RNG, then any unsynchronized content can't
use an RNG, and unsynchronized content is important for improving a game's
experience by using local resources for things that don't really need to be
shared or sent over a network.

For example, maybe in a FPS, part of the non-gameplay-critical graphics use
particle generators for a cool effect that not all players see (because it's
behind a building for some of them and thus doesn't even need to be rendered);
if these generators used a synchronized RNG, then all players would have to do
computations for every particle effect happening anywhere, just so that the
combat and more game-important RNG values would be in synch when they really
need to be.

------
chaosbutters314
For anyone complaining about aoe2 lag, it is mainly just the version on steam.

if you apply the compatibility patch, you can load your steam copy onto voobly
for free and play with next to no lag. The difference is night and day plus
better features with Voobly.

~~~
executesorder66
Thanks. Is there any reason why don't they just fix the steam version?

------
loser777
>For RTS games, 250 milliseconds of command latency was not even noticed --
between 250 and 500 msec was very playable, and beyond 500 it started to be
noticeable.

I wonder if the rise of competitive RTS has changed this guideline. In SC2
people will start to complain if their ping to the server is more than about
130ms, and anything above 150ms starts to become painfully noticeable.

In Brood War being able to play on "LAN Latency" was a always a huge deal---to
the point that unauthorized third party ladder systems enabled players to play
at "LAN Latency" even when battle.net official didn't support it.

~~~
MarkTerrano
Human cognitive reaction times are a lot mroe latent than connections are now.
The quality of play percieved by the players in the studies I did back in the
day was much more about consistency of responsiveness and not that direct
number of milliseconds. Best possible Human reaction time for cognitive tasks
is still around 250msec for most players - it would be great to see updated
information for tournament players (who are a different class of player) on
what their actual perception-to-action time is in their favorite games. AOK
and beyond code used an adaptive scaling system to go faster when the network
would reliably move packets more quickly - so it would auto-adjust to 'lan
speed' (actually a combination of the best render speed of the slowest PC plus
an estimate of the round-trip latency). Also - the command confirmation is not
waiting for RT latency - you are getting confirmation when the command goes
into the local buffer - 'command accepted' \- once that happens it is going to
execute so you get the confirm bark sound from the unit or building queue - or
the movement arrow triggers. The games actually do their command
simultaneously when all machines are executing the turn.

~~~
wongarsu
>Human cognitive reaction times are a lot mroe latent than connections are
now. [...] Best possible Human reaction time for cognitive tasks is still
around 250msec for most players

This is an important realization. Our brain can perceive reasonably fast
actions, but our reaction is much slower. Under good conditions we can easily
tell a 60fps animation from a 30fps animation from a 10fps slideshow, but the
fastest reaction time we can manage is around 100ms (the time one frame is
visible at 10fps).

We are reasonable tollerant to latency because our brain has quite high
latency, and all our actions have to account for that (for example the point
in time where you decide to release a ball you are throwing is very different
from the point it is actually released). On top of that, many real-world
interactions behave similar to latency (e.g. springs). What throws us off is
inconsitent latency, because then we are suddenly not able to predict when to
perform an action in order to have the effect at the desired point in time.

~~~
annywhey
Yeah, there are a few nuances to latency numbers.

The 250ms pure read-react time deals with arbitrary events, but when we can
chunk reactions into a practiced technique our precision goes way up, to
nearly the individual millisecond: thus musicians can play rapid passages with
unusual rhythms in time if they have time to plan and prepare, but they lose
this ability if dealing with unusually high latency(extreme reverb, amp across
the stage, digital audio with huge buffer sizes). The technique, after all, is
based on fast confirming feedback that your execution is correct.

And like you say, "bouncy" latency is even more disruptive. We can adjust to a
small and consistent lag, but inconsistency will degrade any level of skill.

~~~
azernik
Agreed on all counts.

> And like you say, "bouncy" latency is even more disruptive

The technical term for this is "jitter". Networking and telecoms people pay a
lot of attention to this metric, both for the reasons you cite and because
jitter is much more noticeable than high-but-constant latency in voice or
video communications.

------
EamonnMR
A classic. I've been working on doing a minimal Python implementation of some
of the concept here:
[https://github.com/eamonnmr/openlockstep](https://github.com/eamonnmr/openlockstep)

~~~
JanisL
Nice! Is this in a working state?

------
sparky_
For the interested, the unannounced game referenced in the article, "RTS3", is
in fact Age of Mythology, released in 2002.

------
nwmcsween
Would it be possible to apply Delta CRDT's to something like game netcode?

~~~
specialist
I've been wondering the same. Thanks for asking.

------
password4321
I've forgotten all the auto-taunts except "Sure, blame it on your ISP!"

~~~
wild_preference
14: start the game already!

~~~
MarkTerrano
My personal favorite. !14 !14 !14

~~~
BigJono
My friends always used to do:

A) Build a navy! B) Stop building a navy! A) Build a navy! B) Stop building a
navy!

------
addicted
The competitive community is very small, but it's growing (which is amazing
for a game released almost 2 decades ago).

------
banachtarski
Don't try this at home kids. Floating point determinism is a huge problem,
especially with different chips/architectures/compilers out there.

~~~
JanisL
I'd love to hear more about this, any links I can look at?

~~~
banachtarski
[https://gafferongames.com/post/floating_point_determinism/](https://gafferongames.com/post/floating_point_determinism/)

Here's a decent primer on it. In general chips implement floating point math
differently and you could google terms like "floating point register" and
"floating point stack" to get you started. Most importantly, floating point
math is not generally commutative due to precision/rounding.

~~~
JanisL
Thanks!

------
mrkeen
If only it scaled to 300 archers over broadband.

This game is amazing and has aged so well. The only downside is the network
pauses and out-of-sync errors that inevitably happen 40 minutes into a two-
player game.

I can't remember the last time I didn't set the population limit to 75 in an
effort to alleviate network issues.

~~~
cdiddy2
play on voobly instead, its so much more stable

~~~
voltagex_
Why? What has Steam done that makes it so bad?

~~~
mrkeen
No I found the Steam version to be a faithful reproduction of the original. I
had these out-of-sync issues back in the days of passing age2 around as a
folder at LANs.

------
_Codemonkeyism
My first 'network' game was Fire Power on the Amiga - must have felt exploring
a new world writing those early network games.

