
Lessons learned while developing Age of Empires 1 Definitive Edition - deafcalculus
http://richg42.blogspot.com/2018/02/some-lessons-learned-while-developing.html
======
quadcore
_Develop strong out of sync (OOS) detection tools early, and learn how to use
them._

The way lockstep work is that the clients gives themselves a rendez-vous in
the future and agree to compute one turn of the game, given the players' input
of that turn, at that moment. It's very clever. The players input are sent to
every clients and will be computed in the future in a deterministic way. So
every client is computing the exact same game. Now if the clients doesnt
compute the same game given the same players' input, the game is OOS and that
basically should never happen because the game is dead.

To detect an OOS, a client needs to compute a hash of some relevant game data
(ultimately "everything" ends up being represented by the position of the
units (plus a dead state)), every _turn_ and send them with its "end of turn"
message to the server/clients. If clients disagree with the value of the hash,
they are OOS.

The cool thing with lockstep is that you can record every turn inputs (which
is fairly light) and then replay the whole game - because that's already how a
game is played. So, when you have an OOS - or any bug, you can just replay the
game until the OOS and try to fix it. Sometimes it doesnt work though and you
may replay the game without detecting any OOS.

If you dont detect an OOS after a replay, you're basically in deep shit. I
mean _needle in a haystack of c++_ kind of deep shit. Every client needs to
dump _every data of every function that ever ran_ in a log file, and then you
need to reproduce the OOS and then compare the log files and then guess what
caused the data to differ.

~~~
jandrese
Of course if you want your game to update its state at a rate faster than the
slowest ping time of the connected client then you've got problems. On LAN
play you can get away with this, but over the Internet it's not as workable,
especially if some players are on modem.

I remember one game that has crossplay between Linux and Windows brought down
by a scheme like this, because it turns out that Linux and Windows default
floating point libraries had slightly different rounding behavior, and
eventually the pathfinding algorithm would zig on the Windows hosts and zag on
the Linux hosts and desync the clients.

~~~
skittlebrau
You can run your updates faster than the slowest ping. You just need more
command latency. So when you, say, command a unit to move somewhere, there's
an agreed-upon future time when that's going to happen. This time has to be
longer than it takes for the command to get to everyone. If it's not, then you
stall. If command latency is very high, the game feels very unresponsive
though (and tricks of showing local/visual-only confirmation of your commands
only goes so far).

You can't run updates any faster than the slowest machine can run them,
though, or again you stall. This was a bigger problem on Age 1 and 2 where
render rate and update rate were linked. On Age of Mythology and beyond, the
render rate was decoupled from the update rate, so updates could happen more
slowly while animations and so forth could still look smooth via
interpolation.

~~~
a_t48
I used command latency to do multiplayer for an RTS\Tower Defense on mobile
([https://youtu.be/tRAdI0x7L7c?t=300](https://youtu.be/tRAdI0x7L7c?t=300) \-
think Battle Cats but in 3D). All units were AI controlled, and placing a unit
meant it had to go through a spawn animation. It meant the game worked with
ping times in the range of seconds. Add in some positional interpolation and
events for when each unit changed targets and it worked great.

------
nimbius
As with all historic RPG's, The key lesson is in how you craft your C integer
types. One type allows you to cleverly assign various character traits to
specific groups and users, the other turns Ghandi from a benevolent peace
loving icon to a harbinger of endless misery and destruction

[https://www.geek.com/games/why-gandhi-is-always-a-
warmongeri...](https://www.geek.com/games/why-gandhi-is-always-a-warmongering-
jerk-in-civilization-1608515/)

~~~
asavadatti
*Gandhi not Ghandi. It is right there in your URL

------
xrisk
It's a pity they aren't releasing on Steam though. Most players aren't going
to buy the game from the Win10 store.

~~~
gsich
Why not? A MS account is not uncommon.

~~~
zingmars
There is a stigma in the PC gamer community against Microsoft, especially
after Games for Windows. "Steam or go home" sentiments also aren't all that
uncommon. As for the accounts - I really don't know anyone who actually uses
the MS store (whereas pretty much everyone who plays video games has a Steam
account), and I interact with plenty of end-users daily. Maybe it just happens
that the area I live in never adopted it, but I would still say that Steam is
the better platform to try to get exposure on.

~~~
colmvp
Trading one monopoly for another, I guess.

~~~
AlphaSite
A thus far benevolent monopoly who have indicated positive intent and
applicable thought patterns (i.e. the whole piracy is service problem).

~~~
Pica_soO
They could release it on origin...

Joke aside, i assume a month later they will release on steam. The red number
sirens and the fear of pirates have convinced a lot of sailors to rock the
boat.

------
adityab
Looks like they want to avoid changing gameplay as much as possible.

Considering their focus on preserving even minute details of game dynamics...
I wish they'd made an exception for pathfinding - the algorithm in the
original AoE is extremely bad.

In particular, I really wish they'd introduce some improved ways to command
groups of units. IIRC the Rise of Rome expansion introduced double-click group
selection of all similar units, so it's not without precedent. If they change
nothing else, my number one wish would be formations like in AoE II. I can't
stand the way units just swarm like ants when told to move together.

~~~
munificent
It's really hard to change something like pathfinding without revamping the
entire game. A game's balance is like a hanging mobile. Change one aspect, and
the whole thing becomes unbalanced until you adjust all of the other weights
in the system.

For example, if you make the pathfinding smarter, then there's a good chance
you've made all AI enemies effectively more powerful and now all of the level
difficulty is too high. You may also affect the strategy of levels — levels
that used to have carefully crafted chokepoints that the player must notice
may no longer be chokepoints with smarter pathfinding. That in turn might lead
you to want to tweak the design of the level...

~~~
verroq
Except this is a RTS and when players want a unit to get to a place it should
get there without taking a detour across half of the map, especially when the
player can visually see an optimal path.

So no, there are no downsides to better pathfinding.

~~~
s_m_t
consider the following scenario:

In a battle between melee and ranged units the ranged units want to maximize
their surface area touching the ranged units so that all the melee units can
attack at once and quickly change targets once an enemy unit has fallen. The
ranged units want to minimize their surface area so that only the outer layer
of the group will be attacked while every unit on the inside can freely fire
on the melee units.

In an older RTS with poor pathing this meant constantly jockeying your forces
around trying to either get a good surround or keeping rank. Moving your
forces in certain ways could unpredictably change the formation of your units
so great care had to be taken when engaging the enemy. Getting this to work
out was a skill in of itself that took strategy, positioning, the ability to
predict the enemies move, etc. All of this depth could take place between very
small groups of very simple units.

In an RTS with perfect pathing the ranged units always move in a perfect
sphere (minimizing surface area and maximizing volume) and the melee units
perfectly synchronize their movements so they don't trip over each other when
surrounding the enemy. Once it is decided that a group of simple melee units
and ranged units will fight there really isn't much for the player to do
except tell them to get on with it.

As real time strategy games have evolved along with better pathfinding, we've
found out that with all things being equal, perfect pathing makes for very
very boring fights. Active abilities had to be added to give the player
something to do during the fight. Rock Paper Scissors type unit counters had
to be added so that positioning mattered again. More AOE spells and debuffs
had to be added so that another thing would force players to jockey units
around.

Study the design of Starcraft BW and Starcraft II and you can see this very
clearly. Certain fundamental unit matchups were completely broken by improved
pathfinding such that more and more complicated mechanics had to be added to
the game to give them balance and dynamism again.

Of course, this only really applies to C&C, SC, and AOE style RTS games. More
modern RTS series like Men of War and Wargame don't follow this pattern
because the range and lethality of the units typically far exceed their
movement speeds and terrain and cover play a much larger role, but that is a
whole different topic.

------
smacktoward
_> Assume the original developers knew what they were doing. The old code
shipped and was successful. If you don't understand it, most likely the
problem is you, not the code._

This needs to be chiseled on a tablet somewhere.

~~~
khalilravanna
Really great piece of advice. The "every other developer is an idiot"
mentality that can be popular in some people/cultures will almost always bite
one in the ass.

In general, I liked how applicable a lot of the points made in this post apply
to development out side of problem space videogames.

------
indescions_2018
Love reading gamedev post-mortems such as these ;)

Although what author discusses is specific to Age1 engine. Time sync in webrtc
peer networks encounters many of the same pitfalls. With errors tending to
propagate non-linearly as more peers (and more complex scene objects) are
added.

ThreeNetwork - Network sync library for Three.js

[https://github.com/takahirox/ThreeNetwork](https://github.com/takahirox/ThreeNetwork)

~~~
skittlebrau
We got way better at dealing with synced/deterministic simulations over time.
Much better sync logging system, decoupling render rate from simulation rate,
more institutional knowledge/intuition about all the surefire ways to break
determinism (using unsynced random number generator, uninitialized variables,
reading local machine state from sim, using local timers rather than synced
ones, not resetting floating point flags after calling Direct3D, etc). The bad
news was once we were generally avoiding all the common causes as a matter of
course, the sync bugs that remained were non-trivial ones that took lots of
time to diagnose.

------
Avalaxy
I'm surprised they decided to modify the old code, I expected them to make a
clean start and re-create the original experience from there.

~~~
youseecomrade
Way before Counter-Strike: Global Offensive people tried to port the classic
CS 1.6 (or 1.5) behavior to more modern engines or CS versions. From what I
know no project succeeded because even replicating the recoil behavior were
extremely hard. Gameplay features that were actually bugs would have to be
simulated. I imagine it would be ten times harder for a complex game like AoE.

------
grigjd3
Not building custom containers is generally good advice. The chance you build
something that underperforms is noticeable. Other coders won't know what
you've made. There is a fair chance that std has what you need if you're even
just a little bit thoughtful.

~~~
user5994461
Note that the stl containers were far from perfect a few decades ago.

~~~
grigjd3
Amazing what a few decades of development by talented and determined people
can accomplish. Point is, my custom container that I wrote in an hour or so
probably won't be as good.

------
ztoben
Fun read. I've always wondered what kind of experience it takes to get picked
for a project like this. Seems like it would be fun.

------
Tyrannosaur
Cool blog post, but light gray text on white is literally unreadable. How are
people still doing this?

------
lanius
Good thing they decided to keep the gameplay the same, like Starcraft:
Remastered. I never played AoE 1 competitively, but if it's like SC many
bugs/glitches became core gameplay mechanics. Here's hoping the AoE 1 remake
fares better than the one for AoE 2!

------
WadeJohnson
I have no real experience with C/C++ as well as game development but I always
wonder what would happen if old games like these would get recompiled with an,
for example, avx2 flag.

~~~
CJefferson
Very little. Quite a few programs recieves a decent boost when compilers moved
from a 386 default to 486, then Pentium 1. Since then the gains have been much
smaller, unless you use the instructions on purpose.

