Hacker News new | comments | show | ask | jobs | submit login

Develop strong out of sync (OOS) detection tools early, and learn how to use them.

The way lockstep work is that the clients gives themselves a rendez-vous in the future and agree to compute one turn of the game, given the players' input of that turn, at that moment. It's very clever. The players input are sent to every clients and will be computed in the future in a deterministic way. So every client is computing the exact same game. Now if the clients doesnt compute the same game given the same players' input, the game is OOS and that basically should never happen because the game is dead.

To detect an OOS, a client needs to compute a hash of some relevant game data (ultimately "everything" ends up being represented by the position of the units (plus a dead state)), every turn and send them with its "end of turn" message to the server/clients. If clients disagree with the value of the hash, they are OOS.

The cool thing with lockstep is that you can record every turn inputs (which is fairly light) and then replay the whole game - because that's already how a game is played. So, when you have an OOS - or any bug, you can just replay the game until the OOS and try to fix it. Sometimes it doesnt work though and you may replay the game without detecting any OOS.

If you dont detect an OOS after a replay, you're basically in deep shit. I mean needle in a haystack of c++ kind of deep shit. Every client needs to dump every data of every function that ever ran in a log file, and then you need to reproduce the OOS and then compare the log files and then guess what caused the data to differ.




Of course if you want your game to update its state at a rate faster than the slowest ping time of the connected client then you've got problems. On LAN play you can get away with this, but over the Internet it's not as workable, especially if some players are on modem.

I remember one game that has crossplay between Linux and Windows brought down by a scheme like this, because it turns out that Linux and Windows default floating point libraries had slightly different rounding behavior, and eventually the pathfinding algorithm would zig on the Windows hosts and zag on the Linux hosts and desync the clients.


You can run your updates faster than the slowest ping. You just need more command latency. So when you, say, command a unit to move somewhere, there's an agreed-upon future time when that's going to happen. This time has to be longer than it takes for the command to get to everyone. If it's not, then you stall. If command latency is very high, the game feels very unresponsive though (and tricks of showing local/visual-only confirmation of your commands only goes so far).

You can't run updates any faster than the slowest machine can run them, though, or again you stall. This was a bigger problem on Age 1 and 2 where render rate and update rate were linked. On Age of Mythology and beyond, the render rate was decoupled from the update rate, so updates could happen more slowly while animations and so forth could still look smooth via interpolation.


I used command latency to do multiplayer for an RTS\Tower Defense on mobile (https://youtu.be/tRAdI0x7L7c?t=300 - think Battle Cats but in 3D). All units were AI controlled, and placing a unit meant it had to go through a spawn animation. It meant the game worked with ping times in the range of seconds. Add in some positional interpolation and events for when each unit changed targets and it worked great.


As an aside, fixing that problem should be feasible. Linux (or rather libgcc or whatever the case may be) uses an open source floating point implementation (and not virial like the GPL): https://github.com/gcc-mirror/gcc/tree/master/libgcc/soft-fp

With enough finageling you could utilize the libgcc floating point implementation on Windows and make it consistent everywhere. Or you could take the wiser approach of avoiding floats outside of graphics code and using fixed point instead.


Does that mean that games that use lockstep are necessarily client authoritative? Or did I miss something?


I see what you mean. Yes this is exactly that: each client is authoritative, alone in the world - and they just have to trust the other clients will compute the same game.




Applications are open for YC Winter 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: