
Game Programming Patterns: Event Queue (2014) - tosh
https://gameprogrammingpatterns.com/event-queue.html
======
dbatten
Reminds me of the excellent post about the architecture of multi-player in the
original Age of Empires game:

[https://www.gamasutra.com/view/feature/131503/1500_archers_o...](https://www.gamasutra.com/view/feature/131503/1500_archers_on_a_288_network_.php)

You had high-ping 28.8k internet, and you wanted to be able to have immersive
battles in a world with hundreds of units controlled by several players.
There's no way you could send real-time information about the status of every
object in the game. So they didn't. They made the game simulation completely
deterministic and then simply shipped user input around on the network. User
input was processed on a slight (couple hundred milliseconds) delay (to allow
for network transmission speeds) and executed simultaneously in "game time" on
everybody's machine, so the simulation stayed in perfect sync. Voila, no
worrying about lag.

~~~
algorithmsRcool
I recall that Supreme Commander used the same technique, a fully deterministic
sim run in lockstep by all multiplayer peers. For it's era, it had very very
high scale for an RTS game. (thousands of units and enormous game maps (the
biggest of which took 15+ mins to send naval units across)

If I recall correctly, there was another benefit. You could save a multiplayer
game by just having all peers dump the current game state to disk and resume
it later. Since a big SupCom game could take HOURS this was a great feature.

However, there were 2 issues with SupCom's implementation.

1\. It wasn't perfect and sometime hours deep into a big game, you would get a
"desync" error and it was unrecoverable.

2\. Everyone needed to be able to run the sim at the same speed. So the game
could only progress at the speed of the slowest player's computer. And SupCom
was VERY compute intensive.

~~~
planetsmashr
I had no idea that supcom worked that way! Thanks for the insight.

Your comment brings back fond memories of playing huge supcom games at a local
LAN cafe. Those were some of the best times I've had gaming.

~~~
algorithmsRcool
I couldn't count the number of hours I spent playing that game with friends.
Never played anything else like it!

------
bob1029
If anyone is curious, one of the fastest multi-threaded queue implementations
out there is the LMAX Disruptor.

[https://github.com/LMAX-Exchange/disruptor](https://github.com/LMAX-
Exchange/disruptor)

[https://github.com/LMAX-
Exchange/disruptor/wiki/Performance-...](https://github.com/LMAX-
Exchange/disruptor/wiki/Performance-Results)

I've started using a variant of this in my .NET Core projects and have found
the performance to be astonishing.

~~~
SQueeeeeL
Jonathan Blow warns against threaded queues in game development, as normally
simulating your world isn't the bottleneck (rendering is) and it will just
cause a fair bit of unexpected behavior/debugging

~~~
Thaxll
Indeed, lot of games are actually single threaded.

~~~
dkersten
In recent years there has been a trend shift away from this, at least in the
AAA engines, towards a job system. This makes sense: you have a thread per
core and you create jobs to “go wide” when you can. See for example Unity’s
Job system it’s the GDC talk by Naughty Dog from a few years ago.

The big games will also prepare data for rendering in parallel (eg culling and
sorting and whatnot, although much if this is also done on the GPU).

(Going by GDC talks, the rendering teardown articles and just what I see
online from Unity/Unreal. I don’t work in games myself)

~~~
Thaxll
It's true but lot of the main area don't multi-thread well like AI and
physics.

~~~
asdfasgasdgasdg
I don't know why AI shouldn't thread well, assuming there is more than one
actor. As long as they are operating over an immutable view of the game state,
each actor should be able to plan independently and enter its commands
independently. Likewise, there are probably some tricks you can do with
physics. And anyway in most games interactive physics is only done for a few
objects in the game world, and those objects are often not interacting with
each other, at least not physically. You could cluster the objects that can
affect each other and then do each of them single-threaded.

~~~
corysama
> As long as they are operating over an immutable view of the game state

That's a big issue. There is a surprising amount of back and forth between
objects in a single step of gameplay/AI.

And, generally gameplay code tends to be a big mess of wild and ever-changing
requirements from gameplay designers, extreme time crunch and short term (1
game then burn it) goals. Ivory tower software architecture it is not...

Clustering physics into "islands" is common practice though.

~~~
asdfasgasdgasdg
I'm not a game dev but I do know how software can become a mess. I think
engines that are used by many games have a chance to push good practices here.

------
siscia
One think that would make wonder is a company wide event queue.

Just imagine all the things that it would be possible to make automatic if all
the events that happens inside a business environment where in a queue.

And don't even think about how it would be great if the same queue were open
to outside providers.

~~~
Galanwe
That's called an ESB (enterprise software bus), a lot of companies have that,
and there's a lot of softwares already on that space (IBM has quite a good
market share there). Some companies do roll their own though, I've seen quite
a few implementing ESBs on top of 0mq for instance.

My previous company was doing trading and made the outgoing orders available
on the ESB. It's was quite handy since all teams could subscribe to the trade
topic and do their own processing: trade checks, accounting, risk, etc.

It's a bit old school though, I mainly see old corporations with that
architecture. The thing is, it ages quite badly most the time.

Fast forward 5 years, and every single team is publishing stuff on the ESB
"cause it's super easy to publish and consume data from the ESB". Before you
know it, you have a full team dedicated to scaling and maintaining the
monster.

~~~
golergka
Do you mean that these scaling problems hapenned because ESB was misused? Or
would it occur with any architecture or architectures (if different teams used
different instruments) just because of sheer volume of data that needs to be
processed by business logic?

May be I'm bad at phrasing this, but the point is, sometimes you have
perfomance problems because you could write better code, or use better tech,
or tweak requirements a little bit, or do something else a little smarted. But
sometimes you have perfomance problems simply because you just do a lot of
actual work.

~~~
Galanwe
I think these problems occurred because an ESB is too easy to over-use.

Once you have it, it's very tempting to use it for everything. But of course
the more you use it, the more features it needs to have.

Let me illustrate that with a example compiling some of the patterns I have
come to see at companies that overly used their ESBs:

Say Bob is working at a trading company and sets up an ESB where he publishes
the daily trades of the company.

Soon, the trading desk learns about that, and decide that it's actually cool
to use this for trading, they just have to listen on the ESB, and send the
trades accordingly. That's the error pattern 1: making the ESB business
critical.

The next day, the risk team learns about that ESB thing, and decide its very
handy to perform post-trading checks by just listening to the trades flowing
on the ESB. So they setup a system that listens for trades on the ESB, check
that the trade is compliant with some limits, and send an other message on the
ESB to let everyone know that this trade is validated. This is error pattern
2: cascading messages triggered from other messages.

The week after that, the security team learns about the ESB, and decides its
very insecure to let anyone see the trades of the company, so they start
implementing access control on the ESB. This is error pattern 3: now you have
an overly complex layer on top of the ESB to decide who can see what and who
can publish what.

Rince and repeat patterns 1, 2,and 3 for 5 years and here is the situation you
end up with:

\- The ESB is not the easy and handy system it was in the beginning. Since it
has become the de-facto standard for publishing information in the whole
company, it has to support features for _all_ the company use cases. There is
access control to publish per topic, access control to listen per topic,
multiple bindings of varying quality for each technological stack/language
that each team in the company is using. The company of course is not capable /
prepared to maintain a software of this scope, so the ESB is crippled with
bugs that nobody can fix, because, you know, the infrastructure team cannot
fix their groovy scripts using the ESB cause the guy that wrote them left. And
the marketing team has some interns using the excel plug-in but they don't
have time to rewrite them this year. The ESB is now partly un maintained,
because the company relied totally on it without having the capacity /
willingness / foresight of understanding how intricate it can be to update
something that everyone use.

\- The ESB is now very slow, because it was so tempting to publish anything of
various interest on it that everyone did it. The problem is that the ESB is
also critical for the company, so the whole flow of message is now slowly
moving and overflowing, requiring endless tuning and tentatives at scaling it
better. Of course 80% of the messages on the ESB are actually not listened to
by anyone, but since nobody really knows who listens to the published
messages, it's very tempting to just _not_ stop programs from publishing,
ever, because god knows if some random team at the other end of the company
might have a program reading these messages.

\- You most likely have now an IT team dedicated to maintain the ESB. They are
squeezed and pressured by the business teams to keep the ESB fast and easy to
stable without requiring them to recode all the crap they plugged on it. On
the other end, the other IT teams are pressuring them to update the ESB to
support <place your language/stack here>. Of course the ESB team has no
incentive to make any improvement whatsoever to the ESB, because that would
definitely crash most of the crap the less technical teams of the company
plugged to it. But the ESB team is the de facto guardian of the temple of the
ESB, so everyone ends up frustrated by the situation.

\---

I'm not sure I did a good job at explaining the various problems here, but
basically, the one size fits all that ESBs are promoting is often not a future
proof choice.

The reality is that you don't want your whole company coupled to a single
system like that. Otherwise your system will be as good as the worst user of
it.

~~~
mumblemumble
FWIW, I worked at a trading firm that had a very tidy and long-lived ESB that
I found to be a joy to work with.

-but-

My experience there 100% confirms what you say about it not being something
that scales well.

It was a relatively modestly sized firm by headcount, as trading firms go, and
there was a corporate culture of absolutely intense inter-team and inter-
departmental communication. One absolutely would not dream of subscribing to
an event stream without first talking to the team that maintained the program
that published it. Any changes to the event stream - both the grotty little
details about what was being published and how, and the grotty little details
about what was being consumed and how - would be preceded by a discussion
among all the people who were working with it, to make sure that everyone
involved continued to have a complete picture of the interactions involving
it.

That level of communication, which I do believe was essential to the bus's
long term success, just wouldn't scale to a large company. Nor could it have
been maintained at a company that had a more relaxed attitude in general. Nor
could it have been maintained at a company where programmers are allowed to
believe that most of their time at the office can be spent with hands on a
keyboard.

~~~
the_af
> _That level of communication, which I do believe was essential to the bus 's
> long term success, just wouldn't scale to a large company_

What _would_ though? An ESB could be a way to enforce a standard and slow down
gung-ho devs & teams using ad hoc solutions for every problem. But say an ESB
is not the solution -- what is? As the company grows, overhead and friction
become more important. I'm not sure "every piece connecting to every other
piece in whatever way" would help with scale, rather than compounding the
problem...

~~~
mumblemumble
IMO, there is no software solution, because it's not a software problem, it's
a social problem.

As far as specific things to try go, domain-driven design is my personal
favorite off-the-shelf mental framework for dealing with these sorts of
things. Especially the concept of _bounded_ contexts. Embrace Conway's Law;
recognize it's not a criticism, per se, it's also a scaling strategy.

~~~
the_af
Fair enough. I agree partially: it's a social problem. Thing is, _software
engineering_ deals with social problems too: those related to development.

I've never seen DDD successfully used in any company I've worked in, but
that's probably a shortcoming of my own experience. (Likewise, I've never seen
TDD or Agile or lots of things people often mention in their blogs
successfully used. Again, this is probably my own problem!).

addendum: to be fair I've never seen a completely working ESB either. Always a
plan to build or deploy one, never the finished thing ;)

~~~
mumblemumble
I doubt it's your own problem. The thing about methodologies like DDD and
Agile is that they're not just a development practice. They're also (in my
opinion much more importantly) frameworks for how the entire company interacts
with the dev team.

I see them fail more often than not, too, and one thing that's consistent
about every failed implementation I've witnessed first-hand is that non-
developers who are involved in or influence product development weren't
engaged with, bought into, or properly trained in the framework.

------
Const-me
Good article.

I’d like to add that similar queues are widely used in all multimedia, not
just games.

This library [https://github.com/Const-
me/Vrmac/tree/master/VrmacVideo](https://github.com/Const-
me/Vrmac/tree/master/VrmacVideo) uses queues to coordinate 3 threads, one
decoding video by pumping V4L2 kernel queues, another one decoding audio and
playing it with ALSA, and the main thread rendering frames with GLES. Two of
my queues are thin wrappers around mq_send / mq_receive Linux APIs.

~~~
phreack
This is why I love game programming. You think that's what you're learning and
then find out the same techniques are useful all over.

------
nogabebop23
not to be overly pendantic but events and messages mean pretty different
things and have different applications. Same thing with queues and buses.

I suspect the author knows more about this than me but plays a little fast and
loose in their explanations. I feel it's important to have disipline in our
vocabulary so that people learning the material can participate in more
advanced discussions as they grow.

That said, I really appreciate the overall content and using the context of
games for broadly applicable concepts is a great way to promote them (and
funny as generally commercial games have some of the loosest "structure" of
any code I've seen).

~~~
mindcrime
_not to be overly pendantic but events and messages mean pretty different
things and have different applications. Same thing with queues and buses._

I agree with what you say. But I'd add the suggestion that this depends on the
context to some extent. For example, in the world of hardware and electrical
circuits, a "bus" and a "queue" have almost no connection at all. But in the
world of enterprise software, an "enterprise service bus" is often thought of
as basically "a bunch of queues and a message router". Of course that's
imprecise as well, since it ignores orchestration... which just goes to show
even more how fuzzy some of these terms are in practice.

 _I feel it 's important to have disipline in our vocabulary so that people
learning the material can participate in more advanced discussions as they
grow._

I generally agree, but I think this may be a case where "that ship has
sailed." :-(

------
wokwokwok
An aggregate of the previous posts:

[https://news.ycombinator.com/item?id=23205511](https://news.ycombinator.com/item?id=23205511)

This gets posted very often.

...it has some interesting content, to be fair, but I think there not much
left to rehash again.

It hasn’t changed since 2014.

~~~
mindcrime
_It hasn’t changed since 2014._

I'm pretty sure there are a lot of HN users today who weren't here in 2014.

~~~
wokwokwok
ffs. Read the link before you post. Posted in 2014, 2014, 2014, 2014, 2013,
2017, 2018, 2019, 2020.

\-->
[https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...](https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=gameprogrammingpatterns.com&sort=byDate&type=story)

Some kind of link from this gets posted every 2-3 months or so, since it came
out. I appreciate you haven't seen it before... but.... this gets posted _a
lot_ here.

The previous (635 point) thread was literally 3 months ago
([https://news.ycombinator.com/item?id=23203699](https://news.ycombinator.com/item?id=23203699)).

~~~
prerok
Well, I've missed it. There are times I read HN on daily basis and times when
I skip for weeks or even months, so I appreciate the reposting. Especially
because it is an interesting read.

There are a lot of articles being posted and so even if there were a feature
"here is what you missed" I would not have used it :)

------
billfruit
on a related topic, I have often wondered if studying Discrete System/Event
Simulation help with game design and implementation?

Does DES have some deep magical tricks or it is just plain commensense type of
discipline?

------
skapadia
Overall this is a great description of queues and why they are used.

------
bionhoward
Anybody using the Actor Model for games these days?

