Hacker News new | more | comments | ask | show | jobs | submit login
Making an RPG in Clojure (2010) (briancarper.net)
191 points by arm on May 30, 2016 | hide | past | web | favorite | 39 comments

Reading this article is like reading something I might have written myself. I, too, just like the author, have been toying around the idea of Clojure games for a long time. I've made small games (Ludum Dare entries mostly) and started (and subsequently killed) a lot of Clojure game engine projects.

I tried the threading approach using agents (which later evolved into agents inside atoms or was it atoms inside agents? It's been a long time) and came to the same conclusion as the author. I tried to move to a more centralized CES approach, which didn't work out.

It's interesting to see how some people can come down to the same path and achieve the same results (for the good or for the bad). I think my experience with Clojure has been overall positive as far as game development goes, although definitely not empty of woes.

Currently what I'm (very on and off) working on is a Visual Novel/Interactive storytelling engine on ClojureScript, at http://novelette.moe and Clojure is great for it. Feel free to trawl through my github profile (link in my profile) if you're curious about some of my (failed? abandoned?) Clojure gamedev projects (among other things).

Nice work with - http://morg.systems/ld31/ Not a lot of graphics & interaction, but the story telling was good.

I finished the game :-)

Nice, for some reason I have a soft spot for interactive fiction engines! How does it compare to Twine?

I had never heard of Twine before, I just had a quick superficial look at it and wow it looks really nice!

To be honest I don't know how it compares, my engine is closer to something like Ren'Py but I plan to provide more freedom to the writers thanks to the power of functional programming and the DSL-like implementation of Lisp. I haven't formalized on a proper syntax for the scripts yet (although a rough example can be seen on my demo game: https://github.com/Morgawr/ld31/blob/master/src/cljs/game/sc... ), but the idea is to have something like the script of an actual novel. The author can define acts and chapters and have actors enter the scene, interact, define the state of the scene (add/remove sprites, backgrounds, music) and define the transitions from chapter to chapter within the novel.

I've been working on a project to create 2D sprite based game engine in ClojureScript. My goal was to represent the game completely with data and let the engine figure out how to interpret and render it (declaratively). In the end I implemented an entity component system (as other comments have suggested) and used existing javascript rendering libraries to handle rendering (not writing a wrapper for webgl...). You can check it out here: https://github.com/alexkehayias/chocolatier

It's been a ton of fun bringing a functional approach to games using Clojure and I gave a talk about it at ClojureWest this year. https://www.youtube.com/watch?v=TW1ie0pIO_E hope this helps others looking to build games!

For those that are interested in what the current state of Lisp game development is, check out the latest installment of the Lisp Game Jam. The jam page includes lots of information about game development tools for many different Lisps, and the submissions include games written in Common Lisp, Scheme, Clojure, and Emacs Lisp.


> due to multiple threads trying to edit the world ref 50+ times per second.

Of course you get no benefit if you serialize the game through a single ref. Just because you have a fancy feature like agents doesn't mean you don't have to think about how your concurrent code is behaving.

I chose Clojure for the server side code of my MUD. It wasn't for the agents or STM, it was for immutability. Immutability gives me the ability to easily have a consistent snapshot of the world which I can use to perform some operation, such as saving, event propagation, or some AI calculations.

If you don't serialize the game through a single ref (or the equivalent) how do you get the consistent snapshot of the world to perform an operation on?

Event loops are heavily used — basically process input, process physics, process NPCs, render, repeat. (Likely not in that order — I'm not much of a game developer.) This is what the author lands on in the "Success" section.

Speaking of event loops, I feel like they weren't discussed very much at all at my school, despite being at the core of not just most game engines, but most GUIs. I think if I'd embarked on writing a GUI system when I was fresh out of school, I'd probably have done it in a massively threaded way like the developer in the article started their game. An event loop would have somehow seemed too … primitive.

Architecture can heavily affect how serial your process is. By randomly throwing agents at a single ref you can easily turn parts of the process that can be done in parallel into a serial process.

Beyond that, throwing massive concurrency at an inherently serial process tends to give you worse performance vs just recognizing that it's a serial process and not trying to throw concurrency at the problem. This isn't just about fancy concurrency primitives like agents or STM, I've seen plenty of hobby games that ran terribly because they threw concurrency at the problem without analyzing it first, but ended up completely serial because of critical sections in the form of Java synchronization blocks.

As an example of concurrency that you would probably have in every single player game: game logic feeds into render logic, but render logic shouldn't be changing the state of the world. So you can be performing the game logic of game loop #N+1 while the render logic of game loop #N is running. Also, usually, when saving the state of the game all you need is a consistent snapshot at the time. Immutability makes both of these easy.

The rest depends upon the features and design of your game though. If you aren't going to actually try to analyze where you can benefit from concurrency, you're better off not just randomly throwing concurrency at the problem.

Which MUD?

This is great. I've also tried my hand at making video games in Clojure.

For me though, it was a weak attempt at a simple Bomberman clone that would be playable over the network, so that my kids could play it across our different family computers. It was even kid friendly, being about planting timed water balloons, and avoiding getting wet!

It seemed like Clojure would be ideal for this, because the JVM has excellent networking libs and GUI libs for simple 2d graphics.

But I severely underestimated how much time it would take though, so it never really even got to a playable single player state.

The main part I got stuck on was trying to settle on a data format, how to represent tiles and players and items and everything, in a form of immutable data that was easy to both analyze and transform.

The author's idea of mapping a giant 2d string to actual data is genius, although I suspect it wouldn't scale well for an RPG, and for my game it wouldn't have been very helpful since the terrain would be auto-generated (since needing to be somewhat randomized).

It's surprising to see the author here mention using Agents as the original primary building block of the game engine. I've never directly used Agents or found them helpful. To this day I really don't know what they're good for.

A few years ago I created a fully small RPG game in clojure: https://github.com/damn/cdq

Including procedurally generated maps, items and 3 small levels.

I love that the author takes us on a tour of all his blunders, so we can learn from them the way he did.

Far too many "making of" articles present you with a finely polished end product and pretend that its evolution was child's play. That leaves us readers disheartened and reluctant to try similar things.

The article also shows up that there is snake oil in the actors, stm and many other concepts to make concurrency easy and fun. Often you fall back on doing it all in 1 thread because you get better performance that way but then what is the poll of multicore?

Lmax disruptor is different from the others because it is based on engineering (what works on actual hardware) opposed to 'science' (how you imagine it ought to work)

No, it's just that things like actors and STM, and yes, even LMAX are completely overkill for many of the projects out there. Go read up on what LMAX was designed to solve, millions of messages a second. I've worked on several very large projects for huge corporations, where their message load was 1 thousand messages a second peak.

So yea, I don't use STM much, and I never use actors, I use queues quite a bit, but I benchmark them first and use the one that fits best with existing infrastructure and causes the least amount of friction with my client while still getting the job done. It has nothing to do with "actually working on hardware" or "science". Except perhaps that I apply the scientific method in choosing my tech, and that never ends up being LMAX for the jobs I work on.

Actually lmax disruptor is great for all sorts of things, because 10 million messages per second allows fairly fine grained parallelism for data processing and also interactive apps like that app where you have back pressure support that keeps the app from being unresponsive. Real time work means not making commitments you can't keep so if it is clear the system cannot handle a request, drop the message early.

I mainly use the reactor framework, and I have a choice of lmax and a bunch of other queues and also the choice to schedule on the same thread with low overhead.

> lmax disruptor is great for all sorts of things

No, abstractblockingqueue and other general purpose queue implementations are great for all sorts of things. Disruptor has a pretty specific use case - high rate of constant data flow through a system, where it's okay to spin loop threads to consume data. It also requires pinning threads to CPUs to get the benefit gain of cache locality on the ring buffers. It's pretty darned specialised for general use.

Well, based on the original videos Rich did to promote Clojure, I think I remember him claiming that STM was never meant for cases where you wanted parallelism for performance reasons.

STM was intended to make it less error prone to use threads in places where concurrency was the right fit for the problem.

When is parallelism ever needed but for performance reasons? (Honest question, I'm quite a novice when it comes to concurrency)

I maybe didn't word my comment very well. What I meant is that STM is meant to be used in cases where the problem you're solving is most easily expressed in a concurrent way. It's main use case is concurrency, as opposed to parallelism.

For example, a web server is naturally concurrent. You definitely can implement a web server in a single-threaded manner, but you'll end up just faking concurrency anyway. So there are times where you'll want to use concurrency because the problem you're solving is naturally concurrent. In these cases, having an STM is valuable, because it makes it hard to get state changes wrong in a concurrent environment. It is not necessarily a way to make your program faster.

Yeah I remember the benchmarks, didn't look too good. People claimed that with hardware support for transactional memory things could be different. We have that now in the latest Intel CPUs (Skylake for sure), anyone got further information?

I remember seeing games doing actor-like things with message passing on a single thread way back in '02. You could "broadcast" a "damage" message and all the actors that were in the hit region would handle it in however they saw fit. Great way to force decoupling.

Concurrency is somewhat orthogonal(although a nice-to-have), it's more about not doing things that run counter to your hardware.

Over in the games space we don't have the advantage of scaling wide on multiple machines so a couple things that sound heretical are there for good reason.

>You could "broadcast" a "damage" message and all the actors that were in the hit region would handle it in however they saw fit. Great way to force decoupling.

This is great for simple games but it gets hairy really quickly for larger games and it can easily become a mess. There are a few event/callback driven game engines out there and a lot of the most popular AAA game engines provide some kind of event subscription model (I think UE does it, at least, but I never really used it much).

The biggest problem is with figuring out who is in charge of what. Let's say I broadcast the event "player attacks enemy", then the enemy has to receive that event and respond with "enemy plays hurt animation" then "enemy plays dead animation" "enemy broadcast death" and then "player plays acquire XP animation" "player broadcasts level up" etc etc. The order can easily become a mess. And then "player attacks enemy" and "enemy evades attack" and "player plays missed animation" etc etc...

It really becomes very messy for big games, especially if you want to build an game engine entirely based on the broadcasting of events. I think Angel-Engine, now called Angel2D iirc, was mostly based on that and it worked fine for small scale stuff but easily become a mess.

It certainly is easy to understand the logic behind the game that way, it's great for small indie games or game competitions/prototypes, though.

Yeah, I'm trying to remember where I saw it(definitely wasn't UE3, that engine is the definition of tight coupling). It may have been for inter-component communication inside a single entity for a cleaner ECS.

I'm curious, as I'm not in the games industry: what type of model does one typically see? I actually expected it to be something along those lines.

I suspect for bigger, AAA studios, they'd have a Entity Component System.

Would they?

I did read a bit on that, and tried to implement a simple game on it. What I've learned is:

a) it seems to be vapourware; there are some blog posts about it and examples floating around, but that's pretty much it

b) it seems to make sense only for MMO games, which are glorified interfaces to database tables anyway - hence the ECS model, which is about treating your game design as a database problem - fits it well

c) experienced people I talked about think this is a stupid approach, for reasons I'm probably too dumb to understand (except the efficiency ones)

d) in games that are not database-like MMOs, it's getting very hard to figure out what should be a component or a system, how many systems should you have, etc. etd.

e) it seems to seriously complicate implementing game logic - you end up spreading code that belongs together across many different "systems"

Most modern engines, as far as I know, implement a component-driven approach. You can see it in pretty much any kind of public engine like UE or Unity and it works really well. Composition over inheritance any day. I don't really see why you claim that it seems to be vaporware, considering it's one of the most implemented types of game engines currently.

It is true that there are some degrees of limitations and implementations that different game engines go through and none of them is a 100% pure CES engine (something like http://www.chris-granger.com/2012/12/11/anatomy-of-a-knockou... ), where all systems are separated from the components and the entities containing them. Usually you'd have sets of components and systems and whatnot and different hierarchies, but the gist of it is still a Component-driven system.

> It is true that there are some degrees of limitations and implementations that different game engines go through and none of them is a 100% pure CES engine

One thing I vividly remember from learning about ECS is that the main sources on it repeatedly remind you that what UE or Unity do is not ECS, and that the way they understand the concept of "Entity" and "Component" is different. Entity Component System is not what UE or Unity do.

I'm developing NullAwesome[0] with an entity component approach, and so far it seems to be going rather well.

Using an ECS for game design actually has enormous advantages when you are coding a high-performance game in a language like C or C++: you can lay out the components of the entities however you want, so that systems on the hot path have access to as many components as they'll need without blowing the data cache. I'm not actually doing that for this game, but still, it has advantages outside of just MMOs.

[0] https://nullawesome.tumblr.com, https://github.com/bitwize/nullawesome

It sounds like the way that Redux works would be great for this kind of logic.

>> poll of multicore

Intended pun involving threads?

I'd guess the keyboard (from a mobile device?) got "poll" instead of "pool". Or maybe a little typo from the use of writing one and not the other.

On the subject of pools of threads, a few days ago there was a post about the Naughty Dog Engine using fibers (inside threads) for cooperative multiprocessing. They made it so it'd have a cleaner API and be easier to use, but at the end of the talk they said it had also improved performance (because of less context switches).

Also having very little knowledge in games programming (a few experiments in QBasic 4.5 something like 20 years ago, when I was still a child :D ), the most I can get out of this is: multithreading/multiprocessing in games is hard.

Well, I wouldn't call these techniques snake oil, but, just like anything, you can't just throw them at a problem and expect that they will just work. You have to take all of the usual concurrency issues into account (eg contention).

STM might make things simpler, but if many threads are contending for the same data, then they will need to be serialised and while STM might make this serialisation easier for the programmer, it won't automatically make it perform better[1].

For parallel threads to be faster than a single thread, you need the threads to be as independent from each other as possible. You can use actors, STM and such to do this, but its not a silver bullet that just works. You need to apply the correct technique given your situation and requirements.

I think where Clojure largely failed (and I say this as a big-time Clojure enthusiast) is in its promise of making concurrency easy through these magical primitives. Hell, since its release, Clojure has gained more concurrency primitives (futures, core.async, transducers) because the original ones (threads, STM, agents, atoms, vars) weren't enough.

As for a game engine, I would make sure that each entity wrote to its own independent data, so that writes never block each other. That is, they can all read global world state and they can read the previous state of the other entities, but each only ever writes its own data to its own atom (or whatever; I'd have to think about it more to figure out the most appropriate place to store the state). Then I can update all of the atoms in parallel. This is just off the top of my head, so it may need a little refinement. Agents provide a nice async update, but if you have many in flight, they still need to serialise and take turns, losing much or all of the parallelism.

The point is, I don't believe we'll have a magic "throw cores at it" concurrency technique any time soon, we will continue having to think about it and structure our code to keep the cores as busy as possible and synchronisation as low as possible.

LMAX disruptor is indeed a fantastic piece of engineering and it works so well because it eliminates the need for synchronisation (because the updates always happen in a single thread). The awesome "concurrency" from disruptor comes from the design of their ring buffer, which allows many threads to quickly write to it, and the single reader to quickly read, all without synchronisation. Basically, the processing thread can be kept busy without ever having to wait on writers. Fantastic!

My favourite means of parallelism, I think, is still a work-stealing task-based system like what Intel Threading Building Blocks provides, because its relatively easy to think about and deal with, quite flexible (allows firing off tasks for async things, "going wide" for nicely data-parallel things, allows pipelined processing for sequential steps over a collection of data).

But... there's no concurrency/parallelism silver bullet.

[1] STM can of course make sure the locks are as fine-grained as possible, but at the end of the day, if two things write to the same place, then they need to happen serially.

Article is from 2010. Was there ever a follow up?

Unfortunately not, it seems like many factors in his life led to him burning out. He does have a new blog¹ now though.


¹ — http://carper.ca/

That seems to be his last post on gaming/games: http://briancarper.net/tag/66.html

This reminds me of Realm of Racket.

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact