To reference SomeCallMeTim's comment - the way the save game system worked in Grim Fandango was to serialize the entire Lua environment! Stack, all code, etc! And too jamesu - when you patch the game you patch the code that was serialized into the save game! The story behind all of this in Grim is pretty interesting.
If anyone is interested in more details please come to the Game Developers Conference this year as I'm giving a talk on the technical details of how the original game and the remaster works. Oliver Franzke and I will cover both the Grim Fandango and Day of the Tentacle remasters.
What happens when the user wants to save? There's Pluto , which is somewhat slow for complex state. And it doesn't work on LuaJIT code -- and I tend to like to run games on LuaJIT.
And it's opaque: If you're debugging some kind of interaction in the game, you can't just look at the save state to see what's where.
I used to be a big fan of Lua, and this kind of AI and behavior coding was a component of that. But ... I think that, on balance, actually creating a data format that represents the state and "coding" using that plus state machines is actually a more robust and useful way to accomplish the same thing.
Then again you still have this problem even if you power your quest/dialog with data instead.
Without sufficient information it's practically impossible to make a smooth migration.
With a format like that, you have the option of writing your dialogues in code or plain text, and easily switching between the two. The only major downside is that you can't easily define arbitrary behavior that way; every unique action that a dialogue event could trigger would need to be transcribed in your 'save/load' methods, which may feel limiting if you have a lot of hard-to-generalize logic.
It wouldn't be all that slow and I think it would get over most of the overhead in some implementations I've seen of UI/Interaction code for games.
When I said slow, I meant slow to save the entire state of the Lua VM. You can't even do that in C, so it doesn't really apply to the problem I was pointing out.
Say you have an AI that's made 3 decisions and you're in the MIDDLE of a coroutine where it's waiting for the next criteria that will shift it to its next state. So the AI code looks like:
local player = lookForAPlayer()
if navigateToPlayerLocation() then
local quest = givePlayerQuest();
while (!quest:done()) do
waitWithStandardReply("Waiting for you to finish the quest!");
if (quest:resolved()) then
while (true) do
waitWithStandardReply("Thanks for finishing that quest for me!";
else -- quest abandonded
while (true) do
waitWithStandardReply("Thanks for nothing!");
Yes, it's awesome to be able to code AI logic that way. But how do you save the coroutine state? The actual execution state of the coroutine itself, including what line it will resume next, the values of any bound variables, the values of any local variables, are all part of the state of the AI in that case.
In C, you've got functions that may be relocated somewhere else in RAM and potentially stacks to save (if you're using a "real" coroutine library), so just doing a memcpy won't work. What would you memcpy to save the line that the script is paused on? In Lua, it can be done with Pluto, but then you can't use LuaJIT. I see libdill has the ability to use a user-supplied stack, but it doesn't appear to have a "resume" call that takes such a stack memory block, so the best that would happen is that the coroutine would start over from the beginning. Meaning that if you saved the above coroutine in the middle of a quest, the character involved would offer you a new quest when you saw them instead of accepting the result of a successfully allocated quest.
If your object states are all being handled in traditional objects, then you're not using the coroutines in the way described by the article.
But despite that I think coroutines have their place, and this was a nice example that I found when I was looking for inspiration.
(I've been working on a Lua-scripted mail client for the past couple of years, and I'm starting to think more seriously about async-behaviour.)
In particular, when your coroutines are explicit, you can say "queue up this task, this task, and this task, and continue here when they're ALL done." So when you're dealing with several tasks that might take different amounts of time to complete, they can do the work in parallel.
I went from being a major Lua fanboy and posting frequently to the mailing list to drifting away from involvement to abandoning the language, in a relatively short period of time. I even wrote a blog entry about it. 
This is one of the reasons I love lua's coroutines.
You give up:
1. Extra cognitive load every time you need to await.
2. You've got to be more careful when designing some generic higher-order functions (you need to decide which color their function arguments should be).
In return you get:
1. Explicit concurrency. No function that you call unexpectedly suspends behind your back. In some cases this explicitness could be actually important (e.g. when using per-thread/process shared memory).
2. Futures/promises/tasks are first-class values. You could pass them along, save them for later, await on all or any of them and so forth.
3. Theoretically speaking, promises can be more memory efficient than coroutines since you don't need to allocate a full stack for each of them - just the state you need.
"What Color Is Your Function" is misleading: there is no strictly superior solution for this tradeoff.
Futures/promises encode concurrency as a monad and like every monad they are viral and once you start propagating them around, they can be a pain.
But the alternative for not encoding some concern of your program as a monad is to let it pass through an implicit side-channel (I/O, error-handling, global state, built-in scheduler) that is harder to control and can't be treated as first-class object in the language.
Go itself chose to go the explicit with error-handling. It doesn't even uses Monads, so it gets the worst of both worlds in my opinion, but the designers' main concern with exceptions which they managed to avert is still valid: with (unchecked) exceptions you never know which function you call is going to raise an error on you somewhere down the line.
So yes, with Go you'll never have with unexpected throws, but you'll have to deal with unexpected goroutine suspension.
In Go all of your functions will be happily colorless with regards to concurrency but will be colored with regards to error-handling.
Again, both choices here are valid, but they are not so brain-dead simple or universal as this article is trying to put them.
Explicit promises have very nice, composable behaviors that I'm exploiting all the time, and that have no equivalent in the Lua coroutine world.
And once you have propagated them around, I think the ergonomics are just fine; I've taken some pretty hairy callback logic that no one could fully understand without careful study and turned it into async/await code that pretty much anyone could follow.
I think the implicitness of "this could suspend" does frequently cause serious problems; in Lua this shows up as the dreaded "Attempt to yield across a metamethod C call boundary"  bug. Talk about cognitive load! If you're calling C from Lua, and that C code then calls back into Lua, and the latter function yields, you get this error. So you have to be very careful about what is allowed to yield -- and since any function in Lua, at any point in the call stack, can yield, it's easy to accidentally shoot yourself in the foot.
Can you explain why this is unachievable other than by tagging your yield-capable call sites?
The other problem is that even with gevent, what you get is a first-class _coroutine_, not a first class class _continuation_. You have to actually spawn new coroutines (creating new stacks and going through sometimes unnecessary scheduler hoops) to get a first-class coroutine out of a possibly blocking function.
Again, I'm not saying you can't do something like gevent.joinall(gevent.spawn(foo()), gevent.spawn(bar()), but:
1. Many languages don't support this (including Go and Lua, AFAIK).
2. It comes with its own tradeoffs again.
I don't think this is necessarily a property of the primitives. greenlet doesn't support anything like gevent.joinall(gevent.spawn(foo()), gevent.spawn(bar()), but gevent does by adding python code to greenlet. So I'm sure that in the same manner one could do it in Lua using Lua's coroutines. (There might be some concern that we have the wrong primitive: greenlet provides symmetric coroutines and Lua provides asymmetric coroutines. I don't think it matters for this which one we use, but if it does it is simple to implement either in terms of the other.)
That's not to say that the situation is ideal! To a great extent it's good if a language comes with a standard way to do this stuff! But it would be silly to say something like "Lua doesn't have inheritance" because it has a DIY inheritance construction kit instead of an "extends" keyword.
Manually coding up a "state machine" is kind of reinventing the Lua interpreter, no?
Computers are good at generating code. If you want a very fast FSM, they will build it for you from a compact representation (cf. a regex).
But for game dialogs speed is probably a secondary concern. I suspect that a table-driven FSM interpreter with a fully transparent internal state that can be saved / loaded is not very hard to create. Your dialogs' input language becomes the control table, which is possibly no less designer-friendly than Lua coroutines.
An example I see a lot where it needs to happen fast is spatial queries or AI-related dialog. For instance if I walk by someone or something, that has to trigger a query on the dialog engine for someone to say something to the player. It's not just the dialog, but the sound assets, playback, etc. that need to happen and they need to happen in a way that isn't annoying - i.e. player is far away already before dialog starts or has to "wait" in place.
Sometimes as I mentioned, AI also gets thrown into the mix in different ways. The simplest is that you want some piece of dialog to be triggered by some other game states - ex: if player killed dragon, say this, otherwise say that. It seems simple, but when you start throwing in lots of world states like in open world games and want some decent variation and natural dialog that isn't awful and repetitive, it gets harder. Now add on spatial and other requirements for a dialog query and things start to suck. To this end, it's why a lot of dialog systems suck. It's not just laziness, rather sometimes people actually can't get their dialog systems to work fast enough during update cycles and end up dropping features because of the cost vs. benefit for most games.
Dialog seems simple, but is surprisingly complicated in the greater ecosystem of other game systems. But again, this depends on the game and if you just have a side-scroller where you repeat the same 5 catch phrases, then yeah, dialog is simple and speed is not a problem.
That said, I'm in favour of other approaches to this problem - using (i.e. generating) Lua bytecode itself as a way of maintaining state, and then pushing things around while having a good understanding of llimits.h seems to be quite important ..
I have written an entire (popular) game engine that tightly integrated Lua and used custom-generated C++ bindings. I had to dig pretty deep into the Lua VM to accomplish that.
Heck, I found and fixed a bug in the LuaJIT 1.x series -- my fix integrated by Mike Pall, with a credit on the changes page.
The thing is if you encode game state into the program state, like using coroutines for complex, long-running AI tasks, for instance, then there's no way to save or restore that state if you're using LuaJIT. And Pluto saves the state in an opaque manner.
And as another commenter pointed out, if you want to update the game with new behaviors, god knows how that will interact with save states that have huge call stacks in them. And bug fixes? Yeah, since you're saving the whole program state at this point, bug fixes won't be incorporated in the restored image.
Most of the games and engines I've ever worked on that have dialog systems expressly avoid encoding things into the program state as much as reasonably possible. This is not only for restoring state, but also making for a saner experience editing and changing things during development. Very rarely does the format you encode things like dialog stay the same during a dev cycle unless you're using a canned engine and don't make any serious changes.
Using Lua program state itself seems like a way to drive yourself mad by the end of the dev cycle, and make it incredible unfriendly to work with writers and other content creators. People who work with asset pipelines often do a lot of cruel and unusual data conversions and munging to bridge how different people and systems need to work, but eventually things in the actual game engine are best kept as simple data that can be read in and reproduced at any time. Moreover, as a developer, I should be able to load up any state of my game at any point in the game just by providing the necessary input data as possible, and like a pure function, it should produce the same game state every time if possible. Let's also not forget other things like internationalization and localization.
As far as interfacing with Lua, I've mostly done it in the last few decades in a simple way where we handed entity component-style data that is just pure data, i.e. no function pointers and crazy stuff, and either let Lua do the rest with that state, or send pure data back to the C++ when needed. As for the data format itself, I've seen everything from XML to custom binary formats to CSV and Excel used for dialog. I think I'd sooner murder someone if game dialog lived in code, even in a Lua script.
A few reasons why in C++, C, and Lua, and in general in most game programming languages code is not the same as data:
- Tends not to be editor friendly
- Requires heavy meta programming to achieve the same as other formats
- Requires compilation and/or interpretation
- Is not necessarily idempotent depending on compiler options and platform. The same data should be the same cross-platform, and code is almost definitely not because at a minimum level, it can generate vastly different assembly depending on compiler flags or processor targets.
- Purely composite. I would argue that as much as we try, it's hard to compose a lot of code properly and without considerable effort in games. Conversely, composing data can be quite easy.
- Cannot on its own be serialized/deserialized easy while in a runtime state
- Not network transparent
- Cannot be compressed/decompressed in real-time as easily
See my points in other posts in this thread. I am assuming you downvoted me, which is ridiculous and childish, especially given your brief response without any reasoning or evidence despite being contrary to all computer science research.
Composition: see meta-tables. Repeat until completion.
It's telling that all the major game engines and a large portion of papers and talks disagree with you. If Lua was the solution to data-driven development, people wouldn't bend over backwards looking for solutions like entity component systems. Lua is nice, but not suitable for most professional game development as a primary language. As such, it makes it completely unsuitable because Lua data, especially things that depend on runtime state are utterly useless.
Your shipping comment also makes no sense in this context. Tons of other languages have more games shipped than Lua. There aren't too many people relying on Lua for composition and data-driven development in games, because it doesn't work. Even Lisp doesn't work in this context putting speed and other things aside, because of so many reasons I've already listed. Code in the context of game development is not data by the definition of data I am referencing.
For a similar problem of making the AI code for a bullet hell shmup, I thought it was very important to be able to edit the AI code easily to try out a lot of stuff. I've had decent experiences using BulletML and Lua for this, but both of them provide no path forward if I want to update the game and load an old save. Like, because I was not asked to explicitly name every single state in my "AI program", I cannot create a mapping between the possible old states and the possible new states to use when running the updated software against an old save. Then again, the fact that I was not asked to explicitly name every single state in my program feels like a pretty central component of the short iteration time I was after to begin with.
For shmup AI this maybe ends up being acceptable because the game doesn't need to run new behaviors against old save files. But if I did need that, maybe I should be using two systems, one for experimentation and another one for making something upgradable?
"Proper use of byte code generators" -- at that point you're not writing Lua any more, any more than Scala or Clojure targeting the JVM is "writing Java." The article is about using Lua for AI, which is what my comments are about, and my comments all stand.
Arguing about whether the VM is good enough to support a state machine is orthogonal to whether it's a good idea to use Lua coroutines as suggested by the article.
I'm not saying it's the best solution but it's pretty simple and probably easy to understand. I've used a similar approach way back in a networked multiplayer game where players could telnet into the game server and use text based menus to control the game. You can see the relevant Lua code here: https://github.com/dividuum/infon/blob/master/server.lua. I don't want to imagine how this code would look like without the use of coroutines.
There are tons of ways of doing dialog system. Mostly it comes down to your specific game and I highly recommend against generic dialog systems. Rather, the most you can do at a generic level is try to come up with a good way to store dialog and retrieve it in the ways you need. This isn't the best thing I've seen or worked with, but here's an easy to understand and documented method and presentation about it form Valve:
Anyway, I've been programming games and game engines for several decades, and coroutines are always in the list of things that you can throw in the junk pile along with other cool things like continuations, various theoretical data structures, fancy dispatch mechanisms, events/signals, application-wide functional purity, etc. Of course all of these things are useful, but they tend to have problems that make them of limited use in professional game development.
Here's a brief checklist of show-stoppers off the top of my head I tend to go through before introducing anything "creative" into a game, and especially a game engine. Most of the aforementioned items fail one or more of these.
- Performs awful in common cases or when applied to actual game-like conditions vs. theoretical
- Murders cache lines
- Hard or impossible to debug
- Hard to save, load, serialize, deserialize, or snapshot
- Doesn't work over networks
- Doesn't play nice with libraries or data coming from other libraries or languages (ex: 3rd party physics lib)
- Unpredictable execution start or end times
- Non-determinant memory usage or allocation patterns
- Risk of stack-overflows
- Non-portable or problems on specific target architectures
- Requires "religion" meaning it infects all your code or forces you to write all of your code a certain way, everywhere
- Fragments memory
- Long blocking or uninterruptable processing
Conversely, besides the obvious opposites of most of the above, I look for the following when introducing some major data structure or conceptual item like coroutines to a game or game engine:
- Leads to predictable, consistent code and resource usage
- Editor friendly (as a corollary to the above)
- Clear debugging story
- Both simple and easy if possible. No, they are not the same.
- Plays well with the world around it
- Loved by the CPU, GPU, and/or compiler, and produces reasonable assembly on target hardware
- Easy for someone else to jump in and understand the flow and how it works, assuming they are not a moron.
Pretty much these two lists mean that games tend to be made of meat and potatoes things. Simple data structures like arrays, custom allocators, predictable branching and execution patterns, up-front memory allocation, and CPU and GPU loving goodness. Pretty much anything stashing things away and building up massive states or jumping around all over the place, pointer chasing, and so-on should always be a no-go.
Anyway, if you're building a truly simple game, as in a 2 day sort of affair, do whatever you want. If you want to build something even remotely complex that you will work on for awhile, it's best to do things the right way which means detach yourself from everything you think you know about most of computer science and think in terms of pipelines, CPU instructions, caches, and predictability among other things. This often means programming against the grain of your language a bit, throwing out sugar, std libraries, and all kinds of things. It really sucks, but that's the essence of good game programming. I hope one day it will be different, but as long as there's still need to push gameplay and graphical boundaries, professional developers almost always need to squeeze out what they can.
This is one of the worst outlooks on programming I've ever seen. Did you know that any control flow operator such as try/catch is really an application of continuations?
As I said, this is about programming games, primarily in C/C++ or a related language (though applies mostly to others), not programming in general. If you enter into other high-performance domains, you will encounter the same issues. Conversely, other domains such as general application programming do not adhere to this.
Regarding try/catch, I am well aware of low-level implementations in various languages. Did you know that in most game engines, you don't use that much try/catch for this exact reason? If you're littering your game code with try/catch everywhere, you are doing something wrong. Not to say you don't have any, but there are other error handling and return methods or you just let things bubble-up and handle them there. What would you actually do in most try/catch situations deep inside a game loop outside of things where resources can leak horribly like IO?
When you're working on a game, there are of course areas you can compromise. It also as I mentioned depends on the nature of your game. I'm speaking strictly of properly designed, professional games that have to be competitive in terms of features and run at a smooth frame rate while doing many things (i.e. your average AAA console game).
In case you still doubt what I am saying, go look at GDC talks, game dev papers, etc. and you'll see the same thing. As far as someone people around here might be more familiar with as a source, I seem to recall there were multiple times Jonathan Blow mentioning exactly what I did (maybe a talk he gave at Berkley?). Anyway, the point is that as programmers we learn all these cool and fascinating things, but in game dev, you tend to have to let go of "cool" things and spin your thinking a bit towards things like predictability, performance, stability, transparency, and composeability.
- murders cache lines is clearly not important for a dialog system.
the use of lua fixes a lot of things you mention in terms of stability, ease of use by other people, etc.
data driven is the most interesting one.
using lua to drive game logic allows you to be less data driven. especially if the lua code is like in this article -- use of very high level lua functions.
the code now looks more like DSL which is more powerful/flexible then using data driven approaches.
Lua is indeed used to fix many things in game engines, but not really most of what I listed. The top reasons scripting languages tend to be used on top of other languages like C/C++ in games is to avoid lengthy recompilations, provide an entry point for user-editing/plug-ins, serve to some degree as a more RAD tool, and build a DSL for scripting events.
You are correct data-driven is interesting, but your conclusion is 100% wrong. The entire entity component system approach and the general trend towards functional paradigms couldn't be more at odds with you. I can't think of what might be the best resources off the top of my head for you to read, but there's plenty out there on ECS, functional-style C/C++ as related to games, and low-level engine design stuff on older blogs like the one BitSquid wrote years ago.
The entire industry is moving to trying to make games as data-driven as possible. This is as you point out the anti-thesis of what you are saying. Further, your point is really bad advice for any game programmer. Making things less data-driven in general makes it infinitely harder to debug, cache, build editors around, send over a network, save/load, update/migrate, share between threads, and be consistent.
What you are proposing would make you fail an interview with any decent game developer or get you fired. You would drive a tool designer insane as well. Huge chunks of games are built by creating tools with code to manipulate data, not code. That data often lives external to the code entirely, whether it is an excel sheet, custom binary format, custom database, or something else. You want as little as possible in the code and nothing that will affect compilation time. Moreover, you need to be able to also do things like translations and internationalization, not to mention add new assets, hot-swap things during dev, and so on.
In other words, no, building a DSL in Lua is not more powerful as you describe. It could be powerful if it was data-driven, anything else simply would not get used in a professional setting with decent developers. I don't understand why sometimes on HN people want to graft general programming or web dev on to games. That's the only way I can explain this kind of thinking, and even data-driven approaches are becoming more prevalent in these areas.
regarding data-driven vs lua.
my point of view is data == code. except that the guarantees you can provide when using code are less but you get more powerful expressions and flexibility in return. ( ==> turing completeness ) the more you are willing to give up turing completeness you get to the "data driven" side. The stricter and less powerful your data is, the better you are able to guarantee performance and other requirements. however this comes are the cost of loosing flexibility and agility.
i agree. what i am proposing works in a small team (<10) where everybody does everything. this doesn't scale to large teams. (or rather i haven't seen this in working in large teams)
The term component here would be incorrect, rather "system" would be a better term.
There are many variations and approaches to what I am talking about so I'll stick to one very common approach used by a lot of studios and engines today. A typical more cache-friendly update cycle goes like this, calling "update" as a function of time/state on each system:
System A -> System B -> System C -> System D -> etc. (loop and repeat each frame)
This cycle sometimes skips certain systems for either performance or sim reasons. Each system tends to have inner loops and the overall system loop is often broken up a bit into sub-loops or separate loops, ex: render vs. physics for purposes of separate sim-time looping. The idea here is you do a single operation over and over again on the same type of data, avoiding lots of pointer chasing, v-tables, and so on along the way. The code branches very predictably and it becomes easier to avoid loading data into the cache that will blow it up or not fit into a page.
What makes this friendly to a data-driven approach is if you're working mostly with systems that operate on pure data, your update each frame from a purist point of view approaches a functional reduction. In practice, there are certain things that don't fit perfectly into this paradigm, but the idea most engines take is this approach + cut corners when necessary.
I am contending that your point data == code is wholly false. Again, this is well-known in computer science to the point where your argument only stands if you're looking at code from a very low-level point of view. Again, all current game development trends refute what you are saying - almost every year there's one or more GDC speaker sessions about this topic, see the GDC vaults, see Unreal Engine, Unity, CryEngine, etc.
Yes, to a degree code can be data, but it becomes exponentially difficult to manage and process vs. simple representations that tend to be language, runtime, and network agnostic - i.e. data structures. Lua tables and other things inside Lua are not the same (at least not until serialized to a more language agnostic format), especially in the face that in most real game dev situations, you're passing data between tools and languages, and in several different flavors of run and design time. I've done quite a bit of code as data approaches to know that even when it can be done, it has many drawbacks - see object databases, Smalltalk continuation serialization, code gen tools, etc.
Functional languages and several more fringe languages tend to be closer to code == data, the most wildly known and regarding being Lisp-like languages. Typically these languages have a minimal amount of abstractions and power things like first-class macro systems. Even in the face of macros, there are many drawbacks to dealing with code this way in general, and even more in a game. For instance, restoring state exactly as it was and is, and using that state in the face of updates to the game itself are paramount to the point where few games can survive without it otherwise. Data makes this easy.
What you are asserting as more power is exactly the opposite and I'm not sure how I can make this any clearer. Your code can freely operate on data any way it wishes, not the other way around. Code operating on code is generally a bad idea which is why clever meta-programming techniques and code generation are things you would rarely see in any game engine unless it was a hack or a bad implementation.
You are losing little by making things a data representation and gaining much in terms of what I already listed. All the benefits of data are far more valuable for game development and I am not sure what cannot be achieved by forward-loading or pipelining data into an asset pipeline and operating on that data doesn't achieve that this does. Beyond performance, the other benefits like editors, network transparency, easy loading and saving of state, and so on make it easier and faster to develop even small games.
What you are proposing is a solution looking for a problem. I'm not denying it's a fun thing to do, especially if we're talking about a regular "app" rather than a game. Personally, I like coroutines, fibers, CSP, continuations, multiple dispatch, and other fun CS tools but not for game programming in most of the time, and certainly not at the level of an important game system. I am asserting that in the game dev world, treating code as data is a really bad idea, which several other people have also pointed out here. Your solution adds very little, while adding unnecessary complexity to check a "cool" computer science box.
That said, there was a project called lua-colony that attempted the conversion.