> Start with writing new code in Lua (faster to prototype) and then see if you need REALLY to move this code (or parts of it) to C++.
This is the only part of the article I disagree with. I've found that the biggest problem with using Lua in a C++ game is writing code in the "wrong" language, and the mess that can cause. The boundary between Lua and C++ needs solid validation to keep bugs out, and this makes it neither performant nor convenient, so minimising the boundary is important. As soon as code ends up in the wrong language, you generally need to add a large amount of unnecessary boundary code.
An important part of designing a Lua/C++ game for me is having a good mental model of what the "right" language to write any piece of code in is, and only deviating from that when performance mandates.
I think I should clarify this in the article further. What I meant to say is: "after you've implemented the basics, e.g. input handling, basic renderer, file I/O etc. and you've got a basic game running, then you can start with prototyping code in Lua".
I feel like I've fallen into a trap of writing the code in C++ first ("because it's fast!") and then porting it to Lua when I realized that I need to write too much "binding" code.
For example, if you're writing a very simple AI, you can write a FSM/Behaviour Tree implementation entirely in Lua and then write AI logic directly in Lua. This way, you won't need to write binding code between Lua/C++ for these FSM/BT implementations.
And then, after you've written some code with these structures, you can see if you need to port it to C++ for performance or not. And if you do, it will be easier to design API since you already know which parts you need to expose.
One thing that makes prototyping in Lua fast is that the code, which looks simple in Lua, can easily become a pretty difficult template meta-programming code in C++.
Lua is a joy to work with, one of my favorite (scripting) languages.
Having tables as the primary underlying data structure gives it a very LISP-y feel despite having an ALGOL syntax, providing a nice middle ground between the two styles.
Lua really is a joy, and after 40 years of software development is still one of the top ways, in my personal list, to get serious software built.
I've done quite a few projects where I built a library of C modules to do the heavy lifting for some particular purpose (data in/out mostly), glommed these functions into the LuaVM, then built the application logic, very rapidly and smoothly, in Lua - outcompeting others working in my group with their big fancy stacks of stuff, time and time again.
I think its really important for any developer, whether they are encased in the glories of the browser or otherwise, to understand how to add the Lua VM - to anything - and then use that as a means of productive application development.
Its very rewarding once you get the VM glomming happening and can start accessing everything you need from within your own scriptable execution environment. More programmers should learn this technique, imho - it moves us from being effected clients of someone elses' ball of string and wax, to actually causative developers building architecture that really, really works.
Whether you like Lua or not, building-in a VM for your higher-level application logic is just good architecture.
I discovered this technique only a few months ago, and couldn't agree more. I also cannot believe I've missed out on it for so long, having been a software engineer professionally for 13 years now, with 10 years of computer science specific education before that. How is this not taught in every computer science curriculum?
I think it once was taught, but was considered passé in various places during the "Java" revolution, and in general I think there is a degradation of capabilities in most comp-sci programs, a consequence of interference by market players who invest in the schools in order to keep their technologies relevant.
If anyone reading this wants to see what the fuss is, you can get started very easily with antirez' (Redis fame) simple toy project, LOAD81 - which is a great example of just how powerful things can get when you bring Lua into your life. (http://github.com/antirez/load81.git) - note that this is a great project to learn this technique, but if you do anything interesting in the load81 context - please push it back to LOAD81. :)
There's also the book "Creating Solid APIs with Lua" by Tyler Neylon [1,2] which seems to be quite specifically addressing this idea of embedding lua in something so as to implement highly-movable application logic in lua rather than the something you started out with.
Great references, thanks for that! I'll add that book to my library and give it to folks when I recommend they learn the VM+execution environment technique.
It's a shame that Java (and JavaScript) replaced the "Lua with C/C++/whatever" approach. JavaScript is incredibly hard to write in a "correct" way. It has way too many quirks and an incredibly complex tooling system (npm, etc.).
Lua is pretty much a "minimal" language which gets things done. It's not perfect, but it certainly doesn't suffer from the bloat which many other languages have.
I agree with you completely, Javascript has been a step backwards from what could have been, had we been teaching teenagers to glom the VM onto C/C++ layers. This is just a much better approach than 'npm all the things' .. although Lua isn't without its warts: LuaJIT, LuaRocks, 5.1, 5.2, 5.3 ..
Yeah, we're also employing that same architecture to great effect. A really nice side-effect is that you can reconfigure your app easily without recompiling. It really speeds up iteration times.
We can also have a small core team writing C/C++, and then let a wider team loose to build applications with Lua.
"I think its really important for any developer, whether they are encased in the glories of the browser or otherwise, to understand how to add the Lua VM - to anything - and then use that as a means of productive application development."
I completely agree! I feel that I should add thoughts about this into the article.
A lot of people (justifiably) complain that C and C++ are not memory-safe and that it's hard to write readable and safe code in it. Lua allows you to hide the scary parts of C and C++ and write simpler and safer code by default.
Definitely. And I think one of the reasons this approach is so powerful as well as being safer - when done properly - is that it follows the philosophy embodied in the OSI model. By glomming the VM onto a hierarchy of abstraction, one is fulfilling the OSI intention of clean separation of concerns.
This isn't so easy to do in the Javascript jungle, alas.
Happy to read your article, we are very much aligned. I hope your work brings more programmers to the world of Lua ..
I imagine this speeds up development velocity and is easy to adapt to a changing environment.
I think Python is an example of the same pattern, you write your glue code in Python and the high performance code is in C called by C FFI from Python.
I just wonder how we can write code with development velocity of Python or Lua but actually evaluate extremely fast.
Lua is good stuff. Besides games, it is also really useful for hardware/embedded tools. Small, relatively predictable, and easy to bind.
Especially when bringing up and testing new hardware and algorithms, it is incredibly useful to to be able to go from "wouldn't it be useful if we could test <insert scenario here>" to a fully automated system test with a few lines of Lua.
With minimum effort you can get a lot of benefit from using Sumneko's Lua language server in VSCode for near many Lua versions (LuaJIT, PUC-Rio Lua, etc.)
All of the projects serve different purposes, whether you need/love Lua for speed (JIT, for games), low memory usage (AOT, for embedded), or simplicity (all of them).
Great to see sol2 mentioned. We use it in Codea[1] to bind to our game/render engine. The article is right about the lack of safety and inscrutable messages, though. Recently we bound almost the entire iOS Objective-C APIs into Lua, though not using sol2, this used the Obj-C runtime headers and a lot of Lua C-hackery.
Lua seems to be widespread in games, but in fact is a very nice tool to make day-to-day apps user-extendible (with neovim and mpv being a couple of examples).
I have built professional, scientifically sensitive, important apps with Lua.
A recent project involved doing fluidic particulate quantification in an embedded, field portable instrument, which was delivered on time and under-budget - and represented a major revenue source for the customer (40 million dollars worth of orders) when it was shipped. The client was told it "could not be done" with the hardware as specified - not only did we do it with Lua, it has since been widely recognized as the most advanced bit of architecture the company has implemented for its instruments ..
Lua is not just for games. Games are a good place to learn Lua, though.
Out of interest, is there any public information about your project?
I am currently prototyping a Lua environment for instrument control in a regulated environment. I've added several customisations and safety mitigations to make it suitable for use, and it works beautifully. One thing I would like is to be able to showcase examples of use in other products.
If you, or anyone else, have any examples I could point people to, to demonstrate existing real-world uses of Lua, that would be really appreciated.
Many thanks for the reference, this looks like a really interesting device, and it has quite a lot of similarities with the type of system I'm working on (albeit in a completely different application domain).
Currently just starting with a very simple prototype, which is a very basic action scheduler for running analysis workflows. It will become more capable as more system functionality is wrapped and exposed. That ease of wrapping and sandboxing the Lua environment is one of the features I really like about Lua.
Agreed. I see many programs use JS and Python for plugin APIs, while Lua might be a better choice in many cases. It's much simpler and easier to pick up even if you're new to the language.
"The key is to unhinge the gameplay thread from the render/physics threads."
Many simple games don't need it, by the way. A lot of 2D indie games don't do complex enough rendering/physics to warrant a separate thread. This, in turn, makes everything much simpler if your entire game only uses one thread.
Indeed. I have no credentials to speak about such hard problems, my experience is working with high QPS servers written in Lua and a bunch of small indie games. :)
There are luabridge and SOL for Lua to interface with C++. Not used them, both seems gearing towards how to call Lua from C++ code easily, do they also help me to create C++ extensions for Lua to call, I do not know yet.
I have yet to find similar packages for C SWIG can be used to ease the c-binding coding.
Lua with C/C++ works basically like Python with CFFI, but Lua is much simpler and light weight that you can embed into your projects directly. A perfect combination between glue scripting and heavy lifting code that is small and easy to maintain.
Can untrusted Lua code be executed safely, with bounds on time and space (e.g. memory used)? Or if a threshold is exceeded, the host environment is notified?
Time is difficult to do. You can use hooks to have a function called for events like entering a function or executing a new line of code and then decide in that callback if execution should continue. That's fairly expensive and might not catch a tight infinite loop (while true do end).
Another way would be to make a minor interpreter modification to have it count byte code instructions executed and terminate execution that way. But then you might run into problems where calls into native functions might take a long time (e.g. string.rep("x", 1000000000)).
So it's not easy and the only solid approach is to have lua run in some external process and kill that using OS limits. You might get away using a thread instead of a process if you use a custom memory allocator to help with cleaning up resources, but I'm not sure if it's safe to terminate Lua that way.
The following prints 1 through 10 and then exits using Lua 5.3 and 5.4:
local i = 0
debug.sethook(function ()
i = i + 1
print(i)
if i >= 10 then
os.exit()
end
end, "", 1)
while true do
-- nothing
end
I've never used hooks in practice, though, so I may be missing some caveats beyond the obvious.
Regarding time, IIRC the Lua implementation very carefully implements lua_sethook (and related internals) such that you can safely install a hook from a signal handler, at least on common hardware like x86, ARM, etc where pointer assignments are naturally atomic. It can be tricky to make your lua_State context available to a signal handler, though.
Also, unless you isolate the process using system facilities, Lua probably isn't suited to running potentially malicious code as the VM doesn't see enough eyeballs. Looking through the bug history at https://www.lua.org/bugs.html it's apparent that even if you disable all APIs there are occasionally exploitable bugs in the interpreter. I suppose the same is true in practice regarding, e.g., Node--many more eyeballs but the JIT engines are so much more complex that you end up in the same place. (The jury is still out on dedicated WebAssembly engines, but JIT engines are notoriously complex and "memory safe" languages don't help with JIT bugs.)
I don't think that this approach is that good. First of all, you need to write a custom "interface" file. Then, you need to integrate running of swig into your build process, which makes the build complicated.
With sol, it's much easier - you just register function/classes in your C++ code - it's much easier to do than with Lua C API and doesn't require running any "post-processing" steps at all.
Have you tried sol2? It's vastly more capable and transparent. It's also 100% native C++ code, so is just part of your application, with no extra tooling required.
This is the only part of the article I disagree with. I've found that the biggest problem with using Lua in a C++ game is writing code in the "wrong" language, and the mess that can cause. The boundary between Lua and C++ needs solid validation to keep bugs out, and this makes it neither performant nor convenient, so minimising the boundary is important. As soon as code ends up in the wrong language, you generally need to add a large amount of unnecessary boundary code.
An important part of designing a Lua/C++ game for me is having a good mental model of what the "right" language to write any piece of code in is, and only deviating from that when performance mandates.