
How the embeddability of Lua impacted its design (2011) - adamnemecek
http://queue.acm.org/detail.cfm?id=1983083
======
dwarman
I used Lua 5.2 (with 5.1 compatibility enabled) to embed in a C framework for
building and executing data flow graphs. I implemented C .h header to JSON and
back, and also an API wrapper generator, which really helped things along. The
basic features include: co-operative FSM scheduler (using coroutines for sleep
etc), structured FSM Engine, structured data pipes (stucture defined in C
headers and piped in binary, but to Lua they look just like Lua tables
syntactically), Event generation and propagation mechanism, ability to record
and play back data pipe streams, fine grained MIDI parser and generators, and
a bunch of other details. The embedding mechanism is generalised, and so there
is alongside the Lua scripting a similarly complete FORTH drop in language,
for those who might be interested. I think it's a pretty neat system, and
works quite well.

The system had some turbulence during its inception in a now defunct very
small company, but it all reverted to me. I would like to get it up on Github
before I pass on (fairly soon, I fear), but need help fixing all the file
headers, and some minor bugfixing. Volunteers? I have a minimal html
description file if you'd like. But a build help file is needed.

------
rurban
I'm missing the crucial design bits which would influence embeddability:

Do pointers move? How is the GC? How is the FFI? What are the FFI callback
limitations?

With FFI callbacks, calling back into Lua functions and using Lua data you
need either refcounting or a slow mark & sweep. Or just for FFI accessible
data. Of you provide an API for this data to propagate moved pointers back to
C.

~~~
FRex
Pointers do not ever 'move'. You also should never store variables (to
userdata, memory chunks, strings, etc.) by pointers or anything like that
according to documentation because they might be collected at any time. To
stuff like tables or values you never ever get a pointer and having a pointer
to a "Lua value" would make no sense because it's a small struct with just the
type number (in case of nil that's all that matters actually) and then either
an inline value (light userdata, bool, number, etc.) or a pointer to the
actual GC object (thread, string, table, function, internal data structures,
etc.). Lack of exposing internal data structures (there is Buffer in luaL but
it's the only one I can think of and it's so simple and just to collect few
results together it doesn't matter) is probably the reasons why LuaJIT is ABI
and API compatible with Lua 5.1 despite implementing stuff very differently
internally (like values in NaN tagged doubles to keep them in a 64 bit
boundary instead of just placing fields normally in structs and letting
compiler figure it out like stock Lua does).

There is a feature in luaL part of the C API called 'refs' but that's
basically storing stuff in a table array and addressing it by ints and it's
done all using the C API itself, it's not a VM or language feature.

There is also a weak reference system (where table can be marked to have key,
value or both be 'weak' and allow collection if there are no 'strong'
references anywhere to these objects) and that is part of the VM itself (GC
specifically, the table processing code) and has some caveats with stuff like
strings (they are interned in Lua) and it doesn't remove intelligently form
tables serving as arrays.

To call a function from C you push it and its arguments onto the stack (of the
main thread or the coroutine) and execute it with one of API functions
(depending on what you're trying to do)[1], to call it in Lua you 'just' call
it or pass it to pcall or a when making a coroutine.

A C function exposed to Lua like that must implement its 'protocol'[2] which
is getting and returning arguments by interacting with the lua state pointer
so it's an extra effort to wrap them or you can use a third party library
(there are some, especially C++ ones which use templates to automatically wrap
stuff, but I don't use any myself).

C and Lua functions are indistinguishable in most cases, they are called the
same way, etc. but sometimes they differ, i.e. Lua functions can be dumped
with string.dump.

There is no FFI (as there is in C#, JNI, etc.) as such. LuaJIT has an FFI that
parses C declarations and it's quite nice IMO. There are also third party FFI
libs but I know nothing about them. Using that also loses you all the nice to
haves that Lua has like multiple return values (it's customary to return false
or nil and an error message string on errors from many functions and it's
impossible to write an iterator without multiple return values), interacting
with lua state in the function, arg type check calls with automatic errors,
etc.

The GC is a mark and sweep one, in 5.0 it was a stop the world one - two
color, from 5.1 onwards it's incremental tri color with forward and back write
barriers(see [0] for an explanation of what that means and how it works). In
5.2 there was also (off by default) generational one. I have no idea how that
one worked and it was removed in 5.3. LuaJIT also uses mark and sweep from 5.1
but with improvements and for 3.0 there was supposed to be a quad color one.
You of course have full control over it (stop, start, adjust speed, force a
full cycle or a single step, etc.) to complete right now) from Lua and C via
the right function. It does nothing fancy like compaction or copying/moving.

You are also responsible for the memory allocation (in 64 bit LuaJIT it's
taken over and attempting to do that is an error) and the default alloc is
just a malloc wrapper[3]. From 5.2 onwards you also get told during initial
(i.e. not the resize) allocation what kind of Lua object is being allocated
(it doesn't tell you about internal structures though, only the user visible
Lua types) so that might help you make some decisions in the allocator. Lua VM
never allocs memory or touches the system on its own in any way either.

If you want you can remove all the standard libs like os, io, etc. and not use
any of luaL functions (they all are implemented using Lua C API only so they
are 'userspace' and optional, not part of VM, they do not use internals of the
VM itself). Even the require mechanism for importing libs is all userspace
implemented (unlike like Python where it's part of the language, I think).
It's also possible to recompile Lua and change what type the number is (double
so 64 bit by default, plus 64 bit int in 5.3, there is convenient macro to
force both to 32 bit in 5.3, for LuaJIT I'm not sure, it uses NaN tagging in a
double and a handwritten asm to get its speed and compactness and I'm not
familiar with its codebase like I am with the vanilla Lua), add a bit of extra
space in front of the VM for direct instant access from C (space for 1 pointer
by default in 5.3, no space before that) and a bunch of other configurable
things. The codebase of PUC-Rio Lua (i.e. not some third party implementation
like LuaJIT or Havok Script[4] - which I think they don't even sell but that
Google still indexes and they have a testimony on the front page for...) is
also easy to jump into if you know C and very small (c.a. 10k lines if you
ignore the lexer+parser+compiler part, std libs and standalone lua and luac
program sources).

Feel free to ask about anything if I was unclear. I am knee deep in Lua by now
and I've even written about Lua internals for my engineer's thesis (in Polish)
so I'd say I'm quite up there with my knowledge of its working and history and
such compared to average Lua knowing person. I thought about recycling my
notes to write an article about that but I don't really see a point because
I'd largely be rahashing between source code itself (very readable!) and
various resources like no frills intro to vm instructions[5], articles from
the three Brazilian devs themselves, mailing lists, details on LuaJIT page,
etc. If someone is interested about such an article then you can drop me an
email( my email address is visible once you're logged in at
[https://github.com/FRex/](https://github.com/FRex/) ) and if I get a few I
might do it.

[0] - [http://wiki.luajit.org/New-Garbage-
Collector#introduction](http://wiki.luajit.org/New-Garbage-
Collector#introduction) [1] -
[http://www.lua.org/manual/5.3/manual.html#lua_call](http://www.lua.org/manual/5.3/manual.html#lua_call)
[2] -
[http://www.lua.org/manual/5.3/manual.html#lua_CFunction](http://www.lua.org/manual/5.3/manual.html#lua_CFunction)
[3] -
[http://www.lua.org/manual/5.3/manual.html#lua_Allo](http://www.lua.org/manual/5.3/manual.html#lua_Allo)
[4] - [https://www.havok.com/wp-
content/uploads/Havok_Brochures/Hav...](https://www.havok.com/wp-
content/uploads/Havok_Brochures/Havok_Script_2015.pdf) [5] -
[http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInst...](http://luaforge.net/docman/83/98/ANoFrillsIntroToLua51VMInstructions.pdf)

~~~
daurnimator
FYI Lua 5.4 is bringing back a generational collector in a slightly different
style.

Also, what is your thesis?

~~~
FRex
I don't really follow the development of Lua in any way.

Lua 5.3 was very underwhelming to me, I didn't care for the ints and bit
operators _at all_ (and before 5.3 the one number type was a selling point)
and it's annoying how there is such a clear strong split between 5.3 and
LuaJIT because it effectively finalized the fork that 5.2 has sort of started.
Some things (like Love2D, OpenResty and other performance conscious stuff)
stuck with LuaJIT so basically a 5.1 on steroids with few 5.2 features. 5.1
also had the killer feature of incremental GC and came at a nice time and was
the main Lua for almost 6 years so it stuck into gamedev a lot (which is where
I picked Lua from, my hobbist gamedeving).

I also stick to 5.1 by default in my hobby gamedev to be able to bring in
LuaJIT at any time and ignore 5.2 and 5.3 pretty much.

My thesis was about history of Lua a bit (rehashing what's on the main website
plus few Roberto's articles pretty much) and then a lot about how stock Lua
5.1 is implemented (most of it is 1:1 transferable to other stock Luas) like
how bytecode is executed, how does GC work, what does it mean Lua is register
VM and not stack one, how are internal structures laid out, how functions and
closures and threads work, etc. It's (official) English title was "Analysis of
code of original implementation of Lua 5.1" but it is written in Polish
("Analiza kodu oryginalnej implementacji jezyka Lua 5.1"). I don't think it's
available anywhere. That was my engineering (inżynier) thesis which I defended
in February and I'm now doing a master thesis but it's purely mathematical CS
stuff and not related to Lua at all.

~~~
marktangotango
As an aside, there’s stuff on the lua wiki about sandboxing lua, how would you
go about sandboxing lua to limit heap, cpu cycles, and os level api calls? For
lua 5.1? For luajit? An example use case would be letting users upload scripts
to run in game?

~~~
FRex
I never did that actually and it seems involved, hard and very limiting for
what such a script could do.

I dislike talking about touchy high risk stuff like that in general since
someone might think what I say is a list of things to do to be 100% secure and
get burned.

What it says on [0] is a good start I guess: only textual code input, no
bytecode (5.1 has bytecode verification but it was removed in 5.2 because it
had flaws and gave false sense of security, there were examples of breaking
stuff up with malicious handcrafted bytecode), white listing.

To not allow something (like os or debug libs) to be used just don't include
it in the environment for that used supplied function (it does restrict you
quite a bit though I guess...).

For memory you could use your own allocator and enforce a limit there (and
probably set your own atpanic function in 5.1 and longjmp out of it because
you might kill the entire program with the default one when Lua won't be able
to alloc the error message itself, in 5.2 and 5.3 it's preallocated in the VM
so that won't happen).

To prevent runaway Lua loops you can use the count debug hook and throw a Lua
error in it, pcall will get that (just remember to reset it between calls
because it doesn't reset to 0 on its own). Preventing runaway C loops is up to
making sure you don't expose something that can get stuck. You could also set
count to something low or use line debug hook and check the time in it and
error out if the code took too long already but it might slow the code down
too badly (depends on what you do I guess).

That stuff (mem alloc, debug hooks, panic, environments, etc.) is in the docs.

Long strings can also DoS the hashing algorithm and that might totally kill
performance with tables, it's a trivial change to fix that yourself or use one
of the patches[1]. It came up on the mailing list too. I'm sure it happens on
5.1, not sure about latest 5.2 and 5.3 right now and I don't have time to look
into it.

LuaJIT doesn't also let you set the allocator on 64 bit and has some other
issues with out of memory errors[2] (I don't know what they are exactly
about). I guess it's out of the question if you're really paranoid (stock Lua
5.1 is also way easier to read, modify and compile IMO). I'm not very familiar
with LuaJIT.

[0] - [http://lua-users.org/wiki/SandBoxes](http://lua-
users.org/wiki/SandBoxes)

[1] - [http://lua-users.org/wiki/HashDos](http://lua-users.org/wiki/HashDos)

[2] - [http://luajit.org/status.html](http://luajit.org/status.html)

