
Lua 5.4.0 beta - dottrap
http://lua-users.org/lists/lua-l/2019-10/msg00003.html
======
unbendable
We use lua 5.3 on an embedded Platform for scripting and it has been a roller
coaster ride. Not a fun one, unfortunately. You want luasocket? The stable one
is not compatible with 5.3. Packages are sometimes outdated (for years no
updates) and there is no replacement. The lua point releases have breaking
changes. The source code itself is a macro hell which is hard to debug. And
the code is not very readable. The documentation lacks for some topics of you
use the c api. I tried to implement some scheduling for c and lua threads.

Oh and one thing if you must use Windows... you better quit right away. I had
to help my co worker Installing it with luarocks and it is a mess. to be fair
it was easy on my Ubuntu machine.

The thing is lua on embedded has no rival. but god did it cost me some nerves.

~~~
dividuum
> The documentation lacks for some topics of you use the c api.

I'm surprised to hear this. I'm doing a lot of Lua and I always thought the
documentation is very succinct and complete. I really curious when issues you
ran into.

> The thing is lua on embedded has no rival. but god did it cost me some
> nerves.

You might take a look at
[https://bellard.org/quickjs/](https://bellard.org/quickjs/) and
[https://duktape.org/](https://duktape.org/). The latter seems oddly familiar
if you've worked with the Lua C API.

~~~
mikepurvis
My info might be a bit out of date, but something that surprised me when I
looked into it was how little control I had over the embedded lua interpreter
from my outer app. I basically had to hand it control by calling lua_pcall,
and either wait for a result or manage a timeout from another thread.

I expected to be able to do things like tell Lua to run for X milliseconds or
Y opcodes or whatever, but it didn't seem that anything like this existed.

So perhaps my disappointment with the documentation was really just
disappointment that the implementation wasn't set up to do what I wanted in
the way I wanted to do it. But in the course of coming to this realization, I
also found the documentation frustrating and opaque.

~~~
MereInterest
Running for X opcodes isn't too difficult, but isn't immediately
straightforward either. Instead of calling a function with lua_pcall, start a
coroutine using lua_resume. Then, you can have a callback using lua_sethook
and the LUA_MASKCOUNT option, which triggers the callback after a fixed number
of opcodes. Inside the hook, you make a call to lua_yield. That forces the
coroutine that was running to yield control to the outer app. The outer app
can then resume the coroutine if desired.

There was a while that I wanted use this to design a programming-based video
game, where simulated agents worked in real time. Most of the programming
games I have seen use a fixed timeout, and kill any scripts still running
after that time. What I was thinking was to instead have the script constantly
running, and any slow scripts continue running on the next frame (e.g. all
players get 100 opcodes per frame).

~~~
pb82
Thats clever. I used the same approach with lua_sethook to run a hook every X
opcodes and check if time or memory constraints have been exceeded. But
instead of yielding I used longjmp to terminate execution (basically like
raising an exception and catching it a few levels up in the stack).

It's not bulletproof though and the script can still block e.g. when calling
into native code.

~~~
MereInterest
Yeah, native code can block, and in those cases requires additional checks in
that exposed native code. That runs into all the usual multithreading
headaches, and depends on exactly which functions have been exposed to the lua
interpreter.

The advantage to this method over longjmp is that the function call is still
valid and can be resumed at any point.

------
dottrap
Main changes

\- new generational mode for garbage collection

\- to-be-closed variables

\- const variables

\- userdata can have multiple user values

\- new implementation for math.random

\- warning system

\- debug information about function arguments and returns

\- new semantics for the integer 'for' loop

\- optional 'init' argument to 'string.gmatch'

\- new functions 'lua_resetthread' and 'coroutine.close'

\- coersions string-to-number moved to the string library

\- allocation function allowed to fail when shrinking a memory block

\- new format '%p' in 'string.format'

\- utf8 library accepts codepoints up to 2^31

~~~
akklesed
>\- utf8 library accepts codepoints up to 2^31

Interesting. Didn't Unicode restrict UTF-8 to allow encoding only 21 bits?
Does it mean that it can now do 6 byte UTF-8 encodings? What kind of
restrictions did it have before?

~~~
zamadatix
Comparing the documentation seems to verify this easily:

\-
[https://www.lua.org/manual/5.3/manual.html#6.5](https://www.lua.org/manual/5.3/manual.html#6.5)

\-
[https://www.lua.org/work/doc/manual.html#6.5](https://www.lua.org/work/doc/manual.html#6.5)

------
alberth
Just recall that Lua versioning schema allows for breaking changes to point
releases.

This has been both great for Lua as a Lua and one of its biggest challenges.
It's allowed the language be constantly refined and tweaked; however, at the
expense that it's super common for applications to complete break when
upgraded to a point release.

[https://www.lua.org/versions.html](https://www.lua.org/versions.html)

------
fullstop
It's nice and all, but I'll be over here using LuaJIT.

~~~
sam_lowry_
why?

~~~
tyingq
The performance difference is pretty stunning.

[https://luajit.org/performance_x86.html](https://luajit.org/performance_x86.html)

~~~
johnisgood
Indeed. Mike Pall blows them out of the water single-handedly. I wonder if Lua
developers have learnt anything from him with respect to performance. I would
like to see how Lua 5.4.0 would perform in comparison.

~~~
hugomg
It is no surprise that the benchmarks on the LuaJIT website are going to show
it ahead of PUC-Lua, is it? In truth the performance comparison is a bit more
complicated than that. :)

For a pure Lua comparison LuaJIT is typically going to beat Lua by a large
margin. For code that can be JIT compiled you might see ~10x performance
improvements and for code that doesn't you might still see around a ~2x
improvement because the LuaJIT interpreter is written in hand-crafted assembly
language (PUC-Lua sticks to portable and standards-compliant C90).

But things get a bit more complicated if you add foreign C code into the mix.
Under LuaJIT, the JIT compiled code that uses the LuaJIT FFI interface is the
fastest, but interpreted code using the FFI is slower than interpreted code
using the traditional Lua-C interface, and can even be slower than PUC-Lua
using the traditional Lua-C interface. This means that when you are using
LuaJIT you need to be very careful to make sure that your code is the sort of
code that can be JIT compiled. If you hit a "not yet implemented" feature you
can have a big performance degradation.

\------

Nevertheless, Lua 5.4 has brought significant performance improvements
compared to Lua 5.3, specially in integer operations (including for loops).
Another big change was that the `#` is now faster. For example, on my machine
the following loop runs in 1.96 seconds in Lua 5.3, in 1.26 seconds under
LuaJIT, and in 0.96 seconds in Lua 5.4

    
    
        local a = {}
        for i = 1, 1e7 do
            a[#a +1] = i
        end
        print(#a)
    

The new generational garbage collector is also a very big improvement. I've
measured a 1.5x speedup on some GC-dominated workloads, where the final result
was that Lua 5.4 would even edge out LuaJIT, which is still using the old
incremental collector.

~~~
dottrap
> I wonder if Lua developers have learnt anything from him with respect to
> performance.

It cannot be emphasized enough that PUC-Lua and LuaJIT have different goals.
One goal for PUC-Lua is that it must be portable everywhere and be
implementable in pure ANSI C (no platform or architecture specific calls).
LuaJIT in contrast contains much handwritten assembly for the specific
architectures it supports.

PUC-Lua also cares deeply about interoperating with C and hence why the Lua-C
interface is formally part of the language spec, and as the parent thread
mentioned, LuaJIT using the traditional Lua-C interface can be really slow.

That said, PUC-Lua has learned a lot from Mike Pall. It was Pall that helped
them get fully yieldable coroutines across pcall boundaries. I'm sure Pall's
lengthy analysis of unimpressive garbage collection performance in both Lua
5.1 and LuaJIT was read by all of them and a motivation for them to try the
generational garbage collector now finally working in 5.4.

But finally, there is a new project called Pallene (not being pedantic,
formerly known as Titan). It is inspired by all the lessons of LuaJIT. The
name ' _Pall_ ene' is a nod to Mike Pall. Pallene is a statically typed sister
language to Lua that is designed for performance and easy C/Lua bridging, but
in a way that doesn't bring in all the downsides of JIT. The language is glued
at the hip to Lua so there is also easy interoperability between Pallene and
Lua code. Pallene introduces static types not for the sake of type safety, but
for the purpose of inferring what things can be optimized. Pallene brings in a
AOT compiler into the mix and generates optimized binaries. But these binaries
also look like any other Lua/C native library so they are callable from normal
Lua too and the user doesn't know if they were implemented in C or Pallene.

Pallene was previously discussed on HN here:

[https://news.ycombinator.com/item?id=18038619](https://news.ycombinator.com/item?id=18038619)

Also look for the original Titan talk from one of the Lua Workshops a few
years ago.

~~~
hugomg
We chose the Pallene name because it was another one of Saturn's moon names
that sounded nice, and is pronounced the same way in English and Portuguese.
The Mike Pall thing was just a coincidence although we certainly can't deny
his influence, as you have said :)

------
neuland
I use Lua via OpenResty / lua-nginx-module and in personal C projects for
plugins and scripting. Some of these seem useful, like utf8 support and const
variables.

For the multiple user data values feature, I've always found lightuserdata
more useful than userdata, because it's not often that you need just one bunch
of simple memory that can be freed without any other work. Rather, I almost
always have more complex data, such as things that need manual cleanup like
sockets, handles, etc. Or, the structure has pointers to other things that
must also be cleaned up. What I'd like instead is the ability to pass a
destructor callback to `lua_pushlightuserdata` so that it get's called when
the pointer falls off the stack.

I don't use Lua threads or coroutines currently, so I don't have much to add
there. What I do wish is that there were a way to clone or pass objects from
one lua state to another to support parallelism. Basically, I'm thinking of it
like the work Eric Snow is doing on subinterpreters in Python. Passing values
(by copying, no shared memory) between lua instances.

One use case I had is loading a plugin that you then want to run multiple
instances of in parallel with different arguments. Ideally, you wouldn't have
to load the plugin multiple times (therefore calling the module level code
multiple times). So I'd like to copy the entire lua state and then run each
one with different arguments. There's no userdata, coroutines, or lua threads,
so I don't have to worry about things that aren't possible to copy.

I got as far as looking into how to copy functions and then got busy with
other things and stopped. Is there anyone else out there trying to introspect
Lua function structures and copy all the opcodes, upvalues, etc?

~~~
dividuum
I think that's what Lua Lanes if for. Haven't used it myself, but the
descriptions sounds exactly like what you're looking for:
[http://lualanes.github.io/lanes/](http://lualanes.github.io/lanes/)

~~~
neuland
Wow, thanks for sharing! Not sure why I didn't find this when I was looking.
It does look like a very good match to my use case. I'll have to try it out.

------
ecmascript
People that use lua, what do you use it for and why did you pick lua?

~~~
corysama
I’ve used it in many games.

It’s a simple, but powerful language that uses basically two features
(functions and hashmaps) to implement everything. Ex: script files are
implicitly functions. Global variables are in a hashmap. Semantically it is
very much JavaScript without all the surprises. Semi-technical users (artists
and designers) can figure it out and go on to make surprisingly powerful
features.

The binary code is small and fast. The standard library is small and most of
it can be removed easily. Parsing source to bytecode is quick. Loading
bytecode is very fast.

It is easy to embed. It’s easy to contain it’s memory allocations in an arena.
It’s easy to restrict what it can and cannot do. It’s easy to interface with
C/C++. It’s extremely portable.

~~~
OskarS
Your last line there is really one of the big keys. No other language is as
easy to embed and interface with C/C++. The only real exceptions are some
varieties of Lisp, but the "easily embeddable" ones tend not to be as mature,
performant and robust as Lua.

I'm no particular fan of the language, but they do the "embed and interface
with C/C++" better than anything else I've seen.

------
bbmario
Wish they could improve the language syntax. It's so unnecessarily verbose.
Moonscript did a good job improving Lua.

~~~
chc4
Most of it I'm pretty okay with. The one big pain point is lambda definitions:
you _have_ to do `function(x) print(x) end` each time, instead of `\x.
print(x)` or something.

It sounds small, but with callback heavy or embedded DSL code it very quickly
adds up.

------
daurnimator
This is only a beta release; final could still be a year away.....

------
yanex
It seems that quite a lot of people use Lua. I liked the language, it's tiny
and well-made. However, I wasn't happy about the tooling support. IDE plugins,
documentation generators, lint checkers – all of this seem abandoned. Is there
any similar language (embeddable, good C interoperability) with better tooling
besides JavaScript?

~~~
legends2k
[Nim][1] seems to be a potential option.

[1]: [https://nim-lang.org/](https://nim-lang.org/)

~~~
yanex
The language looks really nice. I especially like its strong-typing nature. It
seems, though, that it won't be easy to embed – Nim compiler requires either
gcc or clang.

------
greggman2
This is probably not the right place to ask but ..., given that Lua is
commently embedded in games, anyone know of a sandboxed version? I'd love to
provide a scripting language for user mods to a game but I'd like as much as
possible to not have to trust the mods.

~~~
yanex
You can choose what parts of the standard library scripts allowed to use. Just
exclude os/debug libraries, and that's it. Newer versions of the Programming
in Lua book contain detailed instructions.

------
nomadluap
I'm looking through the new reference manual and there doesn't seem to be any
mention of const variables, or any mention of how the new to-be-closed
variables work. Do you just define the __close metamethod and lua takes care
of the rest?

~~~
dottrap
I haven't tried them yet, but in the work reference manual, look at: 3.3.7 –
Local Declarations and 3.3.8 – To-be-closed Variables

I think the syntax use is: local x <const> = 5 local filename <const> =
"/etc/fstab" local fh <close> = io.open(filename, "r")

------
astatine
We use Lua 5.1 in an embedded system and it works great. We use only a few
standard libraries and have several dozen of our custom libraries exposed as
Lua APIs. Predictable memory usage is really good. It's several years since we
picked the then current version of Lua and have had no reason to upgrade. What
would be the motivations to upgrade to a new Lua version on an embedded system
where there is a working version?

~~~
dividuum
I don't think there's any reason to upgrade unless some of your dependencies
require a higher version and backporting is more expensive that porting over
all your existing code. Since it sounds like you expose libraries with your
own code and you have full control over your ecosystem, I don't think it makes
sense to upgrade. At least in my project I see no reason to switch away from
LuaJIT (so ~5.1/5.2). It works perfectly well and I don't see why that would
change.

------
mikorym
Does this release influence LuaTex? Or rather, maybe, how is it going with
LuaTex?

I've previously opted for LuaTex rather than "base" Tex or XeLaTex and I'm at
the point where I'd happily stick to LuaTex and also learn some of it's more
technical aspects.

------
nikolay
Unfortunately, we only care about LuaJIT, which is lagging behind.

