
A Look at the Design of Lua - creolabs
https://cacm.acm.org/magazines/2018/11/232214-a-look-at-the-design-of-lua/fulltext
======
mion
I studied at PUC-Rio and I met Roberto Ierusalimschy (the main author of Lua)
when he was teaching a course on semantics. I once asked him if I could send a
PR about a feature I wanted in the language and he said something that I never
forgot: "Yes, but I won't use your code. I love that people send me ideas, but
I actually enjoy coding... so I will gladly take your suggestions, though I
will write it myself."

He then explained this "dictatorial" behavior is what allowed him to keep the
implementation simple and concise over the years. At the time he boasted about
the source code being less than 8K LOC, though it has likely increased in
recent versions. (I think it was around version 3.) I'd recommend taking a
look at the source code, it's truly a C masterpiece.

~~~
gota
I'm currently a PhD candidate at PUC-Rio. For context, there's a 'meet the
researchers' round of seminars during the first semesters mainly targeted
towards Master's students (so they can choose an advisor).

Prof. Ierusalimschy's was one of the most interesting seminars in my first
year. His passion is apparent in the way he talks about Lua's history and
current development. Unfortunately I haven't taken any courses with him

As a side note, it's nice to see folks from around here on HN

------
antirez
I've the feeling that Lua became popular mostly based on its implementation,
fast, small and self-contained portable C code, and not for the merits of the
language itself. Certain things are addressed in Lua 5.3, but in general Lua
has several flaws:

\- Before Lua 5.3 there was no integer type, this makes a lot of things harder
than needed in programming, unless you just write high level code.

\- Lua trivially departs from standard syntax like ==, !=, zero-based indexes,
without good reasons, creating a general feeling of confusion.

\- There was an odd idea about what to include by default and what was
optional. The lack by default of bit operations makes the language always in
need of a trivial library.

\- Tables as a single data structure are not ideal for many tasks. Especially
the dualism between tables that are indexed by sequential numbers and tables
acting as maps. It's a lot more kind to your users to give up with the solely
academical ideal of "just one data structure", and give programmers at least
an ordered and a map data type, or at least come up with something better than
that. Lisp attempt at doing most things with lists was much more successful
and clean.

\- The stack-based C API is not comfortable to use.

I could continue to be honest, for instance by default you can't type an
expression in the REPL and see the value returned. However when the task at
hand is to write small scripts for a larger system, the implementation of Lua
offers big enough advantages to make it a worthy choice.

~~~
ansible
> _Tables as a single data structure are not ideal for many tasks. Especially
> the dualism between tables that are indexed by sequential numbers and tables
> acting as maps. It 's a lot more kind to your users to give up with the
> solely academical ideal of "just one data structure", and give programmers
> at least an ordered and a map data type, or at least come up with something
> better than that. Lisp attempt at doing most things with lists was much more
> successful and clean._

I have a different view.

Lua tables are the _foundation_ for just about any conceivable data structure
in CS. While you can do a lot with raw tables in Lua, you can also use them to
build all kinds of things on top of them, like b-trees. Or an object system
(which is a popular hobby among Lua aficionados).

There's a limit to what can and should be built into any base language. Trying
to wedge in more container types can cause a bit of difficulty, in my opinion.
Python is the classic example here, where you have tuples using parens, arrays
using square brackets, and dictionaries using curly brackets. At some level
that is clever, but at another level it is a cause of confusion.

~~~
sulam
Sorry, but tables are awful in practice. I have to iterate all the elements to
get a count? This is especially egregious when I’m using one like a list/array
and need to know how far to iterate. Maybe there’s a better way I don’t know?

~~~
Twisol
If you need something quick-and-dirty, you can maintain an extra `["length"]`
field on your table to cache that information. If you want something more
polished, use a proxy table with custom methods/metamethods mediating access
to your array so that the length is properly kept up to date. This isn't
terribly different from the situation in C with arrays.

Tables are a basic mechanism for implementing any data structure. That is a
qualitatively different statement from saying that tables can _replace_ any
data structure. If you need semantics different from what tables immediately
provide, you'll need to implement the desired data structure yourself -- but
Lua gives you good tools to do that.

> Maybe there’s a better way I don’t know?

You can use `for i, v in ipairs(arr) do ... end` to iterate over an array
without worrying about "how far" to iterate. The `ipairs` iterator will
determine when it hits the end of the array.

~~~
sulam
Yeah, this is fine until I want the last element of an array. Then I have to
iterate it to find it. And writing a loop to get the last element of an array
is crazy-town.

~~~
nikki93
Have you actually used Lua? You don't need to iterate to get the last element
of an array.

~~~
sulam
I have! I’m not an expert at it, though.

------
wcrichton
Lua is a big reason why I'm interested in both systems and PL today. I started
using Lua to build game mods in high school, and I was struck by how simple it
was compared with Java, or even Python that I had tried learning earlier. The
all-encompassing "everything is a table" paradigm really sparked my interest
in the design of systems of computation, and how a broad range of programing
patterns could be expressed with a few primitives.

~~~
gvx
You might find Isle interesting
([https://github.com/gvx/isle](https://github.com/gvx/isle)). It is a small
programming language I've designed and implemented a couple of years ago
heavily inspired by Lua's "everything is a table". In it, a function call
looks like a name concatenated to a table literal, and calling a function
actually consists of creating that table and using it as the environment for
the function.

I still think it contains some neat ideas and concepts, even though I would
never actually use it for something practical.

------
anilakar
Stay away from Lua and use Python, Go, Node, Rust or even C in new projects.
This is going to be a lengthy rant for a comment; apologies in advance.

Having written a ton of networking code in Lua I can tell it has a lot of
design defects that most other programming languages have avoided. Especially
the deeply embedded one-based indexing and lack of a separate integer type
(chars are strings, ints are floats) make it wearing to type any code that has
to parse any binary data. Even bit manipulation requires going through the
FFI. Considering that Lua is often touted as the optimal lightweight solution
in small embedded systems, these features are must-haves.

The builtin string pattern matching library attempts to mimic regular
expressions, but bites the user in the leg every time they attempt to use it
for anything remotely regex-y. Unfortunately most usual string manipulation
seems to revolve around them and the string.gsub function.

The inconvenient string library is also largely irrelevant. The developers
call their language eight-bit transparent which in reality means that the
programmer needs a separate Unicode library, rendering the whole standard
string library mostly useless. One cannot even format UTF-8 text, because the
string library doesn't differentiate between characters and bytes.

The official documentation mostly revolves around C bindings. The actual Lua
language documentation is provided by another web site that looks like it has
been written by an attention-deficit child. Instead of getting to the point,
the articles often first provide a list of ways on how not to implement
things, or they are actually benchmarks of fifteen different ways to call the
standard library to get the same effect. Sometimes, the provided code does not
even work – for example, the given XML parser code crashes when it hits one of
the error paths.

The small runtime and fast VM do not really matter when the language itself is
lacking and requires a metric ton of external libraries to achieve even the
simplest things that are possible in standard C and Rust, which provide the
same level of performance and better type safety.

~~~
jack1243star
These are quite unfair arguments against Lua, especially since that you seem
to be using it for low level networking code and on embedded systems, which
are not really Lua's strong points. (Lua is designed to be embedded as per
being a scripting interface to C and other compiled languages.)

One based indexing is just a design choice, while it complicates
interoperation, I find it justified as Lua tables are actually hash tables, so
it would be weird to model it upon memory offsets. It reminds me that I am
only dealing with a hashmap with integers as keys.

Bitwise and UTF-8 string operations are lacking out of the box, but these are
arguably too specialized to be expected in a general scripting language. The
patterns are a little underpowered compared to other languages, but it is
still quite useful. I suggest looking into LPEG for more complicated
operations. Lua 5.3 do have some functions to deal with UTF-8 codepoints.

Also, regarding Unicode, I'd rather the PL provide only operations on bytes,
and let me decide what "characters" are to be represented. Nearly all PL
providing some Unicode facilities fail in different ways, just let me deal
with ICU myself if I really need Unicode. Also it is a royal pain in the butt
to deal with different encodings, with a language having special "characters"
and "strings" constructs.

I don't understand the complaint on documentation. It is one of the most
practical and clear technical writing that I have seen, if a little terse.
Perhaps you mistook the wiki as the main document? (The Lua language manual is
an one-page HTML page.)

~~~
bsder
> These are quite unfair arguments against Lua, especially since that you seem
> to be using it for low level networking code and on embedded systems, which
> are not really Lua's strong points. (Lua is designed to be embedded as per
> being a scripting interface to C and other compiled languages.)

If this is true, and I actually agree with you, then where is Lua actually
better than an alternative?

And, while I don't share the specific points of the parent poster, I
experienced similar issues with Lua. I found that while the Lua core language
itself is small, I had to compile in so much extra stuff to make it useful
that it wasn't actually small anymore.

And, once you get large enough that you start demanding big ARM Cortex M4
microcontrollers, Python is now in your scope.

~~~
imtringued
Running untrusted code to allow modding of games. Suddenly the fact that you
have a standard library or a complex language with a large amount of features
is no longer a benefit and instead is actually a liability often to the point
that you disable the standard library in lua entirely and only selectively
load parts of it and write your own modding API to allow modification of the
game.

There are entire games built around modding via lua such as gary's mod which
similar to a web browser has to download random code from a server. Then there
are mods like computercraft/opencomputers for minecraft in which direct file
access to the host doesn't make any sense at all because every computer has
it's own filesystem.

I just looked up Python's sandboxing capabilities and I can't find anything.
All I see is answers like [0] that suggest that it's not trivial.

What about other implementations like pypy that claim to have a sandbox? [1] I
can see lots of warnings like "This is not actively maintained."

The lack of mature sandboxing for python makes it impossible to use for the
usecases I've described above.

[0] [https://stackoverflow.com/questions/3068139/how-can-i-
sandbo...](https://stackoverflow.com/questions/3068139/how-can-i-sandbox-
python-in-pure-python) [1]
[http://doc.pypy.org/en/latest/sandbox.html](http://doc.pypy.org/en/latest/sandbox.html)

------
Immortalin
For those of you who are interested, I run a Lua newsletter.

The Lua community is small so every bit helps :)

Shameless Plug:
[https://Luadigest.immortalin.com](https://Luadigest.immortalin.com)

------
SeanLuke
It's worth mentioning the debt Lua owes to Self and Newtonscript. An
extraordinary amount of Lua's syntax and semantics is directly lifted from
these languages. Newtonscript in particular seems to be a template for a great
many of Lua's interesting features.

What drove me away from Lua is its philosophy of providing minimal tools in
which it is _possible_ , but never _easy_ or _convenient_ , to do work in
various programming modalities. Polymorphism, namespaces and packages, arrays,
unicode... these things all can be done, but require significant head-standing
and in many cases a lot of boilerplate. Even proto-style OO, which you would
imagine would be Lua's signature feature, involves lots of rigamarole compared
to languages (like Newtonscript) where it is baked in and elegant.

I kept wanting Lua to pick a pony.

Throughout my usage of Lua, I've consistently gone back to Lisp, where I could
change the language to _make_ inconvenient things convenient when I needed to.

~~~
chaoticmass
Maybe it's because I've mostly used Lua as a tool in my homebrew game engine
side project, but what you said about their philosophy of providing minimal
tools where it is possible to work in different programming modalities is one
of my favorite things about Lua.

------
russellbeattie
As a developer, I've always liked languages that stress
maps/hashes/dictionaries/tables/associative-arrays as their man data
structure.

Off topic, but I've always thought that JavaScript and JSON would have been a
lot better without the relatively artificial distinction between objects and
arrays. Now that there's object shorthand and destructuring, this will never
happen, but it was a nice thought for a while.

~~~
chmod775
Nobody is stopping you from just using arrays for everything - remember
they're also objects.

    
    
        const a = [2,3,…]
        a.prop = "hi"
    

Now whether that's actually something you _should_ be doing is a different
matter...

~~~
russellbeattie
I was thinking more along the lines of using "objects" for everything like
Lua: let foo = {'a', 'b', 'c'}; .

(Also on my JS wish list: Killing the use of const for anything besides actual
constants.)

~~~
anonytrary
> Killing the use of const for anything besides actual constants

You should be using const for _everything_ besides variables that need to get
re-assigned.

~~~
russellbeattie
Heh, I like how you carefully worded that, because normally 'const' would
signify that a variable is immutable in just about any other programming
language in the world. Except in JS, where const was awkwardly jammed into the
language. I guarantee if the spec writers had known how badly const was going
to be abused, they wouldn't have added it.

If you want a strongly typed language, go use C or Java and stop making
JavaScript incomprehensible to all but the most pedantic developers.

~~~
zbentley
> normally 'const' would signify that a variable is immutable in just about
> any other programming language in the world

That's not true in Java, C, Ruby, or Perl (with the 'constant' prag-module),
not really true in C++ (unless you're careful), and . . . generally just isn't
true in most languages.

While a few languages do add recursive immutability properties to their
equivalent of 'const', most languages use the idea of a 'constant' only as a
means of prohibiting _name re-binding_ , not _data modification_. And even
then, many languages that only prohibit const re-binding only do so in an
optional (i.e. it's a warning) way.

Many of those same languages do support immutable data structures either out
of the box or via popular libraries (though in non-compiled language, such
structures often come with a performance cost). They just don't call them
const[ant]s.

Edit: I'm also curious about how you think 'const' is being abused in JS. I'm
not an expert in the language--this is a genuine question; not a veiled
disagreement.

------
imhoguy
To courious web nerds: currently both Nginx and HAproxy have got Lua scripting
modules to extend themselves by implementing e.g. customized filtering,
routing, caching, proxing, storage etc. One can even build a full featured web
app backend in embedded Lua tied to Nginx.

~~~
fosk
Disclaimer: I am CTO of Kong.

If you want to look at a large OpenResty/Lua application from a code
perspective you could take a look at Kong[1] - which is the most popular
public codebase built on top of NGINX/Lua at the moment.

CloudFlare also uses the same stack but most of their codebase is proprietary.

What really makes Nginx/Lua a good combination is primarily LuaJIT[2] support,
a much faster implementation of the Lua VM.

[1] - [https://github.com/Kong/kong](https://github.com/Kong/kong)

[2] - [http://luajit.org](http://luajit.org)

~~~
sgt
What made you choose Lua instead of say, embedding Python into nginx? I can
think of a few good reasons like embeddability, code size, simplicity but it
would be interesting to hear your take on this.

~~~
minxomat
Can't speak for Kong, but I'm currently working on creating a microservice in
lapis (using moonscript). The speed of the underlying OpenResty server, as
well as lapis ergonomics make this really fun. I guess the only practical
reason I can see is the async everything without clutches approach, as well as
the truly impressive TTFB.

For me, day 1 productivity is really high, comparable to Nancy, the framework
I was using before.

------
upofadown
Of particular note; LuaJIT can pretty much directly call C libraries. From
here:

* [http://luajit.org/ext_ffi.html](http://luajit.org/ext_ffi.html)
    
    
        local ffi = require("ffi")
        ffi.cdef[[
        int printf(const char *fmt, ...);
        ]]
        ffi.C.printf("Hello %s!", "world")
    

So while, say, Python is "batteries included", LuaJIT is "all the batteries".

Using C libraries in regular Lua is fairly easy as well, but LuaJIT is
exceptional.

~~~
Sean1708
> So while, say, Python is "batteries included", LuaJIT is "all the
> batteries".

Do you say this because LuaJIT can call shared libraries dynamically? If so,
what about ctypes[0]? Is LuaJIT's FFI just easier to use, or does it do
something that ctypes can't?

[0]:
[https://docs.python.org/3/library/ctypes.html](https://docs.python.org/3/library/ctypes.html)

~~~
ufo
The two most appealing features of the LuaJIT FFI:

1) The syntax. To specify the types of the FFI functions you you use a DSL
that is similar to C declarations. Often you can copy paste function
prototypes from ".h" files with very minimal modifications. With ctypes you
need to specify the types with Python function calls. See "ffi.cdef":
[https://luajit.org/ext_ffi.html](https://luajit.org/ext_ffi.html)

2) Performance. If your code is inside a JIT-compiled trace, LuaJIT FFI calls
have minimal overhead. They compile down to something very similar to what a C
compiler would generate. Ctypes, on the other hand, must "interpret" the FFI
calls which adds more runtime overhead (although admittedly this is also true
for LuaJIT if your code ends up outside a compiled trace).

------
eternalban
Roberto Ierusalimschy, et al., on "The evolution of an extension language, a
history of Lua (2001)" [1], is an interesting read.

The word "configuration" makes repeat performance in that paper, but oddly is
entirely missing from this 2018 paper (by the same authors).

[1]: [https://www.lua.org/history.html](https://www.lua.org/history.html)

------
tonetheman
For game development lua is wonderful. Pico8 uses lua as does love/lua. Both
are wonderful tools to write games.

------
xte
While I'm _not_ a developer nor I like lua I'm curious about a thing: what's
the difference between for instance Guile scheme?

Because they seems closely similar to me and Guile is _far_ more clear and
effective, at least for my programming skills...

------
jokoon
I'm planning to use tcc as a scripting language. Not sure if it's a good idea,
but as long as I don't have to rebuild my code, and as long as it's not slow
like most scripting languages, it sounds like it's good enough.

Although I agree it might be difficult to write scripts in C, in the end it
might just depend on the core functions that are exposed.

I'm not a fan of C, I prefer python, but I guess I will have less issues when
interfacing c code to c++. Python seems quite fat, slow, and too much
different than C++.

------
mitchtbaum
What Lua lacks in terms of performance and broad-scale, low-level library
integration can be made up for within the Rust ecosystem where these exist
along with a well-built Lua wrapper, rLua.

------
epse
The images appear to be dead..

------
Uhrheber
I always perceived Lua of being completely weird, and asked myself why people
would ever want to use it. Especially nowadays, where embedded Python is
everywhere.

~~~
anilakar
On modern embedded hardware, Python is a better alternative in practically
every case. If size and computing power are an issue, either investing in
better hardware or spending time writing C yields better results.

~~~
saagarjha
> On modern embedded hardware, Python is a better alternative in practically
> every case.

How so? Do you have an example where Python outperforms Lua or uses
significantly fewer resources?

~~~
criddell
> or uses significantly fewer resources?

Do you count developer time as a resource?

~~~
scruple
Everyone does, of course, but hardware constraints are a reality, nonetheless.

------
kzrdude
Since the lua authors wrote this article, it can't bring much of a new
perspective on Lua

