
My love-hate relationship with LuaJIT (2015) - walterbell
https://blog.separateconcerns.com/2015-06-18-love-hate-luajit.html
======
williamaadams
I love commentary on LuaJIT, it's like a discussion of the subtleties of an
acient game whose practitioners understand and enjoy, and everyone else
ignores.

My experience with LuaJIT has similarly been love, not much hate though. My
lack of hate stems from the fact that I don't actually use it in production.
I've used it as a learning and teaching tool for the past 4 or 5 years.

What I've learned: Co-routines combined with traditional async IO can make for
some very fast IO bound programs.

JIT based FFI for interop is the cats meow

The guts of operating systems can be easily explained using such a simple
language.

I went as far as creating a collaborative 3D modeling program (ala OpenSCAD)
using LuaJIT.

I've created an entire Windows API wrapper (TINN).

I've explored the guts of Linux (using ljsyscall), and done exploration of
various bits and pieces (lj2procfs).

I've been able to explore and play with any manner of API from dozens of
OpenSource projects and otherwise, simply because LuaJIT (and specifically the
FFI) make it possible to quickly wrap them up in such a way that exploration
with script is easy.

I never worry about the death of LuaJIT, just like I very rarely look at the
guts of my machine that's been running since 2009. The various parts may no
longer be supported, but somehow my machines manages to continue to operate.
Eventually I'll build another one and move on.

With recent changes to TypeScript, and the emergence of the Chakra engine, I
might be tempted to make JavaScript my new thing, but they lack FFI, which I
really love. I wish there were a solution for that.

Go has co/goroutines, which I really like, but I don't think their C interop
story is particularly compelling, and not quite a scripty thing, which I find
to be fairly useful.

I was originally driven to LuaJIT because PUC Lua had a flaw in their OpenGL
binding, and it just wasn't robust enough for my needs. I find no value in
their current versions, because FFI seems to be the most valuable thing to me
I guess.

So, since it's a play thing for me, I find no other language as compelling or
useful, for my needs. Some day, I'll use one of the language tool kits to
convert my most interesting Lua code to another language, hopefully one with a
good FFI. Until then, I'll keep using it.

And, if you've been around the industry long enough, you realize one language
won't do it all for all time. I've long since given up on Objective-C (from
NeXT days), I no longer write 6502 assembly (although still very popular
apparently). I dove into C++ in the past couple of years, and came out with
hives! Rust and Go are on the horizon, and who knows what's after that.

~~~
lukego
Wow! Hard to believe I had never stumbled on your blog until now.
[https://williamaadams.wordpress.com/category/luajit/](https://williamaadams.wordpress.com/category/luajit/)

~~~
williamaadams
heh... I've written a bit about our favorite abandoned language ;-)

------
sebcat
My perception of LuaJIT is that it's Mike Pall's creation (and what a creation
at that!). He stepped down a while ago[1] and since then, at least to me, not
much seem to have happened with LuaJIT. CloudFlare was in talks of project
governance [2], and Cisco has pushed some MIPS support, but I havn't heard
much else. There was a beta release in march this year for 2.1, but that's it.

The fork between PUC Lua and LuaJIT mentioned in the article is a major point
of concern. For good and bad, PUC Lua is not known for backward compatibility.
PUC Lua is at 5.3 and LuaJIT targets 5.1. There have been some breaking
changes since then.

Without a strong community built around LuaJIT, or a corporate backer
investing resources in it, I fear that it will die out eventually. I would be
hesitant to implement something that uses LuaJIT with the project looking the
way it does to me. I would love to be proven wrong though.

[1]:
[https://news.ycombinator.com/item?id=9981802](https://news.ycombinator.com/item?id=9981802)
[2]:
[https://news.ycombinator.com/item?id=10078848](https://news.ycombinator.com/item?id=10078848)

~~~
jgrahamc
We still would like to hire someone to work on LuaJIT. We've found it very,
very hard to find anyone suitable. There are few people who understand LuaJIT
internals and few who want to get into it.

If there is someone reading this who wants to work on LuaJIT full time and
thinks they have relevant background then please contact me.

~~~
lukego
I am confused about why you guys haven't already hired Peter Cawley,
Vyacheslav Egorov, and/or Thomas Fransham.

~~~
chrisseaton
I believe Vyacheslav Egorov already works at Google on LuaJIT.

~~~
lukego
No. He works at Google, and during 2015 he temporarily provided support to
internal LuaJIT users there and made many contributions in the process, but
working on LuaJIT is not his job. (The world world be a better place if it
were!)

Source:
[https://github.com/LuaJIT/LuaJIT/issues/45#issuecomment-2340...](https://github.com/LuaJIT/LuaJIT/issues/45#issuecomment-234073487)

~~~
x0x0
And for companies not named FaceBook, you probably have about 0 chance of
hiring him away from Google where engineers who are really good at building
jits are prized ($$$). I am an acquaintance of a very senior, very very good
jvm engineer who was offered a ludicrous salary by G.

~~~
walterbell
Is Mike Pall independent? I assume many companies have tried to hire him.

~~~
x0x0
I was discussing Vyacheslav. I assume Mike Pall can work for msft/goog/fb if
he wishes to.

------
hesdeadjim
My love hate relationship is entirely with the language itself. It's
incredibly useful as a game scripting language and is extremely easy to bind
native functionality to, but man, the language itself rivals JavaScript for
quirks.

1-based indexing is just a silly departure from decades of ingrained thinking,
dictionaries masquerade as lists but can unintentionally become dictionaries
if you mistakenly put a non-numeric key in them, and the only inheritance is
prototype-based "classes" that are implemented via gross metatable hacking.
Library support is also weak.

I will give it credit though for a robust coroutine implementation.

~~~
chubot
Looks like you got some down votes but I entirely agree. It isn't as bad as
JavaScript, but turning typos into nil rather than raising an exception or
having a compile time error is also annoying, in addition to the other things
you mentioned.

I really wanted to like it for the small and simple implementation, but the
language isn't pleasant IMO.

~~~
lukego
> turning typos into nil rather than raising an exception or having a compile
> time error is also annoying

So true. However, you can disable this behavior and then just forget about it:

[http://metalua.luaforge.net/src/lib/strict.lua.html](http://metalua.luaforge.net/src/lib/strict.lua.html)

------
sobersombrero
I've used luajit. It's unsupported software. If it works, great. If not,
you're screwed. The original Lua may be ten times slower than luajit, but
anyone can understand and debug it. That's worth something.

It's a shame luajit never created a community beyond its creator. FFI sure was
nice.

~~~
mraleph
If one heads over to GitHub to check some recently closed issues and recent
commits one will discover that there are people who can understand and debug
LuaJIT issues.

LuaJIT is not magic, it's a software. Very neat and tightly written but it's
still just a bunch of C/assembly/Lua code.

Written by one human, and thus other humans can understand it too.

~~~
sobersombrero
In _theory_ you're right. But it's dense terse code with very little
documentation. Some people can fix obvious problems around the edges, but no
one would tackle a serious refactoring, porting to more recent version of Lua
or adding a new feature.

------
bachmeier
As appealing as LuaJIT was and is, I could never bring myself to write much
code for it. Even setting aside the possibility of Mike Pall giving up on the
project (which turned out to be correct) the lack of a community around the
language meant there would be difficulty getting questions answered and few
third party LuaJIT packages. Bugs in particular were a major concern: if Mike
Pall didn't find my bug report important, I'd be out of luck.

And as interesting as it is to have a small scripting language be able to run
code so quickly, there are new languages that deliver the same benefits.
Nothing really stands out about LuaJIT to make me feel it is a language I
should use.

~~~
machuidel
Which new languages?

~~~
bachmeier
The benefits I was referring to are being able to write fast, correct code
easily. Other languages that allow one to do that include D, Go, Clojure, and
Scala. If you want a scripting language, you're stuck with LuaJIT for now, but
at least to me that's not an important distinction for anything I'd want to do
with LuaJIT.

LuaJIT comes with warnings like "Callbacks are slow" and "The following
operations are currently not compiled and may exhibit suboptimal performance,
especially when used in inner loops:" which pushes it further down the list
for me.

------
jdonaldson
The Haxe language targets Lua now [1], and some recent commits I've made
enabled 5.1/LuaJIT support. Haxe syntax is significantly different than Lua
syntax, but it's closer to ECMAscript, and most likely more familiar to the
folks here. If the language syntax quirks are a barrier towards targeting
LuaJIT, then check out Haxe.

[1]: [https://haxe.org/blog/hello-lua](https://haxe.org/blog/hello-lua)

------
etiene
I love LuaJIT, but what it needs right now is some sort of Lua5.3
compatibility. Despite being an excellent piece of tech I think this current
state causes some damage to the Lua community because often developers will
write code in a Lua5.1-style so it is compatible with both PUC-Lua and LuaJIT
interpreters. This has made the adoption to newer Lua versions more difficult.

------
jamesu
For me one of the big problems I have with luajit is using it in 64bit
applications. luajit internals need to exist within the first 4gb of memory,
and in some situations this can be a PITA to properly satisfy. There are
solutions, but none I feel too comfortable with using especially cross-
platform.

Given luajit development seems to be stalled, I don't have much confidence
this will ever be fixed.

~~~
corsix
Try building (the `v2.1` branch of) LuaJIT with `CFLAGS=-DLUAJIT_ENABLE_GC64`.
This flag is a relatively recent addition [1], and didn't have JIT support on
x64 until really very recently [2], but it should free you from the 4gb
limitation.

[1]
[https://github.com/LuaJIT/LuaJIT/commit/a92e73023353e59405eb...](https://github.com/LuaJIT/LuaJIT/commit/a92e73023353e59405ebfdb2837b5742d17946a5)
[2]
[https://github.com/LuaJIT/LuaJIT/commit/2868715d80b6ac497a7f...](https://github.com/LuaJIT/LuaJIT/commit/2868715d80b6ac497a7f08393ec325b60d71df8d)

~~~
chanks
Looking at your PR for it
([https://github.com/LuaJIT/LuaJIT/pull/149](https://github.com/LuaJIT/LuaJIT/pull/149))
it sounds as if there are a few issues remaining, but I'm not enough of an
expert to know how problematic they are in practice. Do you think that this
feature is stable enough for production use?

------
coldtea
> _Beyond the purist ideal of understanding the whole stack, this has
> practical implications. When you find a bug in LuaJIT, understanding it and
> fixing it is terribly complicated._

Well, how often people fix bugs in the compiler they use (and haven't written
themselves)? Doesn't seem as much a "practical" consideration as the author
says...

> _Another issue is that PUC Lua and LuaJIT are diverging. LuaJIT implements
> Lua 5.1. Lua 5.2 code can be made to work as long as it does not use _ENV,
> but code that leverages the new features in Lua 5.3 is not supported at all_

Well, not as much diverging as in "going to different directions" as that
LuaJIT implements an older version. Isn't it supposed to eventually catch up?

~~~
catwell
Well, I have hit three different bugs in five years using LuaJIT, two of which
I was the first to report. You cannot consider it as stable as, say, GCC.

Regarding your second point: Mike Pall had no intention to catch up
completely. I don't know how much that has changed under CloudFlare
leadership.

~~~
coldtea
Thanks, I'd expect it to have some bugs (it's a much newer project than GCC,
and even Python or Perl have lots of open bugs), but was missing the second
information that Mike gave up on it. Last time I checked (a few years ago) it
was still going forward.

------
scythe
The problem is that LuaJIT is an engine without a transmission or steering or
braking system (this is a tortured analogy). It _makes sense_ for PUC-Lua to
have no standard library and focus on embeddability. It _does not make sense_
for LuaJIT to do the same thing. LuaJIT's use case looks more like Torch --
scientific computing where a lot of stuff is written in C but you still need
speed for the free-form parts and your target users are in a hurry -- and you
need standard libraries for that. LuaJIT can compete with Go and C# on a speed
basis, but not a usability basis.

Yeah, the actual compiler could use a maintainer, but who looks at LuaJIT and
thinks "clearly, the weak point is the compiler!"?

------
Dylan16807
Does anyone know what percentage of LuaJIT bugs are in the interpreter? You
can still get a large speedup with it while avoiding the most complex code.

Edit: Also what's this about "a machine that does not support LuaJIT" that's
apparently x86-based?

~~~
nmcveity
Not sure about the machine linked in the article, but sometimes the operating
system might prevent you from marking heap allocated pages as executable.

However, in that case you are only prevented from using the JIT compiler and
should still be able to take advantage of the high-speed interpreter included.

~~~
catwell
I wrote that article, just found out it was on the front page of HN today.

The machine linked in the article can not even run the unmodified interpreter
(or at least it could not at the time) because its CPU does not support the
CMOV instruction.

Regarding the bugs: I do think most of them are in the JIT and not the
interpreter. Even using a lower optimization level that the default fixes a
lot of them.

~~~
david-given
What's the processor in the Lima? The website has no technical information on
it whatsoever (even under the 'Tech Specs' sections). Genuinely curious; I
thought all x86 processors made these days supported CMOV.

~~~
catwell
It is a CPU from RDC (Taiwanese manufacturer). It is 486-compatible but misses
some instructions from later Pentium sets.

~~~
david-given
I'm thoroughly bemused as to why they picked it --- unless you're aiming at
some sort of binary compatibility, and given that it won't run Windows I have
no idea what it might be compatible _with_ , surely one of the ARM or MIPS
stable would be more appropriate for this kind of device?

...over here, in _this_ cardboard box, I have an old SheevaPlug: a 1.2GHz
Marvell ARMv5 in a plug-sized box with some flash, an SD card interface,
ethernet, and some USB ports, which appears to be pretty much the same device.
(Of course, it's the software that's the interesting bit.)

------
eggy
I wonder what programs he was writing with Lua in Minix3 if he even was using
them together? He mentions the stuff he wrote for work, and the hobby stuff
later in Lua on Linux.

I also used Minix to learn about OSs back in 1991 by installing Minix 1.5 on
my Amiga. Sad to say after spending an all-nighter getting it working, I
didn't really dive in the way I had intended much after that. Tanenbaum's book
was a real eye-opener for me. The source code and a textbook. I haven't seen
much like it today. Still, I learned a lot about computers in general doing
this, and would recreate the experience in my second dual-boot system, a Mac
PowerPC running MkLinux in 1997!

There's something to be said for reading source code, and recompiling small
OSs even if you just touch upon it. XV6 was re-written, and the source is <
10K LOC [1]. The course book is about 100 pages and the link also includes the
source code in PDF to follow along.

I am also thinking on installing Minix3 on my dedicated linux notebook, but I
am afraid I might botch it because my current system is UEFI/SB=off. I think
you can only use BIOS/MBR with Minix3, although somebody has posted how to
load it from a LiveUSB with Qemu, which you ditch after the install.

[1]
[https://pdos.csail.mit.edu/6.828/2016/xv6.html](https://pdos.csail.mit.edu/6.828/2016/xv6.html)

------
mitchtbaum
"Ravi is a derivative/dialect of Lua 5.3 with limited optional static typing
and an LLVM powered JIT compiler."

[https://github.com/dibyendumajumdar/ravi](https://github.com/dibyendumajumdar/ravi)

~~~
catwell
Ravi is interesting and well known in the Lua community, but it is far from
the performance of LuaJIT. I don't think a LLVM-based JIT compiler will reach
that level of performance anytime soon.

------
ansible
For people who like LuaJIT's FFI, there is this library that brings that same
style to PUC Lua:

[https://github.com/jmckaskill/luaffi](https://github.com/jmckaskill/luaffi)

It isn't terribly fast though.

~~~
justincormack
Its buggy too. Really needs someone to spend some time on it fixing the bugs,
but there clearly is not the demand. Actually I am more interested in a
maintainable static bindings generator.

------
labrador
As someone who doesn't know Lua and after reading this article, can it
properly be said that a scripting language is fast if you write most of your
time sensitive code in C? Why not just C?

~~~
rogerbinns
I do python where your question applies too.

It is significantly quicker to write code in a higher level language, since
the primitives and libraries do more per line of code. The cost is that it
doesn't run as fast. But consider these two scenarios:

1 It takes 2 hours to write the code, and 1 hour for it to run.

2 It takes 2 days to write the code, and 1 minute for it to run

Note that "write the code" includes the actual writing, as well as debugging,
testing etc. The first scenario is the most immediately productive. But the
more you have to run the code, the more the second scenario applies.

Which is why you combine the approaches. Write the parts that aren't
performance sensitive quickly in your high level language, and take longer on
the performance parts in C. That works better than exclusively approach 1 or
2.

~~~
labrador
Thanks, I think you make a good case for using a language like Lua or Python
with an excellent C interface. In some sense, writing everything in C could be
a case of premature optimization.

~~~
rogerbinns
Another benefit of Lua/Python is that it is considerably easier to write
testing and debugging versus C. They support introspection and similar
techniques. A very typical example is something I did a while back.

I wrote the processing in Python first. The core was 10 lines of code and took
about 5 minutes to run on a unit of data. It took about an hour to write, and
was correct.

It then took two days to rewrite in C and was 400 lines of code, executing in
a second or so. Much of the time was finding and fixing minor bugs. But having
the "correct" Python code made it very easy to feed the same test data to both
implementations and verify the output. Writing test code in C would have taken
even longer than just the algorithm!

------
kensai
Since someone mentioned it, how slower is PUC Lua in respect to LuaJIT?
Compromisingly slower? Are there metrics out there or is it a relatively
subjective feeling?

~~~
justincormack
PUC lua is probably the fastest traditional scripting language. LuaJIT is much
faster, in particular if you are interfacing with C, as the ffi has zero
overhead, while the traditional Lua API is relatively slow. Also the
performance is noticeable if you are doing a lot of work in the language too,
generally as fast as C code for many use cases if you tune the code.

~~~
petre
> PUC lua is probably the fastest traditional scripting language

That really depends what you are testing for. For string ops is not the
fastest. For recursive Fibonacci (inefficient) LuaJIT is faster than D and Lua
is faster than Perl.

[http://raid6.com.au/~onlyjob/posts/arena/](http://raid6.com.au/~onlyjob/posts/arena/)

