
Looking for new LuaJIT maintainers - espadrine
http://www.freelists.org/post/luajit/Looking-for-new-LuaJIT-maintainers
======
catwell
I have mixed feelings about this.

On the negative side, Mike Pall is an impressive individual who has
contributed a lot to the field of JIT-compiled languages, so losing him is
obviously bad for the project. LuaJIT is also a hard codebase to get into, I
myself tried to get into it and gave up (I blogged about it last month [1]).
There are probably still parts of it that only Mike understands, I guess we
will see how much in the next months.

On the positive side, this will forcefully increase the bus factor of the
project and take it out of this situation. And maybe we will see it catch up
with Lua 5.3 :)

[1] [http://blog.separateconcerns.com/2015-06-18-love-hate-
luajit...](http://blog.separateconcerns.com/2015-06-18-love-hate-luajit.html)

~~~
ketralnis
> And maybe we will see it catch up with Lua 5.3

For those that think that LuaJIT's 5.1 compatibility isn't too far behind 5.3,
be aware that Lua changes a lot in its minor versions.

To give you an idea, 5.2 introduced changed how scope works (both some stuff
with lexical/global scope, and by redoing environments), introduced the `goto`
statement, finalisers, and changed how `break` works.

5.3 introduced integers (Lua was previously like Javascript with only one
Number type which was backed by a `double`), and utf8 support.

This means that upgrading a Lua version can be a big deal and break all of the
programs written in it. For instance, redis and WoW both still embed 5.1.

~~~
versteegen
Yes, 5.3 is not a "minor version" increment, it's a major version increment on
5.2. That's how Lua version numbering works.

Adding support for 5.3's integer subtype is probably going to be a very large
project, I would guess that annoyed Mike.

People say that LuaJIT's source is difficult to understand, but I've seen
source for several VMs with JIT compilers and this one certainly isn't the
gnarliest; it's decently well commented. It only took me 30 min to add a new
builtin function, with almost no understanding of the codebase beyond skimming
a couple header files. As long as you don't want the JIT compiler to output
asm for your language extensions, it's almost friendly.

~~~
pygy_
Parrotting Mike Pall: having 64 bits integers require 128 bits for
representing Lua values. This would be bad performance-wise because it would
clutter the CPU cache (as it is the case for Lua 5.3 vs 5.2).

Currently, LuaJIT relies on NAN tagging which allows it to represent values in
64 bits.

~~~
versteegen
I noticed that LuaJIT has the LJ_DUALNUM define, which actually internally
adds an int32 subtype which causes numbers which can fit in an int32 to be
stored as such. It does not change the semantics, it's to avoid floating point
ops on (mainly ARM) processors with slow or emulated FPUs. However, I think it
should be possible to extend it without a huge amount of work to implement Lua
5.3 semantics with a 32 bit (rather than the default 64 bit) integer subtype.
I think it would still be useful to a lot of people, personally I would love
to have it.

Increasing TValues to 128 bits would be very undesirable, but since luajit
already supports 64 bit ints through the ffi library I don't see why they
couldn't be stored as boxed udatas/cdatas in the same way.

------
pierrec
This is a good initiative because LuaJIT development has been mostly halted
since May 2014, when the following note was added to the LuaJIT Open
Sponsorships page:

 _" The main LuaJIT author (Mike Pall) is working on unrelated projects and
cannot accept new sponsorships at this time..."_ [1]

Since then, Mike has only made bugfix commits to the project.

I really hope that the features mentioned on that Open Sponsorships page start
happening after these news, and CloudFlare seems like a good candidate to me:
even without knowing how much resources they're ready to offer, I'm willing to
bet it will be an improvement over the current situation.

Everyone was surprised when LuaJIT seized the position of "fastest scripting
language in the west", and it would be awesome if they could manage to keep
that status - or has it been taken over by Javascript already? JS
implementations have been improving dramatically thanks to the competition
between browsers, and V8/SpiderMonkey are implementing features from LuaJIT's
TODO list right now (I'm thinking about vectorized SIMD instructions). A lot
of people use benchmarks as a major factor for choosing a scripting language,
and I'd rather see more projects using Lua/LuaJIT rather than JS as a
scripting language. It will be interesting to see how this turns out. Go
LuaJIT!

[1]: [http://wiki.luajit.org/Open-Sponsorships](http://wiki.luajit.org/Open-
Sponsorships)

~~~
meric

        "fastest scripting language in the west"
    

Ironic - Since Lua's speed is most heavily used in Yahoo! China, Taobao,
Alibaba and programmers hired by Cloudflare to work with Lua worked in those
companies. ([http://agentzh.org/#AboutMe](http://agentzh.org/#AboutMe)) The
use of Lua - openresty with nginx - Nginx is from Russia, too.

    
    
        "fastest scripting language in the east"
    

Might be slightly more appropriate.

But then again - Lua originated from Brazil and heavily used in game companies
in the U.S. Not sure how much they use LuaJIT though.

~~~
mcpherrinm
This is a reference to the trope of quickdraw duels and "Fastest gun in the
west" referring to the western United States in the "Wild West". Not an
east/west of the world like you're thinking.

------
jballanc
Hmm...wonder what this means about the status of the Quad-color bitmapped
arena mark-n-sweep collector Mike was planning for LuaJIT 3.0
([http://wiki.luajit.org/New-Garbage-Collector](http://wiki.luajit.org/New-
Garbage-Collector))?

I remember reading what he had researched so far a couple years back and being
really excited, but it seems like the work was never really continued. I'd be
interested to know if there was some fatal flaw that would've kept the
collector from living up to its potential? or was it just a matter of the
sizable effort required to write a new collector, more-or-less from scratch?

~~~
versteegen
AFAIK, he never moved beyond planning it, unless the GC64 work is also a
prereq for it. E.g. almost no significant updates to the new GC wiki page or
mention of GC in git commits since 2012.

BTW, previous HN discussion of the new GC:
[https://news.ycombinator.com/item?id=4053969](https://news.ycombinator.com/item?id=4053969)

------
deltheil
There is this recent thread[1] where people are looking for resources to help
them understand the project. There are interesting pointers[2] as well as a
part of a PhD thesis[3] that covers it.

[1] [http://www.freelists.org/post/luajit/how-to-read-the-
source-...](http://www.freelists.org/post/luajit/how-to-read-the-source-codes-
of-luajit) [2] [http://www.freelists.org/post/luajit/how-to-read-the-
source-...](http://www.freelists.org/post/luajit/how-to-read-the-source-codes-
of-luajit,1) [3] [http://www.freelists.org/post/luajit/how-to-read-the-
source-...](http://www.freelists.org/post/luajit/how-to-read-the-source-codes-
of-luajit,3)

------
felixgallo
Mike Pall is an international treasure whose work on LuaJIT has been without
parallel in the embedded/gaming world. I hope this move is born of desire and
moving on to great new things rather than any type of necessity.

~~~
CyberDildonics
I've never heard any actual stories of LuaJIT being used in games as opposed
to stock Lua, but I know it was ported to power processors. Have any big games
claimed to use LuaJIT? It would certainly make sense if the memory and gc
pauses were no worse than Lua.

~~~
jpfed
No _big_ games that I know of, but it's certainly used in games; the LOVE
engine embeds LuaJIT.

------
CyberDildonics
Can anyone actually fill Mike Pall's shoes? I always got the impression that
what he was doing came from a very rare combination of JIT and low level
optimization knowledge.

------
buserror
I'm a big fan of Luajit, I think it's ease of use of FFI makes it fantastic to
integrate into bigger projects; all that 'glue' that I haven't had to write !!
Hope someone pick it up; I can understand he wants to move to other things,
and sometime a projects becomes a bit too 'big' and you can feel a bit
shackled to it after a while...

------
dochtman
Wonder what Mike Pall will be up to next...

~~~
logicchains
Maybe he'll find some other language to make an order of magnitude faster.

~~~
dman
Hoping he picks python :)

~~~
CyberDildonics
I think he has written about why Lua was so good for JIT compilation and in
doing so talked about python. I don't remember specifically what he said, but
he had analyzed other languages and there were hurdles that kept them from
being able to achieve the same speed as LuaJIT.

~~~
georgerobinson
I'd like to see cPython without a GIL first. It would be really nice to take
advantage of multi core CPUs without having to use subprocesses.

~~~
vbit
What's the disadvantage of subprocesses?

~~~
ketralnis
There are plenty, but here's an easy example: sockets. File descriptors are
shared across process forks. So let's take a typical web application that
does:

1\. Boot, setting up system wide resources like DB/memcached connections

2\. Start doing work

Then any process forked after step #1 is sharing those sockets and can
accidentally corrupt the work that the parent process is doing. With threads,
you can share the connections through a pool that can be simply mediated
within the process.

Yeah, you can design around it. But with threads, you don't have to.

Here's another: serialisation. If you want to parallelise any processing that
builds up any complex data structures, you have to serialise them between your
parent/child processes which can make it hard to keep the "richness" of the
objects like interdependencies between processing runs. In addition, now your
processing bottleneck becomes how fast your single-threaded parent process can
deserialise the responses (which in Python in particular can really be quite
slow).

And not everything can be serialised. It's not trivial to share sockets with
other processes that have already been started. You can't easily accept a
connection in one process and pass it to another to do the work and send the
response (it's possible, just not as easy as passing an integer around like
you can do between threads).

~~~
renox
Are you talking about Python limitations? Because for 1) if you can put your
sockets 'close on exec' no? For 2) how about using shared memory?

~~~
ketralnis
> Are you talking about Python limitations?

Yes, but:

> for 1) if you can put your sockets 'close on exec' no?

Yes, if you're the one that opened the socket. If you're using a library that
opens that socket for you (e.g. a memcached or postgres library in my
example), you don't necessarily have access to the socket itself. And now
instead of children corrupting the parent's socket, you just have children
with closed sockets. So you still have to solve the inherited-resource-sharing
problem, FD_CLOEXEC isn't magic fairy dust.

> 2) how about using shared memory?

Possible but has its own downsides (you can't lock with regular posix mutexes,
you have to make sure that the memory you're sharing is all in the shared
block so you can't just use malloc, which means that you can't share objects
that you don't control the allocation of, like, say, libraries' objects).

Yes, these are solveable problems. There are lots of other ones, these are
just easy examples. Like I said:

> Yeah, you can design around it. But with threads, you don't have to.

