
State of the Lua Ecosystem (2013) - fuzzythinker
https://speakerdeck.com/catwell/state-of-the-lua-ecosystem
======
fuzzythinker
To get conversations started..

Lapis[1] on Openresty[2] is one of the most simplest, documented, and
performant full web framework stack out there, yet almost no one knows it.

Hello World Open[3] - 2.5k teams, only 17 ( < 0.7% ) teams used Lua. This is
an open competition, which means people can use their favorite language
instead of Java, and less than 1% uses Lua.

The lua community really need to start doing something about this.

1\. [http://leafo.net/lapis/](http://leafo.net/lapis/) 2\.
[http://openresty.org/](http://openresty.org/) 3\.
[https://helloworldopen.com/teams](https://helloworldopen.com/teams)

~~~
_pmf_
> The lua community really need to start doing something about this.

Why? A small community has advantages, too.

~~~
loup-vaillant
Could you name three?

~~~
otikik
\- Can make backward-incompatible changes (see: Javascript, Python)

\- Less trolls & nasty people in general.

\- All the advantages of a niche language (I got my current job because no one
else in my country knows Lua :P).

~~~
catwell
Advantage 1 is the main reason why today's Lua is not as complex as today's
JavaScript, despite Lua being older.

~~~
ANTSANTS
It also significantly helped that Lua's main users could easily use an older
implementation if something they depended on/didn't feel like fixing was
removed. In contrast, breaking backwards compatibility for Javascript would be
unthinkable.

OTOH, there is real fragmentation between 5.1 and 5.2, so not even Lua is
immune to the downsides of breaking backwards compatibility.

------
astrobe_
To me, the fundamental problem is that Lua is designed to be an embedded
extension language. But the vast majority of the publicly available extension
modules are provided as shared libraries. If you have embedded Lua in your
application, you are forced to use dynamic linking to use those modules. And
if you use an OS that doesn't gracefully support dynamic linking (like uClinux
on some platforms), you are screwed.

A second problem is that those modules tend to be kitchen sinks. For instance
if you want UDP networking, and pick one of half-dozen modules that provide
that, nearly all of them also provide other things you don't need. Why do
people choose Lua? Small footprint, often. Why then provide bloated modules?

In other words, the module ecosystem only addresses one use case of Lua - as
an extensible scripting language - and misses all of the others. And the end
result is that it actually has Zero utility for most Lua users.

~~~
catwell
A lot of modules are pure Lua. They may depend on a few critical C libraries
though. But with just LuaSocket, LuaFileSystem and maybe LPEG you can use most
modules.

There are several ways to build those critical Lua modules statically, one
being
[https://github.com/stevedonovan/luabuild](https://github.com/stevedonovan/luabuild)
Even without those, it is not very hard to modify the makefiles to build
statically.

About the bloat: this is certainly true for some modules and not others. As I
said in the talk, everybody uses Lua differently and nobody will use _all_ the
modules and tools available, but there is still some amount of code that can
be shared.

I mean: you could say exactly the same thing about C, and it still has a lot
of libraries.

~~~
virtue3
the Lua interpreter is so small and simple that if you really needed libraries
and couldnt' dynamically link them in you could easily compile them into the
lua interpreter and push the module onto the global table :/. It's really not
hard.

------
fuzzythinker
Video:

[http://2013.capitoledulibre.org/conferences/lua-
workshop/sta...](http://2013.capitoledulibre.org/conferences/lua-
workshop/state-of-the-lua-ecosystem.html)

~~~
bigtunacan
Thanks for sharing. I love Lua. First worked with it about 10 years ago. Still
saddens me that it hasn't seen more widespread adoption.

------
vertex-four
I think that most here are entirely missing the point of Lua. "The Lua
ecosystem" isn't a thing that makes sense. Lua isn't a standalone language -
it's an extension language, and is specifically popular because it doesn't
have much in the way of a standard library or built-in assumptions about its
environment.

If you can't make any assumptions about the environment, you can't have an
ecosystem, because everybody's doing something slightly different. Some might
be running every Lua script in a separate environment, some might have every
Lua script bundled into the same environment, some might be running Lua on a
separate thread so blocking is fine, some might integrate Lua into their
application's event loop library. How are you supposed to build a library that
serves even some reasonable amount of the userbase's use cases?

~~~
ANTSANTS
It goes beyond that, though. One of Lua's goals is to be implemented in 100%
portable C, to the extent that it has no platform-specific #ifdef's (no
#ifdef's at all, actually). This means that to use a lot of incredibly basic
functionality supported on all relevant operating systems (like, as mentioned
in another comment, _changing directories_ ), you need (for vanilla lua and
luajit on platforms that don't support the ffi) a C library dependency, or a
wrapper script (for luajit on the other platforms). This is pretty silly; any
user of a platform without a filesystem would surely know that chdir would not
be supported.

Beyond that kind of indefensible omission, there is much ecosystem
fragmentation and duplicated effort that could be solved by standardizing
tools like a package manager without _requiring_ you to install it on embedded
targets.

~~~
catwell
This "indefensible omission" is solved by using LuaFileSystem. Not having it
in the core language is not such a problem. Another example is sleep(), which
is not portable. You can find a sleep function in libraries like LuaSocket and
luaposix.

Fragmentation, however, _is_ a problem. The Lua authors have often refused to
"bless" (term used by the community when discussing this) libraries. Roberto
Ierusalimschy even said during a Q&A he'd rather "bless" a group of people to
do it instead of him.

The problem is, as several people said in this thread, that it is very hard to
write a library that fits every use case. Even something almost "standard"
like LuaSocket is not used by everybody.

Some people in the community (including me) are trying to reduce the
fragmentation. The authors of the two main ways to manage "packages" (LuaRocks
and LuaDist) are working together, etc. But I think we will never manage to
standardize on One True Library for everything.

~~~
Houshalter
The os.execute function does all this. os.execute("timeout "..seconds) for
example. More complicated for file system operations, but it's possible.

This is significantly less than optimal though.

~~~
catwell
And not portable (or safe, for things like path manipulation). But sure, you
can still use os in small scripts.

------
lxe
Lack of userland packages isn't necessarily a bad thing. A well-documented,
complete standard library (like in Go, for example) takes precedence over
community IMHO.

Even as a Node.js dev, I don't understand the obsession with creating userland
modules, and the value in modularity. Yes, a large library of modules
correlate with community involvement, but it does not necessarily mean
quality.

~~~
ANTSANTS
Lua's designers purposefully try to keep the language's standard library
vanishingly small to reduce portability issues and the footprint for embedded
targets. So for Lua, if no one makes user libraries, you don't have _any_
libraries.

~~~
acqq
The minimalistic core language is OK if the authors don't want to maintain
more. What I miss is some incarnation of the language that would still be more
convenient than Perl or Python (by being a single executable) but have enough
"basic libraries" inside. The last time I investigated I failed to find
something like that.

Once you have Lua with the tons of the packages in different files with the
package managers and stuff you lose the benefit of having a small-size-of-the-
executable language. I understand that the language author doesn't want to be
a part of that.

~~~
darkFunction
Have you tried [http://luapower.com/](http://luapower.com/)

~~~
ANTSANTS
I don't like luapower's approach to package management. Basically it stuffs
the contents of packages in the base of a single directory:

    
    
      luapower/lib.lua
      luapower/additional-files-for-lib/stuff.lua
    

Instead of completely encapsulating each package in its own directory:

    
    
      luapower/lib/lib.lua
      luapower/lib/stuff.lua
    

So instead of being able to just cd to luapower and git clone repo, you need
to use a silly wrapper script that overlays the repos with each other. All of
this presumably because the author wanted to type `require "lib.lua"' instead
of `require "lib/lib.lua"'

~~~
acqq
I like it, I'd like even one step more, of making a ZIP of all of the
packages. I never modified in place any Perl or Python module, and I don't
expect I'd do it with Lua, so having a small portable file set of just a few
self sufficient files I could copy to the another machine is my real wish.

~~~
ANTSANTS
But it's arguably easier to package the other way I described, just stick all
of the relevant directories into a zip file, instead of needing to manually
pick out the "lib.lua" files from the base directory. Literally the only
benefit of this approach is saving a few characters on require statements.

For anyone interested in creating a new language, make your equivalent of
`require "lib_directory"' do the equivalent of `require
"lib_directory/init.lua"' or `require "libname_directory/libname.lua"' so we
don't have this issue.

~~~
acqq
Do you actually do that manually? Why isn't there a script for that? It should
be a part of the automated process. Much better doing it once as an automatic
step than writing everywhere in your sources:

somethingbig/somethingbig

every time you use somethingbig. And that step should also zip the modules, I
believe.

The behavior of require, as I'd like it:

require X first looks in the directory X, for module X. If there is no such,
it opens modules.zip and reads module X from there. If internally in the ZIP
there were a subdir for every module, I don't care, as long as I don't have to
type that fact every time.

modules.zip would be constructed automatically. Whoever wants to tweak some
specific module can unpack it in the separate directory.

~~~
ANTSANTS
Having reread the Lua manual, `require "lib"' can easily be made to execute
`require "luapower/lib/lib.lua"' by adding "luapower/?/?.lua" to LUA_PATH.
This is the way the standard /usr/local/lua libraries are require'd. Now I
cannot think of a single explanation for why luapower works the way it does.

~~~
acqq
I'd prefer modifying require to not need a new entry to the ENV var for every
used module. It is a huge waste of ENV.

~~~
ANTSANTS
[http://www.lua.org/pil/8.1.html](http://www.lua.org/pil/8.1.html)

> To determine its path, require first checks the global variable LUA_PATH. If
> the value of LUA_PATH is a string, that string is the path. Otherwise,
> require checks the environment variable LUA_PATH.

Even if you needed to modify an environment variable, I still think that would
be preferable to creating yet another package manager and fracturing the Lua
userbase further.

~~~
acqq
As a user, I want to "just use it" and the binary to "know" where the stuff of
the module is without me having to hold its hand. Specifically, I don't want
to know about LUA_PATH unless to use it to point to the single dir where I put
my own modules.

~~~
ANTSANTS
Your concern doesn't make any sense. Any package manager _has_ to modify
LUA_PATH in order for require to work with it in that way (unless it installs
libs in /usr/local/lua, which requires admin privileges and isn't portable to
Windows). luapower surely does this already, adding "luapower/?/?.lua" to the
path instead of "luapower/?.lua" is a one-line difference. It's completely
tangential to the criticism I gave.

------
zserge
So, LuaRocks contains just 330 packages and doesn't look like the best package
manager for Lua (and they also have a terrible website as for me).

If a community here wants to make things better - let's meet somehow and
discuss it.

Then if, say, each of 5 Lua developers converts 1 package per day in their
free time then in about three months the whole list of packages will be
converted.

Anybody?

~~~
catwell
Speaker here, funny this got picked up by HN over a year after my talk.

Why makes you think LuaRocks is not the best package manager for Lua? As far
as I can tell, it is.

Things have changed since last year, too! My suggestion of slide 40
("programmatic releases for LuaRocks") has been implemented by merging with
MoonRocks, which is now the official source for packages.

This has accelerated the rate of packaging: as of today there are over 500
packages available.

That being said, if you want to make things better there sure are _lot_ of
things that could be done. If you want to contribute the best place to reach
the community is probably the lua-l mailing list and the more specific
luarocks-developers mailing list.

~~~
zserge
Oh, thanks for a prompt response! I personally got stuck with LuaRocks and
cross-compilation for our MIPS board. Probably it was lack of documentation or
lack of my knowledge. So instead I manually crafted most of the packages we
needed either from scratch or by writing custom makefiles for existing ones
(that was more than a year ago).

The whole idea of rockspec looks very nice to me and I totally support it.

But can I keep a global aka system rocktree and a per-project local trees?
(Pardon my ignorance, I really don't know that). Per-project trees make
deployment a lot easier.

Also, why can't "luarocks" or "moonrocks" be a single executable/script? Isn't
it easier to get user just a download link instead of: "download it here,
unpack, configure, make build, make install, now use it to install moonrocks"?
(Also, I'm not saying that it's a very complex procedure, I know most people
can do it easily, but that leaves an impression that the LuaRocks project is
not very user-friendly).

~~~
catwell
Note: I am not the author of LuaRocks
([https://news.ycombinator.com/user?id=hisham_hm](https://news.ycombinator.com/user?id=hisham_hm)
is), just a user.

Regarding distribution, what OS are you talking about? LuaRocks is available
in Homebrew for OS X and as a package in most Linux distributions. Windows
build looks a bit more painful but I don't have experience with that OS.

Regarding per project trees, there is no such thing as what exists in Python
with virtualenv, but you can certainly do this:

    
    
        $ mkdir my_project
        $ cd my_project
        $ mkdir rocks
        $ luarocks-5.2 --tree="./rocks" install haricot
    

This will install the latest version of Haricot and all its dependencies (in
this case LuaSocket) in the local tree regardless of whether you have them
system-wide. Note that you can change this behavior
([http://luarocks.org/en/Dependencies](http://luarocks.org/en/Dependencies))
but the default dependency mode is "one".

------
sitkack
Luarocks is a clunky package manager. But its biggest crime is that packages
are neither signed nor are they transported over https.

[http://luarocks.org/en/Documentation](http://luarocks.org/en/Documentation)

That said, _some_ of moonrocks goes over http,
[https://rocks.moonscript.org/](https://rocks.moonscript.org/)

The only excusable way for packages to go over http is if they were signed and
the signing chain is secure.

~~~
catwell
There is basically only one person working on LuaRocks. He is also a PhD
student and the maintainer of htop. Given this I think anything is
"excusable".

Package signing is an important feature to have but it requires resources to
set up. If you want to help:
[http://luarocks.org/en/Mailing_list](http://luarocks.org/en/Mailing_list) :)

~~~
fuzzythinker
Wow, didn't know he also authored htop.

After visiting his site[1], discovered gobolinux[2] too!

1\. [http://hisham.hm/](http://hisham.hm/)

2\. [http://www.gobolinux.org/](http://www.gobolinux.org/)

------
zserge
How does LuaRocks implement the cross-compilation? Lua is widely used on
embedded platforms, so I wonder how easy it will be to just take a random
module from the LuaRocks and build/deploy it for some MIPS or ARM board?

~~~
catwell
Pure Lua modules do not need to be cross-compiled.

Simple C-based modules using the "builting" build back-end will probably work
well with cross-compilation just by setting a few variables (CC, LD,
CFLAGS...).

More complex modules will use the "make" or "cmake" back-ends and you will
need to pass specific options to their build systems, which LuaRocks can do.

See
[http://luarocks.org/en/Config_file_format](http://luarocks.org/en/Config_file_format)
for more information.

~~~
zserge
Thanks! Config file looks like a powerful way to customize builds. But am I
right saying that I will have two configs (for host and target) and I will
have to pass --force-config option to toggle them?

~~~
catwell
No, you can use environment variables to choose at runtime (first paragraph of
[http://luarocks.org/en/Config_file_format](http://luarocks.org/en/Config_file_format))

