Hacker News new | past | comments | ask | show | jobs | submit login
Pallene: A statically typed companion language for Lua [pdf] (inf.puc-rio.br)
100 points by eggy on Sept 21, 2018 | hide | past | web | favorite | 37 comments

A few details:

- Pallene is a friendly fork of Titan [1]. The reason for the fork is that the authors of Titan are iterating fast and adding many experimental features to it. The research group at PUC Rio needed a more stable code base and language definition. Improvements from Pallene will be merged back into Titan, which is probably what actual programmers should use in the long run (as of now neither is ready for production).

- There was a presentation of Pallene at Lua Workshop earlier this month. Slides and video should be available at [2] at some point.

[1] http://titan-lang.org

[2] https://www.lua.org/wshop18.html#Ligneul

The talk was great! I'm incredibly impressed with the benchmarks.

Here is a quick link to Pallene's github repo as well, since it's a bit hidden in the paper: https://github.com/pallene-lang/pallene

Gabriel's slides are already available here: https://gligneul.github.io/luaworkshop2018/

For those of us who are only half paying attention, how does this compare to Terra[1]?

Are you using something (e.g. LLVM) for code generation?

Can you generate binaries that are independent of the Lua runtime?

[1]: http://terralang.org/

Terra uses Lua as a metaprogramming language, while Pallene/Titan use Lua as a scripting language.

Terra is designed for high performance numerical computing. As you mentioned, it has no garbage collector, uses C-like datatypes and its executables are completely detached from the Lua runtime.

Pallene, on the other hand, is all about being able to seamlessly interoperate with Lua at run-time. It is possible to directly manipulate Lua tables and functions from a Pallene program. From the point of view of Lua, Pallene modules can be loaded as a C extension module would, except that Pallene is designed to integrate even better with Lua (sharing its garbage collector and having better C API performance).

As the other commenter pointed out, we are currently compiling down to C. However, it is possible that we could switch to LLVM IR in the future.

I had submitted this to HN because I was trying to evaluate using Terra instead of Zig [1] as my C replacement language. I had already become familiar with a particular dual-language environment called Extempore, [2] for livecoding that has a scheme language, and a c-like, s-expression-based language called xtlang that is compiled. You can mix the two in the same programming session, and it really works for me. And then I came upon Pallene.

I like the idea of having a scripting language like Lua, but the clarity of a typed, compiled, non-GC language instead of C or the templates of C++. Terra ticks those boxes. The mix of Lua and Terra seems to be very flexible too from my early explorations. Is Pallene within an order of magnitude of the speed of Terra given no special tricks are needed to pull off a task. Are there any "bad" things Terra does that Pallene keeps within the better side of Lua. I guess I am asking if Pallene is a better fit for a dual-language system based on Lua than Terra other than maximum numerical computing performance. Thanks!

  [1] ziglang.org
  [2] https://extemporelang.github.io/

I talk a bit about this in a sibling comment. Maybe that helps answer your question?

Pallene uses garbage collection, and shares the Lua garbage collector. The idea is to make as easy as possible to call Pallene from Lua and Lua from Pallene. Meanwhile, Terra only uses Lua at compilation time for metaprogramming and template building. At runtime, if you want to call Lua from Terra (or vice versa) you need to use the C API, just like you would do with C.

I wouldn't really try to compare the performance of Pallene with Terra. They are two very different languages, doing different things.

I would just note, for anyone else coming on this, that in Terra you can also call back into Lua. From an implementation perspective, sure it goes through the C API, but as far as users are concerned, it looks pretty much like a normal function call (except as of the 2016 release you have to cast the Lua function to a specific type, but I don't think this is a terribly big deal).

The bigger issue is that as soon as you call from Terra back to Lua you've made a circular dependency, and so you can't then dump an object file and have it be entirely free of the Lua VM. This for me happens to be one of the big selling points of Terra, because I know that I can write as much code as I want and (usually) can still come out with clean object files to link into any old project on the other side.

I definitely get the appeal of a system which is intended to integrate with the same VM. If it weren't for some of the constraints of my current project, I'd definitely be thinking about it, so I'm glad that people are putting effort into exploring both approaches.

Thank you for explaining it more to me.

If I were just interested in numeric computing (arrays as well), I am assuming without benchmarking Terra and Pallene that Terra would win by an order of magnitude, no? Thanks again!

If you were writing naive code, it probably wouldn't be that much. Maybe a 2to 3 times difference. But the thing with Terra is that you can be even faster than naively-written C if you know what you are doing.

I am trying to explore Terra more for this reason, but Pallene has piqued my interest. My guess from the responses and what the authors of Pallene have stated is that Terra is probably better to stick with for numerical work.

They emit C and call out to the system C compiler. They bypass the normal Lua extension mechanism for performance.

I'm first author of the linked paper and am working on Pallene for my PhD. Feel free to ask any questions here!

Would it be a mischaracterization to say that the generated C is basically the source of the Lua interpreter + the statically compilable functions from the Pallene program?

From the point of view of Lua, Pallene works the same as C

If you want to produce a standalone executable, at the end of the day this executable will contain a copy of the Lua VM.

If you want to produce a library that can be "require"-ed from Lua, then you will produce a ".so" shared object file with just the Pallene functions in it.

Sand boxing is what really sets lua apart imo. Will custom allocators and dynamic code loading be supported? Will generated code implement the debug hook counters?

Pallene has been designed to seamlessly interoperate with Lua, not to replace it. So you can still use Lua when you need dynamic code loading or magic debug hooks.

Roughly speaking, from the point of view of Lua, Pallene is a system language like C. It is good for performance and calling C system libraries, but isn't particularly suited for dynamic code loading and sandboxing via interpreter hooks.

(Maybe trivially answerable by looking through the resources, sorry!) -- will Pallene still support `eval`-like behavior (like with Lua's `load` function(s))?

Don't worry, that is still a fine question. Pallene programs are still able to use Lua's "load" to eval dynamically-typed Lua code. But it is less likely that we would implement a function to load Pallene source code at run-time, for the same reasons why it isn't common to dynamically load C source code: Pallene takes longer to compile than Lua, and dynamically-loaded code usually isn't as performance-sentitive anyway.

Cool makes sense. Could actually still be awesome to have a Lua lib or some other runtime lib that can translate Pallene or some other type-annotated Lua format to plain Lua, just for typechecking and ability to use autocomplete etc.

If someone is curious as i was, here's a little bit more context:

Github Repo - https://github.com/pallene-lang/pallene

More about the language - https://github.com/pallene-lang/pallene/blob/master/doc/manu...

I do not understand why Pallene is necessary if performance is the problem. Just use LuaJIT.

I suspect something else is happening here and possibly it has to do with autocompletion/intellisense.

Pallene is indeed approaching this problem more for performance than for autocompletion. (For just autocompletion it would be more appropriate to use a typesystem that is closer to full Lua, like Typed Lua)

For us, the main motivations for the Pallene approach are performance predictability and implementation complexity.

To get maximum performance with a JIT you need to write your code in a way that ensures that the control flow always stays inside the compiled parts of the code. If you are accidentally "too dynamic" then the JIT will fall back to the general bytecode interpreter, with an order of magnitude slowdown. The Pallene type system should ensure that if your code typechecks then it can be compiled to something reasonably efficient.

The type system also lets us use a simpler ahead-of-time compiler instead of a just-in-time one. JIT compilers are notoriously tricky to implement, which in turn means that it is more difficult to evolve them together with the language, or to port them to additional platforms.

When you say "JIT compilers are notoriously tricky to implement" what, I think, you actually mean is "optimizations for dynamically typed languages are notoriously tricky to implement".

If the language you need to compile is Lua - then it does matter whether you are trying to compile it just-in-time or ahead-of-time. Both would be hard to implement if you care about performance (and implementing an AOT compiler that produces fast code would probably be even harder than JIT).

Similarly if your language is like Pallene - which reminds me of Oberon-2 of all things, then you can implement both AOT and JIT with much less effort.

That does not however mean that AOT is trivial to write - as you increase the expressivity of the language and add more ways to write abstract code so does the complexity of optimizing away the cost of those abstractions (eliminating indirections and allocations, resolving polymorphic calls statically, etc)

Finally at some point, even a language like Pallene might benefit from a JIT - because of JITs' ability to see the execution and focus on actually taken execution paths, rather then looking at the program as whole and not knowing which paths would actually be taken.

You hit the nail on the head. With a more "Oberon-like" language we can use textbook optimization algorithms and take advantage of existing compiler backends, like GCC and LLVM. For dynamic languages these techniques aren't enough so more sophisticated JIT compilation seems to be the only option.

I strongly second that. We are more and more former users of LuaJIT who either don't use LuaJIT anymore, or think it is not a good long-term solution, because of those two issues (and absence of support of recent Lua versions).

In addition to that, people working on Titan say it could become an alternative to C for people who write applications in PUC Lua + C:

- It has an FFI so it can be used to implement bindings.

- It can also be used to extract performance-sensitive code from Lua into a compiled language.

- It should support newer Lua versions, being developed by people from PUC / close to PUC. Yay, 64 bit integers and bitwise operators!

Yup, this is the idea. :)

If you don't mind, I'm curious about what applications you currently use Lua for.

This will be very useful for platforms (mostly game consoles) where JIT are prohibited / security risk.

At work, I use Lua 5.3 embedded into a large-ish (a few hundred thousand LOC) cross-platform (runs on Windows, Mac, Linux, iOS, Android) C application as a way to extend it. We have bindings for a lot of the C functions and use Lua for prototyping. We keep the code in Lua in production if possible, or rewrite in C if performance issues force us to.

Personally, I use Lua for a lot of other things including Web development.

Hey, bro. Lua was my first programming language. I started making PSP homebrews before I learned it was actually made here in Brazil.

How is compilation-time execution handled in Pallene? Does it have templating or maybe can it run its functions during compilation with a special token (so it doesn't need a metaprogramming language)?

Currently there is no special support for metaprogramming, but maybe we could borrow some ideas from Terra in a future version...

Please don't use an interpreter for metaprogramming evaluation.

I was so focused on the technical that in pronouncing Pallene as "Pauline" to myself, I finally paused and made the connection LuaJit <- Mike Pall, so Pall...ene. Watch it's probably right in the PDF, and I just read over the connection!

Wow, I hadn't thought of that. Its a complete coincidence, actually!

Where does 'Pallene' come from then, Pallene XXXIII (33), a moon of Saturn?


I have only one question: is it 0-based?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact