But it appears that what they're actually pitching is a simple and flexible code generation environment. It's a way to generate statically-typed code at runtime that targets LLVM but looks nicer than this: http://llvm.org/releases/2.6/docs/tutorial/JITTutorial2.html (C++ code that conjures up LLVM SSA IR directly). You could almost think of this as a high-level API for LLVM code generation and execution that is exceptionally well-integrated into Lua.
For example, in their example where they create a Terra function from a BF program, the equivalent in plain Lua would be to compile the BF program into a Lua program (represented as a big string), load it into the interpreter, and then let LuaJIT JIT it. But with Terra, you can represent the code you're generating symbolically with the "quote" construct instead of having to compile it to a big string. Of course you could just writer a BF interpreter in Lua directly, but if you compile it instead you'll get better performance because you won't pay an interpreter overhead and the optimizer can analyze the program flow to look for optimization opportunities.
[EDIT: removed incorrect criticism about the BF codegen being incomplete]
It's an interesting approach and I look forward to learning more about it.
Since we are primarily using it for dynamic code generation, I haven't done much benchmarking against LuaJIT directly. Instead, we have compared it C by implementing a few of the language benchmarks (nbody and fannkuchredux, performance is normally within 5% of C), and comparing it against ATLAS, which implements BLAS routines by autotuning x86 assembly. In the case of ATLAS, we're 20% slower, but we are comparing auto-tuned Terra and auto-tuned x86 assembly.
Small note, the BF description on the website does go on to implement the '[' and ']' operators below. I just left them out of the initial code so it was easier to grok what was going on. The full implementation is at (https://github.com/zdevito/terra/blob/master/tests/bf.t).
I'm taking a different albeit related approach for dynamic runtime code gen, but either way this is rock solid work, though I'm pretty terrible at deciphering the lua + macro heavy code that is your code examples.
edit: I'm doing something more akin to the Accelerate haskell EDSL approach, with some changes
Just what they've done is a pretty solid. That said, it's not really done as part of a framework for numerics, which just means its a great validation benchmark of their code Gen.
Here's one perhaps relevant paper about LuaJIT for dynamic code generation in QEMU-esque instruction set simulation: http://ieee-hpec.org/2012/index_htm_files/Steele.pdf
It would be better to comparing it to LuaJIT with ffi
Am I right to think that I can generate dynamic libraries (.so) that do not include any kind of interpreter with this?
If I can do this then this may be my dream static / system language...
Yet, it also feels kinda pointless, most Lua use right now is in embedded interpreters in other software, and Terra would be hard to use, since most projects probably won't incorporate it at all.
When it is a new language, you fight for space into another languages turf, your "potential" is illimited, if your language is better, it will win.
This language is obviously made to use with Lua and C at the same time, kind of a bridge of sorts, and thus it has much more limited scope and utility, and many of the uses of Lua even if they might need Terra performance, they cannot shoehorn Terra on their interpreter.
For example for coders of Corona SDK, or WoW, or many other game engines and application SDKs out there that rely on Lua.
How about concurrently? That might be nice to have for asynchronous applications.
There are still some limitations. You'd still have to manage thread synchronization manually, and I think LuaJIT only allows one thread of Lua execution to run at a time, so if your threads call back into Lua they may serialize on that bottleneck.
Is there a way to run Terra with a separate Lua state per thread? So as to not have the problem with serializing when calling Lua from Terra?
Given that LuaJIT has some other drawbacks (e.g. it has memory limitations that PUC Lua doesn't have, due to the details of LuaJIT's NaN-encoding), the usual lesson applies: YMMV, so benchmark... :]
String and memory limitations have yet to bother me at all because anywhere speed matters you get order of magnitude improvements by managing the strings/memory yourself via the ffi. With Terra, it's clear that's the approach being advocated as well. I agree it really bolluxes small benchmarks, especially if the code is written for Lua and not done the "LuaJIT way" with the ffi. Outside of embedded or other exotic environments I think one would be hard-pressed to come up with a real-world workload where Lua PUC outperforms LuaJIT and there's no easy way to turn the tables. There are just far more options for optimization with LuaJIT and more ability to get closer to the metal than you have with Lua PUC.
None of that is to take away from what the Lua PUC guys have accomplished. Like any craftsmen who enjoys his work, I just like to use the best tools. That's LuaJIT in my opinion, and now Terra too.
Why would anyone do this? What's wrong with func, function, def, etc. ?
a terra function denotes a region of code with (slightly) different rules (arrays indexed at 0, use of C apis, etc). Making all of lua low level is interesting, but that's not what this project was going for.
That minor gripe aside, this looks awesome.
-- lua function
-- terra function
function foo(a:int, b:int)
This kind of highly problem-specific optimal code-building is out of scope for any kind of JIT, including luaJIT.
Have a look at Dan Amelang's jitblt:
How is is like lua then?