
DynASM - _qc3o
http://luajit.org/dynasm.html
======
npx
If Mike Pall happens to read this, I feel like he's really selling the world
short by not writing a book about dynamic language implementation. It could
influence generations of programmers! Right now you have to piece together
some of his mailing list posts and try to fill in the blanks, but a more
thorough explanation from him would be invaluable.

~~~
wwwigham
He's also written a large article designing a garbage collector[1], which was
very interesting to read.

[1][http://wiki.luajit.org/New-Garbage-Collector](http://wiki.luajit.org/New-
Garbage-Collector)

~~~
PeCaN
I actually implemented a variation of that once. The quad-color thing is
really brilliant once I understood it.

------
cbHXBY1D
My favorite blog post of all time is Haberman's "Hello, JIT World: The Joy of
Simple JITs" which covers using DynASM. [1]

I wish there was better documentation for DynASM. I wonder what cool
programming projects would be created if DynASM was better known and better
documented.

[1] [http://blog.reverberate.org/2012/12/hello-jit-world-joy-
of-s...](http://blog.reverberate.org/2012/12/hello-jit-world-joy-of-simple-
jits.html)

~~~
lukego
> I wish there was better documentation for DynASM.

"The Unofficial DynASM Documentation" by Peter Cawley is not to be missed:
[https://corsix.github.io/dynasm-doc/](https://corsix.github.io/dynasm-doc/)

~~~
john_reel
None of Cawley’s work is to be missed. Anyone interested in JITs ought to look
into his work.

------
cronjobber
I've used this. If the comparison is with writing an interpreter for whatever
you want done, here's how I remember the tradeoffs.

Using DynASM itself was smooth sailing. It is officially undocumented, but
like the site says, the code is sufficiently well commented.

Generating simple-minded machine code is easy enough (in my opinion) and the
results run decently fast.

I didn't integrate run-time generated functions with exceptions and debugging
(host language was C++). These areas seemed very sparsely documented
(exceedingly sparsely, IIRC, for 64bit Windows) and looked like they would eat
more work than everything else up to that point.

------
timClicks
Mike Pall really is a robot from the future [cf
[http://wren.io/performance.html](http://wren.io/performance.html)]

------
abelsson
A while ago, I spent a fair amount of time evaluating JIT frameworks including
DynASM. The one I ended up using is something called Xbyak
([https://github.com/herumi/xbyak](https://github.com/herumi/xbyak)) which I
think is a bit of a hidden gem. Unlike DynASM, it doesn't require a
preprocessing pass, is better documented and IMHO easier to use.

So if you're in the market for a lightweight JIT engine and target C++/X86,
I'd give Xbyak a whirl too and see which one you like better.

------
rurban
One should really note the major differences dynasm took from the more
abstract and logical jit libraries, which try to abstract platform oddities
from the user. Like LLVM, libjit, asmjit, v8, libgccjit, lightning or nanojit.

It doesn't care how you call a function, it doesn't even care about 2 or 3
address op syntax. All it does is adding simple template logic to let you
handle the platform-specific abstractions by yourself.

And in the end it's easier to add your own prologue or prepcall1 macro, than
relying on a library.

people always looked at the "most portable jit library". dynasm offers no
platform support per se, as all the others do. you have to do by yourself.
it's the least portable of all. but in the end it was much more powerful, and
offered the most ports. you get the asm anyway by dissambling the target
function. with the libraries (such as LLVM) you'd need to translate a high-
level overview to the library API. with dynasm you just need to adjust your
types, prologue, struct field accesses and epilogue. and dynasm is perfect and
practical in doing this, whilst the other libraries try do to much more, with
much more handholding and are much bigger and less effective.

I don't agree with the genius part though. lua itself was pretty much perfect
already, and lua took most of the good parts from earlier dynamic languages,
lisp and self, and esp. Inferno's "Dis VM".
[http://doc.cat-v.org/inferno/4th_edition/dis_VM_specificatio...](http://doc.cat-v.org/inferno/4th_edition/dis_VM_specification)
which was Lucent's faster and smaller competitor of Sun's stack-based and GC-
heavy JVM. it just missed besides a jit a good OO, and luajit didn't add that
neither. e.g. _why the lucky stiff added a nice OO layer on top of lua, plus a
nice simple jit. much simplier than luajit.

luajit just objected to some questionable decisions in lua regarding
performance, and then took over. esp. nan tagging makes a lot of sense, but
the compiler optimizations are also pretty cool. esp. when compared to the
massive LLVM overhead.

luajit is using the slower and bigger 3-address SSA and the optimizations
based on that. but the CPU and the lua bytecode is just using the fast and
natural two address op syntax, where it is not easy to add SSA. This is the
genius part. [http://wiki.luajit.org/SSA-IR-2.0](http://wiki.luajit.org/SSA-
IR-2.0)

~~~
beagle3
Can you link to _why's work on lua and its JIT? My Google fu couldn't find it.
(Unless you are referring to Potion which is entirely unlike Lua)

~~~
rurban
Yes, potion. It's a better rewrite of lua, as luajit. Same op and data layout
as lua, just with a simple method jit (luajit does tracing and optimizations),
plus a msg send based OO.

------
jimmyswimmy
As an electrical engineer I deal with assembly programming all too frequently.
So clearly I am missing something important about the concepts presented here.
Why on earth would I go to the trouble of writing assembly code to put in the
middle of my C and then use another tool to strip the assembly and convert it
to C? Because it turns out that I'm even better at writing C than I am at
assembly. And although the power of assembly is fantastic, I vastly prefer C.

Thanks in advance for any explanation. I'm clearly missing something that
should be obvious.

~~~
bogomipz
I didn't think most electrical engineers would work with Assembly code. Can I
ask what type of electrical engineer you are?

~~~
jimmyswimmy
I'm a jack of all trades design engineer in the R&D space. Assembly is
appropriate for the very low end of the 8-bit microcontroller space.
Especially for timing critical code. Like a bit banged RS-232 port. There are
still micros with only a few kb of flash. Can be hard to fit much of anything
in there. You can do some amazing things in assembly if you're motivated,
which tends to happen when you have a fifty cent budget for a micro, two weeks
left for delivery of prototypes, and code left to write.

------
register
I really wonder where an engineer of his class works. He really seems coming
from an alien civilization.

~~~
watmough
People, including Mike have been looking into whether it would be viable to
clone him.

[https://github.com/LuaJIT/LuaJIT/issues/45](https://github.com/LuaJIT/LuaJIT/issues/45)

------
jsn
I used DynASM in my little weekend project
[https://github.com/jsn/icfp06](https://github.com/jsn/icfp06) , see README
for my experience. In short: I like the idea a lot, but the implementation is
seriously buggy. My experience with GNU Lightning (during the same weekend
project) was much more smooth.

------
bitmadness
Mike Pall is a genius, not much more to say tbh.

~~~
ajkjk
Please say more; as someone who doesn't work in this area I don't know if this
is impressive or important or what.

~~~
matt_m
LuaJIT (of which DynASM is a component) is a JIT compiler for Lua (a dynamic
language), which despite being basically a one-man project was beating
contemporary Javascript compilers on performance and even the Dalvik Java
compiler on Android (making it the fastest JIT compiler in existence for a
dynamic language). Also, it targets multiple platforms (x86, x64, ARM, PPC,
MIPS). It's been in maintenance mode for a while I think.

A couple interesting links about the internals:

[http://lua-users.org/lists/lua-l/2009-11/msg00089.html](http://lua-
users.org/lists/lua-l/2009-11/msg00089.html)

[https://news.ycombinator.com/item?id=2617628](https://news.ycombinator.com/item?id=2617628)

~~~
xaduha
DynASM used in MoarVM too, modern VM for Rakudo (implementation of Perl 6).

------
mikemike
Notable recent use of DynASM: Zend is using it to write a JIT compiler for PHP
8.0.
[http://externals.io/thread/268#email-12706-body](http://externals.io/thread/268#email-12706-body)

------
nmpl
Luajit a wonderfully clever piece of code that only one person understands and
is able to support. Before everyone chimes in that I'm mistaken ask yourself
why Luajit has not evolved to be compatible with the latest Lua 5.3 release.

~~~
copx
Lua was pretty much perfect at version 5.1 (the version LuaJIT supports). Lua
is meant to be a simple language, not one of those languages which accumulate
more and more features and thus complexity over time (I am looking at you C#,
Java, etc.).

So unlike those languages Lua can really be considered "done". Version 5.2 was
a lot like C99, maybe people considered one or two features "nice to have" but
most Lua programmers saw no need to upgrade. Others like Mike Pall even
considered 5.2 inferior to 5.1.

Now we have 5.3 which is really all about one special-purpose feature: 64 bit
integers. That is a feature most Lua users do not need and it comes with a
massive complexity and performance price tag.

Unless you really need 64 bit integer math in your scripts 5.3 is the inferior
language.

I really enjoy that in classic Lua a number is a number, that is such a relief
compared to the hell that is C's countless numeric types and all the potential
bugs resulting from (silent) conversions between them. Lua 5.3 brought that
problem to Lua e.g.

    
    
      a = 9223372036854775000 
      b = 0.5
      c = a + b
      c = c - b
      print(c) // 9.2233720368548e+18
    

The third line silently converts a 64-bit integer variable to a 64-bit
floating point variable, corrupting its value.

And of course Lua 5.3 is slower and needs more memory because of the
additional complexity which must be handled.

Thanks but no thanks.

I mean Lua 5.3 still makes sense _if_ you really need 64-bit integers in your
Lua scripts. But otherwise it is a downgrade.

~~~
pritambaral
> print(c) // 9.2233720368548e+18

I have little experience in lua myself, and I know only that lua is meant to
be a simple language. That said, I can't see from your example what's wrong
with lua5.3. I ran the same code with lua5.1, lua5.2, and lua5.3 – and the
output was identical across all:

    
    
        $ diff <(lua5.1 numcheck.lua) <(lua5.3 numcheck.lua)
        $ diff <(lua5.2 numcheck.lua) <(lua5.3 numcheck.lua)
        $

~~~
copx
In Lua < 5.3 a = 9223372036854775000 is not even valid code. Because only
32-bit integer values are supported.

In Lua < 5.3 the value is immediately corrupted (check the value of 'a' after
the initial assignment) and static analysis can detect that line as a bug.

In contrast in Lua 5.3 it is perfectly fine code and the value of 'a' will be
9223372036854775000 as one would expect. _However_ the moment you - maybe
accidentally - mix that int64 variable with a float64 variable in an
expression you end up with a (silent) bug.

Such bugs will happen. They happen all the time in C and in Lua's case things
are even worse because function parameters do not have types. The
possibilities of hard to find, nasty bugs are endless here. E.g. simply
forgetting the second '/' in '//' in one part of the program can cause bugs in
a completely different part, maybe even in a third party module, and only
under certain circumstances.

~~~
Franciscouzo
"In Lua < 5.3 a = 9223372036854775000 is not even valid code. _Because only
32-bit integer values are supported._ "

That's not true at all, Lua < 5.3 uses doubles for all numbers, which on most
computers is IEEE 754, they can represent integers perfectly up to 2^53

The value is _not_ corrupted, depending on the rounding mode is which number
you're going to get out of that (I get 9223372036854774784)

~~~
copx
>That's not true at all, Lua < 5.3 uses doubles for all numbers

I know that of course, but using doubles for integer math is only safe if you
limit yourself to a certain integer range.

>which on most computers is IEEE 754, they can represent integers perfectly up
to 2^53

False. You get rounding errors after 10^14.

>The value is not corrupted, depending on the rounding mode is which number
you're going to get out of that (I get 9223372036854774784)

The above being an example of such a rounding error.

If I assign the value 9223372036854775000 to a variable I expect said variable
to afterwards have the value 9223372036854775000 not 9223372036854774784. Your
example is exactly what I meant with "corrupted" i.e. the value is changed by
the conversion.

