Re: [ANN] LuaJIT-2.0.0-beta1

[David Manura <dm.lua@...>]

On Mon, Nov 2, 2009 at 11:22 PM, Vyacheslav Egorov wrote:
> it seems that there is no bytecode frontend in Lua JIT now, so it is
> not possible to create Lua JIT backend for metalua.
>
> The only option is to provide a source code backend for metalua. But
> as far as I remember metalua can produce irreducable control flow
> graphs (AST contains `Goto constructs). So translation into source is
> not a straightforward task.

True.  Conceptually, Metalua does these two steps:

  (A) translates a source code string--either plain Lua or Lua with
extended syntax--into an AST of the form given in [1].  This step is
implemented in pure Lua code, and the emitted AST is in the form of
Lua tables.

  (B) translates the AST into Lua bytecodes.  This step is also
implemented in pure Lua code based on Yueliang (a port of the the Lua
compiler to Lua).  However, the output is normally standard Lua 5.1
bytecodes.

So, you should be able to use Metalua under LuaJIT2 to perform steps A
and/or B provided you don't use any Metalua extensions modules
implemented in Metalua without first rewriting them in terms of
standard Lua.  However, I understand you can't run those bytecodes in
LuaJIT2 without perhaps some ugly portability layer.

You may, however, replace step B with your own generator.  This could,
for example, output LuaJIT2 bytecodes if LuaJIT2 added such an
interface for reading those bytecodes.  Alternately you could emit
standard Lua source.  Metalua 0.5 already has a module to do that
latter [2] for standard Lua ASTs.  This module is implemented in
Metalua, but it is easy enough to rip out the Metalua syntax
extensions and replace them with standard Lua.

Now, emitting standard Lua source has a couple issues, each of which
can be addressed in a few different ways.

The first problem is that the Metalua AST [1] has a couple of
extensions outside of standard Lua syntax.  These are `Goto/`Label and
`Stat nodes.  `Goto/`Label are for gotos.  `Stat executes statements
inside of expressions, which can frequently occur upon expanding
syntax extensions and macros.  Though you can achieve `Stat with
closures, the purpose of `Stat is to avoid closures for efficiency
sake.  `Stat nodes perhaps can be avoided.  I had once written a
source code optimizer that can inline function calls [3], and the
primary motivation of that was to eliminate the need for special
bytecode generation for `Stat nodes in Metalua.  Similarly, maybe
LuaJIT2 would negate these efficiency concerns as well.  The problems
handling `Goto/`Label still remain, but I suppose there are ways to
automatically translate gotos in plan Lua code or even patch Lua to
support gotos.  The Clue discussions [5-6] went into that.  This
problem is mitigated, however, in that that not all Metalua extensions
emit `Goto/`Label nodes.  If you avoid those extensions or rewrite
them in such a way not to emit these nodes, you won't have a problem.
The same can be said about `Stat.

The second problem is that standard Lua source does not allow emitting
meaningful debug information like line numbers are symbol names.  This
problem has been mentioned a number of times in the context of Lua
source preprocessing.  You might implicitly achieve this by formatting
your generated Lua code in such a way that line numbers in the
original and generated sources match up, but this might not be always
possible.  There have been suggestions to allow pragmas to be inserted
into the Lua source to alter line numbers somewhat like the C
preprocessor [6], and something similar might be done for symbols.
This is a second reason why Metalua implements its own bytecode
generator.  This second problem is likely easily solved by a patch to
the Lua parser though.

Though I agree it is helpful to think of Lua source as its own type of
portable bytecode, a few Lua syntax extensions as described above
could make it more effective in this regard.

[1] http://metalua.luaforge.net/manual006.html#toc17
[2] http://github.com/fab13n/metalua/blob/0.5-branch/src/lib/metalua/ast_to_string.mlua
[3] http://lua-users.org/wiki/SourceOptimizer
[5] http://lua-users.org/lists/lua-l/2008-07/msg00114.html
[6] http://lua-users.org/lists/lua-l/2008-07/msg00238.html
[7] http://lua-users.org/lists/lua-l/2009-05/msg00288.html