Lua Macros

wiki

A Macro Facility for Lua using Token Filters

The source code for Lua Macro is available here: http://luaforge.net/frs/?group_id=328

lhf's [tokenf patch] (see also [this writeup]) provides a simple but powerful hook into the stream of tokens that the Lua compiler sees. (In Lua, for a given module, compilation into bytecode and execution are distinct phases.) Basically you have to provide a global function called FILTER, which will be called in two very different ways. First, it will be called with two arguments; a function which you can use to get the next token (a 'getter') and the source file. Thereafter, it will be called with no arguments, but will be expected to return three values. (This is confusing at first, and these two functions should probably be given different names.)

The get function returns three values: line,token and value. Token has a few special values like '<name>' (any symbol), '<string>', '<number>', and '<eof>' but otherwise is the actual keyword or operator like 'function', '+', '~=', '...', etc. If the token is one of the special cases, then the value of the token is returned as the third value. (There is an instructive example with the tokenf distribution, called fdebug, which simply prints out these values.)

Token filters read and write tokens one at a time. Coroutines make it possible to maintain complex state, without having to manage a state machine.

LuaMacros

The macro facility described here is pretty similar to the C preprocessor, although it works on an already predigested token stream and is not a separate program through which Lua code is passed. This has several advantages - it is faster (no separate translation phase) and macros can be tested interactively. The disadvantage is that LuaMacro is dependent on a patched version of Lua, and debugging macros can be sometimes a little awkward. (There is a variable macro.verbose which you can set to see the tokens read and written by LuaMacro.)

A simple macro that takes two parameters is this:

macro.define('PLUS',{'L','C'},' ((L)+(C)) ')

(In the first version of LuaMacro, I used token literals inside pair of @'s. Since then I thought this added unnecessary noise to macro definitions; LuaMacro now uses a lexical analyzer to preprocess the substitution string.)

The following is a simple equivalent to a C-style assert, where the actual expression is converted into a string to form the optional second argument of assert() using the 'stringizing' function _STR():

macro.define('ASSERT',{'x'},'assert(x,_STR(x))')

Testing your Macros

Macro definitions need to be in a separate file from the code to be preprocessed, but no longer need to be loaded before your program. Instead, there is a standard macro include. Assuming that the PLUS and ASSERT macros have been defined in plus.lua, then:

--test-macro.lua
include 'plus'
print(PLUS(10,20))
ASSERT(2 > 4)

$ lua -lmacro  test-macro.lua

It is important that the module macro is loaded before the program is parsed, since macros operate on the compile phase.

, they can be tested interactively like this (note that errors are given at the 'correct' line)

D:\stuff\lua\tokenf>lua -lmacro -i
Lua 5.1.2  Copyright (C) 1994-2007 Lua.org, PUC-Rio
> include 'plus'
> = PLUS(10,20)
30
> = PLUS(10)
=stdin:1: PLUS expects 2 parameters, received 1
> ASSERT(2 > 4)
stdin:1: 2 > 4
stack traceback:
        [C]: in function 'assert'
        stdin:1: in main chunk
        [C]: ?

The substitution need not be a string, and may be a function - this is where things get interesting:

macro.define('__FILE__',nil,function(ls) return macro.string(ls.source) end)

The nil second argument indicates that we have no parameters, and the third substitution argument is a function which always receives a table containing the lexical state: source,line and get (the getter function currently being used). This function is expected to return a token list: in this case, {'<string>',ls.source} . Three convenience functions, macro.string(),macro.number() and macro.name(), are available.

In general, the substitution function receives all parameters passed to the macro:

   local mname = macro.name
   local value_of = macro.value_of
   local define = macro.define

   define('_CAT',{'x','y'},function(ls,x,y)
	return mname(value_of(x)..value_of(y))
   end)

This is the only way to handle variable length parameter lists, since otherwise the number of formal and actual parameters must match. Bear in mind that the parameters always come in the form of token lists, which have a particular abbreviated format. For example, {'<name>','A','+',false,'<name>','B','*',false,'<number>',2.3} .

Please note that macro definitions are Lua modules and so you are free to define local variables and functions.

Implementing a Try/Except Statement

You can also define a handler which provides parameters if a macro is intended to be called without a parameter list. This is the third argument to define(). As an actual useful example, here is how 'try' and 'except' can be defined as semantic sugar around pcall():

---- implementing try...except
local stack = {}
local push = table.insert
local pop = table.remove
local global = macro.global
local name = macro.name
local define = macro.define

define('try',{'L1','L2',handle_parms = true},
	' local L1,L2 = pcall(function() ',
	function(ls)
		local L1 = global()
		local L2 = global()
		push(stack,{L1,L2})
		return name(L1),name(L2)
	end)

define('except',{'L1','L2',handle_parms = true},
	' end) if not L1 then local e = L2 ',
	function(ls)
		local t = pop(stack)
		if not t then macro.error("mismatched try..except",ls.line) end
		return name(t[1]),name(t[2])
	end)

So, given code like this:

a = nil
try
  print(a.x)
except
  print('exception:',e)
end

The compiler would see the following code:

a = nil
local _1ML,_2ML = pcall(function()
 print(a.x)
end) if not _1ML then local e = _2ML
 print('exception',e)
end

The smartness of these macros (note that we can here keep track of nested try..except statements) means that we can try out new syntax proposals with a little work, without having to patch Lua itself. And writing macros in Lua is certainly an order of magnitude easier than writing syntax extensions in C!

As an example of more elaborate code generation, here is a using macro which works rather like the C++ statement. There is no true module scope in Lua, so a common trick is to 'unroll' a table:

local sin = math.sin
local cos = math.cos
...

Not only do we get nice unqualified names, but accessing local function references is faster than looking up functions in a table. Here is a macro that can generate the above code automatically:

macro.define('using',{'tbl'},
    function(ls,n)
        local tbl = _G[n[2]]
        local subst,put = macro.subst_putter()
        for k,v in pairs(tbl) do
            put(macro.replace({'f','T'},{macro.name(k),n},
                ' local f = T.f; '))
        end
        return subst
    end)

Here the substitution is a function, which is passed a name token (like {'<name>','math'}), assumes it refers to a globally available table, and then iterates over that table dynamically generating the required local assignments. subst_putter() gives you a token list and a put function; you can use the put function to fill the token list, which is then returned and actually substituted into the token stream. replace generates a new token list by replacing all occurrences of the formal parameters (first argument) with actual parameter values (second argument) in a token list. To use this, put the macro call at the start of your module:

  using (math)

Implementing Type Annotations

A common issue with dynamic languages is this: they are strongly typed, but the actual type at runtime is dynamic. In particular, you cannot look at a function definition and immediately deduce the parameter types, unless somebody has been kind and used comments. We cannot always depend on the kindness of strangers, but it is straightforward to define a set of macros which act as explicit type annotations. We want to define functions like this:

   function rep (String s, Number k)
     return s:rep(k)
   end

   function rep (s, k)
     _assert_type(s,'string','s')
     _assert_type(k,'number','k')
     return s:rep(k)
   end

  function _assert_type(value,typestr,parm)
    local t = type(value)
    if t ~= typestr then
        error(
            ("Argument '%s' expects a type of '%s', got '%s'"):format(parm,typestr,t),
            2)
    end
  end

local subst_putter = macro.subst_putter
local set_trigger = macro.set_trigger
local insert_tokens = macro.insert_tokens
local replace = macro.replace
local mstring = macro.string
local subst = nil
local put
local assert_stmt = macro.lex '_assert_type(arg,tname,_STR(arg));'

local function type_checker_macro(Fname,Tname)
    macro(Fname,{'arg',handle_parms = macro.next_token_grabber('<name>')},
        function(ls,arg)
            local line = ls.line
            if not subst then -- first argument in a list
                subst,put = subst_putter()
                set_trigger(')',true,function()
                    insert_tokens(line,subst)
                    subst = nil
                end)
            end
            -- in any case, put the assertion into the token list
            put(replace({'arg','tname'},{arg,mstring(Tname)},assert_stmt))
            return arg
    end)
end

type_checker_macro('String','string')
type_checker_macro('Number','number')
type_checker_macro('Table','table')
type_checker_macro('Function','function')
type_checker_macro('Boolean','boolean')

This works as follows: we specify macro.next_token_grabber('<name>') as our parameter grabber; it will return the symbol following the macro. The substitution is a function; it just returns the argument it receives, which is that symbol. But the interesting stuff happens as a side-effect; we define a trigger, which fires on the end of the argument list and inserts all the assertions into the code immediately following the argument list.

Notice how families of related macros can be generated with similar code. It would not be difficult to generalize this scheme to handle a proper object-oriented hierarchy.

Implementing List Comprehensions

In PythonLists, FabienFleutot discusses a list comprehension syntax modelled on the Python one.

    x = {i for i = 1,5}

    {1,2,3,4,5}

Such a statement does not actually require much transformation to be valid Lua. We use anonymous functions:

   x = (function() local ls={}; for i = 1,5 do ls[#ls+1] = i end; return ls end)()

However, to make it work as a macro, we need to choose a name (here 'L') since macros are not triggered on arbitrary tokens.

macro.define('L',{'expr','loop_part',handle_parms=true},
    ' ((function() local t = {}; for loop_part do t[#t+1] = expr end; return t end)()) ',
    function(ls)
        local get = ls.getter
        local line,t = get()
        if t ~= '{' then macro.error("syntax: L{<expr> for <loop-part>}") end
        local expr = macro.grab_parameters('for')
        local loop_part = macro.grab_parameters('}')
        return expr,loop_part
    end)

The substitution is pretty straightforward, but we have grab the parameters with a custom function. The first call to macro.grab_parameters grabs upto 'for', and the second grabs upto '}' - nested expressions are automatically handled. Nested comprehensions work as expected:

  x = L{{j for j=1,3} for i=1,3}
  
  {{1,2,3},{1,2,3},{1,2,3}}

A particularly cool idiom is to grab the whole of standard input as a list, in one line:

   lines = L{line for line in io.lines()}

The source code for Lua Macro is available here: http://luaforge.net/frs/?group_id=328

Lua 5.1

-- SteveDonovan