Source Preprocessing

lua-users home

Source code preprocessing (or Source filtering in Perl) refers to a transformation done to the source code before the code gets executed. Some background information is given below:

Source filtering can be used to extend the syntax of a language. It some cases it can be used to improve the efficiency as well.

Lua provides a loadstring functions (similar to eval in other dynamic languages) that can be used to compile Lua code on the fly. This allows a Lua program itself to in some ways filter itself or other code in-process rather than via a separate preprocessor (e.g. as in the C preprocessor).

In the below example, we do a benchmark comparison between a typical method of doing vector multiplication and that implemented using source filtering macro-expansion. This mostly for illustration, and it's not necessarily recommended.

local lua = [[
  -- typical implementation (without filtering)
  local function cross_vector2(u, v, result)
    local u1, u2, u3 = u[1], u[2], u[3]
    local v1, v2, v3 = v[1], v[2], v[3]
    result[1], result[2], result[3] = u2*v3-u3*v2, u3*v1-u1*v3, u1*v2-u2*v1
    return result

  -- test and benchmark
  local function benchmark(func)
    local t1 = os.clock()
    local t2 = os.clock()
    print("time:", t2-t1)
    local vector u = 1, 0, 0
    local vector v = 0, 1, 0
    for n = 1,5000000 do
      --print("DEBUG: u=" .. stringify_vector(u))
    local u = {1, 0, 0}
    local v = {0, 1, 0}
    for n = 1,5000000 do
      -- print("DEBUG: u=" .. table.concat(u, ', '))

-- source filtering implementation
lua = string.gsub(lua, "local%s+vector%s+(%w+)%s*=%s*(%w+)%s*,%s*(%w+)%s*,%s*(%w+)", "local %1_1, %1_2, %1_3 = %2, %3, %4")
lua = string.gsub(lua, "cross_vector%s*%(%s*(%w+)%s*,%s*(%w+)%s*,%s*(%w+)%s*%)", "%3_1, %3_2, %3_3 = %1_2*%2_3-%1_3*%2_2, %1_3*%2_1-%1_1*%2_3, %1_1*%2_2-%1_2*%2_1")
lua = string.gsub(lua, "stringify_vector%((%w+)%)", "(%1_1 .. ',' .. %1_2 .. ',' .. %1_3)")

-- source filter
print("DEBUG[\n" .. lua .. "]")

Results on one system:

time: 0.937
time: 4.078

The above could be made more robust. In particular, additional care should be taken to ensure that the source filtering doesn't occur in unwanted places. For example, in the remote case that your program had

print("cross_vector(u,v,u) = ", cross_vector(u,v,u))

you probably don't want source filtering to occur inside the string. To handle this, your source filter will need to skip over strings. This may require a full Lua parser (e.g. [LuaParse]), or you may even get away with a partial lexer (e.g. LuaBalanced). Luckily, Lua is relatively easy to parse compared to other languages.

Interaction issues may occur if you attempt to use two independent source filters at the same time.

Source filtering is related to code generation. For example, see the usage in LuaList:2006-09/msg00798.html.


See also: LuaTokenParsing, SimpleLuaPreprocessor
RecentChanges · preferences
edit · history
Last edited December 19, 2008 1:04 am GMT (diff)