Lua Token Parsing

lua-users home

The syntax of Lua can be altered by interrupting the token parser and modifying the output. An experimental library ltokens ("A library for lexical analysis of Lua code based directly on the Lua lexer.") is provided by LHF [1]. The following page contains some usage cases for this code which might help with development of the concept. There may be many different solutions to these problems but some usage cases are required to ensure that solutions to practical applications are possible.

No solutions are offered here as the whole token parsing concept is very experimental at the moment. Please create your own page, with a link from here, if you would like to offer solutions to these problems. Free free to add your own usage cases to this list.


Enumerations are useful for grouping values together. Enumeration can be done in Lua quite efficiently by

local SHAPE_SQUARE, SHAPE_CIRCLE, SHAPE_STAR = 0, 1, 2  -- etc.

foo(SHAPE_STAR)  -- usage


Concept: Enum keyword and table of values which only lives at compile time.


print(ShapeType.CIRCLE)  -- usage

C Function Calls

C functions can be registered with Lua (int MyLuaFunc(lua_State*)). This registration involves passing a function pointer and a string function name to Lua (which duplicates the string). To call the function you do a global lookup using a function string name and call the object returned. More descriptive: [2].


Concept: At Lua compile-time a list of the functions exported (much like a link library in C) could be passed to the parser. When we intercept calls to named functions we could insert the function pointer, or function enumeration in order to call the function at runtime.


Switch statement

Arguably Lua doesn't need a SwitchStatement, but it might be an interesting test for ltokens. E.g. (feel free to change the syntax)

switch (value) do
    case 1 do print('one') end
    case 2 do print('two') end
    default do print('default') end

Try-except statement

Lua has exception support, but the interface is functional and low-level, making it cumbersome to use. Perhaps token tinkering would be helpful here.

Type checking

When calling a C binding, or other Lua function, generally objects have to be of a certain type. Lua does not support type checking, but this would be useful for these cases. For example, if we bind a C function,

void foo(int arg1, const char* arg2);
and then we call this from Lua we'll only know if the calling arguments types are wrong at runtime. It would be nice to be able to add optional argument types so if a bound function were recognised it's parameter types could be checked.


See also

RecentChanges · preferences
edit · history
Last edited March 13, 2009 11:53 pm GMT (diff)