Token filter

[Thomas Lauer <thomas.lauer@...>]

I am having an interesting time with token filters but now I've run into
a snag. I probably do something very stupid.... but I don't know what.

I have a module token.lua that implements the simplest of token filters:

function FILTER(get,source)
	print("FILTER",source)
	FILTER=function ()
		local line,token,value=get()
		print("FILTER",line,token,value)
		return line,token,value
	end
end

I have a module testmod.lua:

local g=_G
module('testmod')
function print(s)
	g.print(">>",s)
end

Nothing fancy here. I finally have a program testtoken.lua. Nothing
fancy either:

require('token')
require('testmod')
print("test 1")
testmod.print("test 2")
print("test 3")

This program produces:
FILTER  @.\testmod.lua
FILTER  1       local   nil
FILTER  1       <name>  g
FILTER  1       =       nil
FILTER  1       <name>  _G
FILTER  2       <name>  module
FILTER  2       (       nil
FILTER  2       <string>        testmod
FILTER  2       )       nil
FILTER  3       function        nil
FILTER  3       <name>  print
FILTER  3       (       nil
FILTER  3       <name>  s
FILTER  3       )       nil
FILTER  4       <name>  g
FILTER  4       .       nil
FILTER  4       <name>  print
FILTER  4       (       nil
FILTER  4       <string>        >>
FILTER  4       ,       nil
FILTER  4       <name>  s
FILTER  4       )       nil
FILTER  4       ;       nil
FILTER  5       end     nil
FILTER  6       <eof>   nil
test 1
>>      test 2
test 3

The token filter filters the require()'d testmod but not the source file
itself (the file that require()'d token). Which is okay as a token
filter that may filter itself is probably not a clever idea. So up till
new all is fine and dandy.

I now change the first line of testtoken.lua, to comment out the
require('token'). Instead I call lua with '-l token', which should be
broadly the same, or so I thought:

lua -l token testtoken.lua

However, I get this surprising error message:

FILTER  @testtoken.lua
FILTER  2       <name>  require
FILTER  2       (       nil
FILTER  2       <string>        testmod
FILTER  2       )       nil
FILTER  3       <name>  print
FILTER  3       (       nil
FILTER  3       <string>        test 1
FILTER  3       )       nil
FILTER  4       <name>  testmod
FILTER  4       .       nil
FILTER  4       <name>  print
FILTER  4       (       nil
FILTER  4       <string>        test 2
FILTER  4       )       nil
FILTER  5       <name>  print
FILTER  5       (       nil
FILTER  5       <string>        test 3
FILTER  5       )       nil
FILTER  6       <eof>   nil
FILTER  0       <eof>   nil
FILTER  0       <eof>   nil
test 1
lua: testtoken.lua:4: attempt to index global 'testmod' (a nil value)
stack traceback:
        testtoken.lua:4: in main chunk
        [C]: ?

The require('testmod') seems to be 'ignored'. Or rather, as the listing
shows (the last two lines starting with 'FILTER  0'), the file seems to
be empty...?!

I have investigated the token filter C sources and it seems that
require()'ing a file should work as a new closure with its own LexState
is produced whenever a new file is about to be parsed.

So I am clearly overlooking something vital here... but what?

-- 
cheers  thomasl

web : http://thomaslauer.com/start