lhf's token filter patch

[Gunnar Zötl <gz@...>]

Hi there,

I played aroung a little with Luiz' token filter patch (see
http://www.tecgraf.puc-rio.br/~lhf/tmp/macros.tar.gz), compiled and 
linked against 5.1w6 on linux. First let me say: I want this :-))) It
is so cool, and a very nice addition to luas meta-mechanisms.

Might I suggest a name change for the filtering function to be
redefined, something like TOKEN_FILTER, to make the connection a bit
clearer. Maybe even prefix it with an underscore, to make it
consistent with other special variables (_G, _VERSION).

And something else I noticed: The lexer is not very forgiving. If you
use the wrong get() function (which, under certain circumstances also
happens in Luiz' sample macros), lua throws a memory exception. A
little example to show how to trigger this:

$ lua -l macros
Lua 5.1 (work6)  Copyright (C) 1994-2005 Tecgraf, PUC-Rio
> a b
>>> parsing stdin
a b stdin:1: '=' expected near 'b'
> c
>>> parsing stdin
Speicherzugriffsfehler

a.k.a. memory fault. The reason for this is, when a parsing error
occurs within a line entered interactively, Luiz' macro expander does
not recognize this because it never sees the <eof> token terminating
the line. Thus he reuses the previous get function, which is not valid
any more, resulting in a kablooie. Writing the filter function a bit
differently gets rid of this, but I really think lua should not
explode on you when calling the wrong function ;-)

Gunnar