Re: Macros and Token Filters Revisited

[Jerome Vuarand <jerome.vuarand@...>]

2010/6/12 steve donovan <steve.j.donovan@gmail.com>:
> Hi all,
>
> Since a few recent posts contained sentences like 'this calls out for
> a macro' I thought I'd review the state of play for macros based on
> token-filtering.  (Syntactical macros as implemented by Metalua are
> technically superior, although harder to write;  this is a custom
> compiler step generating bytecode).  Lexical macros using the
> token-filter patch don't have an explicit compile step, since the
> lexical stream within the Lua compiler is directly modified.  But you
> do need a patched Lua.
>
> lhf's token-filter patch was always intended as proof-of-concept;
> I've done a somewhat more sophisticated version here (patch against
> 5.1.4):
>
> http://lua-users.org/lists/lua-l/2010-02/msg00325.html
>
> There is an explicit API provided for token filtering (no dependence
> on global variables like FILTER) and C token-filters can be
> dynamically loaded.  Although, only _one at a time_. In principle one
> could chain token filters but it's awkward and could be rather slow;
> the next iteration will explore the possibilities and trade-offs
> involved.

One problem is that a filter output must be a valid Lua token stream
(ie. the compiler can compile it), while the input of a filter may not
be (and for most useful filters it isn't, hence the very purpose of
the filter). So chaining filters may be of little use, unless the
upstream filter is more laxist (ie. let go through token chains that
the compiler wouldn't accept).

> It is true that a patched luac could be used to generate valid Lua
> bytecode, which could then be distributed.  This is a little awkward -
> Lua bytecode is not intended to be platform-independent.  (One of the
> changes to Lua in Adobe Lightroom was to make the bytecode loader
> platform-independent)

While saving bytecode reduces portability, what prevents you from
re-serializing the output of the token filter in source form ? Are
there token that could be emited by a token filter, but couldn't go
through the unmodified Lua lexer ?