Re: What's up with token filters (Re: New operators?)

[Rici Lake <lua@...>]

David Given wrote:
> It occurs to me that a reasonably elegant way of implementing non-trivial
> syntax modifications to Lua would be to have a totally separate, off-line
> compiler --- most likely written in Lua --- that reads in a Lua chunk,
> transforms it somehow, and then outputs it again as vanilla Lua source that
> can be passed to the real interpreter.

I don't think that would be adequate. There are a number of useful and
interesting constructions which can be implemented by generating VM
code, but cannot be implemented (efficiently) as source code transforms.

One example is the : call syntax, already discussed somewhere in this
longish thread (and not just because the VM provides OP_SELF to
implement the construct). You could, I think, just barely implement
that with a source transformation which broke expressions into the
equivalent of three-operator instructions, allocating temporaries as
necessary, but you would still end up with unnecessarily bulky compiled
code in many cases.

Having an assembler for the Lua VM would be a plausible option, though,
but it would require rather more explicit definition of the VM
than Lua presently offers.
> This way we gain all the advantages of the amazingly fast stream-based
> compiler, plus all the advantages of a complex AST-based compiler (which can
> also do code optimisation), plus we maintain a clear distinction between the
> Lua core language and the extended language (because they're implemented in
> separate programs!).
>
> If run-time compilation of extended language chunks was required, it ought to
> be relatively straightforward to override load() to run the script through the
> off-line compiler first.