lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]


Luiz Henrique de Figueiredo wrote:
>> Would it be computationally expensive to have the tokenizer generate
>> three different tokens (maybe more if you want to distinguish [[]] from
>> [=[]=]), and just have the parser treat them equally ?
>
> This would defeat the goal of token filters, which is to use the Lua lexer
> unchanged. Plus it would complicate filters as well. It is doable and not
> expensive, but you'll have to patch the lexer. (Plus I find it ugly...)
>

I agree. However, there is nothing stopping a token filter from
recognizing constructs like:

   L"foo"

   utf8"foo"

etc., and translating those into other string constants (instead of
letting Lua translate them as function calls).