lua-users home
lua-l archive

[Date Prev][Date Next][Thread Prev][Thread Next] [Date Index] [Thread Index]

On 29 Sep 2006, at 19:24, Gunnar Zötl wrote:

Nick Gammon wrote:

NG> The lexer can be told whether or not to consider a sign as a separate
NG> token, I have written ones that do that.

NG> For example, after getting a symbol (x in your example), it is not
NG> syntactically correct for it to be followed by a number (eg. x 5) so
NG> the next "+" must be an operator token, not the start of a number.

actually, the unit recognizing such things would be the parser. The
lexer just knows how to recognize tokens. However, if it is the parser
recognizing negative numbers, and not the lexer, then the parser can
easily be written in a way to recognize such stuff.

Actually I think NG was suggesting an approach where the parser, having eaten an 'x' say, changes the lexer state so that the next '+' is tokenised as part of a number, not as the token '+'. All the compiler writing manuals tell you not to do this, but in real compilers it's pretty common. Isn't that what <FOO> is for in lex?