Re:lex patch

[Philippe Lhoste <PhiLho@...>]

> recent posts discussed EOL-differences. The lexical analyzer in Lua
> has similar problems with 'foreign' EOLs. Another problem is that it
> is locale specific. I've backported some of my changes of Sol's lex
> to lua-4.1-work4. The patch is attached. Here's a list of changes:
> 
> - Accepts "\n", "\r", and "\r\n" as end of line.

That's good, I suggested that some time ago, Luiz thought it wasn't
necessary because he assumed that while transfering a text file from a computer to
another, it was translated to the platform's EOL (unless I
missunderstood/forgot his exact argument). Which is obviously false, eg. when unzipping a file.

> - Accepts CTRL-Z as end of file.

They are becoming rare (Dos times?), but this is useful as well.

> - EOLs in long strings and escaped EOLs in strings are converted
> to a single "\n"

? I am not sure to understand fully.

> - Has its own ctype macros to make identifiers locale independent.

Well, originally, identifiers *are* voluntarily locale dependent, at least
it is documented... But as Visual Basic shown, if it is friendly for a locale
developper (which can type identifiers in its own language), it is
problematic for exchanging source code over nations.
Now, perhaps we should keep the things the way they were, at least for
backwark compatibility, but warn the users of the issues of using accents and
diacritical signs outside strings. It is more a problem of discipline then.
Now, perhaps I didn't understood your patch... (I didn't read the code!)

> These changes make sure that a Lua script gives exactly the same
> compiled code on any system, regardless of EOL convention of locale
> setting. The next changes (also included in the patch) are for
> convenience:
> 
> - Adds "long comments": --[[ comment ]] (see Lua-Wiki power patches).

I believe Lua authors have already implemented it (if not released). I want
to implement its lexing in SciTE.

> - The '#' comment is allowed not only in the first line but in all
> lines. That way you have less problems preprocessing Lua scripts
> with cpp (inserts #line directives).

Why not.

> - Changes the "\123" escape from decimal to octal as it is done in
> _all_ other languages I know.

Mmm, I don't like it.
Lua don't have to be like "all other languages".
Octal is an old survivance of the PDP days, and isn't used much outside this
notation. Most programmers are unconfortable with it (wild guess :-).
Of course, it is confusing (I remember Luiz having made the confusion in a
line of the Lua 4.0 code :-) but it is clear and easy to use.
And more annoyingly, this patch breaks backward compatibility.

Regards.

-- 
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--
Philippe Lhoste (Paris -- France)
Professional programmer and amateur artist
http://jove.prohosting.com/~philho/
--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--=#=--

GMX - Die Kommunikationsplattform im Internet.
http://www.gmx.net