Re: future of annotations in Lua?

[Philippe Verdy <verdy_p@...>]

Also the current Lua syntax is inconsistent in another part:

The "_expression_ statements" in Lua currently only allow expressions that are simple function calls (with the form "a(b...)" or "a b..." if it is currified), but it currently forbids expressions using unary or binary operators, like "a + b" or "(a + b)" or "a[b]", as valid statements, even if they are intrinsicly a function call, to "__add(a, b)" and "__getindex(a, b)" in these examples.

And I see no fundamental problem in extending "expressions statements" in Lua to any valid Lua _expression_ (even if this has no effect such as using a simple number or string, or "-1" with the unary operator, or "1+2" with a binary operator). It does not change the syntax complexity at all, but just simplifies it with no additional cost and without changing any existing semantics or runtime and debugging behavior.

Le sam. 8 juin 2019 à 07:35, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

Also another goal of Lua (also _javascript_) is that the language must be usable in an interactive way (e.g. in a developer or debugger "console"). And this forbids using syntaxic constructs in the LALR definition that require rollbacking an already processed "reduce" action (which would have caused some action to be already taken and executed.

And this is where C++ fails completely (even if its syntax is LALR): it can only compile program units as a whole (by making using highly contextual shift-reduce conflict resolutions and building an entire "astract syntax tree" (AST) before even trying to compile it), and not in interactive way, so it is not usable as a "scripting language". So a C++ parser can only be used in a first pass and "execution" or "compilation" (of the AST) can only occur in a completely separate stage.

On the opposite, a Lua (or _javascript_) parser is not required to build an AST (it may still help in a compiler, but not in a interactive developer console), it can compile and run the code directly while parsing it, and even if the program unit is not complete. And this qualifies Lua (or _javascript_) as a valid scripting language.

If Lua had made the ";" mandatory at end of every "_expression_ statement" (instead of making it optional), there would not even be any conditional shift/reduce resolution, the Lua parser would use only trailing function calls, with no cost at all and no recursive stacking, and never requiring to build any AST.

Le sam. 8 juin 2019 à 06:55, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

The syntaxic analysis in Lua is made with the constraint that it can be parsed with a LR(1) parser as much as possible (with a single token of look-ahead). But actually Lua is LALR because it must also support a few rollbacks for resolving a few shift/reduce conflicts.

Implementing these rollbacks adds a cost in the parser because before it cannot be written simply using trailing function calls: these functions have to return a validation status that must be tested, so these functions inserted in the parser will stack up in memory with all their existing local context (unlike trailing function calls that are easily compiled by Lua as simple jumps.

If we add syntaxic features to Lua, we should keep this in mind and not transform Lua into a Fortran-like language (whose parsing is extremely complex because of its optional whitespaces between all keywords and identifiers, and very tricky to write and test with enough coverages) or in a C++-like language (also very tricky).

Le sam. 8 juin 2019 à 06:41, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

But if I write:

  a = b

  ;(break 'here')

  c = d

it is still functionally equivalent (in core language) to the 3 statements (including the empty statement);

  a = b

  ;

  c = d

but simply not to:

  a = b

  c = d

In that last case, there's an ambiguity because "b c" may potentially be a currified function call "b(c)".

That ambiguity already complexifies the existing Lua parsing because the compiler MUST rollback after parsing "a = b c" when it sees the next "=" (there cannot be two assignments in the same statement).

This is the known case in Lua syntax where there's a "shift/reduce conflict" in Lua, that is not always resolved simply as a "shift" (which still has the priority), but where the compiler MUST support rollback at this point in order to be able to retry with a "reduce", and why the empty statement ";" was necessary to the syntax (because of the priority given to the "shift" which may still be valid up to some later point in the source stream of tokens in which case there will be an inner reduce later, that cannot be safely rollbacked and that cancels any prior candidate shift/reduce rollback point)

Le sam. 8 juin 2019 à 06:30, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

Example of use of an annotation on an empty statement:

   a = 1

   ;(break 'here')

   b = 2

here "(break 'here')" is the "break" annotation which may have optional parameters possibly currified, entirely written between parentheses because "break" without them is a core langage statement.

If the annotation is ignored (because there's no debugger using it and the code is just run with a core engine) then it is equivalent to the 3 statements:

   a = 1

   ;

   b = 2

with the same empty statement, not necessary here because "1 b" is not a valid currified function call "1(b)" as a constant number is not an object and has no function members, so it is equivalent also to the two separate statements:

  a = 1 b=2

Le sam. 8 juin 2019 à 06:04, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.

Le sam. 8 juin 2019 à 05:55, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.

There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any _expression_ (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.

Le sam. 8 juin 2019 à 05:44, Philippe Verdy <verdy_p@wanadoo.fr> a écrit :

Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".

Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <mobile@majumdar.org.uk> a écrit :

> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards