Re: ANN: luaSub 0.41 syntax front-end

[Fabien <fleutot+lua@...>]

On Jan 23, 2008 12:10 PM, Asko Kauppi <askok@dnainternet.net> wrote:

However, doing all the necessary precautions can be made in many
macro systems.

Everything computable can be done with any Turing-complete language. But to make it practical, you need the proper abstraction level; if a system requires to reimplement half of metalua to get a comparable expressiveness, it's abusive to claim that it "can be made".
 

In particular, luaSub seems to me syntax and scope
aware enough, that it shouldn't be much harder there. Just to make
sure, luaSub's mods are made in 100% Lua, not in a "low level language".

I mean low level in the Alan Perlis sense: "A programming language is low level when its programs require attention to the irrelevant ".

With the wrong libraries and APIs, all languages can be low level. In the mod samples provided with luasub, you're still fighting with literal syntax matters when you're trying to perform semantic operations. That's the wrong level of abstraction: you've got more than enough to chew with semantics, so you need to be completely freed from concrete syntax concerns.

A luasub mod essentially plugs callbacks into the token stream, and its "access" to the AST is limited to having some named entry points where interesting bits of the tree are about to be parsed; it doesn't really give a direct grasp on the only right mental model (trees), and you can't describe non-trivial search and transform on them (at least not in a robust, readable and maintainable way). When splicing trees together, mods are directly piping token streams together, that's only marginally higher level than string processing; try to do some capture analysis on the fly and your code will become absolutely ugly (and proably wrong as well).

For instance, you want to think of an upvalue as <<a reference to a bound identifier that's separated from its binding statement by at least one function declaration>>, and that's mostly the way you'd describe it with metalua's walk.id. You're doomed if you try to think of it in terms of:

<< a list of ids that's after a "local" or "for" or "function"+"(" or "local"+"function" keyword, then followed by more "function" keywords than "end" keywords related to a "function" one, and with no "local"/'for" keyword introducing an homonym function in between >>.

It is theoretically possible to use such abstractions (although I'm myself really not confident in the stream-oriented description above), the way it's theoretically possible to write a complex web application in pure C. Your code will become unexploitable very soon. too soon to write anything useful and robust IMO. For that level of power/abstraction, I think m4+Lua or Luma offer similar capabilities under a much better better interface.

I think most people will flock around once the solutions are so simple they hardly see them anywhere.

Again, I think the limiting factor is not ease (nor compilation time), but trust in robustness. There's no point in making an "extensions design for Dummies" system, because nobody wants to use an extension written by a dummy. It won't be easy to write an extension; what should be easy is:

• reading an existing extension, to convince oneself of its robust design. The extension's code has to inspire confidence and make the design principles clearly visible.
• use a trustworthy third party extension with confidence (this includes absence of exceptional non-working cases, no nasty interference with other extensions, no capture issues, clear syntax error messages, confidence that no syntax error will be silently accepted...)

You know there's a significant gap, in expected code quality, between a single-use program and a published library. IMO there's an even wider gap, in terms of quality expectations, between a runtime library and a language extension.

Besides, writing in the same language as everyone else has an enormous software engineering value, as noted by Steve. It means that simple syntax hacks are condemned to be virtually never worth it. If you take the try...catch example above, which I'd consider borderline-useful, it can make sense because:

• You can trust that the finally-bock will be executed no matter what. No need to think twice about it, no exception, no need to reengineer a barely readable design pattern (in which you can miss a typo), just trust 'finally'.
• You have, with structural pattern matching, an efficient and readable way to analyze an error, and to decide to catch it or not. Not unreadable nested 'if' statements, except on pathological cases.
  
• the catching code is readable: no need to write relaunching code when you can't treat an exception, it falls through transparently. Without ever forgetting the finally-block, of course.
  
• You can confidently nest try...catch blocks, or put try-block into catch-blocks... it just works.
  

A new paradigm is intended to free the user's mind for more important problems. If the syntax supporting a paradigm comes with traps and exceptions to be careful about, it failed to free the user's mind, so it's worthless. If a syntax hack is just "convenient" instead of changing the way one thinks about his programs, it should remain unwritten.

-- Fabien.