Re: ANN: luaSub 0.41 syntax front-end

[Fabien <fleutot+lua@...>]

About luasub/metalua comparison:

First and foremost, both systems don't have the same purpose: metalua's core is about compile-time meta-programming, i.e. it tries to provide the best possible abstraction level and manipulation toolkit to analyze, modify and generate programs as data. One of the features required to do so is a dynamically extensible parser, which means that metalua has all the syntax tweaking features you might want, but that's just the gateway to metalua, not its core.

Your typical metalua extension will spend 80% of its code doing tree matching or code walking on AST, reading and generating trees, and maybe have a dozen lines or so of syntax extension glue at the end, to offer easy access to the features from source files. The focus is on code as data, not on syntax: syntax is boring, and is a very bad representation for automated manipulation, so you want to get rid of it in your extension's design ASAP. As a rule of thumb, if the core of your extension contains references to concrete syntax matters, you've screwed up somewhere.

As you noticed, metalua is a more complex framework to master than a "simple" preprocessor, and there certainly is a learning curve to follow before you can do interesting stuff. The thing is, most of these interesting stuff are not practically doable with a simple preprocessor. If you want to polish a syntax extension to the point where it's as reliable as a native Lua feature, you'll always have to fight some corner cases which require advanced code analysis. So metalua makes it possible to write robust macros, and the price to pay is, quick and dirty hacking without reading the manual is difficult.

Apart from robustness when provided with correct code, you want a compiler to provide helpful error messages when fed with syntactically incorrect sources; that's something most preprocessors fail miserably. This, together with macro robustness, is a strong requirement if you want to get distributable, reusable extensions. Trust in the extensions' robustness, not milliseconds saved on compilation time, make a meta-compiler usable or not.

A couple of examples of why writing usable macros is not so easy: if you're writing a "do...end as function" or "try...catch" extension, you want it to react correctly to embedded return statement, and ideally to coroutines. If you decide not to handle those, you want to emit a proper error message, at compile time for static stuff such as returns, at runtime for coroutine stuff.

If you're writing a runtime type checking extension, you want to give type to variables, and re-check their content every time they are re-assigned. You want to do that without being fooled by variable names capture. You want to check the values returned by each return statement in a function, without being fooled by local functions and their own return statements. You want your type assertions to live in a separate namespace, and still be expressive enough to support polymorphism, dependent types etc.

In most extensions, you want to avoid accidental variable capture. This includes internal capture (the kind you usually get with C macros), but also external captures, where it's the user's code that captures the macro's variable. For instance, if your macro uses the global function type() and I decide to use it within a block "do local type = 'foo'; $CALL_THE_MACRO() end", I want it to work, and without requiring me to know how your macro is implemented. This is just another instance of lexical scoping, and it is as important for macros as for regular code, if not more.

(End of examples)

A couple more words about compilation speed: either you're working on some embedded device with a CPU speed < 10MHz, and then you'd better precompile your Lua scripts on a development PC, or you have a PC less than 15 years old, and you can compile your source files on-the-fly without a significant overhead. With a lower-end notebook and the standard meta-libraries precompiled, I've never felt a significant compilation overhead. I really can't see the purpose of shaving more compilation time, especially at the expense of code maintainability. The same remark applies to memory consumption: there are cases where you care about the VM's consumption, but in such cases you precompile anyway so compiler consumption is not relevant.

About "Cons: non-Lua looking syntax": the very point of an extensible language is that you can make it map your problem space better (you've probably read Paul Graham's Lisp evangelism about this). Meta-programming is definitely a complex task that deserves significant syntactical support; I understand why some people want to stick to vanilla syntax whatever the task at hand, but those people don't want a static meta-programming system, period. If you want something that looks like Lua, stick to plain Lua, it's the global optimum by definition.

Finally, in your list of preprocessors, you have forgotten Luma, which is IMO the best simple approach (metalua being the best comprehensive one). It's essentially a term rewriting system, built on simple and quickly understood principles. It doesn't let your macros look like parts of the core language, which is consistent with the fact that it doesn't give you the tools to make them as robust as core features. So within its application domain, it has a consistent and usable design.

-- Fabien.

PS: if you're not bored with my logorrhea yet, you can read my PoV on metalua vs. luma at http://luaforge.net/forum/forum.php?thread_id=2556&forum_id=1470
.