Re: Simple configurable expression parser (in pure Lua)

[Lorenzo Donati <lorenzodonatibz@...>]

On 22/06/2022 20:54, Lars Müller wrote:
> Having heard your full use case I would suggest jotting down all your
> expressions in a stupidly-simple-to-parse notation (S-expressions,
> anyone? or perhaps a Lua DSL?) and then generating all the other formats
> out of that (C(++), LaTeX, perhaps Lua). Evaluating the expression trees
> in Lua should be trivial either way.
>
> -- Lars

Thanks for the idea. It didn't occur to me, and it might be an option in 
other situations.

However, there is a big drawback: the readability of the "source" will 
be greatly hampered. Keep in mind that all my tools are aimed at 
simplifying and speeding-up my workflow while designing a test or a 
handout for my lectures.

Therefore I like to write the text essentially as a latex fragment
with some embedded Lua sprinkled all over. Part of the underlying engine 
that allow me to do so is a modified (by me) version of the Lua template 
processor by Rici Lake found on WIKI pages, which has served me 
extraordinarily well for ages (even in other applications, that is).

This makes the "source" very readable, because I can enter something 
like this in a file that will be parsed as a Lua-based DSL:

----------------------------------------------------------------

Section { pt = 9, ncols = 2, shuffle = shuffle_answers }
[==================================================[
Consider an ATMega328P MCU. We want to configure pin PD4
as a digital output having a high logic level.
Which of the following C code fragments does what's required?


# function EmitAnswer( text )
#   return ([[
#   \begin{minipage}{\textwidth}[outer=c]
#   \begin{lstlisting}[ style = Cstyle ]
#%s
#   \end{lstlisting}
#   \end{minipage}
#   \vspace{0.5 em}
#   ]]):format( text )
# end

]==================================================]
{
   CorrectAnswer
   [==================================================[
   ${EmitAnswer[[
      DDRD |= (1u << PD4);
      PORTD |= (1u << PD4);
   ]]}
   ]==================================================]
   ,
   [==================================================[
   ${EmitAnswer[[
      DDRD &= ~(1u << PD4);
      PORTD |= (1u << PD4);
   ]]}
   ]==================================================]
   ,
-- (Other alternative answers stripped)   ,
}

----------------------------------------------------------------

This is a "declaration" of a Section in a test (in this case a 
multiple-choice question), where the text of the question is a Lua 
string whose content is LaTeX, which in turn embeds more Lua code!!!
Then a set of possible answers follows.

BTW, in this simple example the function EmitAnswer does little (just 
making possible answers format uniform, without repeating boilerplate 
Latex). In other cases it does much more, e.g. randomizes the results 
according to some parameters or calculates the correct result of some 
algorithm or expression. Lots of different cases, really. Anyway there 
are multiple levels of metaprogramming at play (with quite some hidden 
configuration and interactions of function environments).

In this case those C code fragments didn't need to be processed 
(evaluated as C, I mean). However, if any of those expressions needed to 
be evaluated, with your solution I'd lose any readability of the 
"source" of the test.

Take this as another example:

----------------------------------------------------------------

Section { pt = 9, ncols = 6, shuffle = shuffle_answers }
[==================================================[
What value is stored in the variable x?
\\
# local c1 = VERSION_PARMS.c1
# local c2 = VERSION_PARMS.c2
# local function HEX( n )
#   return "0x" .. BaseConvertFormatted( n, 16, 4, 4 ):trim()
# end
#
# local expr = ([[(%s|%s) & ((1u << 4) | (1u << 5) | (1u << 
6))]]):format( HEX(c1), HEX(c2) )
# local expr_lua = expr:gsub( '1u', '1')
# varcontent =assert( load( "return " .. expr_lua ) )()
#
\begin{lstlisting}[ style = Cstyle, xleftmargin = 2 em ]
   uint16_t x = ${expr};
\end{lstlisting}
#
# function EmitAnswer( x )
#   local ans = ([[$\mathrm{%sh}$]]):format( BaseConvertFormatted( x, 
16, -4, 4, "~~" ) )
#   return ans
# end
]==================================================]
{
   CorrectAnswer
   [==================================================[
   ${EmitAnswer( varcontent ) }
   ]==================================================]
   ,
   [==================================================[
   ${EmitAnswer( varcontent << 1 ) }
   ]==================================================]
   ,
   [==================================================[
   ${EmitAnswer( varcontent << 2 ) }
   ]==================================================]
-- (Other alternative answers stripped)   ,
}

----------------------------------------------------------------


Here you can see that an expression `expr` entered in C (with 
"metaparametrization") is being "massaged" in order to be evaluated in 
Lua, and then it is output as C.


Therefore your approach would really slow me down, since even if I 
devise a smart notation to represent an expression, I don't really 
reason that way. I really want to be able to write down a plain 
Lua/C/whatever expression (which is what my brain can process visually 
in a fast and reliable way). All the other "weird" processing should be 
done by the tool.

Otherwise I would never be sure to have entered the "right" expression, 
and I would always need to recompile the Latex output to see the final 
PDF to be sure. This will really add steps to my workflow instead of 
simplifying it.


Thanks again for the suggestion, though.

Cheers!

-- Lorenzo.