> An option to NOT discard the comments (or perhaps a secondary api call?) ???sounds like a great little 'tweak' to the core library that would provide a desirable benefit. How much effort would that so called 'little tweak' be (just curious, not volunteering)?
Not much effort at all, it seems. Here are the diffs in llex.c from Lua 5.3.4:
< read_long_string(ls, NULL, sep); /* skip long comment */
> read_long_string(ls, seminfo, sep); /* skip long comment */
> return 'C';
< next(ls); /* skip until end of line (or end of file) */
> save_and_next(ls); /* skip until end of line (or end of file) */
> TString *ts;
> ts = luaX_newstring(ls, luaZ_buffer(ls->buff),
> seminfo->ts = ts;
> return 'c';
The patched code returns 'C' for a long comment and 'c' for a short
comment, with the contents of the comment in seminfo->ts. There is a
small bug that the first char in long comments is dropped but it is
probably easy to fix; I haven't tried.
I just added support for this to proxy.c in ltokenp and it works: