Re: Lua C API can't be used from other languages without C wrappers

[Russell Haley <russ.haley@...>]

On Mon, Feb 19, 2018 at 11:29 AM, Tim Hill <drtimhill@gmail.com> wrote:
>
>
>> On Feb 18, 2018, at 7:03 PM, ThePhD <jm3689@columbia.edu> wrote:
>>
>> Perhaps tangential to the discussion, but longjmp / setjmp has bitten me a lot too. From not running C++ destructors to making cleanup code get skipped and generally surprising behavior, if there's any mechanism I'd avoid in any API -- even with plain C where destructors and stack unwinding isn't a thing -- it's these.
>>
>> Unfortunately, that means your entire API has to be rewritten from the ground up to propagate errors properly to every external function, meaning the internals have to be written to handle it. This is its own challenge, and is extremely hard to do properly throughout an entire API.
>>
>
> +1 .. the C function calling model in Lua is very easy to use for trivial C functions, and very hard to use for non-trivial ones.
>
> By non-trivial, I mean any C function that must manage dynamic resources during its execution, such as memory, file handles etc. These must be cleaned up before the C function returns to Lua, and this is where the problems start. I’m not talking about C functions that maintain state BETWEEN calls (userdata can handle that case), I mean ones that allocate temporary resources just for the duration of the C function call.
>
> The problem, os course, is that if you call almost any Lua function while executing the C function, Lua can throw an error and silently demolish the C stack. This leaks any resources that the function is using, a very nasty bug indeed.
>
> It seems to me there are only two solutions to this:
>
> (A) Allocate a full userdata that stores (or references) the allocated resources. Then setup a meltable for the userdata with a __gc() C function that can cleanup (idempotently) all the resources.
>
> (B) Create a proxy C function that Lua calls instead of the real C function, which then allocates the resources before using lua_pcall() to call the real C function. This allows the C proxy to catch any Lua errors and perform cleanup before returning to Lua.
>
> Both are cumbersome to use from C code, and are tricky to get right. Both add significant overhead to the C function at run-time. Both are inelegant. Solution (A) also suffers from not performing cleanup until the next GC.
>
> I find it interesting that PiL is totally silent on this subject, and somewhat worrying since without either of the above strategies, code will silently leak resources, a type of bug that typically only shows up under stress (e.g. low memory) and is very hard to isolate.
>
> I won’t suggest a solution here (several occur to me), but I do think this is one of the weaker points of Lua.
>
> —Tim

I'm reading PIL right now to learn about the C API, with an a focus on
using it in .Net and C++ (and Free Pascal hopefully one day too!). It
occurred to me last night that the C API has the same design decisions
as filesystem support and user input via readline: "Not our problem".
This was hammered home for me when I watched ThePhD's Lua Workshop
presentation on sol2 and how clean and beautiful it is; but it took a
DOCTOR to put that together. The counter point can be seen in .Net
implementations that leak memory even in their own tests. Another
counterpoint can be seen in Javascript Interpreter Shell (jsish.org)
that uses the TCL model. Jsi provied C structures and methods for
interacting with the embedded interpreter.

I wanted to use Lua to *simply* things, not manage a stack. I have to
say Lua integration has been a bit of a heart breaker. After the PhDs
presentation, I'm now wondering about ditching the C API all together
and re-writing .Net integration using Sol2 instead. Ah, dreams...

Russ

>
>
>
>
>
>