Re: Lua C API and 1:1 lua_state : os threads

[Andrew Starks <andrew.starks@...>]

On Mon, Mar 10, 2014 at 1:01 PM, William Ahern
<william@25thandclement.com> wrote:
> On Mon, Mar 10, 2014 at 10:50:53AM -0500, Andrew Starks wrote:
> <snip>
>> So, we've had to do all of the things that others do as well; moving
>> Lua values between states being the most obvious example. We are using
>> a message passing pattern, of course, and are very happy with how Lua
>> is working out, so far.
> <snip>
>> Does anything come to mind when one thinks of improving the C API, Lua
>> and managing multiple states between operating system threads[2]?
>>
>> --Andrew
>>
>> [1] We thought about a multi-state xmove that worked across lua_states
>> and that didn't support lua functions or c closures. But then we
>> thought that might be better as a separate library...
>
> One issue with moving objects between different global states is handling
> panics and rewinding the locking state. In my project (cqueues) I use
> message passing over sockets for most communication. But on thread creation
> I need to copy the startup routine and arguments.
>
> I have a carefully choreographed (and documented) block of code to do the
> correct thing and safely return the error to the caller. But I first
> serialize the arguments before passing them off to the new thread, so the
> only panic or error I have to deal with is from the new VM.
>
> If you try to write xmove for separate global states, and especially if you
> implement table copying (without serialization), then you'd have to worry*
> about panics and errors in both the source and destination VMs. That gets
> really hairy when you toss OS thread synchronization into the mix.
>
>
> * Well, I guess you're not required to worry about it. AFAICT, Lua Lanes
> just punts the issue. But in complex environments with complex code, it's
> not uncommon to hit memory limits (e.g. via process ulimit), or having to
> deal with tables with metatable routines that might blow up inside the copy
> operation.
>


This is exactly what we do. The only time that we would use
"super_xmove" is when we initialize the new lua_state from the
arguments of the first one. We are using nanomsg right now, but could
be using any message passing facility for inproc.

I'm a huge fan of:

https://github.com/moteus/lua-llthreads2

...which uses the same approach. The "fan" part comes from the totally
comprehensible error messages that I get in the remote thread. They're
like... normal.


--Andrew