Multi-thread Semantics (Re: Selenophobia)

[William Ahern <william@...>]

On Thu, Mar 30, 2017 at 12:55:59PM -0400, Sean Conner wrote:
> It was thus said that the Great pierre once stated:
<snip>
> > The threading will be great too... but I understand it will prevent lua 
> > to be compiled on some systems... may be, at least, a parallel_for in 
> > the language with a default implementation to a normal for, and a link 
> > to tbb where available?
> 
>   At what point is is cheaper to create the threads vs. just doing
> everything sequentially?  A thousand elements?  A million?  Ten?  
> 
>   Also, once you allow threading access to a single Lua state, you have the
> Python problem of a global interpreter lock.

That presupposes a particular implementation. One of the ideas I've had in
the back of my mind involves special copying and moving semantics. You
create a table with a special mode, like "t" for (t)hread-safe. Any key or
value object you add to the table is copied, including tables, strings, and
userdata, inheriting the "t" mode or an analagous attribute. (Anything read
from the table is copied out, dropping the attribute.) You could than pass
that table and all its subobjects to another Lua interpreter, which takes
ownership of everthing.

Once passed, the root table is then marked as inaccessible in the original
interpreter. The implementation of such tables would probably need an
additional level of indirection so all references could be invalidated. Or
maybe instead of a table for the root there's a new type of container
object--basically a pointer to a table.

The idea is to split the burden between the Lua VM and user code. It allows
keeping the Lua interpreter simple (hopefully) and performant, but makes
common multi-threaded patterns easier to implement by enforcing and
automating safe semantics. Because message passing is considered by many
academics to be the safer and saner approach to concurrency, particularly
with threading parallelism, we can think of the limitations of shareable
reference semantics as features. ;)

Lua would have to add a new metamethod, __copy, for userdata. Maybe also
__move, but technically the interpreter needn't care whether the __copy
metamethod implemented move semantics. For userdata it all looks the same
from the perpective of the Lua interpreter.

This assumes that the different Lua interpreters share the same underlying
allocator. That's the case by default. But even if allocation were an issue,
the special container table could be associated with a particular Lua
interpreter (in addition to the executing interpreter), inheriting its
allocator parameters for use during copy operations.

I suppose there'd also need to be a counterpart object in the receiving
thread for moving the root object. There could be FIFO buffering, but
needn't be. Signaling is another issue, but that can be handled entirely by
the application code. All Lua needs to expose is a send and/or receive
interface.