Re: Is it possible to run Lua threads in OS threads safely?

[Norman Ramsey <nr@...>]

 > I have noticed that you write "mostly reliable" followed by the scary note
 > "occasionally computations locks up on mysterious ways"... do you mean
 > dead-locks?

Yes.  Computation halts.  I'm running on Linux, and I have to
terminate the computation by sending INT to the process.  To get it to
stop properly, I have to send INT twice.

 > I also noticed that you mention upvalues as a way to pass data between
 > threads, but based on the documentation one could also use the send/receive
 > methods, right?

Yes.  I am using send and receive extensively.

 > (searching the internet, I found that someone mention the fact that if
 > there is a table as an upvalue then newproc fails, and this may not be
 > immediately clear to the developer, is this correct?)

It definitely fails.  I seem to remember that the error was relatively
easy to diagnose.

 > Why do you write that the concurrently model is at the same time "simple"
 > but "it took a while to figure it out how to use it effectively?"...it
 > seems it is not simple after all or maybe it is "too simple" and you bumped
 > into some unintended behavior?

I am accustomed to a richer concurrency model somewhat in the style of
Tony Hoare's CSP or John Reppy's Concurrent ML.  What took time was
figuring out how to use luaproc's model effectively even though it
lacks features that I was accustomed to use.

 > Yes I guess serialization (of tables) is one of the most required
 > functionalities :) It would be useful to have such a method in the table
 > library for that.... I guess it is not there because of the choices one can
 > make (shallow vs deep copies, etc - the mesh of references can be really
 > complex)

I've created a github gist that has my implementation of serialization
and also of a "work crew" abstraction.  There are undoubtedly missing
dependencies, but some of it may be useful.  Documentation below.

https://gist.github.com/nrnrnr/0cc1d8848ffae7f99f8ed2b332f19124


Norman


=============== Overview of module luaproc.workcrew ===============

Completes a job of 'work', where work is an abstract type.
A job is represented by a table containing these fields:

  { work      : work list
  , unpack    : function(work) returns scalar, ...  -- defaults to table.unpack
  , state     : a value pickle option -- defaults to empty table

  , worker    : function pickled   -- take result of unpack, return scalar list
  , collector : function pickled   -- merge scalar list from worker into state

  , nworkers  : int option         -- how many worker threads to create
  , uid       : string             -- unique identifier
  , status    : function(work)     -- optional, for side effect
  }


Function types:

  worker    : function (work) returns results
  collector : function(state, results) 
  status    : function(work) 

If I write the unpickling operation with vertical bars,
workcrew.run(job) is equivalent to the following

  local state = |job.state|
  for _, w in ipairs(job.work) do -- parallel
    job.status(w)                                    -- atomic
    local results = { |job.worker|(job.unpack(w)) }  -- not atomic
    |job.collector|(state, table.unpack(results))    -- atomic
  end
  return unpickle(pickle(state))

The first result returned by job.unpack must not be `false`.

The job and state tables will be copied multiple times, but only one
thing is mutated: the state table associated with the collector
thread.  

Picklers and unpickler can be found in luaproc.serialize.
===================================================================

luaproc.workcrew.run = function(job) returns state
  As described in the overview, create a work crew, run the job's work,
  tear down the crew, and return the final state:
    
    local state = |job.state|
    for _, w in ipairs(job.work) do -- parallel
      job.status(w)                                    -- atomic
      local results = { |job.worker|(job.unpack(w)) }  -- not atomic
      |job.collector|(state, table.unpack(results))    -- atomic
    end
    return unpickle(pickle(state))


================================================================


luaproc.serialize.ofstring = function(string) returns value
  Undoes tostring, so ofstring(tostring(v)) is isomorphic to v.

luaproc.serialize.pickle = synonym for tostring

luaproc.serialize.picklechunk = function(string) returns pickle
  Pickles a sequence of statements ending in return.
  When unpickled, produces the value returned.

luaproc.serialize.picklemod = function(string) returns pickled value
  Given string x.y.z, returns "(require 'x.y')['z']", used
  to pickle a function that lives in a module.
    
  Also accepts arguments function(modname, membername)

luaproc.serialize.tostring = function(value) return string
  Return a string that, when fed to ofstring,
  reconstructs an isomorphic value.
  Input must be composed of scalars and tables,
  no functions or userdata.

luaproc.serialize.unpickle = synonym for ofstring