Re: [ANN] Helper Threads Toolkit

[Javier Guerra <javier@...>]

On Friday 10 March 2006 9:20 pm, Vijay Aswadhati wrote:
> Looks squeaky clean. Have not played with it yet though.

really? i still have to write some docs...

> There are some use cases that I don't understand how I could handle
> using this package.
>
> I have C++ tasks (that I wish I could make them invocable from Lua)
> that can be canceled or it's state updated from one or more tasks.

i think of two approaches:

1) extend the toolkit: add a Cancel(task) function, and rename Finish(task) to 
State(task, ...). also add a send-as-event(task) callable from the work() C 
function, that would put the task in the out queue before the end of work().  
that way, the Lua code would be 'notified' by the appearance of the task in 
the output queue, and the State(task, ...) would get any data it needs, 
without stopping or finishing the task.  one-shot tasks would be deleted 
after State(task) if the task was in the TSK_DONE state.

2) separate the background work from its Lua interface:  spawn your work 
threads with your own API, not using Helper Threads; but add helperfuncs to 
get any event.  those tasks are 'done' when there's a corresponding event by 
the work thread.

something like this:
-------
in = helper.newqueue()
out=helper.newqueue()
-- several threads, so the events won't block
evt_th1 = helper.newthread(in,out)
evt_th2 = helper.newthread(in,out)
evt_th3 = helper.newthread(in,out)
evt_th4 = helper.newthread(in,out)

recorder = newrecorder(params...)
streamer = newstreamer(params...)

recdone_tsk = isdone (recorder)
in:addtask (recdone_tsk)
strmdone_tsk = isdone (recorder)
in:addtask (strmdone_tsk)

isfill_tsk = get_recorded_data (recorder)
in:addtask (isfill_tsk)
isempty_tsk  = nil

while true do
  local data = ""

  evt = out:wait()

  if evt == isfill_tsk then
    data = helper.finish(evt)
    isfill_tsk = getrecordeddata (recorder)
    in:addtask (isfill_tsk)

  elseif evt == isempty_tsk then
    helper.finish(evt)
    local isempty_tsk = streamsomedata (streamer, data)
    in:addtask (isempty_tsk)

  elseif evt == recdone_tsk then
    rec_endstatus = helper.finish(evt)
    recorder = nil
    if streamer == nil then
      break
    else
      kill (streamer)
    end

  elseif evt == strmdone_tsk  then
    strm_endstatus = helper.finish(evt)
    streamer = nil
    if recorder == nil then
      break
    else
      kill(recorder)
    end
  end

end

--------------------

getrecordeddata(recorder) would finish when there's some data, but the 
recorder wouldn't finish.

streamsomedata would finish when the streamer can accept more data, but before 
it starves (in fact, it could just append a list, so it doesn't need to be a 
helper func....).

isdone() would finish when the given work is done (either because there's no 
more input, or because it was cancelled), and it's finish method returns any 
end status.

newrecorder(), newstreamer() and kill() are usual C functions that create and 
kill your working threads.

i think (2) is more general, but needs more work from the end developer (you).

for (1) to work, i would add a cancel() method to the ops structure, but i'm 
not sure (yet) if there's a race condition if a running task signals an event 
(appearing in the out queue while still in the TSK_BUSY state), and finishes 
before (or while) the Lua code calls state(task)

for (2) to make sense, it's needed to differentiate "worker threads" from 
"helper threads". a thread that stays up working for prolonged time, getting 
and generating data is a 'worker thread', and out of my toolkit scope.  
'helper threads', OTOH, are a trick used to show a non-blocking API to 
blocking processes.

in this case, it's possible to use helper threads as a mechanism to 
communicate with worker threads.  the advantage is that it's (relatively) 
easy to use blocking semaphores and cond variables to get the events and then 
notify asynchronously to the Lua code.

until now, i had thought that worker threads were too much case-specific to 
bother supporting them, but looking at your description, i think it wouldn't 
(shouldn't?) be hard to do.

thanks for the feedback!

-- 
Javier

Attachment: pgpP7XoOTeNe_.pgp
Description: PGP signature