RE: Options for Asynchronous Callbacks

["Thijs Schreijer" <thijs@...>]

> -----Original Message-----
> From: lua-l-bounces@lists.lua.org [mailto:lua-l-bounces@lists.lua.org]
> On Behalf Of steve donovan
> Sent: dinsdag 27 maart 2012 17:41
> To: Lua mailing list
> Subject: Re: Options for Asynchronous Callbacks
> 
> On Tue, Mar 27, 2012 at 3:10 PM, Luís Santos <jasonsantos@gmail.com>
> wrote:
> > My current preferred design for such things is to create a userdata
> > with the event queues and have lua poll it and dispatch.
> 
> There is a fairly elegant way to do this with the Windows API,
> actually.  There are APCs (Asynchronous Procedure Calls) which can be
> queued on the main thread, which is sleeping with SleepEx().  This
> will return if anything is in the queue, and then we go back to sleep.
> 
> So that will be the next experiment; all Lua code running on one
> thread.  But Peng Zhicheng mentions the debug hook method, which could
> fit into this scheme.  Only one way to find out!
> 
> Multithreading is hard, as Barbie the Software Engineer says ....
> 
> steve d.

Long post ahead... 

If I understand it correctly the issue is similar to these;

Multithreading, callbacks from unknown threads
 - http://thread.gmane.org/gmane.comp.lang.lua.general/89324/focus=89324 

Lua, multithreading and Qt 4.8
 - http://thread.gmane.org/gmane.comp.lang.lua.general/89092

I implemented a queue as mentioned by Luís in my darksidesync code and
besides polling added an option to get notified by a UDP packet so a
select() call would return. See https://github.com/Tieske/DarkSideSync 

As per my previous post, I don't think the debug hook is actually
threadsafe. The APC's are nice but while the Lua state is busy, and doesn't
enter sleepex, there is no room for callbacks getting executed (and it's not
cross platform if that is important). This is really a nasty problem to
solve.

The next step I had in mind for darksidesync was adding block and unblock
functions to the library. Every call to a C function that would possibly
block would then first call unlock() and upon returning call lock(). This
would be similar to the Lua_Lock and Lua_Unlock macros, except that these
are used for almost every c boundary and mine would only be used for
potentially blocking calls; io.read, socket.select, etc.
Whenever unlock() is used, the darksidesync lib would then be free to have
callback threads pass on into the lua-state, or even, when data is already
delivered and waiting in the queue, spawn its own new thread to deliver the
queued data.

I have not created the code yet, but do see the benefits;
 - instead of many locks/unlocks by lua_unlock and lua_lock, it would only
add the overhead to the potentially blocking calls (which is far less) (see
this post; http://lua-users.org/lists/lua-l/2005-08/msg00571.html though I
wouldn't want the bytecode counter as that would increase locking issues
again)
 - interruption/thread changing would be more Lua coroutine like, because
larger Lua blocks would get executed, and the boundaries are determined by
the code, not by pre-emption. Resulting in far less synchronization issues.
 - can be done cross platform

Downside;
 - I'd have to modify the default IO library regarding blocking calls,
update LuaSockets in a similar way, etc. to use the new lock() and unlock()
functions


Thijs