Re: Options for Asynchronous Callbacks

[Gaspard Bucher <gaspard@...>]

On Wed, Mar 28, 2012 at 8:55 PM, Thijs Schreijer <thijs@thijsschreijer.nl> wrote:

> From: lua-l-bounces@lists.lua.org [mailto:lua-l-bounces@lists.lua.org] On

Behalf Of Gaspard Bucher
> Sent: woensdag 28 maart 2012 18:07

> To: Lua mailing list
> Subject: Re: Options for Asynchronous Callbacks
>

> Some of you who attended the Lua Workshop might know of my struggles to
> use one of the proposed models here (mutex based locking) and that I
rewrote
> all the code in lubyk to use coroutines and sockets.
>
> Blog post on the different architectures: http://lubyk.org/en/post381.html
>
> I first chose the "many threads with a global lock" architecture because
it
> does not require any central scheduler or event loop and because it seemed

> very easy to use. Blocking sockets ? add a thread. Callback based API ?
add
> a thread. GUI ? add a thread. You kind of understand the song here.
>

Gaspar,

Your post mentions the global lock approach and the coroutine approach, but
did you also try the Python GIL model as mentioned by Mike;
http://lua-users.org/lists/lua-l/2005-08/msg00571.html ? Or even better,
that model without the bytecounter (to reduce sync issues to the coroutine
level)?
If so, what where your results?

Thijs

In fact my model was what you call the "Python GIL" (selected lock/unlock) and some of the issues were related to finding where to unlock the lock (ScopedUnlock was my friend and enemy). I could not make it work, at least not with the working time I have (it requires very deep knowledge of all the code paths in external APIs). I did not even consider locking in all C transitions (what you call the "global lock") for performance reasons. I was really struggling to find a proper alternative until I understood that all my selected locks (IO related) could be fused together in a single "select" without any of the threading nightmares that I had been through (no deadlock risk).

The "select" model is not perfect and totally trivial though:

1. It makes garbage collection of threads difficult because they are all accessible from the main scheduler (OS threads can simply die on garbage collection).

2. Rescheduling involves some hassle with the event queue (OS threads can be interrupted and timers restarted).

3. You have to be careful not to yield across C or pcall boundaries (this involves some coroutine wrapping instead of pcall).

4. Working with libraries that do not provide a file descriptor can be difficult (Avahi is a good example).

Good luck !

                                                               Gaspard