Thanks for the discussions during lw11 (closures vs mutex)

[Gaspard Bucher <gaspard@...>]

I have just finished rewriting all the core parts in Lubyk (lk.Thread, lk.Timer, lk.Socket, zmq.Socket, mDNS) to use a centralized scheduler and coroutines [1] instead of using OS threads and a global mutex.

The continuations model works very well for sockets:

1. reading data:

2. not read-ready ?

3. ... yield(filedescriptor, 'read')  ----> this will add the coroutine to the select

4. recv()

The hardest parts in this rewrite were:

1. Find a way so that the scheduler does not hold too tight on the coroutines so that they get garbage collected (we do not want users to have to worry about cleaning up). The solution was to attach a finalizer [2] to the thread so that the coroutine cannot be accessed from the scheduler and we never wake half dead zombies.

2. Find a way to move all posix sockets and zmq sockets inside Qt's event loop. This was made possible by creating a 'poll' object and tricking the scheduler (when Qt is loaded, the poll operation becomes a yield).

3. Hopefully, mDNS (Bonjour) use filedescriptors so this was easy to adapt.

I am pretty proud of the final result because it does not crash, it is easy to stop (one thread) and automagically changes from the zmq based (more precise) scheduler to Qt's event loop when the dylib is loaded.

I would never have achieved this without the deep discussions during the Lua workshop and the flexibility of Lua. So thanks a lot to those who spent some time enlightening me in Frick or on the mailing list and to the authors for such a nice programming language...

Cheers,

                                                               Gaspard

[1] http://bit.ly/qpAPgk (lk.Scheduler)

[2] http://bit.ly/pUiI0h (finalizer in lk.Thread)

And finally, something that never happened with the previous scheduling model:

==== lk.crypto          ( 2 tests): OK

==== lk.DavServer       ( 1 tests): OK

==== lk.Thread          ( 1 tests): OK

==== lk.Dir             ( 1 tests): OK

==== lk.Environment     ( 2 tests): OK

==== lk.FileResource    (14 tests): OK

==== lk.Finalizer       ( 2 tests): OK

==== lk.Inlet           ( 6 tests): OK

==== lk.InletMethod     ( 5 tests): OK

==== lk.path helpers    (20 tests): OK

==== lk.Morph           ( 6 tests): OK

==== lk.Mutex           ( 3 tests): OK

==== lk.Node            (13 tests): OK

==== lk.Outlet          ( 3 tests): OK

==== lk.OutletMethod    ( 4 tests): OK

==== lk.Patch           (13 tests): OK

==== lk.Process         ( 2 tests): OK

==== lk.ProcessWatch    ( 2 tests): OK

==== lk.Properties      (12 tests): OK

==== lk.RemoteProcess   ( 2 tests): OK

==== lk.Scheduler       ( 5 tests): OK

==== lk.Service         ( 1 tests): OK

==== lk.Socket          ( 6 tests): OK

==== lk.Thread          ( 7 tests): OK

==== lk.Timer           ( 4 tests): OK

==== zmq.Pub/Sub        ( 3 tests): OK

==== zmq.Push/Pull      ( 3 tests): OK

==== zmq.Req/Rep        ( 1 tests): OK

==== zmq.Socket         ( 1 tests): OK

==== mdns               ( 1 tests): OK

Success! 146 tests pass (731 assertions).