Re: Coroutine library scheduler

[Andrew Starks <andrew.starks@...>]

On Mon, Feb 24, 2014 at 4:14 PM, Sean Conner <sean@conman.org> wrote:
>
> It was thus said that the Great John Hind once stated:
> > I do not have a suggestion for anything that already exists, but I do
> > think something really powerful could be done with a coroutine scheduler
> > for Lua.
> >
> > For example, I would like to be able to say "yield this coroutine until
> > some real-time criterion is met" ( yield(for 10 seconds), or yield(until
> > packet received OR 20 seconds has elapsed) ).
>
>   I've written such code, to turn event based programming (using poll() or
> epoll(), depending upon the underlying operating system [1].  One thing to
> remember:  Lua coroutines are cooperative, not preemtive [2].  For the code
> I've written, given that it's mostly network driven, I explicitely ignore
> the fact that any coroutine could hog the CPU indefinitely (that is:  I have
> to make sure I have no infinite loops---I'm not going to attempt to
> implement preemptive threading).  But, any routine that could potentially
> "block" now becomes a point of scheduling, which means, I have to supply my
> own read() and write() calls (or even sleep()).
>
>   (For the record, I recently did a test and figured out that a coroutine on
> 32 bit x86 system is around 900 bytes---much smaller and lighter than a
> pthread)
>
> > But that coroutine scheduling idea is just so technically sweet ...
>
>   The scheduler I have is simple:  I'm either waiting for an event, or I'm
> not.  As such, I have two lists, one of a list of coroutine waiting for an
> event, and a list of coroutines that are ready to run (because their events
> have arrived).  Simple round-robin scheduling.  I found no need for anything
> more complicated than that. [3]
>
>   -spc (I can post code if anyone is interested)
>
> [1]     Unix variants, I should note.
>
> [2]     Yes, you can do preemtive Lua coroutine, if you want performace to
>         drop like a stone.  To do it safely, you have to hook into the debug
>         layer and trap either intructions (each routine to run for X
>         instructions), lines, or functions.  I personally don't recomend it.
>
> [3]     Really, any more complicated than that, and you know what?  You're
>         writing a kernel.
>



I thought I should follow up with more than "Lumen looks nice!"...

Whenever I think about multi-threading Lua, my brain leaps to message
passing patterns.

Given that, llthreads + zmq or nanomsg + some kind of serializer /
deserializer  = a pretty powerful tool set. Add a coroutine scheduler
and you have a something that can handle events, idling, etc.

That's kind of what Lumen is about. It has tasks, the ability to kill
a thread, timeouts, pipes, etc.

This is from the readme file:

```lua

    local sched=require 'lumen.sched'

    -- task receives signals
    sched.run(function()
        local waitd = {'an_event'}
        while true do
            local _, data = sched.wait(waitd)
            print(data)
        end
    end)

    -- task emits signals
    sched.run(function()
        for i = 1, 10 do
            sched.signal('an_event', i)
            sched.sleep(1)
        end
    end)

    sched.loop()

```

The tests folder is helpful as a tour guide, as well.

Sometimes parts are a bit... out of sorts, but this is to be expected
with something that is under development. It's *documented* (gasp)
pretty well and its design seems very reasonable. It is based on
SierraWirelsse's luasched[1].

Also, it comes with baked in support for nixio, tcp/ip and possibly others.

-Andrew

[1] https://github.com/SierraWireless/luasched/