Re: Is it possible to yield/resume the main thread?

[Francisco Olarte <folarte@...>]

Stefan:


On Sat, Jul 4, 2020 at 1:14 AM Stefan <ste@evelance.de> wrote:
>
> Am 03.07.2020 um 18:49 schrieb Francisco Olarte:
> > Anyway, you do not have to use the main coroutine to run your thread,
> > why don't you just use an alternate one and lua_yield on whatever
> > lua_state is passed into your C callback?
> >
> > If I'm not too confused just try:
> >
> >> int main()
> >> {
> >>     L = luaL_newstate();
> >>     luaL_openlibs(L);
> >>     lua_register(L, "cb", luacb);
> >
> > MAINL = luaL_newstate(), openlibs and register in mainL, then do...
> >
> >     L = lua_newthread(MAINL);
> >
> > All the rest mainly unchanged.
> >>     const char script[] =
> > ...
> >    }
> >> close_state:
> >>     lua_close(L);
> >
> > And  change this to lua_close(MAINL) here.
>
> I tried this approach as well and it gives the same result - it
> skips the 'Y'.

You are totally right, I forgot you where using:
>>>>
lua_State* L;
int luacb(lua_State* T)
{
    return lua_yield(L, 0);
}
<<<<

Which, IMO, is a recipe for disaster, I should have noted it and added
"change lua_yield to use the state passed to it, lua_yield(T,0)".

The fact that coroutine.yield does not take a co parameter like resume
or status, clearly hints to the state used in lua_yield not being used
to "direct" the yield, but just for all the other stuff the api does.
I did not check that too much because it never occurred to me someone
would try to yield from a "hidden" ( global, not passed in ) pointer
in a callback.

> The 5.1 manual is more direct about what yield does:
> "When a C function calls lua_yield in that way, the running coroutine
> suspends its execution, and the call to lua_resume that started this
> coroutine returns."
>
> But is this still true? Or can the CFunction yield a different thread
> as well?

You might be able to do it, but IMO you are in for a lot of pain.

Lua yield is a primitive for asymmetric coroutines, you do not "yield
to a coroutine", in Lua , you just yield to your parent. If you want
symmetric ones you can build a kernel for that, in a hundred lines
supporting it, it is a classical programming exercise, basically you
use the main thread as a supervisor, the only one calling resume, and
make it be the only one calling resume and bridging amongst the other
coroutines, and optimize from there.

> After yielding the 3 different threads in the callback (the one passed
> as parameter, the main thread, the extra thread created as you
> suggested) on 5.1.5, 5.3.5, 5.4.0 and LuaJIT2.1 it looks like they
> all do different things, including crashing, printing a random double,
> or ignoring the second yield.

I think this is because you are constantly invoking undefined
behaviour, but I'm not that versed.

> To me it seems that it is not safe to yield directly into the
> "ground layer" C.

That I take as granted, especifically using an external state. Even in
lua it is not that safe to blindly yield. Normally when I'm doing
coroutines, typically to linearize flows on event driven systems, I
run everything inside a coroutine, typically defined in the lua side,
I just use the main thread to dispatch and I normally never
resume/yield in C ( even handlers go to main, main identifies handler,
resumes it, handlers does it work. Async callbacks in C just fire some
stuff and return an identifying token to lua, which the lua wrapper
typically stores somewhere and return, its caller eventually yields in
lua, main thread returns and a later event reconstructs the token and
calls the main thread which uses the token to resume the appropiate
coroutine to handle it ).


> > I've been using coroutine packages since CP/M and MSDOS, even written
> > some for those, and I've normally found "do not run app code in the
> > main coroutine" to be the easier thing. In fact package I wrote having
> > the resume/yield style did not have a main coroutine.
> Thanks for the interesting story! Are there multiple threads or
> processors involved or is it fully cooperative?

It's CP/M (8080) / MSDOS (8086), no multicore there. Anyway, when I've
done this can of things in some of this things my coroutines where in
a scheduler which worked more or less like a lua state, you could
invoke it from different threads but only one at a time. I normally
did not run coroutines for work splitting, but for synchronization of
logic.

In recent ones I tend to not write any of them anymore, there are too
many good coroutine packages to bother. What I have this days is
typically a thread drinking from a message queue running a lua state
with coroutines, a thread pool doing http, voip and other stuff
getting the work to do from another queue and posting results to lua.
Lua does high level control logic, it's fast enough for all I do once
I offload the waiting or slow stuff ( sometimes I offload preparsing
of complex documents), and I use it specifically because coroutines
are easy and just work.

Francisco Olarte.