Re: co-routines [was Re: Unicode]

[Edgar Toernig <froese@...>]

[MULTIPLE STATES]

Roberto Ierusalimschy wrote:
>
> Notice that, in my implementation, a thread (or stack) is not a new type,
> and is not subjected to GC: The external code must destroy (close) it
> explicitily.

The same as of Sol-0.42.

> (Of course, you can pack it inside a userdata, and set a GC tag method for
> that.)

And this is exactly the case I meant.  Each coroutine is implemented as
a userdata object that holds the state.  In the first pass the userdata
is collect (but the state was still present), the gc-callback frees the
state and in the next pass the objects of the state are freed.

> > I noticed two points: you need something to pass data from one state to
> > another. Your solution with lua_ref(L1)/lua_getref(L2) is IMHO not correct.
> > You may raise an error in the wrong state.
>
> You have to work really hard to get an error calling lua_getref.

A Lua-stack overflow.  And the error will be deadly!

> But I agree that it is not "polite" to do calls over a foreigner state.

Not only "not polite".  Between two API calls (i.e. lua_getref(L2, x)
lua_setglobals(L2)) the other thread may run and you have just put him
some garbage on the stack...

> A "passvalue" function is already in my "to think" list (although I guess
> that for that case we need a "sendvalue" instead of a "receive" one).

I needed the sendvalue too.  But just during the setup phase where I'm sure
a stack overflow cannot happen (the stack is new and empty).  So I've
misused receivevalue.

> > Another point is that I have globals for each state.  At the moment
> > (with the standard API) it seems impossible to change globals for
> > other states without a new API function.
>
> Each thread has its own "globals". lua_setglobals operate over the
> given thread, so you can change globals as you like. (Again, you may
> need to call that function over the "wrong" thread, but again you have
> to work hard to get an error over lua_setglobals.)

Lua_setglobals cannot raise an error.  But you have to get the table
to the other state.  Same problem as above.  IMHO that's a generic
problem.  Maybe a function similar to luaD_runprotected is required to
perform a couple of operations atomically within the other state and it
will forward errors to the local state.



[LOCKING]

> > Sounds like the "Big Kernel Lock" in first Linux SMP kernels.  The problem
> > was that it scaled very bad.  Basically you limit the use of Lua to one
> > state at a time.
>
> I don't think this is really a "Big Kernel Lock". Every time Lua calls C it
> unlocks, so even a loop like «while 1 do a=sin(i) end» will be preemptive.
> (OK, you cannot do busy waiting, like «while not a do end») And, as I said,
> you can register an empty call hook, so that every function call that Lua
> makes it also unlocks; [...]

On SMP only 1 CPU may be within Lua.  On single CPU systems every thread
fights for the lock and you get a thread ping pong.  IMHO, it does not
scale.  But maybe most people just want pthreads and do not care about
scalability.

> I also like coroutines, And, using a call hook as a dispatcher, we can get
> almost "real threads" with coroutines.

As a bonus you get all this nasty locking stuff in the high level language :-)

> But coroutines block the whole program in I/O operations (yes, you
> may use select, but not always, and not for output).

Why not for output?  Especially in the defined environment you find in
Lua it would be reasonable (not in ANSI-C though).  And then you have
what most people call "real threads".

> Moreover (again, I agree this is sad) most OS offer an "official"
> implementation for threads, but not for coroutines.

And even more sad, the only (regular) OS I know that supports coroutines
(they call it different, though *g*) is the one I will never work with.

> Even the stack needs lock, because another thread may be doing garbage
> collection.

Uh, yes, the GC would be harder.  It wants to run exclusive in one thread
and wants to lock everything.  You would need something to synchronize all
threads...

> And the more locks you have, the bigger the chance of subtle
> errors; and the performance gets worst too (lock/unlock are not that
> cheap).

Guess why I wrote "Good luck in mutex hell" once ;-)  Afaik, locks
are reasonable fast as long as there's no contention.  But to get
everything right is IMHO kind of masochism ;-)

I guess, a lot of people are satisfied with the single lock.  I do
not need pthreads, only simple coroutines.  And these seem to work
well with the interface.

Ciao, ET.