Re: green-threads in lua

[folays <folays@...>]

"Ivan Gagis" <igagis@gmail.com> writes:

> Hi Patrick,
> 
> lua docs for lua_yield() says that:
> This function should only be called as the return expression of a
> C function, as follows:
> 
>      return lua_yield (L, nresults);
> 
> so, I'm not sure it is possible to use lua_yield() from within the debug
> hook.

It is (i did it myself) but only on some conditions, one of the most
restrictive being that you cannot yield inside iterator (doing so will
raise an error the coroutine, it will not be resumable). There are other
restrictions but most of them (if not all) can only be triggered if you
expose specific features to the Lua code, allowing it to use them:
- setting and usage of metamethods
- call C functions : you can (hardly) workaround that by proxying with a
Lua function, the C function and the Lua proxy being under your control,
you can write a scheduler, pause execution in the C function if you want,
registering state of operations of the C function, and returning a flag 
to the Lua proxy for making it yielding, thus with the registeted state of
what the C function was doing, you can resume back the processing.
- pcall: probably can be workaround with the combination of the workaround
for C functions above and a new coroutine inside the new "context".

But iterators are problematic: If what you're trying to achieve is a
sandbox, then with the stock Lua, you cannot prevent untrusted code to
create and use iterator, you can still set a debug hook but if it is called
inside a "metamethod/C-call boundary" you will either have to give a new
slice time to the coroutine or kill it by yielding. Thus, you won't be able
to pause the coroutine (for scheduling and giving a slice time to another
coroutine) without killing the first.

So, with Lua 5.1.3 the current state is that you cannot make a scheduler
(with time slices and all) able to handle execution of multiple, possibly
longtime running Lua script, without having to kill them if you want to
schedule another while the debug hook triggered inside an iterator (rather
than just lowering the coroutine priority in your scheduler and schedule
another one). Again, that's under the assumption that you really want to
safely run (multiples,sandboxed,unstruted,unknown Lua code) at the same
time, inside the same Lua State, both allowing them to run a large amount
of time and scheduling them to make them run smoothly and independently.

Please correct me if i'm wrong, i would really like to be :)

Don't get me wrong, i like Lua and use it extensively, and it already does
a lot.

Now, you may want to know that there are some way to have what's called a
fully resumable VM:
- http://lua-users.org/wiki/ResumableVmPatch
- LuaJIT 1.0 (Coco fix this)
- http://lua-users.org/wiki/LuaFiveTwo , the Coco's patch is under
consideration
- LuaJIT 2.0, i believe that i've read in the past (maybe on the roadmap)
that support for yielding everywhere could be dropped, until more recently
where i believe i've read Mike Pall telling that support for this is
scheduled again.

Personally i've been working on a from-scrath, in C, small (less than 2k
lines for now) HTTPd featuring Lua comparatively to PHP, with all aspects
described above (multiples coroutines per Lua State, scheduled, sandboxed
and independent). It works well except when the instruction hook trigger
inside an iterator: The poor coroutine is killed. Or i could use real C
threads with multiple Lua states (which would probably work really well).
But i like the idea to wrote a Lua coroutines scheduler, and being able to
have a http://localhost/status.lua page describing the status of the
coroutines (much like the STATE column displayed by BSD's `/usr/bin/top').

So now, i put my project in pause, waiting to see what wonderful features
will provide Lua 5.2 and LuaJIT 2.0 :)

You probably don't care my experiences with Lua but the goal is to make
authors know what the features are wished, with purposes in mind.

> I have not tried this but if it works the it will be a possible way to
> implement the green threads in lua, thank you for the idea.
> 
> But anyway, I do not think that using the debug abilities for normal work is
> not a very good solution, and not the fastest one.
> So, I think that introducing the capability of executing specified number of
> instructions in the next version of lua will be a good language improvement.
> 
> Thanks,
> Ivan

-- 
folays