Re: Coroutines and blocking IO

[Jon Smirl <jonsmirl@...>]

On 1/26/06, Javier Guerra <javier@guerrag.com> wrote:
> On Thursday 26 January 2006 4:19 pm, Jon Smirl wrote:
> > > implement read()=EAGAIN, call out to a registered callback if present,
> > > epoll_wait(), get event, return.
> >
> > Note that if the filesystem doesn't implement O_NONBLOCK you simply
> > won't get the EAGAIN return from the read call. If you code assuming
> > O_NONBLOCK is implemented the same code will work if it is not.
>
> right, the low level facilities are in place. but, since most of the unix
> culture is "let it block, fork if you need to multitask", most high level
> libraries expose only the blocking behavior.

You can have APIs that block without supporting a non-blocking API
still keep lua working. Down inside the C implementation of the
library create a native IO thread for the problem subsystem.  The IO
thread goes off and does the blocking IO leaving the main thread to
keep running lua code. When the non-blocking IO completes the IO
thread messages the main thread. On the next lua schedule it picks up
the completed IO from the IO thread and returns it to the lua code.

Just because libraries are using IO threads doesn't imply that lua is
aware that native threading is happening.

> the challenging part isn't writing the libraries, it's designing good
> conventions that would let us join IO from several libraries and tie them all
> with a single select()/poll()/callback, whatever.

This is the key. There can only be one poll() happening in the system
so this must be coordinated. Check out libevent.
http://www.monkey.org/~provos/libevent/

It would also make sense to provide a sample implementation of the
native IO thread scheme so that it also gets implemented in a
consistent manner.

> a small proposal: writing a thin C library that wraps FILE* or fd- like
> objects in a userdata and provides a few generic Lua functions: select(),
> poll(), event(), callback(), or something like that.  not the opening,
> data-handling, closing, etc, funcitons, but the blocking,events,callback kind
> of things.
>
> then, any IO Lua library could use this C library to create this objects and
> let them all be multiplexed together.  right now, the file library and the
> LuaSocket library don't know of each other, select() is in LuaSocket, LuaSQL
> does only blocking IO, and so on.
>
> let's write the scaffolding!
>
> > lua should definitely provide a run-time implementation of sendfile()
>
> no, that should definitely be in a library, not the language.

I meant that as a standard library instead of having everyone write it
on their own.

--
Jon Smirl
jonsmirl@gmail.com