Re: A Batteries API specification (was Re: LuaDEAL - Lua DEad Alive Libraries)

[Sean Conner <sean@...>]

It was thus said that the Great Sean Conner once stated:
> It was thus said that the Great Sean Conner once stated:
> > It was thus said that the Great Lorenzo Donati once stated:
> > > 
> > > For that some kind of "mandated/blessed" API specifications would be needed.
> > 
> >   Okay, let's see if the community can just come together to define the
> > minimum viable module for directory functions. 
> 
>   Okay, second take.  I've looked at LuaFileSystem, LuaPosix, PenLight and
> my own org.conman.fsys modules and I've come up with a set of minimal
> directory functions.  I've gone ahead and decided to use the method of error
> reporting that the existing Lua functions (like io.open()) use---namely,
> return a falsey value, a string with the error message, and an integer
> representing the error as a value.  I don't like it, but I'm going with the
> majority view point here.

  Some discussion about the second take on directory functions but no real
major challenges to the API, so why don't I now discuss an event API.  

  I looked at the following modules:

	luasocket
	luaposix
	se.zash.poll
	lua-epoll
	org.conman.pollset (bias alert)

  These were the modules I found that were *only* provided support for
events, and not other unwanted functionality.  It's been brought up that a
major problem for Lua modules are the lack of composability and one of my
intentions with this thread is to make modules with a base level of
functionality and focus so they can be used as needed to build upon. 

  Of the modules I did study, there doesn't seem to be an overall common
method to how they were implemented.  luasocket only supports the select()
function, and it's API is:

	readset    = { fd1 , fd2 , fd3 }
	writeset   = { fd1 , fd2 , fd3 }
	r,w[,errs] = select(readset,writeset[,timeout])

  Returned is a list of file destriptors ready for reading, another list of
file descriptors ready for writing, and a set for priority (or out-of-band)
data (it is *not* for errors, at least on Linux).  The error returned is one
of two strings:

		"timeout"
		"select failed"

  It's up to the caller to manage the file descriptor lists, and the file
descriptor lists are just that---file descriptors and not the FILE* object
return from io.open().  The timeout value is a double, representing the
number of seconds to wait for, if not given, wait forever.

  luaposix uses poll() instead of select(), and its API also requires the
caller to manage a list of file descriptors:

	list =
	{
	  [fd1] = { events = { POLL_IN = true , POLL_OUT = true } , revents = {} },
	  [fd2] = { events = { POLL_IN = true , POLL_PRI = true } , revents = {} },
	}

	ret[,errs,erri] = poll(list[,timeout])

  And here, the timeout is an integer represnting the number of milliseconds
to wait for.  The passed in list is modified to fill in the revents fields,
listing the file descriptors that have pending events.  Errors are returned
as both a message string and an integer.

  se.zash.poll is based around select() or epoll() (Linux) and here, the
caller does not have to manage a list of file descriptors.  This API is
an odd mixture of ease and clumsiness:

	okay[,errs,erri]        = poll.add(fd[,readable[,writable]])
	okay[,errs,erri]        = poll.set(fd[,readable[,writable]])
	okay[,errs,erri]        = poll.del(fd)
	fd,readevent,writeevent = poll.wait([timeout])

  It's easy to add, remove and modify the events for a given file
descriptor, but pulling the events out requires repeated calling as it only
returns the next ready file descriptor (and two flags, one for a read event,
one for a write event) and not a list.  The timeout value is again a number
representing the number of milliseconds to wait.  The readable and writable
parameters are booleans, indicating which events you are interested in
receiving for the given file descriptor.

  lua-epoll() is based around epoll() (Linux) is similar to the one above,
as it was written by the same person.

  And finally org.conman.pollset, the one I wrote.  This supports select(),
poll(), epoll() and kqueue() (it selects the appropriate backend at compile
time), and is more like the se.zash.poll/lua-epoll module than the others
but does have some differences:

	pollset     = require "org.conman.pollset" -- returns a function 

	set,erri    = pollset()		-- returns a userdata
	erri        = set:insert(fileobj,events[,val])
	erri        = set:update(fileobj,events)
	erri        = set:remove(fileobj)
	list[,erri] = set:events([timeout])

  The fileobj parameters is anything that can respond to fileobj:_tofd()
call, which is a function that returns a file descriptor that can be used to
pass to the event function.  I did it this way to abstract out what it is
I'm passing in (either a socket or a file from io.open() [1]).  The events
parameters is a string similar to the mode string for io.open().  The
various letters represent:

	r	- select read events
	w	- select write events
	e	- select error events (not supported across all backends)
	p	- select priority data (is this even a thing these days?)

  The list that is returned is an array of events, of the following
structure:

	{
	  {
	    obj      = fileobj, -- possibly, see below
	    read     = true,    -- read event
	    write    = true,    -- write event
	    error    = true,    -- error event
	    priority = true,    -- priority event
	  },
	  { ... }
	}

  The obj field that is returned will be the fileobj passed in to
set:insert(), OR the optional val parmater (which can be any Lua value). 
For instance, I use the val parameter to pass in a function to handle the
event in my event loop, but it can be a coroutine or table or any other Lua
value.  If not given, it defaults to the original fileobj.

  I briefly checked out cqueues (the only framework out of half a dozen or
so) to see how it handled events, and it doesn't really surface an API for
events to Lua (unless I am mistaken, and I will admit I ould be).  I didn't
check any other framework beacuse I was unsure if they surfaced an API or
not, and I was just interested in an API, not a framework.

  I will admit a bias here---I think the API I have is probably the best of
the lot and with some minor modifications to the error reporting this is the
one I'll be promoting in this thread.

Usage:        set,err = org.conman.pollset()
Desc:         Return a file descriptor based event object
Return:       set (userdata/set) event object, nil on error
              err (integer) system error (0 - no error)

Usage:        set._implemenation (string) informtional only.  Will be:
                      * 'epoll'
                      * 'poll'
                      * 'select'
                      * 'kqueue'
		      * some other implementation defined value

Usage:        okay[,errs,erri] = set:insert(file,events[,obj])
Desc:         Insert a file into the event object
Input:        file (?) any object that has metatable field _tofd()
              events (string) string of single character flag of events (see above)
              obj (?/optional) value to associate with event (defaults to file object passed in)
Return:	      okay (boolean) true if success, false on failre
              errs (string) error message
              erri (integer) error number

Usage:        okay[,errs,erri] = set:update(file,events)
Desc:         Update the flags for a file in event object
Input:        file (?) any object that reponds to _tofd()
              events (string) string of events (see above)
Return:	      okay (boolean) true if success, false on failre
              errs (string) error message
              erri (integer) error number
Note:         file must have been added with set:insert(); othersise
              results are undefined

Usage:        okay[,errs,erri] = set:remove(file)
Desc:         Remove a file from the event object
Input:        file (?) any object that reponds to _tofd()
Return:	      okay (boolean) true if success, false on failre
              errs (string) error message
              erri (integer) error number
Note:         file must have been added with set:insert(); otherwise
              results are undefined

Usage:        events[,errs,erri] = set:events([timeout])
Desc:         Return list of events
Input:        timeout (number/optional) timeout in seconds, if not given, function will block until an event is received
Return:       events (table) array of events (nil on error), each entry is a table:
                      * read (boolean) read event
                      * write (boolean) write event
                      * priority (boolean) priority data event
		      * error (boolean) error event
                      * obj (?) value registered with set:insert()
              errs (strint) system error message
              erri (integer) system error value

  
  I don't have Windows, so I can't say for sure how Windows IOCP works, but
I'm hoping they can fit into the scheme I've outlined above.  Thoughts? 
Comments?  Criticisms?

  -spc

[1]	Yes, I do add a function to the FILE* metatable for this purpose.
	It's not part of org.conman.pollset since it's not always needed.