Re: A Batteries API specification (was Re: LuaDEAL - Lua DEad Alive Libraries)

[Sean Conner <sean@...>]

It was thus said that the Great pocomane once stated:
> On Thu, Jan 30, 2020 at 9:52 AM Sean Conner wrote:
> >   Generally under POSIX, if it has a file handle, it can be used for event
> > processing.  More and more modern POSIX systems are making non-file objects
> > available via file descriptors (like signals under Linux).  Right now, I"m
> > aiming more for a "widely supported event processing" model, which now is
> > mostly file-related.
> 
> Ok. I was hoping it was a more general purpose "Event notification
> system", but it is fair enough to start with the most common cases.
> 
> > > I am not discussing the implementation here, if there is the need for
> > > a "Fake file descriptor" internally, it can be done [1]. In this
> > > spirit I would substitute "File descriptor" and "_tofd" with something
> > > else.
> >
> >   I'm open to suggestions.  I don't like "_tofd" myself, but I'm not sure
> > what a better name would be.
> 
> The fact is that a file descriptor does multiple different things (it
> both controls the stream and provides the stream) so no name will be
> satisfactory. 

  I don't follow.  A file descriptor on POSIX is a integer value that has
meaning for the operating system, not user code [a].  It's largely a token
used to reference an open file that you hand back to the operating system to
have it perform operations on said file (reading data, writing data, maybe
setting some option about how to read/write the data).  The standard C
library further abstracts this out to a "FILE *", which on a POSIX system
contains a file descriptor, plus some other data to support the C
abstraction to files.

[a]	It's actually an index into a per-process array of files if you
	really want to be pedantic about it.

> E.g. for the monitoring you could speak of "Event
> source" (_eventsource()), but it is weird when you realize that it is
> the same object you write to. Maybe it is better to make the reference
> completely opaque at this point,  with somthing like "_rawresource()".
> 
> >         events = nfl.SOCKETS:events()
> >         for _,event in ipairs(events) do
> >           event.obj(event)
> >         end
> 
> It seems to me that all the examples would be practically identical
> with the return format I proposed. But it let you to write also
> 
> monitor:insert(sock1, 'rw')
> events = set:events()
> if events[sock1] and events[sock1].read then
>   -- do something
> end
> 
> that it is a more direct than to loop on a list and to check if the
> object is the wanted one.

  I might have failed to fully describe the output from getting the events
list.  It doesn't return a set of *all* the event sources leaving the user
to scan through the list looking for ones with activity, it returns a list
*of only the sources that have activity during that call*.  If I set up
event monitoring for 10 sources and call set:events(), I might get back a
list of two events, but both those events have been triggered.

> However for me this is a very minor issue.
> 
> > > Moreover, I think that keeping it simple is more valuable than your
> > > trick to substitute an optional var to Obj. If a user code really need
> > > it, it can keep the map in a table by its own.
> >
> >   Honestly, I didn't find the current implementation all that difficult, and
> > I did it four different times (select(), poll(), epoll() and kqueue()).
> 
> I think that also a very small over-structure should be avoided if its
> implementation in the user code has the same complexity:
> 
> myhandler[obj] = function()
>   --...
> end
> 
> events = nfl.SOCKETS:events()
> for _,event in ipairs(events) do
>     myhandler[event.obj](event) -- instead of event.obj(event)
> end

  But I have to keep track *anyway*.

  When I call io.open(), I don't get back a file descrptor, but a userdata,
which itself contains the standard C 'FILE *' object, which contains a valid
target for events.  If a open a socket, I will get back some userdata which
contains some abstraction of the socket (on POSIX, a file descpriptor),
which contains a valid target for events.  I have to track these
userobjects, in addition to the operating system level object anyway. Since
I have to track this information *anyway*, it's just as easy to implement
arbitrary data as it is the userdata.

> > > [1] However, probably every lua-side standard specification will need
> > > some c-side specification. And probably, some internal details have to
> > > be exposed there.
> >
> >   What did you have in mind?
> >
> 
> Nothing, if you are thinking just to file descriptors. But I would
> like to see a "Event Notification Module" (ENM) that could in
> principle work with any kind of event. Something like libuv, but much
> simpler, and extendable by other C modules.

  Um ... libuv is just a wrapper for select(), poll(), epoll(), kqueue(),
IOCP, et. al, which is what I'm describing.  Also, epoll() (a Linux specific
function) allow allows handing of signals and file system events; kqueue()
can do that but also process specific events, which I think IOCP (Windows)
can do as well.  

  It's not like I'm explicitely limiting the events to *just* sockets or
specicial files, but that's the common ground among all the event handlers.

> I do not have any proposal, neither for the API. If I have to put
> together something, I would say to register a callback somewhere, e.g.
> in an hypothetical Gamepad Module (GM):

  Yeah, I described using callbacks with my example.

> // pseudo-C
> enm_register_scheme("gm", my_gm_callback);
> 
> Then in lua you could:
> 
> local evsrc = enm.newmonitor"gm://1"
> local s = enm.newpollset()
> s:insert(evsrc, 'r') -- my_gm_callback is called
> s:insert(enm.newmonitor"tcp://aserver:1234", 'r')
> local event = s:wait(0.1) -- my_gm_callback is called (the tcp handler is too)

  On POSIX, this will most likely have to be some form of descriptor; on
Windows some type of handle (which is really the same concept as POSIX file
descriptors---some integer number that has meaning for Windows but is
otherwise an opaque value to usercode).

> How does s:wait work? If there were just files and sockets, it would be a
> select/poll/whatever. But with the GM? Thread pools and worker queues? I
> think the important here is just to specify a C API that let enm_wait to
> act as it was a select/poll/whatever, delegating to my_gm_callback the raw
> work.
>
> Sorry for the very raw state of my ideas.

  So raw they seem like pure speculation and uncertainly about how this
stuff works.
 
> Finally, I think it is important to start stating what should contain
> the standard library. E.g.:
> - File system interaction

  I've already covered directory handling.

> - Socket

  I wanted to wait until the event stuff is cleared up before going into
sockets.

> - Event notification module

  What this is about.

> - Crypto/TLS (I would prefer an embeddable library without external
> so/dll, like bearssl)

  Again, you are missing my point---I'm trying to figure out what the API
should look like---it's up to a module to implement the API using whatever
underlying library/system has to offer.  There are several Lua modules using
OpenSSL; my own TLS wrapper uses LibreSSL (and in particular, libtls).  Feel
free to implement the API using bearssl, but don't mandate a particular
implementation.

> - Object oriented programming

  Why?  

> Other (?):
> - Logging
> - Audio
> - GL/glext/GLFW/image-loading
> - What else?

  So, start describing.  Come up with an API.

  -spc (Who's beginning to think Lorenzo is correct about my efforts)