Re: Suggestion: Lua 5.3 -- a facility for creating and manipulting Lua file handles (C API)

[Sean Conner <sean@...>]

It was thus said that the Great Sir Pogsalot once stated:
> 
> There are many situations where it would be advantageous to be able to
> create file handles that could use both your own I/O functions and the
> standard io.read()/io.write().
> 
> One example is that I have created my own unfinished/shoddy binding[6] to
> BSD sockets because I could not get "low enough" with luasocket[7].
>  luasocket represents sockets as its own userdata type, not as a Lua file
> handle -- but I wanted the ability to call recv() on my socket type and
> also be able to io.write() to it (I managed to get this working! \o/).

  I did a similar thing by creating my own socket libary [11], and I too,
wanted a way to use the standard Lua IO calls on it.  I ended up with the
following:

	net  = require "org.conman.net"
	fsys = require "org.conman.fsys" -- [12]

	address = net.address('127.0.0.1','tcp',80)
	socket = net.socket('127.0.0.1','tcp')
	socket:connect(address)
	fp = fsys.openfd(socket:fd(),"rw")
	-- fp has the type as if I called io.open()

  I thought about removing socket:fd(), but it proved too useful for my own
interface over select/poll/epoll() [13][14] (which I call a "pollset") that
requires the use of a Unix file descriptor, not a FILE * or a socket type. 
My rational for that went a bit like this:

	A pollset is usually used for network activity, so I could include
	the code for that with the net module.  *BUT* there are cases where
	it would be useful for certain devices like TTYs and serial ports,
	which can be opened using Lua's io.open() call.

	But if I'm using a pollset to manage TTYs (or serial ports) I'd
	rather not include a bunch of network related code, so there's a
	reason to keep the pollsets out of the network module, and put them
	into the file system module [12].

	But if I'm doing a bunch of network related stuff, I don't
	necessarily need the file system stuff.  Also, if I'm doing file
	system related stuff, I might not even use the pollset module, and
	thus, why include the code there?

  Thus, both lines of thought lead me to make pollsets their own module. 
But I didn't really want the pollset module to have to understand the
internal structure of my sockets; in fact, I wanted it to be as independent
as possible.  But pollsets are defined for file descriptors.  So in my
opinion, the best solution is to pass in file descriptors to the pollset,
and leave obtaining said file descriptors to other code.  So that's why my
network module has socket:fd(), and I created a function in my file system
module to return the file scriptor of a Lua file (a FILE *).  

  Getting back to the network module.  I initially defined socket:read() and
socket:write() functions, but they didn't work the same as Lua's file:read()
and file:write() functions, because of the semantics of packets [15].  So I
renamed them socket:recv() and socket:send() (based on the recvfrom() and
sendto() kernel calls).  This works well for both UDP and TCP (yes, you can
use recvfrom() on TCP sockets) and an implementation of Lua's file:read()
(50 lines of Lua) and file:write() can be written on top of this, although I
did avoid including this in the actual module since a majority of the work I
do is with UDP.

> luasec[8] wraps luasocket objects to provide encryption on network
> connections (basic gist -- I'm not a security person).  It calls a method
> (socket:getfd()) to retrieve the file descriptor from the luasocket C
> object and then returns a dummy object after it has "wrapped" the socket.
>  In this way, luasec doesn't have to know the implementation details of the
> socket object in luasocket.  I, however, think it would be great if luasec
> could assume it were a standard Lua file handle.
> 
> luaposix[9] is another well-known project with the ability to retrieve the
> file descriptor number from a Lua file handle (fileno()[10]).

  All abstractions leak.  

> In Lua 5.3 I would love to have the ability to create and manipulate Lua
> file handles without creating my own copies of internal functions from
> liolib.c and lauxlib.h.  They are very simple objects (as shown above), and
> some projects would benefit from having an official way to create this type
> of userdata so that they become more interoperable with other third-party
> projects.  In my [unfinished] lsock project I have shown that by
> representing sockets as Lua file handles I can make use of my network
> bindings and use what is available in the io library (shameless promotion,
> maybe).

  The code to create a LUA_FILEHANDLE isn't that bad though:

	static int fsys_openfd(lua_State *L)
	{
	  FILE **pfp; 
	  
	  pfp  = lua_newuserdata(L,sizeof(FILE *));
	  *pfp = NULL;  /* see comments in fsys_pipe() */ /* [16] */
	  luaL_getmetatable(L,LUA_FILEHANDLE);
	  lua_setmetatable(L,-2);
	  
	  *pfp = fdopen(
	                luaL_checkinteger(L,1),
	                luaL_checkstring(L,2)
	        );
	
	  if (*pfp == NULL)
	  {
	    lua_pushnil(L);
	    lua_pushinteger(L,errno);
	    return 2;
	  }
	
	  lua_pushinteger(L,0);
	  return 2;  
	} 

> fileno() is part of C89 and C99, but fdopen() is POSIX,

  Sorry, but fileno() is POSIX, not C89/C99.  

> Anyway, that is what I am thinking about.  No more :getfd() -- file
> descriptors should not be known from the Lua side of things (imo) -- and
> then everyone could benefit from the common Lua file handle.
> 
> Thoughts?

  It mostly works, but the Lua :read() and :write() modules break down
(in my opinion) in the presence of UDP (or rather, non-stream based network
protocols).

  -spc

> Sources:
> 1: http://www.lua.org/manual/5.2/manual.html#luaL_checkudata
> 2: http://www.lua.org/source/5.2/liolib.c.html
> 3: http://www.lua.org/source/5.2/lauxlib.h.html#luaL_Stream
> 4: http://www.lua.org/source/5.2/liolib.c.html#io_fclose
> 5: http://www.lua.org/source/5.2/liolib.c.html#newfile
> 6: https://github.com/Pogs/lsock
> 7: http://w3.impa.br/~diego/software/luasocket/
> 8: https://github.com/brunoos/luasec/wiki
> 9: https://github.com/luaposix/luaposix
> 10: https://github.com/luaposix/luaposix/blob/master/ext/posix/posix.c#L848
> 
> PS: I'm going to be so mad if there's a typo in here somewhere...

[11]	https://github.com/spc476/lua-conmanorg/blob/master/src/net.c

[12]	https://github.com/spc476/lua-conmanorg/blob/master/src/fsys.c

[13]	https://github.com/spc476/lua-conmanorg/blob/master/src/pollset.c

[14]	Under Linux, the underlying implementation uses epoll(); under other
	Unix systems it uses poll() and if those are unavailable, there is a
	final implementation that uses select().

[15]	TCP lends itself to :read() and :write() semantics, but not UDP,
	which is more packet oriented.

[16]	fsys_pipe() is a bit more complicated because of the potential
	errors when creating the two FILE* and leaking resources.  I'm not
	sure if your proposal would have made that function any easier to
	write or not.  But the code is in [12].