Re: Suggestion: Lua 5.3 -- a facility for creating and manipulting Lua file handles (C API)

[Sir Pogsalot <sir.pogsalot@...>]

In response to Coda:

I don't agree that opening up file handle creation to the C side of Lua would mean we have to forgo an appropriate level of security.  Lua does have a fair bit of type checking code to make sure its receiving file handles in the io library... The easiest way to fool it is to take any userdata you want and set the metatable to the "FILE*" table in the registry -- if you use the right io function you can cause a segfault if it looks at non-existent members of the userdata.  This would be easier in Lua 5.1 because you have newproxy() to create a zero-sized userdata. :-)  Still -- *SECURITY* :D

In response to Sean:

Unfortunately I am not too familiar with the ins and outs of using poll() or epoll() -- select() was the simplest for me to grasp and I did that by copying fd sets into tables and back.  Don't have much more to say about that...  :x

I was kind of a snob when I made my socket binding.  I saw the io library and how friendly it was compared to recv() and send() and I desperately wanted to fool those functions into working on my sockets.  I didn't want to wind up writing my own compatibility layer to the io functions or writing functions that were very similar -- I wanted to use THOSE functions with no shim.  I thought if I could orient my bindings so that they took Lua file handles and derived the necessary other type (fd on Linux, HANDLE on Windows) then I would be walking down Easy Street :]  For the most part it worked -- I have not had to duplicate any io functions and this means other libraries can operate on my sockets without knowing their not regular files.  Otherwise they error just like an invalid operation on a file would.

What I did in code I haven't yet committed is I backed the Lua file handle table with my lsock networking table so if it can't find :recv() in the first it chains to the 2nd.  It's sneaky but I'm not sure I'd do it in production because it means modifying the __index of the "FILE*" table.

I have actually [successfully] written to a UDP-packeted socket with io.write(), but I didn't do more experimenting to see if io.read() worked just as well.  What I loved most about using Lua file handles is that if the underlying socket doesn't support an operation or has already disconnected or the operation fails for whatever reason, the io.read/write functions will just error with: return nil, strerror(errno), errno

My favorite part is that my error handling code matches how an error would be propagated in the io functions.  I swear I got the biggest thrill out of seeing it fail to write to a socket sometimes.

I think I'll go back on what I said about :getfd().  I don't like fds being known to the Lua side of things, but a fair number of libraries expect that method to exist.  It is somewhat common, but I think it only exists because people are representing files as different types (bad).  :>

The main focus of my socket bindings is that I was trying to be as true to the C prototype as possible so it would be easy for beginners to take a C networking tutorial and translate it to Lua without much guessing or digging through documentation.  Sometimes I feel like details about luasocket are hard to find...

I like your socket option code in net.c though, it's much friendlier than what I tried to do by adding every defined CONSTANT I could and having a get/setsockopt().  That's the only part of my code I really am not happy with yet.  I still need to get it to compile in Visual Studio, though...

I'm jealous that you got netlua_interfaces() implemented ;-)  I need to give lsock similar functionality.  OH THE JOYS OF HAVING YET ANOTHER SOCKET LIBRARY :]

-peace-

On Fri, Nov 15, 2013 at 10:47 PM, Sean Conner <sean@conman.org> wrote:

It was thus said that the Great Sir Pogsalot once stated:

>
> There are many situations where it would be advantageous to be able to
> create file handles that could use both your own I/O functions and the
> standard io.read()/io.write().
>
> One example is that I have created my own unfinished/shoddy binding[6] to
> BSD sockets because I could not get "low enough" with luasocket[7].
>  luasocket represents sockets as its own userdata type, not as a Lua file
> handle -- but I wanted the ability to call recv() on my socket type and
> also be able to io.write() to it (I managed to get this working! \o/).

  I did a similar thing by creating my own socket libary [11], and I too,
wanted a way to use the standard Lua IO calls on it.  I ended up with the
following:

        net  = require "org.conman.net"
        fsys = require "org.conman.fsys" -- [12]

        address = net.address('127.0.0.1','tcp',80)
        socket = net.socket('127.0.0.1','tcp')
        socket:connect(address)
        fp = fsys.openfd(socket:fd(),"rw")
        -- fp has the type as if I called io.open()

  I thought about removing socket:fd(), but it proved too useful for my own
interface over select/poll/epoll() [13][14] (which I call a "pollset") that
requires the use of a Unix file descriptor, not a FILE * or a socket type.
My rational for that went a bit like this:

        A pollset is usually used for network activity, so I could include
        the code for that with the net module.  *BUT* there are cases where
        it would be useful for certain devices like TTYs and serial ports,
        which can be opened using Lua's io.open() call.

        But if I'm using a pollset to manage TTYs (or serial ports) I'd
        rather not include a bunch of network related code, so there's a
        reason to keep the pollsets out of the network module, and put them
        into the file system module [12].

        But if I'm doing a bunch of network related stuff, I don't
        necessarily need the file system stuff.  Also, if I'm doing file
        system related stuff, I might not even use the pollset module, and
        thus, why include the code there?

  Thus, both lines of thought lead me to make pollsets their own module.
But I didn't really want the pollset module to have to understand the
internal structure of my sockets; in fact, I wanted it to be as independent
as possible.  But pollsets are defined for file descriptors.  So in my
opinion, the best solution is to pass in file descriptors to the pollset,
and leave obtaining said file descriptors to other code.  So that's why my
network module has socket:fd(), and I created a function in my file system
module to return the file scriptor of a Lua file (a FILE *).

  Getting back to the network module.  I initially defined socket:read() and
socket:write() functions, but they didn't work the same as Lua's file:read()
and file:write() functions, because of the semantics of packets [15].  So I
renamed them socket:recv() and socket:send() (based on the recvfrom() and
sendto() kernel calls).  This works well for both UDP and TCP (yes, you can
use recvfrom() on TCP sockets) and an implementation of Lua's file:read()
(50 lines of Lua) and file:write() can be written on top of this, although I
did avoid including this in the actual module since a majority of the work I
do is with UDP.

> luasec[8] wraps luasocket objects to provide encryption on network
> connections (basic gist -- I'm not a security person).  It calls a method
> (socket:getfd()) to retrieve the file descriptor from the luasocket C
> object and then returns a dummy object after it has "wrapped" the socket.
>  In this way, luasec doesn't have to know the implementation details of the
> socket object in luasocket.  I, however, think it would be great if luasec
> could assume it were a standard Lua file handle.
>
> luaposix[9] is another well-known project with the ability to retrieve the
> file descriptor number from a Lua file handle (fileno()[10]).

  All abstractions leak.

> In Lua 5.3 I would love to have the ability to create and manipulate Lua
> file handles without creating my own copies of internal functions from
> liolib.c and lauxlib.h.  They are very simple objects (as shown above), and
> some projects would benefit from having an official way to create this type
> of userdata so that they become more interoperable with other third-party
> projects.  In my [unfinished] lsock project I have shown that by
> representing sockets as Lua file handles I can make use of my network
> bindings and use what is available in the io library (shameless promotion,
> maybe).

  The code to create a LUA_FILEHANDLE isn't that bad though:

        static int fsys_openfd(lua_State *L)
        {
          FILE **pfp;

          pfp  = lua_newuserdata(L,sizeof(FILE *));
          *pfp = NULL;  /* see comments in fsys_pipe() */ /* [16] */
          luaL_getmetatable(L,LUA_FILEHANDLE);
          lua_setmetatable(L,-2);

          *pfp = fdopen(
                        luaL_checkinteger(L,1),
                        luaL_checkstring(L,2)
                );

          if (*pfp == NULL)
          {
            lua_pushnil(L);
            lua_pushinteger(L,errno);
            return 2;
          }

          lua_pushinteger(L,0);
          return 2;

        }

> fileno() is part of C89 and C99, but fdopen() is POSIX,

  Sorry, but fileno() is POSIX, not C89/C99.

> Anyway, that is what I am thinking about.  No more :getfd() -- file
> descriptors should not be known from the Lua side of things (imo) -- and
> then everyone could benefit from the common Lua file handle.
>
> Thoughts?

  It mostly works, but the Lua :read() and :write() modules break down
(in my opinion) in the presence of UDP (or rather, non-stream based network
protocols).

  -spc

> Sources:
> 1: http://www.lua.org/manual/5.2/manual.html#luaL_checkudata
> 2: http://www.lua.org/source/5.2/liolib.c.html
> 3: http://www.lua.org/source/5.2/lauxlib.h.html#luaL_Stream
> 4: http://www.lua.org/source/5.2/liolib.c.html#io_fclose
> 5: http://www.lua.org/source/5.2/liolib.c.html#newfile
> 6: https://github.com/Pogs/lsock
> 7: http://w3.impa.br/~diego/software/luasocket/
> 8: https://github.com/brunoos/luasec/wiki
> 9: https://github.com/luaposix/luaposix
> 10: https://github.com/luaposix/luaposix/blob/master/ext/posix/posix.c#L848
>
> PS: I'm going to be so mad if there's a typo in here somewhere...

[11]    https://github.com/spc476/lua-conmanorg/blob/master/src/net.c

[12]    https://github.com/spc476/lua-conmanorg/blob/master/src/fsys.c

[13]    https://github.com/spc476/lua-conmanorg/blob/master/src/pollset.c

[14]    Under Linux, the underlying implementation uses epoll(); under other
        Unix systems it uses poll() and if those are unavailable, there is a
        final implementation that uses select().

[15]    TCP lends itself to :read() and :write() semantics, but not UDP,
        which is more packet oriented.

[16]    fsys_pipe() is a bit more complicated because of the potential
        errors when creating the two FILE* and leaking resources.  I'm not
        sure if your proposal would have made that function any easier to
        write or not.  But the code is in [12].