Leaking file locks in Lua 5.3-beta and automatic cleanup of temporary resources

[Philipp Janda <siffiejoe@...>]

Hi!

To avoid problems with NUL-bytes when reading lines, Lua 5.3-beta reads 
single bytes in a loop instead of using `fgets` like previous Lua 
versions did. For better performance Lua uses unlocked I/O when 
available (i.e. on POSIX machines, though MS Windows has a similar API: 
`_lock_file`, `_unlock_file`, and `_getc_nolock`). I found the following 
piece of code in `liolib.c`:

    luaL_buffinit(L, &b);
    l_lockfile(f);
    while ((c = l_getc(f)) != EOF && c != '\n')
      luaL_addchar(&b, c);
    l_unlockfile(f);

The problem I see here is that `luaL_addchar` could `longjmp` out of the 
loop when memory allocation fails without unlocking the file object. One 
solution to this specific problem would be to read into a character 
array on the C stack and unlock the file stream before doing a 
`luaL_addlstring`. Or you could rely on the implementation of 
`luaL_Buffer`s, and unlock the file just before the buffer has to grow.

Anyway, something like this can also happen with other temporary 
resources like memory buffers, temporary files, etc., and the usual 
approach I use is to allocate a userdata (with a suitable `__gc` 
method), and let the GC do the cleanup if the function exits with a 
`longjmp`. Obviously that doesn't work well with locks or other 
resources that need immediate cleanup. C++ has RAII for this, so I was 
wondering whether it makes sense to add functions for automatic resource 
cleanup to the Lua C API. The way I imagine it, Lua would store an extra 
function pointer and a `void` pointer for every call to a 
`lua_CFunction`, and call those stored function pointers (up to the next 
`setjmp`) in reverse order before doing a `longjmp` in case of an error.

The API could be something like:

    lua_setdestructor(lua_State*, void (*destructor)(void*), void*);
    lua_rundestructor(lua_State*); /* calls and unsets destructor */

And the snippet from `liolib.c` would look like:

    luaL_buffinit(L, &b);
    l_lockfile(f);
    lua_setdestructor(L, fileunlock, f);
    while ((c = l_getc(f)) != EOF && c != '\n')
      luaL_addchar(&b, c);
    lua_rundestructor(L);

with

    static void fileunlock(void *ud) {
      l_unlockfile((FILE*)ud);
    }

What do you guys think?

Philipp