The following code, in Lua 5.3.5,
local io = require 'io'
local mt = getmetatable(io.stdin)
mt.__gc = nil
io.tmpfile()
collectgarbage()
collectgarbage()
collectgarbage()
collectgarbage()
-- END
prevents the built-in IO library's finalizers from running on all the handles created subsequently (in the same Lua state), and at least some of the handles created previously (at a minimum, all the standard handles). This can be most easily seen by setting breakpoints at newprefile() and at f_gc() in liolib.c. When running the code above in a fresh Lua instance that is (gracefully) closed after the script returns, newprefile() is called four times (three times for each of the standard IO handles and one for the temp file) and f_gc() is not called at all. When the line
mt.__gc = nil
is commented out, f_gc() is called four times as expected.
I believe this (and the below) is a problem because Lua code should not be able to make C libraries leak resources.
The ultimate problem is in
static void createmeta (lua_State *L) {
luaL_newmetatable(L, LUA_FILEHANDLE); /* create metatable for file handles */
lua_pushvalue(L, -1); /* push metatable */
lua_setfield(L, -2, "__index"); /* metatable.__index = metatable */
luaL_setfuncs(L, flib, 0); /* add file methods to new metatable */
lua_pop(L, 1); /* pop new metatable */
}
Here, the metatable has the __gc method directly as a key (via flib), and the metatable is modifiable. The latter is what makes this a problem.
I write this up in details because I think a very similar pattern is used by many other (third party) libraries. For example, LuaSocket (latest GitHub at the time of writing; it would be nice to have accurate versioning there):
local so = require 'socket'
local mt = getmetatable(so.tcp())
mt.__gc = nil
collectgarbage()
collectgarbage()
collectgarbage()
collectgarbage()
-- END
A breakpoint at meth_close() never fires when that code runs (do a full life-cycle of a clean Lua state to be sure), yet it does when mt.__gc = nil is commented out. The code that creates the metatable is more complicated than in liolib, but ultimately it shares the same trait, being a modifiable metatable allowing the user code to remove (or replace) the __gc method.
In Programming in Lua 4, pages 333 - 336 and especially Listing 32.2 (the latter essentially identical with the code in PiL 1) explain how to use __gc and metatables when dealing with userdata. Unfortunately, that particular example, as explained in the last paragraph on page 334, is special in the sense that the userdata are never accessible to the user code, so its metatable cannot be manipulated like in my examples above. If those userdata were accessible to the user, that would have the exact same problem, making it possible for the user code to interfere with finalization. So in that sense the example is probably misleading.
The more complicated example in the next section 'An XML Parser' does not have the special circumstances of the previous example. The userdata are accessible to the user, and their metatable (see Listing 32.9) is modifiable, thus seemingly susceptible to the same pattern of abuse. The text around that example seems to consider the GC part already addressed so it does not focus on this at all, missing the chance to warn the reader of the pitfalls.
I do not think the problem is addressed in the manual (__metatable is, in my opinion, not mentioned where it should, nor is its purpose stressed at all). With the sample code in PiL and well-known Lua libraries having this problem as illustrated above, we seem to have a well-established unsafe paradigm and the question is how we can get out of this trouble.
Cheers,
V.
