Re: Make "package.loaded" weak

[William Ahern <william@...>]

On Sun, Feb 19, 2017 at 08:12:39PM +1100, Daurnimator wrote:
> On 19 February 2017 at 20:09, Daurnimator <quae@daurnimator.com> wrote:
> > On 19 February 2017 at 18:06, Daurnimator <quae@daurnimator.com> wrote:
> >> e.g. a module that registers a metatable with `luaL_setmetatable`
> >> inside of it's `luaopen_` function.
> >
> > Oops, I meant luaL_newmetatable here.
> 
> And now reading the docs
> (http://www.lua.org/manual/5.3/manual.html#luaL_newmetatable) I
> noticed that luaL_newmetatable *doesn't* do anything if the key
> already exists.
> This is slightly better for the reloading situation: it means that
> things won't break as I mentioned earlier with luaL_checkudata.

IIRC, the problem with cqueues that you ran into before was that some
cqueues Lua script modules interpose C-based methods to provide enhanced
behavior (particularly yielding for Lua 5.1). That interposition caused
headaches when reloading because when the metatable already existed the
internal routine short-circuited. The cqueues' metatable loading code did

  if (luaL_newmetatable()) {
    luaL_setfuncs();
  }

instead of

  luaL_newmetatable();
  luaL_setfuncs();

However, the Lua script-based code unconditionally interposed methods,

  local _accept; _accept = socket.interpose("accept", function ()
     _accept()
  end)

On every reload (which in your case was part of some testing framework,
IIRC) the interposition stack kept growing, sometimes resulting in odd or
broken behavior. (No infinite loops; I think side effects sometimes
conflicted.)

The short-term fix was to unconditionally reload C routines into metatables
in the C code, just as the related Lua code unconditionally interposed
methods. Long-term fix requires, IMO, rethinking some ancillary issues, not
just namespacing and interposition.

The reason I originally short-circuited was because of metatable
cross-dependencies. Particularly for the optimization of caching a metatable
as an upvalue, a submodule might need to initialize the metatable for a
related submodule at load-time. Rather than loading the submodule outright,
which is awkard from C (especially before Lua 5.2's luaL_requiref, but also
if require is overloaded to do network I/O or otherwise yield), I just
called the internal C routine that specifically loaded the particular
metatable. That routine, which might be invoked indirectly many times, was
written to be a noop after the first invocation. The design of
luaL_newmetatable seems to have contemplated this conditional behavior.

Side note on reloading: I originally wrote cqueues modules to be able to
reload themselves without having access to the original .lua file.

  -- foo.lua
  local loader = function(loader, ...)
    local foo = {} -- foo module
    foo.loader = loader
    return foo
  end
  return loader(loader, ...)

I did this so that if an application wanted to create new Lua VMs after
sandboxing itself (e.g. after entering a chroot jail), then foo.loader could
be serialized and copied into the new VM.

  local loader_s = string.dump(foo.loader)
  ... -- create new Lua VM and copy serialized code
  local loader_f = loadstring(loader_s)
  package.loaded["foo"] = loader_f(loader_f, "foo")

C-based modules would already be available inside the sandbox as POSIX
requires that implememtations cache and index shared libraries by file path,
and the implementations I tested didn't care whether the file remained
accessible or not.

The benefit of this approach was that it worked without relying on or
modifying the application environment; it's entirely self-contained. The
disadvantage is that it's confusing and only works for modules that follow
the pattern--i.e. basically only internal modules.