Re: More about packaging (fwd)

[RLake@...]

I am sorry I missed this interesting
discussion, and I apologise for joining it a bit late. (I just moved and
my ISP was lamentably slow in reconnecting me to the internet.)

1. I personally feel quite strongly
that library loading should not automatically insert anything in the global
namespace, whether the library in question is a Lua library or a C library.
Furthermore, the end user should be protected from knowing details like
"what is the entry point of the library in a {.dylib, .so, .dll} file?"
So I would like a simple uniform interface like this:

local string = require "string"
coro = require "coroutine"

It does not then bother me that the
name I use in the module is "standard"; the require statement
includes the standard name.

2. I also believe that it is necessary
to distinguish between two use cases:

  1) Module A requires Module B
in order to install itself.
  2) Module A will require Module
B during execution of its functions.

This allows for modules to have circular
dependencies, which seems to be useful. There was a long discussion about
this some time ago; although I am not a big fan of circular dependencies,
I can see that there are circumstances where they are unavoidable.

3. Finally, I believe it important that
it be easy to sandbox module tables. As I have mentioned before, this requires
that each sandbox have its own table. It is entirely legitimate to override
a module function (or other member), or to add new ones; but sandboxes
need to be protected from each other doing this.

One of the wonderful things about Lua
is that all of this machinery is quite simple to implement. However, as
Diego points out, a little standardisation would help a lot in implementing
it.

At a bit more length:

1. Module tables vs. module instantiators

Internally, modules are always generated
by executing a function, either a Lua chunk or some lua_CFunction, typically
luaopen_*(L). I'm calling this the module instantiator. Currently, the
calling conventions for instantiators vary considerably between proposals;
my suggestion is that the standard calling convention be that the instantiator
is passed a table as its first parameter, which it fills in and returns.
The module system then caches both instantiators and tables. The basic
logic of require is "if the table has been cached, return it; if the
module has an instantiator, use it to create the table; otherwise, find
the Lua or C library and get the instantiator from that. A rough implementation
is found below.
 
A simple example in Lua looks like this:

--module complex

local math = _MODULE.math -- see below

return function(complex, name)
  function complex.new(real, imag)
return {r = real, i = imag} end
  function complex.add(a, b) return
{r = a.r + b.r, i = a.i + b.i} end
  function complex.abs(a) return
math.sqrt(a.r^2 +a.i^2) end
  -- etc.
  return complex
end

In C, this looks about the same (considerably
simplified)

/* ... some code at end of message ...
*/

LUALIB_API int luaopen_complex(lua_State
*L) {
  luaL_newmetatable(L, "Complex");
  /* Written out here, but the
simplification to luaL_openlib() is obvious */
  lua_pushliteral(L, "new");
lua_pushcfunction(L, complex_new); lua_settable(L, 1);
  lua_pushliteral(L, "add");
lua_pushcfunction(L, complex_add); lua_settable(L, 1);
  lua_pushliteral(L, "abs");
lua_pushcfunction(L, complex_abs); lua_settable(L, 1);
  /* ... */
  lua_settop(L, 1);
  return 1;
}

The second argument ("name")
is provided for autoloaders. This is a slight change from the current autoloader
behaviour, where a global is passed to the chunk with the name. Functionally,
there is no difference, though.

Note that in the first case, executing
the Lua chunk returns an instantiator. In the second case, the instantiator
is luaopen_complex. So in both cases, we have an instantiator function
to play with. Static libraries (and dynamically generated modules, for
that matter) can be registered by adding their instantiator function to
the instantiator cache. 

The rough implementation below assumes
that C module entry points will always be in the form luaopen_<name>,
whether or not they are static; this seems like a simple convention and
avoids issues with hypothetical OSs which don't like multiply defined dynamically
loaded symbols.

(Edgar is quite right that basic version/sanity
checks should be performed prior to calling the entry function.)

The advantage of instantiators is that
they can be re-executed in different sandboxes, in order to fill independent
module tables.
   
2. Global namespace

I am not so much concerned here about
name collision; as Diego (I think) pointed out, modules need to have unique
names one way or another. The problem is that there is no reliable way
for require() to know which environment table to insert the module into.
Furthermore, internal references within the module may or may not be in
the same environment as the caller of require(), so hacks like using getfenv
to walk up the stack are going to lead to unpredictable results. This is
particularly an issue for foreign ("C") modules, which have a
different environment than any sandbox. Returning the module table as the
result of the require() is much simpler all round.

3. Preloading / circular dependencies

The code below creates _MODULE with
a __index metamethod which creates the module table, ready to be filled
in. This can be used directly for the case where the library only needs
to exist, and will not be called until later. The module tables are then
created with a __index metamethod which acts as a trigger; this is based
on the implementation in LTN 11.

4. Multiple Lua modules in a single
C library.

Some of the base library .c files register
more than one Lua module; a couple of them also register globals. In order
to get the code presented below to work with this, it will be necessary
to at least create separate luaopen_* functions for the different Lua modules.
I don't think that is a huge change; the remaining changes to luaL_openlib()
actually simplify it.

[CODE SAMPLE 1 -- basic module loading
system]
It was easier to write this in Lua,
but it needs to be part of the base library in order to get proper integration
of static and dynamic libraries. Rewriting it in C would be tedious and
arguably unnecessary; it is not going to be of much use in a Lua compiled
without at least the byte-code loader, so it can be compiled into the lua
executable as bytecode.


-- make a weak keyed table
local function Weak() return setmetatable({},
{__mode = "k"}) end

-- maps module tables to instantiator
functions
local MODULE_MAKER = Weak
-- maps module tables to their names
local MODULE_NAME = Weak

-- Find the module. The precise definition
of find_in_path might vary,
-- depending on OS interfaces, but a
simple implementation would be
-- to try all the possibilities one
at a time.

local function find_module(name)
  local filename = find_in_path(name,
LUA_PATH)
  if filename then 
    return assert(assert(loadfile(filename))())
  end
  filename = find_in_path(name,
LUA_DYNPATH)
  if filename then return assert(loadlib(filename,
"luaopen_" .. name)) end
  error("Could not find module
'"..name.."'")
end

-- An unloaded module lazily instantiates
itself; they are created with this
-- metatable.
-- require() forces the instantiation
by referencing a module member.
-- I left out the code to verify that
the module does not require itself. 
local mod_meta = {}
function mod_meta:__index(key)
  local instantiator, name = MODULE_MAKER[self],
MODULE_NAME[self]
  if instantiator == nil then
    -- We don't have an instantiator
    instantiator = assert(find_module(name))
    MODULE_MAKER[self] = instantiator
  end
  -- disable the trigger. This
should actually change the trigger so that
  -- an error is signalled if the
module requires itself, and then set it
  -- to nil after the instantiation
  setmetatable(self, nil)
  -- should do more error checking
  instantiator(self, name)
  return self[key]  -- the
trigger has been turned off so rawget is not necessary.
end

-- The _MODULE table uses the following
metamethod to automatically create a new table
-- for so-far-unreferenced modules.
The unloaded modules are given the metatable above.

local use_meta = {}
function use_meta:__index(name)
  local mod = setmetatable({},
mod_meta)
  self[name], MODULE_NAME[mod]
= mod, name
  return mod
end

_MODULE = setmetatable({}, use_meta)

function require(name)
  local mod = _MODULE[name]  --
get the module table
  local _ = mod._VERSION  --
trigger the load if necessary
  -- it doesn't matter if mod._VERSION
doesn't exist. But it should :)
  return mod
end

-- Finally, we need a way to register
static modules (whether foreign or Lua):

function register_instantiator(name,
instantiator)
  local mod = _MODULE[name]
  MODULE_MAKER[mod] = instantiator
end

-- A C stub for registering static libraries
on startup:

LUALIB_API void luaL_registerlib(lua_State
*L, const char *name, lua_CFunction instantiator) {
  lua_getglobal(L, "register_instantiator");
 /* should check that this worked */
  lua_pushstring(L, name);
  lua_pushcfunction(L, instantiator);
  lua_pcall(L, 2, 0);  /*
should do something on error here */
}

luaL_registerlib(L, "string",
luaopen_string);
luaL_registerlib(L, "table",
luaopen_table);
/* ... */

[CODE SAMPLE 2 ---- partial C code for
Complex example]
#include <math.h>
#include <lua.h>
#include <lauxlib.h>

typedef struct Complex {lua_Number r,
i;} Complex;

static int pushComplex(lua_Number r,
lua_Number i) {
  Complex *rv = lua_newuserdata(L,
sizeof(Complex));
  luaL_getmetatable(L, "Complex");
  lua_setmetatable(L, -1);
  rv->r = r;
  rv->i = i;
  return 1;
}

static Complex *checkComplex(lua_State
*L, int narg) {
  Complex *rv = luaL_checkudata(L,
narg, "Complex");
  if (rv == NULL) luaL_typerror(L,
narg, "Complex");
  return rv;
}

static int complex_new(lua_State *L)
{
  lua_Number r = luaL_checknumber(L,
1);
  lua_Number i = luaL_checknumber(L,
2);
  return pushComplex(r, i);
}

static int complex_add(lua_State *L)
{
  Complex *a = checkComplex(L,
1);
  Complex *b = checkComplex(L,
2);
  return pushComplex(a->r +
b->r, a->i + b->i);
} 

static int complex_abs(lua_State *L)
{
  Complex *a = checkComplex(L,
1);
  lua_pushnumber(L, sqrt(a->r
* a->r + a->i * a->i));
  return 1;
}