experience implementing a module system

["John Belmonte" <jvb@...>]

Recently I had a go at implementing a module system in Lua.  I'll describe
how it works and mention the limitations with Lua I ran into.

My motivation was to make a better module system than what is presented in
the Lua faq.  The problem with that system is name clashes.  If you are
using a global table to represent a module called "string", for example, you
have to be careful that no code in your project uses the same global name
for another purpose.  Furthermore the module name must be hard-coded and
repeated throughout the module definition, which does not allow for easy
renaming.

As for the new design I created, I'll start with an example:

    --
    -- test.lua - this is a file using a module
    --
    dofile( "ModuleSys.lua" )
    local modz = import( "modx.lua" )  -- note how module can be "renamed"

    modz.func1()  -- in global space must use plain module name

    function test()
        %modz.func1()  -- when not in global space must use upvalue
    end


    --
    -- modx.lua - this is a file defining a module
    --
    dofile( "ModuleSys.lua" )
    local m  -- the local module table... convenient to just use "m"

    m.var1 = 10

    function m.func1()
        %m.var1 = %m.var1 + 1  -- must use upvalue outside of global scope
        %m.var2 = %m.var1/2
    end

    return export( m )  -- chunk must return like this


Using "export(m)" exports the whole table m and the result is just as
efficient as what is presented in the faq.  Optionally you may specify a
list of export variables and read-only variables.  However this adds a level
of indirection so module access is not as efficient.

    -- export func1 and var1 from table m, with var1 read-only
    return export( m, {"func1", "var1"}, {"var1" } )

This system also allows nesting of modules along the lines of Python:

    --
    -- modxx.lua - a module with nested modules
    --
    dofile( "ModuleSys.lua" )
    local m
    m.nested = import( "modx.lua" )

    -- ...

    return export( m )

So what went wrong with this design?  The problem relates to upvalues, which
are being used to emulate local namespaces for the modules.  Lua only allows
upvalue access to the next outer scope.  This prevents the following:

    function m.func()
        local test1 = function()
            %m.hey = 10  --error
        end
    end

It's the same problem we run into when we use an upvalued-table to emulate a
closure for function objects.  There is a workaround, but it can hardly be
considered acceptable:

    function m.func()
        local m2 = %m
        local test2 = function()
            %m2.hey = 10
        end
    end

I think that with the recent issues brought up regarding the upvalue
concept, it's an area of the language that warrants research into
improvement.

At first I had thought that Lua had lexical (static) scoping and that the
upvalue syntax simply gave you access to this.  But it turns out this is not
the case, upvalues just let you peek into the enclosing scope at runtime.

So is lexical scoping (binding variables at parse time) incompatible with
the goals of Lua?  Without having any way to implement namespaces Lua will
always be haunted by the name-clashing problem, which in turn will
discourage the community to build and share modules.

-John