Re: Packaging and importing

[Wim Couwenberg <w.couwenberg@...>]

> If there is anything you would like to be able to do when you package
> your libraries, but feel you can't do with our proposal, this is the
> right venue to talk about it. Chances are we can figure out how to do it
> so that everything is compatible.

I've dug in to the proposed require implementation since we needed it
for deployment of our app.  I have some remarks (aka questions :-) ).

1.  For developing extension modules in C we apply a scheme where part
of the module is in C (let's say a DLL, but could also be statically
linked) and another part is in an accompanying Lua script.  So we have
basically two files (DLL, Lua) per module.  Under the current proposal
it is not possible to cleanly deploy such a module:

  1.a  The lua part in itself is *not* a module so should not be
       loaded through "require".  (It is a "private" script.)

  1.b  The entry point in the C module that gets called by require
       does not know the path from which it could luaL_loadfile the
       Lua script.

  1.c  I'd rather not put the Lua part inside the DLL (as resource),
       partly because we'll move to Linux with our app but mostly
       because it defies easy scripting during development and adds
       unnecessary steps to the build process.

For example, my recent Fiber module is set up as a single C file,
without accompanying Lua script, because of this deployment issue.  I
see two options for require that would cleanly allow to load such
dichotomous modules:

  1.a  Pass the exact path to the module as a second argument to the
       module entry point (the required name being the first), which
       can then luaL_loadfile the Lua part from the same location.

  1.b  Let each entry point decide for itself which arguments it likes
       to receive.  loader_C (or any other loader) could return a
       closure with appropriate upvalues.  Or a loader could return
       not only the module entry point but also any arguments (to save
       us an auxiliary closure) like so (pseudo code snippet of
       "require"):

           local function call_opt(entry_point, ...)
             if entry_point then
               return true, entry_point(...)
             end
           end

           local done, result
           for _, loader in ipairs(package.loaders) do
             done, result = call_opt(loader(name))
             if done then break end
           end

           
2.  I find it odd that the call <module(modname)> stores its table (if it
didn't exist already) in package.loaded[modname], since this name is not
necessarily "require"-ed anywhere.  Suppose we do <require(name)> and
this leads to a <module(modname)> call.  Then I'd expect that module
would store the table at package.loaded[name] (i.e. the inner-most
required name).  As an example, suppose we develop a package "complex"
but decide to see it as an extension of "math".  Then the complex
package could look something like:

    module "math"

    function complex(re, im)
      -- return a complex number
    end

    I = complex(0, 1)
    
    -- etc. etc.

Under the current implementation, <require "complex"> would simply
return true instead of the math table.  The other scheme *does* need
some rule to enforce that a <module(modname)> call consistently
returns the same table.  We could use the globals (_G) for that or
something like a package.modules table.  (Or, better still, a table
that is private to the "module" function, since "require" is the
public interface to load a package/module.)


3.  I use the opposite order in calling "require" and "module", so
instead of

    local aap = require "aap"
    module "noot"

I use

    module "noot"
    local aap = require "aap"

because it handles possible cyclic dependencies better (aap could
simply be "true" instead of the aap package table in the first snippet
above).  Are there any unexpected (=unwanted) consequences?


4.  (Maybe this should be item 1. :-) ).  For deployment of stand
alone apps, the require philosophy of simply trying a list of loaders
is very simple and flexible, especially since you can control the
specific loaders that will be used.  For example, in a release
deployment we package many C modules into a single DLL simply by
adding a loader that checks for some fixed prefix (say "aap.noot.") in
the required name to a select the appropriate DLL.  Couldn't be much
simpler than that...  :-)  (Of course, we also pass the full package
path to the entry point in this custom built loader, as suggested in
1. above)

HTH (in one way or other :-D ),
Wim