Re: Binary Modules for Lua [was Re: External modules]

["Thatcher Ulrich" <tu@...>]

On Nov 21, 2002 at 09:42 -0500, Peter Loveday wrote:
> Thanks for preparing this to actually get this idea off the ground!
> 
> Shame this isn't for 5.0; I can't really evaluate it properly as I
> don't use Lua 4 at all.  However, some initial comments on what I
> can see:
> 
> Perhaps using the extension .dll on windows is not the best idea;
> Something like .luamod or .luaext or something might clarify the
> purpose of the file.  Or maybe some other portugese word for
> something else astronomical that sounds catchy :)

Hm, I see your point, although I think there's benefit to sticking
with ".dll", as people familiar with Win32 will know that it's a
shared library.  luamodule() does assume a prefix of "lua", so running
this code under Win32:

	loadmodule("SDL")

will actually try to load the file luaSDL.dll.  So for any module "XX",
the Win32 binary would be called "luaXX.dll".  Ignacio added some text
to the wiki page that explains all this.

> Never having used tolua, I'm unsure how this source works.  I
> presume this .pkg file is passed to tolua, along with the SDL
> headers, and it produces a binding?

Yes, that's basically it.  Actually, the .pkg file in the case of
luaSDL is just a hand-cleaned version of the SDL headers.  It's very
similar to the way SWIG works.

> So, as far as I can gather, a module only exports two symbols;
> lua_version() and lua_import().

Right.

> Firstly, lets change these names... it would be better if anything
> with lua_ on the start were a call to a standard lua_ library
> function.  Perhaps luamod_ or something.

Yeah, I think you're right about this.  lua_* is too generic.  I think
I'll change it to "luaLM_version" and "luaLM_import".

> I presume the host app calls lua_version() on the mod, to determine
> if it matches sufficiently.  How do we determine this?  Can we
> assume a major version is always compatible? eg 5.0 will be
> compatible with 5.2, but not 6.0 ?

It looks for an exact string match between LUA_VERSION in the
interpreter, and the string returned by the module's lua_version.

> Then lua_import() is called, passing in the lua_State * to set up
> for.  Is there any need for a means to clean up a state?

Ignacio wrote some code for this, but it's not being used yet.

>  I suppose the module can
> install some global object with a gc hook, but that doesn't gaurrantee any
> order of clean up (ie there might still be module objects waiting to be
> cleaned up, which means the module can't close itself down yet)

Unloading is not 100% straightforward; personally I'm inclined to
procrastinate on the whole issue :) I don't think it's a must-have
feature, but it would be good to figure it out at some point.

> One issue that comes to mind here, is that a module must be capable
> of being loaded by more than one state in the same Process (possibly
> on different threads) at the same time.  ie, the code must be
> re-entrant, such that one can say something like:
> 
>     lua_State *A, *B;
>     ...
>     lua_import(A);
>     lua_import(B);

Good point... I guess the requirement here is that modules must not
use static state; they should only use state which is associated with
a lua_State (e.g. they can "safely" store things in Lua tables &
variables).  But I think there will be exceptions to this, depending
on the module.  For example, two instances of a SGVAlib binding may
not be able to safely coexist, because there's typically only one SVGA
card per computer.

> Namespaces: Perhaps the module should return a table to use as it's
> namespace, such as:
> 
>     sdl = loadmodule("luasdl")
> 
> This of course doesn't confine the module to _only_ use this table, it is
> more of a convention.  If something needs to be global, it could still do
> so.

Yeah, I think namespacing needs to be a recommended convention, but
not enforced as a requirement.  I'm not sure about having loadmodule
return a table though; that might make it hard to use tolua's "module"
keyword (for example) which takes a compile-time name to use as the
enclosing table.

Although, it's easy enough to assign tables on the Lua side.  I think
we need a convention to determine the name of the table where a module
puts its stuff.  I think it's sensible to just use the name of the
module, so for example loadmodule("SDL") puts its stuff in a table
named "SDL".

> global metatable: If the module changes the 'global' metatable, it
> must do so cleanly.  i.e. not just replace the calls, but pass to
> the existing metatable where appropriate to allow other modules to
> still function.

Sounds sensible.  Recommendations here should be documented as part of
the "advice to module authors".

[snip]

> > * I'm DEFINITELY NOT proposing that "loadmodule" be built into the
> >   standard Lua interpreter.  I think this community standard can
> >   function just fine as a consensus specification, centralized on some
> >   pages at lua-users.org, without altering the standard Lua
> >   distribution in any way.
> 
> While it doesn't need to be built in, at least having a standard
> approved as _the_ Lua approved way of doing it would encourage
> people to write modules.  After all, we can all add this to our
> interpreters, but unless people actually produce modules in this
> fashion it means nothing.
> 
> I understand the desire to keep the Lua code platform independent;
> but I cannot help but think this will limit the acceptance of this
> as a standard if it doesn't work in a standard Lua binary.  Getting
> people to accept Lua is already hard enough, and saying you have to
> download this other package with loadmodule() support for each
> platform then re-build Lua before you can test with module xyz is
> not much worse than saying you have to statically link xyz.

Well, Lua does not actually have *any* official binaries (if I'm
interpreting the note on www.lua.org correctly -- it says "these
binaries are not part of the official distribution", but I'm not sure
if that refers to all the binaries, or only the user-contributed
ones).

Anyway, here are a couple reasons why loadmodule shouldn't be in the
default binaries:

1. security risks -- loadmodule opens up some holes

2. some Lua users have strenously objected to non-ANSI code in the
   official distribution; I don't think "officialness" is crucial
   enough to make it worth antagonizing existing users.

> Nonetheless, either way this is a vital standard to have in place; I
> am finding it increasingly difficult to answer user questions on
> whether they can do feature X in Lua, because it means I have to
> find and link appropriate code in, which is almost never for the
> right version of Lua, and not always under a clear license regarding
> commercial use.  At least loadable modules would be directly
> available to end users, and as such the licensing is not my concern
> with each and every one.

Exactly.  Also, having a single central place for loadmodule binaries
should be sufficient, if it's well known, maintained, and a critical
mass of modules get produced.

> Perhaps my situation is a little different from most Lua users; in
> my case the 'end-user' does not imply some other programmer who is
> using an embedded Lua to create internal scripting, but the actual
> end-user of our application.  These people are not programmers, or
> even technical in many cases, yet most of them can manage a little
> scripting (some more than others).
> 
> It is with these people in mind that I must consider any extension
> mechanism; and this makes it important that such a system does exist
> (it is impossible for these people to link static modules with the
> application, as has been suggested as being adequate for
> extensions).  I can understand this rationale in embedded O/S
> applications, or even internal scripting, but it simply does not
> work in a user-accessible scripting system.
> 
> So I am very keen to have this work as a standard, and work simply.

Yup, I agree with all of the above -- I appreciate the comments and
encouragement!

-- 
Thatcher Ulrich
http://tulrich.com