Re: RES: package system modification (module loaders)

[duck <duck@...>]

> All the above. I had a look at recently posted LuaThreads package ..
> but to build it required modifying the Lua sources (via a macro
> if I recall).

That's special to LuaThreads: the core needs to be recompiled with 
luathread.h #included throughout. This serves to #define some special 
macros which allow core support for pre-emptive threading (in particular, 
it inserts real locking code for thread synchronisation inside the Lua 
core machine -- see luastate.h). 

For example, in a regular Lua build the macros lua_lock and lua_unlock 
compile away to nothing. The macros are used inside the core, 
something like this: 

lua_pushnil (lua_State *L) {
  lua_lock(L);
  setnilvalue(L->top);
  api_incr_top(L);
  lua_unlock(L);
}

A Lua made ready for LuaThreads converts these dud locks and unlocks into 
pmutex_thread_lock/unlock()s for pre-emptive thread safety. .

So a LuaThreads-ready Lua is just fine even if you never load or link in 
the threading library, except for needing pthreads.

[This could be a load of rot. Diego, please correct any rubbish spoken
above.]

> This is a likely problem for many system packages: sockets,
> signals, exceptions, Apache module, blah blah etc ..

The "rebuild" is, as Diego notes, a convincingly necessary core-tweak 
*specific to pre-emptive multithreading*. (There is LuaTask, which is a 
pure add-on library, but it lets you run whole Lua programs in parallel, 
not to start Lua functions as new threads of execution. So it's not at all 
the same thing.)

LuaSocket, for instance, requires no such rebuild. It's a pure add-on lib
in C and Lua.

>Clients expect the subcomponents 
>to 'just work' including Lua.

I've taken to statically including everything :-) I added a library called 
lstatic.c, in which I stick any special "staticising" code I need, and I 
openlib() it via LUA_EXTRALIBS. It's very cheesy, just looks like this:

--cut--

/* This file should be edited as required to assist in      */
/* statically managing libraries written to be used         */
/* more naturally in dynamic fashion, e.g. luasocket.       */
/* If you need to register C libraries into package.preload,*/
/* or to expose library functions for calling from Lua,     */
/* or to load and run Lua library code, do so here and call */
/* luaopen_lstatic() via LUA_EXTRALIBS.                     */

LUALIB_API int luaopen_lstatic(lua_State *L) {
    /* Make things "5.1-style" */
#   include "compat-5.1.b2c"
    /* Expose initialisation functions for C libs later loaded via Lua */
    lua_register(L, "luaopen_lsocket", luaopen_lsocket);
    /* Do special preconfig, inc. package.loaded & package.preloaded */
#   include "lstatic.b2c"
    /* Load any Lua libraries we want statically bound */
#   include "socket.b2c"
    return 1;
}

--cut--

The #included files here are just Lua source converted into C-runnable 
code blocks with bin2c. So first I magically run compat-5.1.lua from right 
inside by binary, and then do any other initialisation I prefer to do in 
Lua than in C (laziness). For example, the lstatic.lua file alluded to 
above looks like this:

--cut--

--Helper code to load up dynamically-flavoured C/Lua libraries
--that are statically built into the executable. Helps ensure
--that pure C libraries appear in package.loaded (to provide
--5.1-style compatibility), that the compat-5.1.lua code is
--loaded (providing 5.1-style require() and the new module()
--function), and that Lua libraries which initialise C code
--can find that C code in package.preload.

--Register linked in immediate-init non-standard libraries

package.loaded.thread = thread
package.loaded.mime   = mime
package.loaded.lfs    = lfs
package.loaded.w32reg = w32reg

--Register linked in libraries initted from Lua code

package.preload.lsocket = luaopen_lsocket

--cut--

(FYI, w32reg is a Win32 registry-access library I have almost written,
designed to make registry access "as easy as variable assignment", as
MS have done in VB and VBS. It isn't, errrr, quite finished yet since
I discovered this list, where I have learned new stuff instead.)

bin2c is great because it lets you build Lua code right into your
binary, though it means the code is in memory twice (once as a 
data string in the interpreter and once as a bytecode string inside the 
Lua machine).

The "data string" can be Lua source or compiled Lua, but since luac 
depends on the standard libraries, which depend on my lstatic, I need to 
"make" once with bin2c run on the Lua source, and then again (having 
bootstrapped luac) with bin2c run on the output of the compiler. This 
because the compiled code is usually quite a bit smaller than the source.

Ideally, above, I'd load socket.lua out of data glued to the end of the 
binary itself (why I was so interested in srlua and any proposed 
descendants earlier). For now I just tweak my lstatic.c to 
lua_loadbuffer() extra Lua modules at startup.