Re: [Experiment] Module system based on static hashing

[Sean Conner <sean@...>]

It was thus said that the Great Sean Conner once stated:
> It was thus said that the Great Stefan once stated:
> > Hello,
> > 
> > I've conducted an experiment to speed up Lua state creation with static
> > hash tables generated by GNU gperf and got some interesting results.
> > 
> > Introduction
> > ============
> > 
> > Lua states offer a very light-weight way to execute independent scripts,
> > which is a much desirable feature for programs that execute a large
> > number of them (e.g. web servers).
> > 
> > Unfortunately the standard library is too small for many tasks and
> > adding modules by hand is quite a hassle. Furthermore are dynamic
> > libraries a platform-dependent mess.
> > 
> > The goal of this experiment was to find a way to add many more functions
> > to Lua a) without using dynamic loading and b) without slowing down the
> > creation of new states.
> > 
> > luaL_openlibs loads all functions, tables and values such as print,
> > string, _VERSION, math.pi etc. that make up the standard library into
> > the Lua state so that the script can access them via table lookups.
> > 
> > But rarely does a script use ALL of them and the more functions get
> > added, the more unnecessary work luaL_openli has to do.
> > So, the less unused Lua values get loaded into RAM, the better.
> > 
> > OK, so what if we don't actually load them and just set a metatable
> > with a __index metamethod that fetches the values as the script needs
> > them? The script won't notice absent values it doesn't use -- Great!
> 
>   I have a similar issue at work, although it's not speed reasons but
> installation reasons (the less files to install, the better).  To that end,
> I created what I call the Kitchen Sink Lua executable, which includes *all*
> the modules we use at work.  At startup, I load all the luaopen_*() calls
> into package.preload (these are listed in a luaL_Reg[] arraay) and add a
> special loader to package.searchers to load the modules in Lua (that are
> compiled via luac, compressed [1] and stored in the executable; the loader
> finds the appropriate data and decompresses it when lua_load() is called).
> 
>   I do call luaL_openlibs() but upon reflection, my Lua state initialization
> code (which is only called once---I don't create tons of multiple states)
> could just be (sans error checking):
> 
> 	static const luaL_Reg preloadtable[] =
> 	{
> 	  { "coroutine" , luaopen_coroutine },
> 	  { "table"     , luaopen_table },
> 	  { "io"        , luaopen_io },
> 	  { "os"        , luaopen_os },
> 	  { "string"    , luaopen_string },
> 	  { "math"      , luaopen_math },
> 	  { "utf8"      , luaopen_utf8 },
> 	  { "debug"     , luaopen_debug },
> 	  /* other pre-installed C-based modules */
> 	  { NULL        , NULL }
> 	};
> 	
> 	/*-----------------------------------------------
> 	; create our state with the bare minimum required
> 	; for a Lua state. 
> 	;-----------------------------------------------*/
> 	
> 	L = luaL_newstate();
> 	luaL_requiref(L,"_G",luaopen_base,1);
> 	luaL_requiref(L,"package",luaopen_package,1);
> 	luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_LOADED_TABLE);
> 	luaL_setfuncs(L,preloadtable,0);
> 	lua_pop(L,3);
> 	
>   This (not checked, just off the top of my head) populates a Lua state with
> the functions NOT in a module (like dofile(), pairs(), select, etc.) and the
> package module, which I believe is the minimum required [2]. This does
> assume that all scripts call require on basic modules like os and io. 
> Preloading modules written in Lua is left as an exercise to the reader.  
> 
>   Would something like this work?

  So I decided to try this and get some timings.  The code is very
simple---the base version basically does:

	L = luaL_newstate();
	lua_close(L);

The minmin version does:

	L = luaL_newstate();
	luaL_requiref(L,"package",luaopen_package,1);
	luaL_getsubtable(L, LUA_REGISTRYINDEX, LUA_PRELOAD_TABLE); // oops
	luaL_setfuncs(L,preloadtable,0);
	lua_pop(L,2);
	lua_close(L);

(this assumes we can require stuff like pairs(), next(), assert(), etc.)
And the min version is the code I'm quoting above (but loading
LUA_PRELOAD_TABLE instead of LUA_LOADED TABLE, my bad).  

And the full version:

	L = luaL_newstate();
	luaL_openlibs(L);
	lua_close(L);

The work was done with Lua 5.3.5 on a slightly older 32-bit Linux system and
the times are in nanoseconds (using clock_gettime(CLOCK_MONOTINIC)).  So
with that said, the results of my simple test:

[spc]lucy:/tmp/fl>./bare 10000
new:     19999.100000
lib:         0.000000
close:    5302.000000
memory:      2.000000
[spc]lucy:/tmp/fl>./minmin 10000
new:     23124.800000
lib:     38415.200000
close:   11407.900000
memory:      5.000000
[spc]lucy:/tmp/fl>./min 10000
new:     21497.600000
lib:     60993.500000
close:   14968.400000
memory:      6.000000
[spc]lucy:/tmp/fl>./full 10000
new:     20842.000000
lib:    174061.400000
close:   35268.400000
memory:     14.000000
[spc]lucy:/tmp/fl>

Each was run 10,000 times and the runtime (and memory, obtained from
lua_gc()) averaged. No special compiler options were used but I think this
is enough to give us some ballpark figures.  The base (just state creation
and deletion) used 2K and gives us a baseline to compare the rest.  The
minmin case (just loading package and leaving the rest to be require()ed)
took almost three times as long and amost 3 times the momory.  The min case
took almost four times as long, and 3 times the memory.  And the full case
... well ... 9 times longer and 7 times the memory.  

  -spc (Code available upon request)

> [1]	I would get better compression by not pre-compiling the Lua code,
> 	but the code I have works, and the difference is not enough to
> 	actually worry about it.
> 
> [2]	I'm not sure how to make functions like assert(), getmetatable(),
> 	etc. avaialble via require().  It could be done I suspect, but would
> 	take a bit of code.