Re: [Experiment] Module system based on static hashing

[Stefan <ste@...>]

Am 12.04.2020 um 21:29 schrieb Roberto Ierusalimschy:
>>  * require is really, really slow
> 
> Does this conclusion apply to the case when the module loader is
> already available in 'package.preload'?

No, I used require "as-is" (i.e. it loads and unloads the .so every
time, as strace will reveal). Which takes a lot of time, unsurprisingly.

> More generally, did you compare your implementation with the technique
> of registering all modules in 'package.preload' and then requiring only
> the ones needed?
> 
> -- Roberto
> 

I expanded the tests now to cover this.
(
https://github.com/evelance/lxlib/tree/09bf1392a74713fb7bd393b99c8aaa7ba3482f4f
)
The test script is such:

-- local b64 = (lhf and lhf.base64) or require "base64"
-- local mathx = (lhf and lhf.mathx) or require "mathx"
local mathx, b64 = math, { encode = function(s) return s end }
local coro = coroutine.create(function()
    local i = 0
    while true do
        i = i + 1
        coroutine.yield(mathx.cos(i))
    end
end)
local x = 123
for i = 1, 100 do
    x = math.sin(x) + select(2, coroutine.resume(coro))
end
return b64.encode(tostring(x))

The base64 and mathx libraries are preloaded with:

// base64
lua_getglobal(L, "package");
lua_getfield(L, -1, "preload");
lua_pushcfunction(L, luaopen_base64);
lua_setfield(L, -2, "base64");
lua_pop(L, 2);
// mathx
lua_getglobal(L, "package");
lua_getfield(L, -1, "preload");
lua_pushcfunction(L, luaopen_mathx);
lua_setfield(L, -2, "mathx");
lua_pop(L, 2);

Now they are available for require, but the script doesn't
yet use them. The results look like this:

Script doesn't use preloaded libraries
$ /usr/bin/time -v out-preload
Create states     -  1.0722s
Load libraries    - 11.5264s
Load bytecode     -  0.5074s
preload           -  0.2457s
Execution         - 12.8259s
memory usage per state: 22KB
full gc cycle     -  1.0079s
Cleanup           -  1.8520s
...
User time (seconds): 26.49
Maximum resident set size (kbytes): 2729652

Script doesn't use libraries
$ /usr/bin/time -v out-lx
Create states     -  1.0943s
Load libraries    -  0.1933s
Load bytecode     -  0.4935s
Execution         - 10.8626s
memory usage per state: 8KB
full gc cycle     -  0.3105s
Cleanup           -  0.9013s
...
User time (seconds): 12.99
Maximum resident set size (kbytes): 1078072

If the first two lines of the script get commented in,
the script now actually uses the libraries:

Script uses libraries
$ /usr/bin/time -v out-preload
Create states     -  1.0800s
Load libraries    - 11.4073s
Load bytecode     -  0.6092s
preload           -  0.2317s
Execution         - 15.1144s
memory usage per state: 25KB
full gc cycle     -  0.9813s
Cleanup           -  2.2180s
...
User time (seconds): 28.51
Maximum resident set size (kbytes): 3085820

Script uses libraries
$ /usr/bin/time -v out-lx
Create states     -  1.1002s
Load libraries    -  0.1815s
Load bytecode     -  0.5016s
Execution         - 11.3895s
memory usage per state: 9KB
full gc cycle     -  0.3824s
Cleanup           -  0.6131s
...
User time (seconds): 13.23
Maximum resident set size (kbytes): 1124768

The preloading is very fast (it only adds a CFunction
to a table), and it is obviously much faster than
dynamic loading.
But it still pulls all values into the RAM, even if they
are not needed. This influences the memory footprint:
 * Preload but don't use libraries: 2.73GB
 * Preload and use libraries:       3.09GB (+13%)
 * Static table not accessed:       1.08GB
 * Static table and accessed:       1.12GB (+4%)

It is possible to omit luaL_openlibs and "require" only
those that are actually needed, but then the user needs
to explicitly open the string library to get the string
metamethods as well.
Omitting half the libraries will ca. halve the time and
memory needed by luaL_openlibs, too.

The goal of the static table is to increase the size of
the standard library (functions that are directly accessible,
like in PHP for example) without sacrificing Lua's fast
startup time and small memory footprint.