Re: Quality-of-Life patches

[Sean Conner <sean@...>]

It was thus said that the Great Viacheslav Usov once stated:
> On Fri, Feb 25, 2022 at 8:09 PM Sean Conner <sean@conman.org> wrote:
> 
> > > When you have
> > > preload-costly embedded libraries, or just really many embedded
> > > libraries and many Lua VMs in your application, you have to consider
> > > the cost of preloading, it may have a noticeable effect on
> > > performance.
> >
> >   Having done this for work (well, not embedding luasocket, but I have a
> > custom build of Lua with 81 modules built in, 34 of which are in C, and 47
> > in Lua), I haven't noticed any appreciable different in speed from running
> > the normal Lua interpreter.
> 
> Performance is not just speed, it is also storage.

  It is.  And that's why, when I include Lua code in an executable, it's
compressed, and in my custom loader, I uncompress the data as it's required.

> > Lua modules in Lua have to be "compiled" in
> > either case.
> 
> This remark makes me wonder what your baseline really was.
> 
> Certainly the cost of compiling 47 Lua modules vs not compiling them
> at all should be measurable. Then, of course, this cost should be
> viewed in the perspective of what else your Lua states are doing.
> 
> I was thinking about a situation when a Lua state (VM) may end up
> using only a small number of preloaded modules, and when its "main"
> payload might be relatively light and short-lived. If all your Lua
> states always use all of the preloaded libs, or stay on doing some
> heavy lifting, then, sure, the act of preloading has a negligible
> run-time cost.

  Now I do have some apps at work, written in Lua, with all the modules (and
app code) compiled in---the one program I did describe is one used mostly
for testingand quick scripts for use in our environment where it would be
annoying to install Lua, plus the modules, on any given system.

  Now there are two levels to compiling.  The first is making the actual
executable.  Attached is a sample of my Gopher server written in Lua [1]. 
The build process will compress the Lua code and generate object files from
said compressed data.  These are *only* the modules required to run the
project.  The modules written in C are also compiled and statically linked
into the executable as well.  This is the first level of compilation.

  The second level is when a Lua module (or the app itself) is requested. 
It is then compiled by the Lua VM.  This has to happen regardless of how the
Lua code is packaged.  So first, running my gopher server via lua:

[spc]lucy:~/source/port70>time lua port70.lua 
usage: port70.lua configfile

real    0m0.011s
user    0m0.005s
sys     0m0.006s
[spc]lucy:~/source/port70>

And from the "included all as an exectuable":

[spc]lucy:~/projects/gemini/misc/C>time ./port70
usage: ./port70 configfile

real    0m0.009s
user    0m0.007s
sys     0m0.002s
[spc]lucy:~/projects/gemini/misc/C>

  It's a noticible difference in runtime, but the embedded version is
slightly quicker to run (probably because it avoids the read() system call
to get the script data, even if said data needs to be decompressed).  The
Lua executable size on my system is 208,255 bytes, while the embedded
executable is 1,119,059 bytes.  So five times the size, which might be
significant for your use case.  On the system I ran the tests on, a clean
build took about 20 seconds to compile, but that includes compiling Lua,
LPeg, all the C-based Lua modules, and compressing and convering all the Lua
modules to object files, and could be considered a one-time cost (changing
just one file reduced the rebuild to less than a second).

  And just to be complete, on my job laptop, lua is 296,288 bytes in size,
my custom kslua [2] is 1,607,506 bytes in size.  And timings for lua:

[sconner]belial:~/repo/ecid/tests/bin>time lua /tmp/x.lua 
Hello, world!

real    0m0.009s
user    0m0.003s
sys     0m0.004s
[sconner]belial:~/repo/ecid/tests/bin>

and kslua:

[sconner]belial:~/repo/ecid/tests/bin>time ./kslua /tmp/x.lua 
Hello, world!

real    0m0.011s
user    0m0.003s
sys     0m0.006s
[sconner]belial:~/repo/ecid/tests/bin>

  In this case, it's a bit slower.  Again, it's noticible, but here, lua is
just slightly quicker to run (and here, it may be due to not having to load
any modules, but taking the time to load package.preload[] and adding the
custom loader is .002s longer?  Wierd.  This is the newer Apple M1
architecture, so perhaps some instruction cache issues?  I don't know,).  Is
it bothersome being slower?  Actually, I hadn't noticed until I actually
measured it, so it doesn't really bother me.  But again, your milage may
vary.

  -spc

[1]	https://github.com/spc476/port70

[2]	Which stands for "Kitchen Sink Lua", for includes all modules,
	including the kitchen sink.

#include <stdio.h>
#include <stdlib.h>

#include <lua.h>
#include <lauxlib.h>

#include "preloadlua.h"
#include "modules.h"
#include "orgconman.h"

extern int luaopen_port70_getuserdir (lua_State *);
extern int luaopen_port70_setugid    (lua_State *);

extern unsigned char const c_port70[];                     extern size_t const c_port70_size;
extern unsigned char const c_port70_cgi[];                 extern size_t const c_port70_cgi_size;
extern unsigned char const c_port70_handlers_file[];       extern size_t const c_port70_handlers_file_size;
extern unsigned char const c_port70_handlers_filesystem[]; extern size_t const c_port70_handlers_filesystem_size;
extern unsigned char const c_port70_handlers_http[];       extern size_t const c_port70_handlers_http_size;
extern unsigned char const c_port70_handlers_sample[];     extern size_t const c_port70_handlers_sample_size;
extern unsigned char const c_port70_handlers_url[];        extern size_t const c_port70_handlers_url_size;
extern unsigned char const c_port70_handlers_userdir[];    extern size_t const c_port70_handlers_userdir_size;
extern unsigned char const c_port70_mklink[];              extern size_t const c_port70_mklink_size;
extern unsigned char const c_port70_readfile[];            extern size_t const c_port70_readfile_size;
extern unsigned char const c_port70_safetext[];            extern size_t const c_port70_safetext_size;
extern unsigned char const c_port70_utf8util[];            extern size_t const c_port70_utf8util_size;

/***********************************************************************/

static luaL_Reg const c_preload[] =
{
  { "lpeg"                  , luaopen_lpeg                  } ,
  { "org.conman.clock"      , luaopen_org_conman_clock      } ,
  { "org.conman.errno"      , luaopen_org_conman_errno      } ,
  { "org.conman.fsys"       , luaopen_org_conman_fsys       } ,
  { "org.conman.fsys.magic" , luaopen_org_conman_fsys_magic } ,
  { "org.conman.math"       , luaopen_org_conman_math       } ,
  { "org.conman.net"        , luaopen_org_conman_net        } ,
  { "org.conman.pollset"    , luaopen_org_conman_pollset    } ,
  { "org.conman.process"    , luaopen_org_conman_process    } ,
  { "org.conman.signal"     , luaopen_org_conman_signal     } ,
  { "org.conman.syslog"     , luaopen_org_conman_syslog     } ,
  { "port70.getuserdir"     , luaopen_port70_getuserdir     } ,
  { "port70.setugid"        , luaopen_port70_setugid        } ,
  { NULL                    , NULL                          }
};

static prelua_reg__s const c_prelua[] =
{
  { "org.conman.const.exit"            , c_org_conman_const_exit            , &c_org_conman_const_exit_size            } ,
  { "org.conman.const.gopher-types"    , c_org_conman_const_gopher_types    , &c_org_conman_const_gopher_types_size    } ,
  { "org.conman.net.ios"               , c_org_conman_net_ios               , &c_org_conman_net_ios_size               } ,
  { "org.conman.nfl"                   , c_org_conman_nfl                   , &c_org_conman_nfl_size                   } ,
  { "org.conman.nfl.tcp"               , c_org_conman_nfl_tcp               , &c_org_conman_nfl_tcp_size               } ,
  { "org.conman.parsers.abnf"          , c_org_conman_parsers_abnf          , &c_org_conman_parsers_abnf_size          } ,
  { "org.conman.parsers.ascii.char"    , c_org_conman_parsers_ascii_char    , &c_org_conman_parsers_ascii_char_size    } ,
  { "org.conman.parsers.ascii.control" , c_org_conman_parsers_ascii_control , &c_org_conman_parsers_ascii_control_size } ,
  { "org.conman.parsers.ip-text"       , c_org_conman_parsers_ip_text       , &c_org_conman_parsers_ip_text_size       } ,
  { "org.conman.parsers.iso.char"      , c_org_conman_parsers_iso_char      , &c_org_conman_parsers_iso_char_size      } ,
  { "org.conman.parsers.iso.control"   , c_org_conman_parsers_iso_control   , &c_org_conman_parsers_iso_control_size   } ,
  { "org.conman.parsers.mimetype"      , c_org_conman_parsers_mimetype      , &c_org_conman_parsers_mimetype_size      } ,
  { "org.conman.parsers.url"           , c_org_conman_parsers_url           , &c_org_conman_parsers_url_size           } ,
  { "org.conman.parsers.url.gopher"    , c_org_conman_parsers_url_gopher    , &c_org_conman_parsers_url_gopher_size    } ,
  { "org.conman.parsers.utf8"          , c_org_conman_parsers_utf8          , &c_org_conman_parsers_utf8_size          } ,
  { "org.conman.parsers.utf8.char"     , c_org_conman_parsers_utf8_char     , &c_org_conman_parsers_utf8_char_size     } ,
  { "org.conman.parsers.utf8.control"  , c_org_conman_parsers_utf8_control  , &c_org_conman_parsers_utf8_control_size  } ,
  { "port70"                           , c_port70                           , &c_port70_size                           } ,
  { "port70.cgi"                       , c_port70_cgi                       , &c_port70_cgi_size                       } ,
  { "port70.handlers.file"             , c_port70_handlers_file             , &c_port70_handlers_file_size             } ,
  { "port70.handlers.filesystem"       , c_port70_handlers_filesystem       , &c_port70_handlers_filesystem_size       } ,
  { "port70.handlers.http"             , c_port70_handlers_http             , &c_port70_handlers_http_size             } ,
  { "port70.handlers.sample"           , c_port70_handlers_sample           , &c_port70_handlers_sample_size           } ,
  { "port70.handlers.url"              , c_port70_handlers_url              , &c_port70_handlers_url_size              } ,
  { "port70.handlers.userdir"          , c_port70_handlers_userdir          , &c_port70_handlers_userdir_size          } ,
  { "port70.mklink"                    , c_port70_mklink                    , &c_port70_mklink_size                    } ,
  { "port70.readfile"                  , c_port70_readfile                  , &c_port70_readfile_size                  } ,
  { "port70.safetext"                  , c_port70_safetext                  , &c_port70_safetext_size                  } ,
  { "port70.utf8util"                  , c_port70_utf8util                  , &c_port70_utf8util_size                  } ,
  { "re"                               , c_re                               , &c_re_size                               } ,
};

/***********************************************************************/

int main(int argc,char *argv[])
{
  lua_State *L;
  int        rc;
  
  L = luaL_newstate();
  if (L == NULL)
  {
    fprintf(stderr,"Cannot create Lua state\n");
    return EXIT_FAILURE;
  }
  
  preload_lua(L,c_preload,c_prelua,sizeof(c_prelua) / sizeof(prelua_reg__s),argc,&argv[0]);
  
  lua_pushcfunction(L,preload_lualoader);
  lua_pushliteral(L,"port70");
  lua_call(L,1,1);
  
  if (lua_type(L,-1) == LUA_TFUNCTION)
    rc = lua_pcall(L,0,LUA_MULTRET,0);
  else
    rc = -1;
    
  if (rc != 0)
    fprintf(stderr,"lua_pcall() = %s\n",lua_tostring(L,-1));
    
  lua_close(L);
  return rc;
}