Can I save size by dumping a table of chunks? (and, how do I do it?)

[Charles Nicholson <charles.nicholson@...>]

Hello everyone-

I'm writing a program, let's call it "A", that loads + dumps lua scripts in a way similar to luac.  A loads each lua script from disk, brings it into lua with luaL_loadbuffer, and then dumps it to a binary payload with lua_dump().  Each binary payload is stored at a known place in a larger binary file (that has other data in it), which is passed on to another program later, let's call that second program "B".  B creates a lua table, loads it up with these chunks from the binary file, and runs them at specified times.

A's output binary file is getting quite large, and the majority of the size is the dumped lua scripts, so I'm investigating ways to make it smaller.  One thought I had was that perhaps if program A could pre-create the table in the structure that B uses, there would be size savings in eliminated redundancy.  Each of the input lua scripts have the same 5 or 6 strings again and again, so I was thinking that if string pooling happens, perhaps that would reduce the size.  Also, if each dump contains any other redundant boilerplate, that would be eliminated if they all got dumped in one big chunk.

Is this a valid assumption?  If not, please stop me here.  :-)  If so...

My approach so far has been this:

Program A has on its lua stack a table, constructed via the C API, that has integers for keys and chunks for values.  It creates the table like this (this isn't verbatim, it's trimmed + simplified to show the issue):

=================================================

lua_newtable(lua);  // 'lua' is a newly-opened lua_State*

for (int i = 0; i < count; ++i)

{

    luaL_loadbuffer(lua, luaScripts[i], std::strlen(luaScripts[i]), luaScriptFilenames[i]);

    lua_rawseti(lua, -2, i);

}

std::vector< char > dumpedPayload;

lua_dump(lua, &WriteLuaToStdVector, &dumpedPayload);

=================================================

and that lua_dump() of course fails because the top of the stack is of type LUA_TTABLE, and not LUA_TFUNCTION.

I looked at string.dump() and a few of the various table serialization utilities on the Lua wiki, but if possible I'd really like to preserve the source lua script filenames + debug info.  Can I create a Lua closure (from C or Lua) that declares this table as a local, and returns it?  Ideally the individual files would be in chunk form here, I'd rather not include the script source code in the big binary file.  

One other constraint is that I can't simply merge the chunks, I saw some utilities for that, because each chunk redefines variables in the global scope.  This works in program B because it always calls lua_setfenv() on the chunk with a new env table before lua_pcall().

I started looking into manually assembling a lua function using luaF_newproto() and friends, but that seems very involved and I'd like a second opinion first :).  It also feels like there's an easier way to do this.

Thanks in advance for reading this far, and any advice, direction, flames you can offer would be greatly appreciated.

Best,

Charles Nicholson