Re: Fun with serializing Lua functions (was Re: Sharing userdata among stats.)

Functions are an interesting problem. Yes, you can get the bytecode (string.dump()) but there are issues:

This is a bit OT, but I was also working on a multithreading problem yesterday, and ended up rolling a rather dirty C-hack to solve the following problem:

Given a Lua 5.2 function with upvalues that are only strings, numbers, booleans, nils, or _ENV, create a new closure that contains at most one upvalue (equal to the parent function's _ENV). Any non-_ENV upvalues should be converted to constants, using the value currently stored in the upvalue.

My excuse for writing this (rather odd) function was that string.dump is a great tool for moving simple functions from one thread to another; but, 'prepping' a function for transfer via string.dump can be harder than it needs to be, if said function happens to include a lot of upvalues that are simple types. In the case of such upvalues, just converting the upvalue ref to a constant ref is usually the behavior I want, and writing a function that hacks into the guts of Lua and does exactly that is relatively straightforward (in C, anyhow).

If anyone wants a copy of the code, let me know.

Similarly, of course, if anyone wants to tell me what a fool I am for solving this particular problem in this particular way, constructive criticism is always welcome :)

-Sven

On Fri, Jun 30, 2017 at 7:48 PM, Sean Conner <sean@conman.org> wrote:

It was thus said that the Great Sean Conner once stated:
>
> Functions are an interesting problem. Yes, you can get the bytecode
> (string.dump()) but there are issues: 1) the default serialization of
> functions will only work on the same architecture and 2) does not contain
> the current values of any upvalues. 1) is not an issue with your usecase,
> but 2) is. Again, that can be worked around by not only encoding the
> function (via string.dump()) but also including any upvalues, which can get
> quite interesting (oh, an upvalue might be math.floor() or _G itself!).

So I played around with this today, and I created a github repository for
those interested:

https://github.com/spc476/LuaFunctionSerialize

Right now, the code only supports Lua 5.1, and is not meant for any real
production (it's just a "proof-of-concept" right now). But it works. At
least, for the one complicated recursive function I've tested so far.

1) It uses string.dump() to get the bytecode
2) It serializes any upvalues [1]
3) It only serializes the environment if it's the global environment
4) It can handle any normal Lua global value (like io.stdin or getmetatable)
5) It will probably fail horribly if given a Lua function written in
C (I did not check for this condition)

So there you have it. Something to play around with.

-spc

[1] bools, numbers, tables and functions. Tables can have cycles.