Re: Serializing Lua functions (was Re: status and perceptions of luaproc)

[William Ahern <william@...>]

On Sat, Aug 08, 2015 at 09:39:01AM +1000, Daurnimator wrote:
> On 8 August 2015 at 08:38, Sean Conner <sean@conman.org> wrote:
> > It was thus said that the Great Rena once stated:
> >>
> >> The thread libraries I've used run a separate Lua state in each thread and
> >> provide some type of communication channel between them. This is a nice
> >> simple model, but maybe not the most efficient. In particular, while it's
> >> usually possible to pass most types of Lua objects between threads, tables
> >> can only be recursively copied, functions have to be dumped and recompiled
> >> (and can't have upvalues), and userdata can't safely be passed around
> >> unless it's designed to be referenced by multiple Lua states.
> >
> >   Actually, you can serialize Lua functions with upvalues.  I've done it
> > (just not shown the code).  In fact, I was able to serialize quite a bit,
> > including tables with circular references.  The only things I couldn't
> > handle were coroutines, user data and Lua functions written in C [1].
> >
> >   As it stands, the function serialization is only good across the same
> > architecture (everything else I could serialize was CPU independent) and I
> > started working on fixing that by diving into the "dump format" used by Lua.
> >
> >   -spc (I kind of lost interest when I realized I didn't have a need for
> >         this)
> >
> > [1]     At least not directly.  I could "serialize" the function io.write(),
> >         but I did so by reference, assuming the receiver side could resolve
> >         such a reference.  I could even handle known userdata types like
> >         io.stdout by again, using references. [2]
> >
> > [2]     Strings basically.  I just passed the name along.
> >
> >
> 
> The lua 'full' serialisation project was 'Pluto' this has been
> succeeded (for 5.2+) by Eris.
> https://github.com/fnuecke/eris
> They have the concept of "special persistence". see
> https://github.com/fnuecke/eris#special-persistence
> Which allows you to add a '__persist' field to a metatable.
> 
> I think this is a good approach, and have been meaning to add such a
> field to my lua libraries.

For message passing of userdata a __move and/or __copy metamethod would probably make more
sense.

Persisting often doesn't make any sense in an IPC scheme. You can't persist
descriptors (the values would copy, but the descriptor itself might be
invalid when you unpickle the data). Then you have things like pointers
which are relative to specific memory addresses, including the address of
the particular userdata. For many situations have to enlist the help of the
underlying application code in copying the data.

I've been thinking of adding support for __move and __copy to cqueues'
thread module. cqueues supports socket descriptor passing (using Unix
sendmsg/recvmsg), but you can't pass SSL state that way. Moving the
low-level socket userdata across the Lua VMs would make that trivial.
(But then if the SSL object has Lua functions installed as callbacks for
things like verification, ALPN, etc things get messier. Which is one reason
it's still on the back-burner.)