Re: Object "Forwarding" Between States

[Sean Conner <sean@...>]

It was thus said that the Great Scott Morgan once stated:
> I'm thinking about ways to share objects between states, something I've 
> not needed to do before, and have come up with a possible technique:
> 
> State L1 has the actual object `obj`
> State L2 is given a 'forwarding' object `fwd`
> 
> A call in L2, e.g. `fwd:foo("test")`, picks up the name of the method, 
> and it's arguments, than passes them into L1's state, calling the method 
> on `obj`. Results are then returned to L2 as expected, with non-basic 
> Lua types being wrapped in their own forwarding objects.
> 
> You could expose plain functions this way too.
> 
> So, given that basic outline, a couple of questions:
> 
> Q1 : Has anyone ever done this? (Is there already a lib)
> Q2 : Is this viable? There's obvious marshalling overhead, and limits on 
> passing types, but if those aren't an issue, does it seem okay? Worst 
> problem may be thread syncing.
> 
> Scott

  How generalized do you want the solution to be?  The more specific you
make the solution, the easier it is.

  I have investigated this (serialization, about five years ago), and
looking specifically to what you asked for, the first thing to watch out for
is, as you say, thread syncing.  Aside from any threads, if L1 wants to send
data to L2, then:

	if L1 wants to send, but there is no recpient ready, it has to block
	if L2 wants to receive, but there is no sender ready, it has to block

  That's for the specific case of two Lua states.  If there are more states
involves, you have to solve two specific subcases:

	one sender L1
	two receivers L2 and L3
		Who does L1 send to?  

	two senders L1 and L2
	one receiver L3
		Who does L3 receive from?

from which one can generalize to:

	given X senders, and Y receivers, how to map senders to receivers?

  Once that is solved, how do get get data from L1 to L2?  Leaving aside if
all the states are in the same process or not, there are still issues to
address.  The basic data types of Lua, nil, boolean, number and string, are
easy to deal with.  A Lua table, as long as it contains only those base
types, and no cycles, is also pretty straightforward to deal with.  If you
can limit the problem to base types and non-cyclic tables, then you are
probably good to go.

  A more generalized approach---one that can handle just about anything
thrown at it, is much harder.  First complication, tables with cycles:

	x = {}
	x[x] = x -- self-referential table from Hell

  Next up, functions.  The first stumbling block are function upvalues. If
you want to pass a function, you'll need to deal with upvalues as well,
which also means dealing with the possible global state of Lua itself (_G
via _ENV).  I'll get back to this in a bit.  The other complication is how
the funciton is written---Lua or C?  If it's Lua, it can be dumped to a
string, but you'll need to somehow flag it as a "function" and not a
"string".  If it's a C function, again, I'll get back to this in a bit.

  Now you have userdata, and there are two types---light user data and full
user data.  The former is just a pointer to something Lua has no control
over and as such, could be just moved over to the other state as is.  No
real issues there.  But the full user data is something else.  The base Lua
interpereter only has one type of userdata, which is a FILE* (represents an
open file), but the number and nature of userdata is open ended.

  The last value type is a Lua thread.  I never got around to figuring out
how to serialize a thread, and I'm not sure if it's even possible via the
Lua API.

  I came up with solutions to these (except for Lua threads), and the
solution was for a generalized serialization method, not the limited scope
mentioned here.  I used CBOR [1][2], as it has ways of adding semantic
tagging to values, and extentions to handle self-referential tables [3].  I
used the semantic tags to deal with known Lua values, so if I was sending
the user data type of io.stdin, what would get encoded is the string
"io.stdin" but tagged as a standard Lua value.  Functions were strings that
were either tagged as a name (like "math.floor") or a dumped Lua function (I
don't recall how I dealt with functions written in C---probably assumed it
existed on the other side and sent just the name).  I also handled upvalues
as I could already handle just about anything else in Lua. It's not much,
but I do have a bit of code that can be viewed [4].

  I hope this solidifies some of the issues for you.

  -spc

[1]	Concise Binary Object Representation <http://cbor.io/>

[2]	https://github.com/spc476/CBOR

[3]	The above table ends up encoded in CBOR as the bytes 

		D9 01 00 D8 1C A1 D8 1D 00 D8 1D 00

[4]	https://github.com/spc476/LuaFunctionSerialize