Re: Feature request: userdata slice

On Aug 19, 2015, at 1:33 AM, 云风 Cloud Wu <cloudwu@gmail.com> wrote:

I’m not clear what your actual problem is, as you seem to be suggesting a solution without really saying what the underlying problem you are trying to solve is. What I *think* you are saying is that you wish to expose two distinct userdata values that have the same lifetime from a GC standpoint, but I’m not clear why you need to do this.

I think this is a common case.

For example, we have a tree based document in C, (maybe an xml document, a scene object in 3d engine, or a window in GUI system). and then we want to access a part of it.

We want to write like this in lua :

local t = obj.x.y.z -- obj is an userdata

We can't create separated userdata stores the separated part of obj , because only one object in C. So create a proxy userdata for obj.x , obj.x.y and obj.x.y.z is a solution, each proxy referenced a part of obj.

It's not enough, we also need create a weak table in obj to cache "x" "x.y" and "x.y.z" proxy userdata, because we can't returns different proxy userdata for obj.x when we call obj.x more than once. (obj.x should be unique, when we need use it as a table key later)

If this tree based document is large and deep, we need create too many small full userdata to access each part of it. It's very expensive now , and not necessary to clone the whole tree from C to lua with lua table.

We had try to use lightuserdata to simulate the idea (userdata slice) above, but gc doesn't mark light userdata, we should be very careful to manage the lifetime. and lightuserdata has no user type, it's very dangerous to work with other library.

As I said in my earlier post, the original motivation for userdata was to allow Lua to provide opaque storage for C libraries. The operative word here is “opaque” .. Lua can store the data and manage its lifetime (via the GC), but not access it. This provided a string guarantee that the C library could rely on the integrity of the data in the userdata when called, since Lua code could not corrupt it.

What you are asking for is transparent userdata; that is a block of userdata that can be processed by Lua code. There are several different approaches to this:

1. “Lift” the entire data into Lua, marshalling it into a Lua-like form (say, a table), and then re-pack it back into binary form when you have finished processing it. This requires some kind of unpacking/packing API, written by you (and probably in C). One advantage is that you can process the data using Lua constructs. The disadvantage is the pack/unpack overhead, plus extra memory involved.

2. Develop a custom C library that provides whatever low-level manipulations of the data you want (enumerate contents, edit values etc), and write Lua code that uses this API. This is probably more efficient that (1) though it needs some overhead when making the calls to the C library.

The slice technique you discuss appears to confuse ownership and lifetime management of the dataset being manipulated. If you have only one object in C, then by definition the lifetime of the object is controlled either by C (in which case you should use a light userdata) or by Lua (a full userdata). If you need to create subsidiary objects that share the same lifetime as the “master” object, then you need proxy userdata that can coordinate with the full userdata to track the lifetime of the object.

—Tim