Determining if _ENV is the first upvalue of a closure?

For various reasons we need to dynamically move Lua functions (actually closures) across lua_State boundaries. This is of course easy using dump()/load() pairs, portability of the bytecode is not a problem since this is all occurring in-process on a given architecture.

Of course, dump()/load() doesnt handle any upvalues for the closure (ok, function) being dumped, no surprises there. However, since we DO need to transport some upvalues with certain functions, we serialize them and send them along with the dumped function. This all works fine, after loading the closure at the receiver we simply de-searialize the upvalues are install them in the loaded closure.

The problem occurs with _ENV. Of course we don’t serialize the environment (nor could we), so we skip the first upvalue. However, not all functions *have* _ENV as the first upvalue, since the compiler only emits this if it is used in the function.

The problem then, is determining which functions have _ENV as an upvalue and which do not. I’m unable to find a deterministic way to do this. I can’t look at the type of the first upvalue, since some of our upvalues are also tables. I can’t look at the name of the first upvalue, since this can be stripped out with debug information. I *can* look to see if the first upvalue is the same table as the distinguished environment, but (argh) some functions have custom environments and hence a different environment. I can look to see if the table has _G (and if it is a self-reference to the table), but while this is a good hint it is of course not deterministic as a regular table could have this (though it would be odd).

Clearly the compiler knows is it emitting a closure (ok, function prototype) with _ENV as the first upvalue, since it makes sure _ENV is always the first upvalue. But it doesn’t seem to leave any breadcrumbs that I can access from the C API to determine this for a given function.

Any ideas anyone???

—Tim

P.S. Incidentally, since load() simply assumes that the first upvalue is always _ENV, this can lead to some surprising behavior in Lua. For example:

function foo()

local x = 10

return function return x end

end

f1 = foo()

f2 = load(string.dump(f1))

printf(f1())

print(f2())

The first print() prints 10, but the second prints “table: …”. This is correct in a sense, but I think most people might expect nil (which you would get if the inner function referenced any global variable or function). It’s also a problem for sandboxes, since it means you can reach back past a local _ENV environment and reach the distinguished environment unless you disallow load().

—Tim

The problem occurs in determining WHICH upvalue to serialize. If the function in question does not reference a global, then its the first upvalue. If, however, the function has one or more global references, Lua inserts _ENV as the first upvalue, bumping the other upvalues along one to make room. I cannot find a deterministic way, from the C API, to determine if a given closure has _ENV as its first upvalue or not. I can’t look at the names of the upvalues, as these may be stripped with debug information.

The compiler clearly knows if it’s emitting a function with _ENV as the first upvalue, since it takes care to ensure it is always the first upvalue in the function. However, this information doesnt seem to make it into the compiled chunk anywhere (that I can see).