I checked the attached script and examined the source code of Lua.
[...]
when the function returns NOTHING (either an explicit `return' statement or implicitly),
Lua generate this opcode:
RETURN 0 1
when the function returns some value (e.g. nil), Lua generate this opcode:
LOADNIL 8 0
RETURN 8 2
in the latter case, the VM still adjust the stack before calling the return hook, but since
the returnning values are loaded at the top of the stack, the active stack frame holding the
local variables is not destroyed, so your hook can now get the expected results.
Many thanks for that detailed analysis. (BTW, as Philipp already found
out, the problem also happens when we return a local variable, because
Lua optimizes that case and returns the variable directly from where it
lives, not from the top of the stack.)
-- Roberto