Re: Interesting interaction between debug hooks, the garbage collector, and "open" VM instructions

[Mike Pall <mikelu-0902@...>]

Patrick Donnelly wrote:
> This is caused by the top of the stack being reduced below the top
> slot for the running function. The string and table will both be
> collected (during the hook) by the GC before the SETLIST operation is
> executed. What follows is a patch which corrects the problem.

As Peter explained this cannot happen with bytecode generated by
the Lua parser. Patching the interpreter is only a band-aid (the
patch is flawed, too). It's necessary to fix the bytecode verifier
to prevent injection of malicious bytecode.

On short inspection this turns out to be a non-trivial problem.
The bytecode doesn't contain enough information to easily
determine the maximum used stack slot at all points in the
bytecode stream. The debug info helps a bit (lifetimes of locals),
but this is not always available and incomplete (lifetimes of
temporaries are missing).

E.g. take the following two functions:

local function foo(...)
  local a,b,c,d = {},{},{},{}
  return ...
end

local function bar(...)
  local a,b = {},{}
  do local c,d = {},{} end
  return ...
end

And the resulting bytecode:

foo()
  1  NEWTABLE 	1 0 0
  2  NEWTABLE 	2 0 0
  3  NEWTABLE 	3 0 0
  4  NEWTABLE 	4 0 0
  5  VARARG   	5 0
  6  RETURN   	5 0

bar()
  1  NEWTABLE 	1 0 0
  2  NEWTABLE 	2 0 0
  3  NEWTABLE 	3 0 0
  4  NEWTABLE 	4 0 0
  5  VARARG   	3 0
  6  RETURN   	3 0

It's tempting to try to determine the maximum used stack slot by
searching backwards from the VARARG opcode. This is 4 for foo(),
so the VARARG starting at slot 5 is ok. But the VARARG starting at
slot 3 in bar() would be rejected even though the bytecode is
valid. One needs a full data-flow analysis to discover that none
of the potentially overwritten slots are used later on. Oh well ...

--Mike