Causing the currently running function to be garbage collected

["Peter Cawley" <lua@...>]

If a function's upvalues pointed to all the instances of that function
on the stack, and assuming that the function does not exist in any
tables, then if that function set all of it's upvalues to nil, and
then a GC collection, then it would cause itself to be collected. This
then causes all kinds of fun (i.e. segfaults) as the current
environment table has probably just been collected as well. In
straight Lua, it is impossible to have this situation arise, however,
if bytecode manipulation is performed, then a function's upvalues can
point to all the instances of that function. The following example
performs this: (assumes 32-bit little endian, online at
http://codepad.org/JRsc9hLD)

function Evil()
  local Experimental, _2

  Experimental = function()
    -- Erase all references in the stack to this (currently running) function
    Experimental = nil
    _2 = nil -- (this line only does so after bytecode manipulation)

    -- Do some cycles of garbage collection to free ourselves, and
some allocations to try and overwrite the memory
    for i = 1, 10 do
      collectgarbage "collect"
      alloc()
    end

    -- A segfault will probably now have occured
  end

  Experimental()
end

-- Do some bytecode manipulation of the Evil function
Es = ('').dump(Evil)
Es = Es:gsub("(\36..."      -- OP_CLOSURE
          .. "%z%z%z%z"     -- Use local 0 as upvalue 0
          .. "%z%z)\128%z"  -- Use local 1 as upvalue 1
          ,
             "%1\0\1")      -- OP_CLOSURE, using locals 0 and 2 as
upvalues 0 and 1
                            -- (local 0 is the Experimental function,
local 2 is where the function is placed for the call)
Evil = loadstring(Es)

-- Function to trash some memory
function alloc()
  local t = {}
  for i = 1, 100 do
    t[i] = i
  end
end

-- Run the evil
Evil()