Some dirty tricks to patch the OP_TAILCALL opcode.

[Peng Zhicheng <pengzhicheng1986@...>]

Hi list,

I remebered some time a go there was a discussion about disabling tailcalls so that the stack traceback
message can show every function in the call path. I now have another idea.

after reading the code of the Lua VM dispatch routine, I found that it might be posible
to patch the OP_TAILCALL instruction to the OP_CALL instruction: the RA and RB has the
same meaning for both instruction, but the OP_CALL use the RC to adjust the returned value.

further investigation into the code of the parser (and code generator) shows that RC for the
OP_TAILCALL instruction is  always set to 0(MULTRET), and more importantly, each OP_TAILCALL instruction
is followed immediatly by an OP_RETURN instruction, which is always MULTRET, too.

so I did some experiments, and it seemed work!

the attached code implents a module containing two functions:
  detail(for de-tailcall): to replace a function's OP_TAILCALL with OP_CALL
  retail(for re-tailcall): to recover the patched function.

CAUTION:
I used Lua's internal data structures from non-public headers,
so BE VERY CAREFUL if you want to experiment with it.
pay close attention to the compiler flags so it is binary compatible

I've no idea whether it has other not predicated effects.
could someone please check that or prove that?


the test code looks like this, at least it gave me expected result under Lua 5.2.3:

-----------8x------------------------------------------------------------------
function f (x) if x then return g() end return g() end
function g() return h() end
function h() return error("here") end

function go()
   return xpcall(function (...) f(...) end, debug.traceback)
end

_, msg1 = go()

local tailcall = require "tailcall"

tailcall.detail(f)
_, msg2 = go()
tailcall.detail(g)
_, msg3 = go()
tailcall.detail(h)
_, msg4 = go()

-- recover one function
tailcall.retail(g)
_, msg5 = go()

-- recover all
tailcall.retail(f)
tailcall.retail(h)
_, msg6 = go()

-- show the result
print(msg1, msg2, msg3, msg4, msg5, msg6)
-----------8x------------------------------------------------------------------


best regards.


/**
 * demo experiments patching OP_TAILCALL
 * pengzhicheng <pengzhicheng1986@gmail.com>
 * 
 * code put in public domain
 */
 
#include <lua.h>
#include <lauxlib.h>

/**
 * CAUTION:
 *    this module uses Lua's internal data structure.
 *    be ABSOLUTELY sure you understand what it does before using it.
 */
#include <lstate.h> /* for CallInfo */
#include <lopcodes.h>
#include <lobject.h>

static struct Proto *protoarg (lua_State *L) {
   /* index2addr(L, 1) */
   CallInfo *ci = L->ci;
   StkId o = ci->func + 1;
   /**
    * the macor `clLvalue' may not check type if the lua_assert not define.
    * so I explicitly check the type of the argument.
    */
   if (!ttisLclosure(o)) {
      luaL_argerror(L, 1, "only Lua function can be de-tailcall-ed");
      return NULL;
   }
   LClosure *cl = clLvalue(o);
   return cl->p;
}

/**
 * this function iterate through the proto's instruction array
 * and change each OP_TAILCALL instruction to an OP_CALL
 * 
 * to be able to recover the patched instructions later,
 * the `lineinfo' debug info is abused to mark the patch position,
 * so you MUST NOT call this function on stripped functions.
 */

static int detail (lua_State *L) {
   struct Proto *p = protoarg(L);
   Instruction *code = p->code;
   int sizecode = p->sizecode;
   int *lineinfo = p->lineinfo;
   int sizelineinfo = p->sizelineinfo;
   int i;
   for (i = 0; i < sizecode; ++i) {
      Instruction instr = code[i];
      if (GET_OPCODE(instr) == OP_TAILCALL) {
         code[i] = SET_OPCODE(instr, OP_CALL);
         /* ASSERT(i < sizelineinfo); */
         lineinfo[i] = -lineinfo[i];
      }
   }
   return 0;
}


static int retail (lua_State *L) {
   struct Proto *p = protoarg(L);
   Instruction *code = p->code;
   int sizecode = p->sizecode;
   int *lineinfo = p->lineinfo;
   int sizelineinfo = p->sizelineinfo;
   int i;
   for (i=0; i < sizelineinfo; ++i) {
      if (lineinfo[i] < 0) {
         Instruction instr = code[i];
         /* ASSERT(GET_OPCODE(instr) == OP_CALL) */
         code[i] = SET_OPCODE(instr, OP_TAILCALL);
         lineinfo[i] = -lineinfo[i];
      }
   }
   return 0;
}


static const luaL_Reg funcs[] = {
   {"detail", &detail},
   {"retail", &retail},
   {NULL, NULL},
};

int luaopen_tailcall(lua_State *L) {
   luaL_newlib(L, funcs);
   return 1;
}