Re: Hooks, tailcalls and tracing LuaJIT

[Benoit Germain <bnt.germain@...>]

2011/4/4 Luiz Henrique de Figueiredo <lhf@tecgraf.puc-rio.br>

> But note that the Lua debug interface is rather brittle, anyway.
> It mixes too many different concerns, it's not designed to be
> extensible, it works at the wrong abstraction level and it's
> incomplete (doesn't report yields, errors). It may work for simple
> tasks, but a full-blown debugger or trace analyzer needs way too
> much guesswork and workarounds. Even setting a simple breakpoint
> is excessively complicated.

We welcome suggestions to improve the debug interface.

I have one regarding breakpoints:

Native debugging is done by replacing the instruction by a special NOP that causes an exception that invokes the debugger.
When the user steps after having hit the breakpoint, the instruction that the debugger stored for this breakpoint is executed so that program behavior is not altered.

I'd like very much a similar feature in Lua: it would require a new opcode (let's call it OP_BREAK), two API calls to set/clear a breakpoint, and an application-provided handler.
When the user sets a breakpoint, the underlying opcode is returned by the set breakpoint API call so that the application can store it. Then it is replaced by OP_BREAK in the function proto.

Whenever the VM encounters OP_BREAK, it calls the handler than can do whatever it wants, and should return the opcode that was stored for the breakpoint's location.
The VM then processes this opcode normally and continues execution. In the same manner, one can remove a breakpoint by providing the opcode that was stored.

This would much reduce the overhead incurred when testing for breakpoints in the line hook. Just like native debugging, the VM execution would be slowed only when it actually encounters a breakpoint.
I'm not sure that Lua should provide the opcode storage facility, as the application might want to store more information regarding a given breakpoint (hit count, break condition...)

There is also the problem of _expression_ evaluation within a context while the VM is suspended in a line hook / OP_BREAK handler. When a breakpoint is hit, if I wanted to evaluate an _expression_ in the debugger, be it for break condition evaluation or simple _expression_ inspection I would do it that way:

* create a thread with lua_newthread
* change its environment with an empty one that uses a metatable with __index to access the global VM's environment
* get the list of locals and upvalues in the current call frame, add them in the thread's environment (-> as globals)

* compile the _expression_ in the thread, run it, fetch the results, display them

Maybe having some helper in the API that would prepare the way for easy (and properly sandboxed) _expression_ evaluation would be welcome?

-- 
Benoit.