Re: Hooks, tailcalls and tracing LuaJIT

[Geoff Leyland <geoff_leyland@...>]

On 4/04/2011, at 11:36 PM, Luiz Henrique de Figueiredo wrote:

>> But note that the Lua debug interface is rather brittle, anyway.
>> It mixes too many different concerns, it's not designed to be
>> extensible, it works at the wrong abstraction level and it's
>> incomplete (doesn't report yields, errors). It may work for simple
>> tasks, but a full-blown debugger or trace analyzer needs way too
>> much guesswork and workarounds. Even setting a simple breakpoint
>> is excessively complicated.
> 
> We welcome suggestions to improve the debug interface.

I think to some extent, "what Mike said": doesn't report yields, errors

It would be nice if thread switches were reported, and if pcall and xpcall frames and errors were reported.  It'd also be nice to have some way to query the thread stack, and if there were a quick way to get the stack depth (either real or imagined, depending on your view of tailcalls).

I'm not sure I'm up to suggesting improments, but I can tell you what I've had to work around...

In order to handle threads, I look for calls to C functions named "resume" and "yield", but there's a few tricks:
 - resume doesn't always switch threads (if the thread has finished), so if I see a resume, I then have to watch the current thread until the next line event, and see if that's in a different thread.
 - yield doesn't get called at thread exit.  In this case you can assume a yield if there's a return to a nil stack frame.

Alternatively I could almost get threads to work just by watching calls and *returns* from a function called resume and avoid the trickiness with yield, except that LuaJIT doesn't report returns from C calls, and, of course, there's coroutine.wrap.

In order to handle coroutine.wrap, I assume that *any* call to a c function not named one of yield, resume, error, pcall or xpcall could be a wrapped function, and then watch to see if the thread has changed at the next line event.

Compared to that, pcall, xpcall and error are relatively easy - I just watch for calls to C functions with those names.  It's worth remembering that coroutine.resume has an implicit pcall, but a wrapped coroutine does not.

If users choose to change the names of any of these magic functions, then all bets are off.

Oh, and it would also be nice if there was a way to set a hook for all threads from Lua, instead of having to monkey patch coroutine.create and coroutine.wrap and hoping that there aren't any existing threads that a user is wanting you to trace (which you could manage by monkey patching yield and resume..., but you get the idea).  In this particular case I'm using a C hook anyway, but I have a fallback Lua hook in case the C hook isn't compiled.

Cheers,
Geoff