Re: [LuaJIT FFI] [Windows] "bad callback" error in MsgWaitForMultipleObjects() proof-of-concept

[Duncan Cross <duncan.cross@...>]

On Fri, Dec 23, 2011 at 8:43 PM, Mike Pall <mikelu-1112@mike.de> wrote:
> Duncan Cross wrote:
>> What's happening is that the application is giving a "bad callback"
>> error shortly after the application starts, especially when moving the
>> mouse over the window.
>>
>> If I turn off JIT with jit.off(), it seems to stop happening.
>
> The problem is that a FFI callback cannot safely be called from a
> C function which is itself called via the FFI from JIT-compiled
> code. In your case this is the call to MsgWaitForMultipleObjects.
>
> I've put in a lot of heuristics to detect this, and it usually
> succeeds in disabling compilation for such a function. However in
> your case the loop is compiled before the callback is ever called,
> so the detection fails.
>
> The straighforward solution is to put the message loop into an
> extra Lua function and use jit.off(func).
>
> --Mike
>

Hi Mike,

Thanks very much for this prompt explanation!

Are you certain it's MsgWaitForMultipleObjects() calling that WNDPROC
callback itself though? My reading of the documentation for that
function [1] is that it only waits until there is an event to process,
it doesn't do anything about it - hence the usual calls to
TranslateMessage/DispatchMessageA afterwards in the loop. I tried
printing debug.traceback() in that callback - I see instances of
CreateWindowExA(), ShowWindow(), UpdateWindow(), PeekMessageA() and
DispatchMessageA() on the stack, but not MsgWaitForMultipleObjects().

(I realise this may not affect the explanation/solution much, but in
case it is significant I thought I better query this.)

[1] http://msdn.microsoft.com/en-us/library/windows/desktop/ms684242(v=vs.85).aspx

-Duncan