Alien gotcha: Passing the result of alien.callback() directly to a struct field

[Duncan Cross <duncan.cross@...>]

Hi List,

I have only just started experimenting with the excellent Alien
library [1]. At the moment, all I'm trying to do is get an extremely
simple Win32 GUI application going, using only low-level calls to
system DLLs. (Just as a proof-of-concept - I do know that there are
much more convenient GUI packages available!)

Anyway, I've just come from debugging a crash that was perplexing me
for a while, and I thought I'd mention it here in case anyone is
interested. Every time I ran the application it crashed with no error
message after a few seconds. It turned out the problem was this line:

 classdef.lpfnWndProc = alien.callback(MyWindowProc, { ret = "long",
abi = "stdcall"; "pointer", "uint", "uint", "long" })

...here, "classdef" is an object created with alien.defstruct() that
is intended to mirror the "WNDCLASSEX" structure [2], and MyWindowProc
is a "WindowProc" function [3] implemented in Lua. What I only just
realised is that the assignment to classdef.lpfnWndProc is not
actually storing the object returned by alien.callback() --  the
__newindex metamethod is just converting the callback to a pointer,
and storing *that* in the underlying buffer -- and so, it is
immediately eligible for garbage collection. When it *does* get
collected, Windows explodes the next time it tries to use the
callback.

So, remember to make sure your callback objects are kept alive
somewhere else! I imagine there is probably some way that alien
structs could be changed so that they keep callbacks alive, but I
haven't tried looking into the Alien source myself to see how tricky
that would be to do.

-Duncan

[1] http://alien.luaforge.net/
[2] http://msdn.microsoft.com/en-us/library/ms633577%28VS.85%29.aspx
[3] http://msdn.microsoft.com/en-us/library/ms633573%28VS.85%29.aspx