Progess applying tolua++ ideas to SWIG!

[Don Hopkins <dhopkins@...>]

I've been studying the tolua++ and SWIG wrapper code, to get a better 
understanding of what's going on.
Now I've taken a first crack at extending SWIG to integrate C++ objects 
with Lua, using the "peer" object idea (using setfenv) from tolua++, 
that lets you write Lua "peer" classes that extend the C++ objects, 
override handlers on a per-instance basis, and makes it easy for C++ to 
call back into Lua handlers.

I've made some typemaps and modifications to Lua SWIG runtime to support 
delegating to scripted Lua peer classes (so you can define callbacks in 
the peer classes, as well as in the individual peer objects).
If you're interested, I'd be glad to send you my changes to try out. I'm 
still experimenting, and would appreciate any feedback, please!

It requires the classes I'm wrapping to have a static variable "static 
LuaRef peerClass" and an instance variable "LuaRef obj", where I've gone 
"typedef int LuaRef" so SWIG can apply typemaps to them. The static 
LuaRef peerClass is a reference to the peer class to use for the C++ 
class, and the instance variable LuaRef obj is a reference to the 
userdata that represents the C++ instance.
Currently the lifetimes of the objects are controlled by the application 
instead of the Lua garbage collector, so the C++ objects have to know 
how to clean up their Lua references when they get destroyed.
When my application initializes, before it returns any object references 
to Lua, I define some peer classes corresponding to the C++ classes. 
First there is a shared Base_Peer class that handles creating the peer 
object, which all the other peer classes inherit from.


------------------------------------------------------------------------
-- Peer Classes.


Base_Peer = {};
Base_Peer.__index = Base_Peer;


function Base_Peer:newPeer(obj)
   peer = {
       obj = obj,
   };
   gui.Message("Base_Peer:newPeer setting metatable. self: " .. 
tostring(self) .. " obj: " .. tostring(obj) .. " peer: " .. tostring(peer));
   setmetatable(peer, self);
   return peer;
end


Screen_Peer = {};
setmetatable(Screen_Peer, Base_Peer);
Screen_Peer.__index = Screen_Peer;
gui.Screen_peerClass = Screen_Peer;


I'm using the same table for the metatable and the class method table, 
to keep it simple.
The "newPeer" function is a class method that's meant to be called on 
the subclasses of Peer (Base_Peer is abstract). It's called with self 
equal to the peer subclass, an obj equal to the userdata of the C++ 
object we're wrapping. It makes a new peer object, and sets its 
metatable to the peer class, and returns it.

The newPeer method is called by the wrapper code when it needs to return 
a userdata, to create a Lua object of the right class to peer with it. 
The peer (lua table) is stored in the env slot of the userdata. The Lua 
class to wrap a C++ object is stored in a class variable of each C++ 
class, called "peerClass", which is accessible from Lua. That's what the 
"gui.Screen_peerClass = Screen_Peer" does: it tells the C++ wrappers 
what Lua class to wrap the C++ object in.

The exciting magic all happens in the SWIG runtime, in a new function I 
added called SWIG_Lua_PushPeer. It takes a pointer to a C++ object, a 
pointer to its "obj" reference slot (where to cache the reference to its 
userdata), its peerClass reference (from its class, to tell which Lua 
class to peer with), and a swig_type_info so SWIG_NewPointerObj has 
something to chew on. SWIG_Lua_PushPeer is called from a typemap I wrote 
to return C++ objects to Lua. (Actually it's a macro that writes 
typemaps to return C++ objects to Lua).

Here are some typemaps for passing LuaRef's back and forth between Lua 
and C++, and storing them in C++ members, as well as passing a Lua stack 
index in and out of a function (but not storing it in a member, because 
that wouldn't make much sense).

This SWIG macro %def_lua-obj_typemap takes a class name argument, and 
defines a typemap for returning that C++ class to Lua, via the magic 
SWIG_Lua_PushPeer function, passing it everything it needs to know to 
create a peer object and attach it to the C++ object.


////////////////////////////////////////////////////////////////////////
// Typemaps for storing integer LuaRef's in members, that let
// you get and set Lua objects from the scripting language.
// Uses Lua references to store integer references to Lua
// objects in C++ objects. If the reference we're setting
// already refers to an object, then we unreference the
// old reference so it gets garbage collected.
//
// Note: You should declare the following typedef:
// typedef int LuaRef;
//
// Note: The C++ object's destructor should clean up
// all its references, by calling LuaUnRef(&obj).


%typemap(in)
   LuaRef
%{
   // Make a reference to the value.
   lua_pushvalue(L, $input);
   $1 = luaL_ref(L, LUA_REGISTRYINDEX);
%}

%typemap(memberin)
   LuaRef
%{
   // If there's a valid reference there already,
   // then unreference it so it gets gc'ed.
   if (($1 != LUA_NOREF) &&
       ($1 != LUA_REFNIL)) {
       luaL_unref(L, LUA_REGISTRYINDEX, $1);
   }
   // Now set the reference.
   $1 = $input;
%}

%typemap(out)
   LuaRef
%{
   // Return the value of a reference.
   if (($1 == LUA_NOREF) ||
       ($1 == LUA_REFNIL)) {
       lua_pushnil(L);
   } else {
       lua_rawgeti(L, LUA_REGISTRYINDEX, $1);
   }
   SWIG_arg++;
%}


////////////////////////////////////////////////////////////////////////
// Typemaps for passing a Lua stack index into a function that knows
// about Lua.
//
// Note: You should declare the following typedef:
// typedef int LuaStackRef;
//
// Note: You can't store a stack index in a member, but a function
// can take it as a parameter and do something useful with it.


%typemap(in)
   LuaStackIndex
%{
   $1 = $input;
%}

%typemap(out)
   LuaStackIndex
%{
   lua_pushvalue(L, $1);
   SWIG_arg++;
%}


////////////////////////////////////////////////////////////////////////
// Macro for defining typemaps for returning C++ objects to Lua,
// which have a Lua object reference member called "obj", that
// stores the original Lua wrapper table. The first time we return
// the C++ object, we make a Lua object, and store a reference to
// it in the C++ object's "obj" member. Subsequently we check the
// C++ object's "obj" field, and return the same Lua object wrapper,
// instead of creating a new one each time.
//
// Note: The C++ object's destructor should clean up
// all its references, by calling LuaUnRef(&obj).


%define %def_lua_obj_typemap(CLASS)
   %typemap(out)
       CLASS *
   %{
       SWIG_Lua_PushPeer(
           L,
           (void *)$1,
           &$1->obj,
           CLASS::peerClass,
           SWIGTYPE_p_ ## CLASS);
       SWIG_arg++;
   %}
%enddef


////////////////////////////////////////////////////////////////////////


So then I can define a typemap that wraps my Screen class like:


%def_lua_obj_typemap(Screen);


And from then on, whenever I return a Screen from C++ to Lua, it will be 
automatically paired with a peer by SWIG_Lua_PushPeer!

It calls SWIG_Lua_PushPeer to push a peered C++ userdata on the Lua 
stack, to return it to Lua. The first time this is called, it creates 
the peer object, and caches a reference to it in the C++ object's obj 
member passed as the int *pobj. This function works independently of the 
C++ class, by taking a generic void pointer, an int pointer, a Lua peer 
class reference int, and a swig type info. All the class-specific stuff 
is factored out into the typemap that calls SWIG_Lua_PushPeer. (That's 
why you have to call the macro for each class you want to wrap, to 
generate the typemaps to wrap each class automatically.)

Here's a good example of its use in the wrapper for the "static Screen 
*Screen::curScreen" class variable getter, which shows a typical call to 
SWIG_Lua_PushPeer:


static int _wrap_Screen_curScreen_get(lua_State* L) {
 int SWIG_arg = -1;
 Screen *result = 0 ;

 result = (Screen *)Screen::curScreen;
 SWIG_arg=0;

 SWIG_Lua_PushPeer(
   L,
   (void *)result,
   &result->obj,
   Screen::peerClass,
   SWIGTYPE_p_Screen);
 SWIG_arg++;

 return SWIG_arg;

fail:
 lua_error(L);
 return SWIG_arg;
}


And here's SWIG_Lua_PushPeer itself:


void SWIG_Lua_PushPeer(
   lua_State *L,
   void *ptr,
   int *pobj,
   int peerClass,
   swig_type_info *type)
{
   // Pushes a peered userdata on the stack,
   // creating the peer Lua object if necessary,
   // and caching a reference to the peer in pobj.
   if (ptr == NULL) {
       // If the pointer is NULL, then return nil.
       lua_pushnil(L);
   } else if ((*pobj != LUA_NOREF) &&
          (*pobj != LUA_REFNIL)) {
       // If the C++ object already has a Lua object,
       // then return the Lua object referenced by obj.
       lua_rawgeti(L, LUA_REGISTRYINDEX, *pobj);
   } else {
       // If the C++ object doesn't yet have a Lua object,
       // then make one, stash a reference to it in pobj,
       // look for a peerClass, and if defined, then
       // call the peerClass's "newPeer" method to create
       // a Lua peer object, and attach it to our userdata's
       // environment.

       // stack: ...
       SWIG_NewPointerObj(L, ptr, type, 0);
       // stack: ... userdata
       lua_pushvalue(L, -1);
       // stack: ... userdata userdata
       *pobj = luaL_ref(L, LUA_REGISTRYINDEX);
       // stack: ... userdata
       if (peerClass != LUA_NOREF) {
           // stack: ... userdata
           lua_rawgeti(L, LUA_REGISTRYINDEX, peerClass);
           // stack: ... userdata peerClass
           if (!lua_istable(L, -1)) {
               lua_pop(L, 1);
               // stack: ... userdata
           } else {
               // stack: ... userdata peerClass
               lua_pushliteral(L, "newPeer");
               // stack: ... userdata peerClass "newPeer"
               lua_gettable(L, -2);
               // stack: ... userdata peerClass function
               if (!lua_isfunction(L, -1)) {
                   lua_pop(L, 2);
                   // stack: ... userdata
               } else {
                   // stack: ... userdata peerClass function
                   lua_pushvalue(L, -2);
                   // stack: ... userdata peerClass function peerClass
                   lua_pushvalue(L, -4);
                   // stack: ... userdata peerClass function peerClass 
userdata
                   lua_call(L, 2, 1);
                   // stack: ... userdata peerClass peer|nil
                   if (!lua_istable(L, -1)) {
                       // stack: ... userdata peerClass nil
                       lua_pop(L, 2);
                       // stack: ... userdata
                   } else {
                       // stack: ... userdata peerClass peer
                       lua_setfenv(L, -3);
                       // stack: ... userdata peerClass
                       lua_pop(L, 1);
                       // stack: ... userdata
                   }
                   // stack: ... userdata
               }
           }
       }
   }
}


I also modified SWIG_Lua_class_get and SWIG_Lua_class_set in the SWIG 
runtime, to know about delegating index events to the userdata's env 
(the peer object).

I added this code at the end of SWIG_Lua_class_get to try delegating to 
the peer object, if it exists:


   /* STACK: userdata "attname" metatable nil */
   lua_pop(L, 2);
   /* STACK: userdata "attname" */

   // Added by dhopkins to delegate to the peer class in the
   // userdata's environment.

   lua_getfenv(L, -2);
   /* STACK: userdata "attname" peer|nil */
   if (lua_isnil(L, -1)) {
       /* STACK: userdata "attname" nil */
       lua_pop(L, 1);
       /* STACK: userdata "attname" */
   } else {
       /* STACK: userdata "attname" peer */
       lua_pushvalue(L, 2);
       /* STACK: userdata "attname" peer "attname" */
       lua_gettable(L, -2);
       /* STACK: userdata "attname" peer val|nil */
       if (lua_isnil(L, -1)) {
           /* STACK: userdata "attname" peer nil */
           lua_pop(L, 2);
           /* STACK: userdata "attname" */
       } else {
           /* STACK: userdata "attname" peer val */
           lua_remove(L, -2);
           /* STACK: userdata "attname" val */
           return 1;
       }
   }


And I added this code to the end of SWIG_Lua_class_set, to try 
delegating to the peer object, if it exists:


   // Added by dhopkins to delegate to the peer class in the
   // userdata's environment.

   lua_getfenv(L, -2);
   /* STACK: userdata "attname" value peer|nil */
   if (!lua_istable(L, -1)) {
       /* STACK: userdata "attname" value nil */
       lua_pop(L, 1);
       /* STACK: userdata "attname" value */
   } else {
       /* STACK: userdata "attname" value peer */
       lua_pushvalue(L, 2);
       /* STACK: userdata "attname" value peer "attname" */
       lua_pushvalue(L, 3);
       /* STACK: userdata "attname" value peer "attname" value */
       lua_settable(L, -3);
       /* STACK: userdata "attname" value peer */
       lua_pop(L, 1);
       /* STACK: userdata "attname" value */
       return 1;
   }

   /* STACK: userdata "attname" value */


I also made some utilities for calling back from C++ to Lua, and 
unref'ing a LuaRef:

Here's how LuaCallback is used in the OnSize handler (of the superclass 
Tracker, which Screen and other classes inherit from). LuaCall takes an 
object reference and a string callback name. It tries to look up a named 
callback function on the object, and calls that function with the 
varargs parameters automatically converted to Lua objects pushed on the 
stack according to the format string. The string can also specify the 
return value types after a ">". This may look familiar because I ripped 
the code off from one of the Lua tutorials, and added the "r" option to 
pass reference arguments, so "rii" means an object reference (obj is 
passed as "self") and two integers (width and height).


void Tracker::handleSize(
   int width,
   int height)
{
   LuaCallback(
       obj,
       "onsize",
       "rii",
       obj,
       width,
       height);
}

Here's how LuaCallback is defined (it's built on some lower level 
functions LuaCallV and LuaCall, that follow):


// Function should already be on top of Lua's stack.
bool LuaCallV(
   const char *sig,
   va_list vl)
{
   int narg, nres; // Number of arguments and results.

   narg = 0;

   // Push the arguments.
   while (*sig) {

       switch (*sig++) {

           case 'd': // Double argument.
               lua_pushnumber(
                   L,
                   va_arg(
                       vl,
                       double));
               break;

           case 'i': // Integer argument.
               lua_pushnumber(
                   L,
                   va_arg(
                       vl,
                       int));
               break;

           case 's': // String argument.
               lua_pushstring(
                   L,
                   va_arg(
                       vl,
                       char *));
               break;

           case 'r': // Reference argument.
               lua_rawgeti(
                   L,
                   LUA_REGISTRYINDEX,
                   va_arg(
                       vl,
                       int));
               break;

           case '>': // End input argument values, begin output result 
pointers.
               goto endwhile;

           default:
               //error(L, "invalid option (%c)", *(sig - 1));
               return false;

       }

       narg++;

       luaL_checkstack(
           L,
           1,
           "too many arguments");
   }

endwhile:

   nres =
       strlen(sig); // Number of expected results.

   // Do the call.
   if (lua_pcall(
           L,
           narg,
           nres,
           0) != 0) {
       //error(L, "error running function: %s",
       //      lua_tostring(L, -1));
       return false;
   }

   nres =
       -nres; // Stack index of first result.

   // Retrieve results, output through pointer arguments.
   while (*sig) {

       switch (*sig++) {

           case 'd': // Double result.
               if (!lua_isnumber(
                       L,
                       nres)) {
                   //error(L, "wrong result type");
                   return false;
               }
               *va_arg(vl, double *) =
                   lua_tonumber(
                       L,
                       nres);
               break;

           case 'i': // Integer result.
               if (!lua_isnumber(
                       L,
                       nres)) {
                   //error(L, "wrong result type");
                   return false;
               }
               *va_arg(vl, int *) =
                   (int)lua_tonumber(
                       L,
                       nres);
               break;

           case 's': // String result.
               if (!lua_isstring(
                       L,
                       nres)) {
                   //error(L, "wrong result type");
                   return false;
               }
               *va_arg(vl, const char **) =
                   lua_tostring(
                       L,
                       nres);
               break;

           case 'r': // Reference result.
               *va_arg(vl, int *) =
                   luaL_ref(
                       L,
                       LUA_REGISTRYINDEX);
               break;

           default:
               //error(L, "invalid option (%c)", *(sig - 1));
               return false;

       }

       nres++;
   }

   return true;
}


bool LuaCall(
   const char *func,
   const char *sig,
   ...)
{
   bool result = false;

   va_list vl;
   va_start(
       vl,
       sig);

   // Get the globally defined function by name.
   lua_getglobal(
       L,
       func);

   if (!lua_isfunction(L, -1)) {
       goto done;
   }

   result =
       LuaCallV(
           sig,
           vl);

done:

   va_end(
       vl);

   return result;
}


bool LuaCallback(
   LuaRef objRef,
   const char *attName,
   const char *sig,
   ...)
{
   bool result = false;

   va_list vl;
   va_start(
       vl,
       sig);

   if ((objRef == LUA_NOREF) ||
       (objRef == LUA_REFNIL)) {
       goto done;
   }

   // STACK: ...

   // Get the reference to the object.
   lua_rawgeti(
       L,
       LUA_REGISTRYINDEX,
       objRef);
   // STACK: ... userdata|nil

   if (!lua_isuserdata(L, -1)) {
       // STACK: ... nil
       lua_pop(L, 1);
       // STACK: ...
       goto done; // No userdata.
   }

   // STACK: ... userdata
   lua_pushstring(
       L,
       attName);
   // STACK: ... userdata "attName"
   lua_gettable(L, -2);
   // STACK: ... userdata function|nil
   if (!lua_isfunction(L, -1)) {
       // STACK: ... userdata nil
       lua_pop(L, 2);
       // STACK: ...
       goto done; // no callback function
   }

   // STACK: ... userdata function
   lua_remove(L, -2);
   // STACK: ... function

   result =
       LuaCallV(
           sig,
           vl);

done:

   va_end(
       vl);

   return result;
}


void LuaUnRef(LuaRef *refp)
{
   if ((*refp == LUA_NOREF) ||
       (*refp == LUA_REFNIL)) {
       *refp = LUA_NOREF;
       return;
   }

   luaL_unref(
       L,
       LUA_REGISTRYINDEX,
       *refp);

   *refp =
       LUA_NOREF;
}