Re: Callbacks, upvalues and memory management

[Mark Hamburg <mark@...>]

The first bit of advice is "don't use refs".

But then what should one use?

If you can count on the Lua side being able to own the lifetime for the C++ objects -- i.e., they would never get collected within Lua while still being referenced from C++, then the following construction will work:

	Lua userdata
		--> C++ object (deleted when the userdata __gc metamethod gets invoked)
		environment table --> callbacks and references to other Lua side data

To get from C++ back to Lua, you then maintain a fully-weak table mapping the address of the C++ object as a light userdata to the full userdata proxy on the Lua side. So, to get to your callback, you do a lookup in this table to get the full userdata, get it's environment table, and then lookup the callback there.

Okay. But what if the C++ objects need to live on beyond the Lua data? The first thing I will assume is that you then have some form of shared lifetime management for these objects such as reference counting. Now, however, you have the issue that the Lua side links can go away before the object does which means that you need to be prepared for that to happen. This will show up in the form of the lookup in the fully-weak map from the C++ object to the Lua full userdata returning nil. Effectively, Lua has forgotten about the button and so the on_click isn't going anywhere. So, if you are building your view tree, you want to make sure that Lua can see it or somehow get from the root of the tree to the members without passing through C++. (This sort of system can work just fine. I've worked on a couple systems now where the logical view tree existed in Lua data structures and then mapped over to a physical view tree on that native side.)

But if that doesn't work, then you need to do further work when adjusting the reference counts on the native objects. Basically, you need a way to tell Lua not to collect an object if it's reference count is greater than 1. (The 1 is for the reference from the full userdata.) The easiest way to do this is to add logic to the code that increments and decrements the reference count to add the object to a set of preserved objects when the count transitions from 1 to 2 and remove it from this set when it transitions back. This can make reference counting more expensive but it only kicks in on the 1->2 and 2->1 transition. You also now need to do more analysis to make sure you aren't creating uncollectable cycles though there aren't inherent risks the way there are with something like refs. Basically, you just have to be aware when looking at the structure for reference count cycles that the Lua side is now potentially a big connection point between objects. (This also becomes a mess to write when dealing with multithreaded code which may now need to grab access to the Lua universe, but that's another matter. It's also horrendously complex if you need to reference the same C++ object from multiple Lua universes, but at that point if you also needed environment tables, things were probably getting horrendously complex anyway.)

Mark