Re: problems upgrading from 5.0.3 - 5.1 with lua_newthread luaL_ref lua_gc and lua_resume

[Rici Lake <lua@...>]

On 31-Oct-06, at 12:47 PM, subatomic wrote:

> Thanks Rici,
>
> I did get it working using lua_setfield (basically the same as 
> lua_setglobal) as a line-by-line replacement for luaL_ref.   I use 
> sprintf( name, "%x", threadptr ) to create a unique name for it.  This 
> seems to create the reference count increment to keep the thread from 
> being GC'd...   Really strange.

Actually, Lua uses a tracing garbage collector, not reference counts.

You'd probably find it faster to use a lightuserdata as the key:

lua_State *t = lua_newthread(L);
lua_pushlightuserdata(L, t);
lua_insert(L, -2); /* flip the top two stack elements */
lua_rawset(L, -3);

But keeping them all on the main stack strikes me as a better solution.

Alternatively, you could put your own reference table on the stack at a 
known slot (perhaps 1), and then use luaL_ref with that table instead 
of the registry.

> Any ideas what changed between 5.0 and 5.1 to cause luaL_ref not to 
> work?  Seems funny that luabind would cause the issue, I'm using the 
> same version (cvshead) of luabind as I was using in 5.0.3...  So if 
> luabind is somehow stomping the luaL_ref results now, it should have 
> stomped them before right?

I think the indexes changed, I'm not sure. If luabind is reproducing 
the behaviour of luaL_ref instead of just using the interface (for 
"efficiency"), it's possible that it is not getting it right. I don't 
know much about luabind.

> >> The comment about having to collect actors before their thread is
> >> deleted suggests that something is wrong with your finalizer model, 
> if
> >> you actually experienced the crash to which you refer.
>
> Could you point me to some information about "finalizer models" ?   
> I'm not sure what to look for on this, and searching google doesn't 
> turn anything up.

Sorry, that was a bit telegraphic.

The problem with the luaL_ref model is that objects don't have 
references to other objects. They have indexes into the ref registry, 
which are opaque to the garbage collector. Consequently, the garbage 
collector will be more conservative than necessary; it often won't be 
able to collect cyclic pseudo-references.

The issue with finalizers is that they have to be run in the right 
order; if object A references object B, then B cannot be finalized 
while A is live. If B also references A then there is a cycle; that's 
not a problem for the garbage collector (except as noted above, when 
the references are opaque), but it is a problem for running the 
finalizer. It's not obviously safe to run the finalizers in either 
order.

In practice, it is probably the case that some order will work; even 
though A holds a reference to B, for example, it's finalizer may not 
actually use that reference. There is no way the garbage collector can 
know this, though, and the fact that the references are not real makes 
the situation worse. Often the finalizers are not run at all until you 
close the lua state; at that point, Lua will run all the finalizers 
even if the objects appear to be live. It runs them in reverse order of 
creation (newest object first). That can cause a crash if the newer 
object's finalizer expects the older object to still be around, and 
that's what I meant by your "finalizer model".

Since all of that is probably hidden inside luabind, I don't know what 
to suggest to fix it, though.

Lua 5.1 userdata have an environment table which can be used to hold 
real references to dependent objects; I've found that works a lot 
better than the luaL_ref model, in many cases.

Many people use complicated schemes involving weak pointers, etc., to 
work around the finalizer issue; it's generally not pretty. However, in 
a lot of real applications, the problem can be avoided by removing the 
dependencies. For example, if an object A includes a FILE* which needs 
to be closed when the object is garbage collected, instead of using a 
finalizer attached to A itself, it is possible (with Lua 5.1 
environment tables) to create a dependent object A' which does not 
refer to anything, but which includes the FILE*. If A holds the only 
reference to A', then A' will be garbage collectable whenever A is, but 
A' can be finalized without respect to ordering considerations. This 
strategy can at least make finalization dependencies clearer, and that 
is often enough to solve finalizer cycles.