Re: Weak tables and userdata finalization

[Mark Hamburg <mark@...>]

On Sep 2, 2011, at 8:51 AM, Mark Hamburg wrote:

> On Sep 2, 2011, at 8:44 AM, Mark Hamburg wrote:
> 
>> I believe I just worked through a bug in my project caused by weak caches together with userdata finalization, but I wanted to confirm that my analysis was correct and not just the result of sleep deprivation.
>> 
>> * Lua 5.1.4
>> * Two userdatas each with __gc metamethods. Let's call them UD1 and UD2.
>> * We also have a table T.
>> * T references UD2 -- e.g., T.x == UD2
>> * T is accessible in a weak cache C (getmetatable( C ).__mode == 'kv') and C[ 'key' ] == T
>> * UD1 has an environment that also references T -- e.g., getfenv( UD1 ).y == T
>> 
>> So, here is what I think is happening to me that I want to confirm is possible.
>> 
>> * The GC finds that neither UD1 nor UD2 (nor T) are accessible via strong links, so it queues UD1 and UD2 for finalization.
>> * As part of queueing UD1 for finalization, it marks its environment so that that remains available for the __gc metamethod and this in turn marks T.
>> * T then survives in the cache.
>> * We fetch T out of the cache.
>> * This doesn't stop UD2 from being finalized which then makes us unhappy when we go to use use UD2 via T.x.
>> 
>> Except that writing this down, it doesn't jive with my recollection of what the GC does. In particular, I would have expected the reference to T in C to have been cleared even if T is kept alive by the finalization logic.
>> 
>> The symptom, however, is clearly that we go to use UD2 and it has already been finalized. UD1, on the other hand, was just a theory.
>> 
>> Any other insights into how we could end up running through weak caches and reaching an object that has been or will be finalized? I know about the dangers with keys on weak tables since those are identified as potentially being needed by the __gc metamethod, but I am not iterating weak tables anywhere in a way that would discover such finalized keys.
> 
> Maybe I just confirmed my original theory and demonstrated that my recollection was wrong:
> 
>  udsize = luaC_separateudata(L, 0);  /* separate userdata to be finalized */
>  marktmu(g);  /* mark `preserved' userdata */
>  udsize += propagateall(g);  /* remark, to propagate `preserveness' */
>  cleartable(g->weak);  /* remove collected objects from weak tables */
> 
> Basic lesson: Weak tables plus finalization are a dangerous mix?

Though does this then get worse? I had assumed that if I had an object that could be queued for finalization on the value side of a weak table, that entry would get cleared in the GC pass so one could recognize such objects by testing an appropriate mapping. For example, our proxy system for Objective-C objects uses a weak table mapping light userdata Objective-C object addresses to Lua proxies so that we generate the same proxy for a userdata object if we need to publish it to Lua multiple times. It would be really bad if we could end up using a proxy that had been scheduled for finalization. Now, I haven't seen this happening, but the above code snippet certainly suggests that it's possible. Please tell me I've missed something as opposed to just being extremely lucky...

Mark